2D digital human real-time interaction method and device, equipment, storage medium and product

Abstract

The invention discloses a 2D digital human real-time interaction method and device, equipment, a storage medium and a product, relates to the technical field of computers, and discloses a 2D digital human real-time interaction method, which comprises the following steps: in response to a user input instruction, determining a reply audio and a digital human image corresponding to the reply audio; based on a preset time frame sequence, transmitting each frame of reply audio to an audio pipeline through a preset audio writing process, and transmitting each frame of digital human image to a picture pipeline through a preset picture writing process; based on the time frame sequence and a preset video coding process, performing video coding on each frame of reply audio in an audio pipeline and each frame of digital human image in a picture pipeline to obtain a target digital human video; and playing the target digital human video to realize interaction between the digital human and the user. According to the method, the digital human video clip is generated in real time, so that the fluency of the digital human video is improved, and the effects of real-time driving of the digital human and fluency of the digital human video are realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment, storage medium and product for real-time interaction of 2D digital humans. Background Art

[0002] Digital human technology integrates a variety of artificial intelligence technologies such as computer graphics, motion capture, natural language processing, speech recognition and synthesis. Digital human technology can create virtual characters with human characteristics, including appearance, behavior and interaction capabilities. Digital humans can exist in the non-physical world and are created and used by computer means. Digital humans can not only simulate human appearance, but also simulate human behavior, such as walking, talking, and expression changes.

[0003] In the related technology, the text / voice input by the user is usually input into a large language model to obtain the corresponding text answer, and then the audio is generated from the text through TTS technology, and the audio is used to drive the digital human. However, this method has the problem that the digital human is difficult to drive in real time and the digital human video will be stuck.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a 2D digital human real-time interaction method, aiming to solve the technical problems in related technologies that digital humans are difficult to drive in real time and digital human videos may freeze.

[0006] To achieve the above purpose, the present application proposes a 2D digital human real-time interaction method, the 2D digital human real-time interaction method comprising:

[0007] In response to a user input instruction, determining a reply audio and a digital human image corresponding to the reply audio;

[0008] Based on a preset time frame sequence, each frame of the response audio is transmitted to the audio pipeline through a preset audio writing process, and each frame of the digital human image is transmitted to the image pipeline through a preset image writing process;

[0009] Based on the time frame sequence and the preset video encoding process, each frame of the reply audio in the audio pipeline and each frame of the digital human image in the picture pipeline are video encoded to obtain the target digital human video, wherein the video encoding process, the audio writing process and the picture writing process are executed in parallel according to the time frame sequence;

[0010] The target digital human video is played to realize the interaction between the digital human and the user.

[0011] Optionally, the step of determining the reply audio and the digital human image corresponding to the reply audio includes:

[0012] Determining a reply audio based on the user input instruction;

[0013] Extracting audio features corresponding to the reply audio of each frame;

[0014] Dividing the audio features according to a preset time length to obtain divided audio features, so that each frame of audio includes a preset number of continuous audio features;

[0015] Based on the divided audio features, a corresponding digital human lip image is generated, and based on a preset digital human base map and the digital human lip image, a digital human image corresponding to the reply audio is generated.

[0016] Optionally, the step of dividing the audio features according to a preset time length to obtain the divided audio features includes:

[0017] Respectively copying the first feature and the last feature in the audio feature to obtain a corresponding number of first copy features and last copy features;

[0018] Expanding the first copy feature to before the first feature, and expanding the last copy feature to after the last feature, to obtain an audio feature after edge expansion;

[0019] The audio features after edge expansion are divided according to a preset time length to obtain divided audio features.

[0020] Optionally, the step of generating a digital human image corresponding to the reply audio based on a preset digital human base map and the digital human lip shape image comprises:

[0021] Obtain the lip position information of the digital person;

[0022] Based on the lip position information, the lip-shaped corresponding area in the preset digital human base map is replaced with the digital human lip image to obtain the digital human image corresponding to the reply audio, wherein the digital human base map is preloaded into the memory.

[0023] Optionally, the method for real-time interaction of 2D digital humans includes:

[0024] During the interaction between the digital human and the user, a first video of the digital human in a first state and a second video of the digital human in a second state are determined, wherein the first state refers to the initial silent state of the digital human, and the second state refers to the listening state of the digital human or the answering state of the digital human;

[0025] Based on the first video, determining a corresponding initial state frame;

[0026] Adding the initial state frame to the front of the first frame and the back of the last frame of each digital human video to obtain a modified digital human video, wherein the digital human video includes the first video and the second video;

[0027] The modified digital human videos are spliced ​​to obtain a spliced ​​digital human video, and the spliced ​​digital human video is played to achieve digital human state switching.

[0028] Optionally, the step of adding the initial state frame before the first frame and after the last frame of each digital human video to obtain a modified digital human video includes:

[0029] Based on the initial frame, generating a corresponding intermediate frame through a frame insertion algorithm;

[0030] The initial state frame is added before the first frame and after the last frame of each digital human video, and the intermediate frame is added between the first frame and the initial state frame, and between the last frame and the initial state frame of each digital human video to obtain a modified digital human video.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a 2D digital human real-time interaction device, the 2D digital human real-time interaction device comprising:

[0032] A determination module, configured to determine a reply audio and a digital human image corresponding to the reply audio in response to a user input instruction;

[0033] A writing module, used for transferring each frame of reply audio to an audio pipeline through a preset audio writing process based on a preset time frame sequence, and transferring each frame of digital human image to an image pipeline through a preset image writing process;

[0034] An encoding module, used for performing video encoding on each frame of reply audio in the audio pipeline and each frame of digital human image in the picture pipeline based on the time frame sequence and a preset video encoding process, so as to obtain a target digital human video, wherein the video encoding process, the audio writing process and the picture writing process are executed in parallel according to the time frame sequence;

[0035] The playing module is used to play the target digital human video to realize the interaction between the digital human and the user.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a 2D digital human real-time interaction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the 2D digital human real-time interaction method as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the 2D digital human real-time interaction method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the 2D digital human real-time interaction method as described above are implemented.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] In the related art, the text / voice input by the user is usually input into a large language model to obtain the corresponding text answer, and then the audio is generated from the text through the TTS technology, and the digital human is driven by the audio. However, this method has the problem that the digital human is difficult to drive in real time and the digital human video will freeze. In contrast, the present application responds to the user input instruction, determines the reply audio and the digital human image corresponding to the reply audio; based on the preset time frame sequence, each frame of reply audio is passed to the audio pipeline through the preset audio writing process, and each frame of digital human image is passed to the picture pipeline through the preset picture writing process; based on the time frame sequence and the preset video encoding process, each frame of reply audio in the audio pipeline and each frame of digital human image in the picture pipeline are video encoded to obtain the target digital human video, wherein the video encoding process, the audio writing process and the picture writing process are executed in parallel according to the time frame sequence; the target digital human video is played to realize the interaction between the digital human and the user. It is understandable that the present application proposes to execute the video encoding process, the audio writing process and the picture writing process in parallel according to the time frame sequence. On the one hand, the synchronization of the picture and the audio is maintained to avoid excessive backlog of single-modal data in the picture pipeline or the audio pipeline. On the other hand, the synchronous execution is performed according to the time frame sequence to realize real-time generation of video clips at a fast speed, thereby improving the smoothness of the digital human video, and further realizing the real-time driving of the digital human and the smoothness of the digital human video. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flowchart diagram of the first embodiment of the 2D digital human real-time interaction method of the present application;

[0044] Figure 2 This is a flow chart of the modules of the real-time interaction method of 2D digital human in this application;

[0045] Figure 3 This is a flow chart of the information processing module of the 2D digital human real-time interaction method of this application;

[0046] Figure 4 A flowchart of a response generation module of the 2D digital human real-time interaction method of the present application;

[0047] Figure 5 A schematic diagram of the flow of a digital human video generation module of the 2D digital human real-time interaction method of this application;

[0048] Figure 6 A schematic diagram of the process of obtaining audio features of the 2D digital human real-time interaction method of this application;

[0049] Figure 7 This is a schematic diagram of the video encoding process of the 2D digital human real-time interaction method of this application;

[0050] Figure 8 A flow chart of the second embodiment of the 2D digital human real-time interaction method provided in this application;

[0051] Fig. 9 A schematic diagram of adding transition frames to a video in the real-time interaction method of a 2D digital human in this application;

[0052] Fig.10 A schematic diagram of video stitching of the real-time interactive method of 2D digital human in this application;

[0053] Fig.11 A schematic diagram of the digital human state switching process of the 2D digital human real-time interaction method of this application;

[0054] Fig.12 A schematic diagram of the reply interruption process of the 2D digital human real-time interaction method of this application;

[0055] Fig.13 This is a schematic diagram of the module structure of the 2D digital human real-time interactive device according to the embodiment of the present application;

[0056] Fig.14 This is a schematic diagram of the device structure of the hardware operating environment involved in the 2D digital human real-time interaction method in the embodiment of the present application.

[0057] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0059] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0060] The main solution of the embodiment of the present application is: in response to a user input instruction, determine the reply audio and the digital human image corresponding to the reply audio; based on a preset time frame sequence, pass each frame of reply audio to the audio pipeline through a preset audio writing process, and pass each frame of digital human image to the picture pipeline through a preset picture writing process; based on the time frame sequence and the preset video encoding process, perform video encoding on each frame of reply audio in the audio pipeline and each frame of digital human image in the picture pipeline to obtain a target digital human video, wherein the video encoding process, the audio writing process and the picture writing process are executed in parallel according to the time frame sequence; play the target digital human video to realize the interaction between the digital human and the user.

[0061] In this embodiment, the 2D digital human real-time interactive device is used as the execution subject. For the convenience of description, it is hereinafter referred to as "device".

[0062] In the related technology, the text / voice input by the user is usually input into a large language model to obtain the corresponding text answer, and then the audio is generated from the text through TTS technology to drive the digital human with the audio. However, this method has the problem that the digital human is difficult to drive in real time and the digital human video will be stuck.

[0063] This application provides a solution to achieve real-time driving of digital humans and smooth digital human video effects.

[0064] It can be seen from the above embodiments that the present application proposes to execute the video encoding process, the audio writing process and the picture writing process in parallel according to the time frame sequence. On the one hand, the synchronization of the picture and the audio is maintained to avoid excessive backlog of single-modal data in the picture pipeline or the audio pipeline. On the other hand, the synchronous execution is performed according to the time frame sequence to realize real-time generation of video clips at a fast speed, thereby improving the smoothness of the digital human video, and further realizing the real-time driving of the digital human and the smoothness of the digital human video.

[0065] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a terminal system, etc. that can realize the above functions. The following takes a 2D digital human real-time interactive device as an example to illustrate this embodiment and the following embodiments.

[0066] Based on this, the present application embodiment provides a 2D digital human real-time interaction method, referring to Figure 8 , Figure 1 This is a flow chart of the first embodiment of the 2D digital human real-time interaction method of the present application.

[0067] In this embodiment, the 2D digital human real-time interaction method includes steps S100 to S400:

[0068] Step S100, in response to a user input instruction, determining a reply audio and a digital human image corresponding to the reply audio;

[0069] It should be noted that user input instructions refer to instructions input by the user in voice or text. After receiving the user's instructions, the device parses and understands them, determines the user's needs, and finally generates an audio response based on the content of the user's instructions. This usually involves speech synthesis technology, that is, converting text into speech (Text-to-Speech, TTS).

[0070] In the specific implementation, in order to make the digital human's response more vivid and intuitive, the device also needs to generate a digital human image that matches the audio response, which specifically involves facial animation technology, that is, dynamically generating the digital human's facial expressions and lip shapes based on the audio content and semantic information.

[0071] In the specific implementation, the overall goal of this application is to realize a 2D digital human real-time driving system that supports interaction. The user side can interact with the digital human by inputting voice or text. The digital human will switch between the three states of silence, speaking, and listening, and respond to the user's input accordingly. The overall technical process of this application includes four stages: 1. User input processing stage; 2. Reply generation stage; 3. Digital human generation stage; 4. Digital human state control stage. Among them, in the user input processing stage, the device receives voice or text input from the user, and for voice input, the corresponding text information is generated through the voice recognition algorithm. Subsequently, the device analyzes the text input to determine whether the digital human control instruction is triggered, and if so, the corresponding instruction information is generated.

[0072] Furthermore, in the response generation stage, the device first generates a corresponding text response by calling a large language model, and then generates an audio response from the text through TTS technology as the input for driving the digital human in the next stage. For instruction information, the instruction is passed to the next stage.

[0073] In the specific implementation, in the digital human generation stage, for the audio answer passed in the response stage, the audio features are first extracted, and the corresponding character lip shape is generated according to the audio features through the lip shape inference network. Then the generated lip shape is pasted back to the digital human base map, and finally the generated digital human image stream and audio stream are encoded into video clips. The remote client downloads the corresponding video clips as needed.

[0074] Furthermore, in the digital human state control stage, the digital human state can be switched between three states: silent, speaking and listening according to the current scene and user instructions. When switching, the picture can transition naturally without the phenomenon of character teleportation, picture jump and so on.

[0075] In the specific implementation, the solution adopted by this application mainly includes four modules: information processing module, reply generation module, digital human generation module and digital human state control module. The overall flow chart is as follows: Figure 2 shown.

[0076] In the specific implementation, refer to Figure 3 , the information processing module is responsible for receiving and processing the voice or text input from the user, and generating the corresponding text or instruction information. Specifically, the input from the user side includes two types, voice input and text input. For voice input, an audio recording switch will be set on the user side interface to control the start, pause and end of audio recording. After the user's audio input is obtained through the acquisition device, the text content is extracted through the voice recognition algorithm; for text input, the text is directly passed to the instruction discrimination stage after acquisition. The purpose of the instruction discrimination stage is mainly to identify whether the text contains specific control instructions, such as: "Please speak louder", "Speak slower", "Stop talking", etc. For the case where a control instruction is detected in the instruction discrimination, the corresponding control instruction is generated and passed to the reply generation module together with the text; for the case where there is no control instruction, the text is directly passed to the reply generation module.

[0077] In the specific implementation, the answer generation module is responsible for receiving the text and control instructions from the information processing module and generating an answer in audio form. The processing flow of this module is as follows: Figure 4As shown, the reply text is generated by calling the language model, and the user's speech in text form is passed in and the reply text is obtained. The reply audio is generated by calling the TTS technology. For control instructions containing audio control instructions, such as adjusting the volume, speaking speed, pitch, timbre, etc., the corresponding request parameters are updated first. Subsequently, the updated audio parameters and the reply text are used together to generate the reply audio through TTS. The module output content includes the generated reply audio and control instructions, which are passed together to the digital human generation module.

[0078] In the specific implementation, the digital human generation module is responsible for receiving the reply audio and control instructions passed in by the reply generation module, generating a digital human video clip with lip shape matching the audio, and transmitting it to the remote server. The processing flow of this module is as follows Figure 5 As shown, the specific digital human generation process refers to the subsequent steps.

[0079] In a specific implementation, the step of determining the reply audio and the digital human image corresponding to the reply audio includes:

[0080] Based on the user input instruction, the reply audio is determined; the audio features corresponding to each frame of the reply audio are extracted; the audio features are divided according to a preset time length to obtain divided audio features, so that each frame of audio includes a preset number of continuous audio features; based on the divided audio features, a corresponding digital human lip image is generated, and based on a preset digital human base map and the digital human lip image, a digital human image corresponding to the reply audio is generated.

[0081] In a specific implementation, the user inputs a command or question through voice or text, and the device parses the user input and generates a corresponding audio response. The device analyzes the generated response audio and extracts the audio features of each frame (or time period). These features include Mel-frequency cepstral coefficients (MFCCs), Mel-frequency spectrum energy features, etc., which can represent the key information of the audio signal.

[0082] Furthermore, the device divides the extracted audio features according to a preset time length (preferably 40 milliseconds) to ensure that each frame of audio contains a certain number of continuous audio features. This is done to ensure that the features of each frame of audio are complete and can be used for subsequent lip shape generation.

[0083] In a specific implementation, the device uses the divided audio features to generate a digital human lip shape image corresponding to these features. The device combines the generated lip shape image with the preset digital human base map (i.e., the digital human facial model, excluding lip shape changes) to generate the final digital human image. This image will show the digital human's facial expression and lip shape when speaking, synchronized with the reply audio.

[0084] In the specific implementation, in order to improve the computational efficiency and avoid repeated calculations, the device adopts the method of selecting sliding windows when extracting audio features. Audio feature extraction mainly includes two steps, audio feature pre-extraction and time window feature selection. In the pre-extraction stage, the entire audio data is input into the HuBERT feature extractor to obtain the features of the entire audio.

[0085] For example, for the image of the xth frame, the features corresponding to the time window from the x-1th frame to the x+7th frame are selected as the audio feature input for the lip shape generation stage of the xth frame. By selecting the time window, it is avoided that the audio features of the 9 frames before and after need to be calculated to obtain the audio features of each frame, which reduces repeated calculations and improves calculation efficiency.

[0086] In the specific implementation, the input of the digital human lip shape generation stage is the extracted and divided audio features, which generates the digital human lip shape image at that moment. The generation network is based on DINet. In order to speed up the calculation process, the data and network model parameters are both in half precision during the inference process, which improves the calculation efficiency without significantly affecting the lip shape effect.

[0087] In a specific implementation, the step of dividing the audio features according to a preset time length to obtain the divided audio features includes:

[0088] The first feature and the last feature in the audio feature are copied respectively to obtain a corresponding number of first copy features and last copy features; the first copy feature is expanded to before the first feature, and the last copy feature is expanded to after the last feature to obtain the audio feature after edge expansion; the audio feature after edge expansion is divided according to a preset time length to obtain the divided audio feature.

[0089] From the extracted audio features, the first feature (i.e., the very beginning part) and the tail feature (i.e., the very end part) are copied respectively, and the corresponding number of the first feature and the tail feature are copied as needed to obtain the first copy feature and the tail copy feature. The device expands the first copy feature to before the first feature, that is, adds the copied first feature to the beginning position of the original audio feature sequence; similarly, expands the tail copy feature to after the tail feature, that is, adds the copied tail feature to the end position of the original audio feature sequence. The purpose of this is to create a certain buffer area at the beginning and end of the audio feature sequence, which may be necessary in some audio processing tasks, such as providing a smoother transition when generating digital human lip images.

[0090] In a specific implementation, the device divides the audio features after edge expansion according to a preset time length (for example, 40 milliseconds per frame) to obtain a series of divided audio features. This division is usually used for subsequent feature analysis. For example, in speech recognition or audio synthesis, each frame feature can correspond to a short time segment of the speech signal. That is, the purpose of the above process is to add additional feature data at the beginning and end of the audio feature sequence, which helps to improve the performance of the audio processing algorithm, especially when it is necessary to consider the smooth transition of the starting and ending parts of the audio signal. In this way, it can be ensured that each frame of audio contains a preset number of continuous audio features.

[0091] In the specific implementation, refer to Figure 6 In this embodiment, the entire audio feature is first filled by edge extension, 1 frame is filled forward, and 7 frames are filled backward to obtain the filled audio feature. It can be understood that the number of filled frames can be set by oneself, and the sum of the number of filled frames forward and the number of filled frames backward is equal to the preset number of frames - 1, and the 1 frame is 1 frame of the current frame.

[0092] In a specific implementation, the step of generating a digital human image corresponding to the reply audio based on a preset digital human base map and the digital human lip shape image comprises:

[0093] Obtain the lip position information of the digital human; based on the lip position information, replace the lip-shaped corresponding area in the preset digital human base map with the digital human lip image to obtain the digital human image corresponding to the reply audio, wherein the digital human base map is preloaded into the memory.

[0094] In the specific implementation, in order to increase the speed of image generation, this application adopts the method of pre-loading the digital human base map into the memory. After obtaining the driven digital human lip image in the generation stage, the lip corresponding area of ​​the digital human base map can be replaced with the driven lip image according to the pre-saved lip position information, so as to obtain the full-body image of the digital human at the current moment more quickly.

[0095] Step S200, based on a preset time frame sequence, each frame of reply audio is transmitted to an audio pipeline through a preset audio writing process, and each frame of digital human image is transmitted to an image pipeline through a preset image writing process;

[0096] It should be noted that the time frame sequence is a predefined time sequence used to synchronize audio and image data. Each frame of audio and image should be processed and displayed at a specific time point in this time sequence.

[0097] In a specific implementation, the audio writing process is a process for processing audio data, which writes each frame of reply audio (i.e., the audio generated by the system in response to the user) into an audio pipe in a preset time frame order, wherein the audio pipe is a channel for transmitting audio data. Similarly, the picture writing process is responsible for writing each frame of the digital human image (i.e., the visual representation of the digital human synchronized with the reply audio) into the picture pipe in a preset time frame order, wherein the picture pipe is responsible for transmitting image data. The audio pipe and the picture pipe are two independent transmission channels, respectively used for the transmission of audio and image data. In practical applications, these pipes need to work synchronously to ensure that the playback of audio and images is synchronized. The device sends the processed audio frames and image frames into their respective pipes in a predetermined time sequence for further processing or direct output to the terminal device.

[0098] Step S300, based on the time frame sequence and the preset video encoding process, perform video encoding on each frame of the reply audio in the audio pipeline and each frame of the digital human image in the picture pipeline to obtain the target digital human video, wherein the video encoding process, the audio writing process and the picture writing process are executed in parallel according to the time frame sequence;

[0099] In specific implementation, the video encoding process is a process that combines audio and image data and encodes them into a video file. Video encoding is the process of compressing video signals into a specific format (such as H.264, H.265, etc.) for easy storage and transmission.

[0100] In a specific implementation, the video encoding process, the audio writing process, and the image writing process are performed simultaneously, which means that they all work in the same time frame sequence, but are executed independently. This parallel processing can improve efficiency because it allows audio and image data to be processed simultaneously instead of one after another. The device merges the reply audio in the audio pipeline and the digital human image in the image pipeline through the video encoding process, and encodes them into the final video file, namely the target digital human video. This video file contains synchronized audio and image data and can be used for playback or further processing.

[0101] In the specific implementation, the input of the video encoding and transmission stage is the generated digital human image and the reply audio, which are encoded and transmitted to the remote client. In order to improve the encoding efficiency and achieve real-time generation effect, Figure 7 As shown in Figure 2, a video generation solution based on multi-process and pipeline is designed. Figure 7As shown, three pipelines are designed for the data encoding process: data pipeline, picture pipeline and audio pipeline. The data pipeline is used to receive the digital human image and audio data generated by the generative model, and then the picture and audio clip data corresponding to each frame are respectively passed into the picture pipeline and audio pipeline. This design can maintain the synchronization of pictures and audio, and avoid excessive backlog of single-mode data in the picture / audio pipeline. Subsequently, the picture writing subprocess is responsible for converting the digital human image obtained from the generative model into binary data and writing it into the picture pipeline. To avoid pipeline data backlog, the time interval for writing pictures into the pipeline is close to the duration of each frame of the video (40 milliseconds). The audio writing subprocess is responsible for segmenting the audio data according to the duration of each frame of the video, converting the audio data corresponding to each frame into binary form and writing it into the audio pipeline. The video encoding subprocess receives binary data from the picture pipeline and the audio pipeline, and encodes it into several video clips (usually 8 seconds in length) through ffmpeg. In order to improve the efficiency of video encoding, this application sets the value of the encoder parameter "-preset" to "ultrafast", sets the key frame interval to 25 frames, and sets the quality factor crf to 18. After each video segment is generated, the server sends the name of the video segment and a list of download addresses to the client, and the client can select the video segment to download and play as needed.

[0102] In a specific implementation, the above process describes an efficient multimedia processing flow in which audio and image data are processed in parallel and merged and encoded into a video in a preset time frame order. This approach ensures that the audio and image parts of the video remain synchronized, providing a smooth and natural viewing experience.

[0103] Step S400: Play the target digital human video to achieve interaction between the digital human and the user.

[0104] In a specific implementation, the device plays the target digital human video on a suitable device, such as a computer, a smart phone, a tablet computer, or a dedicated display screen. The playback process involves decoding the video file and displaying the image at a suitable frame rate, while playing the audio synchronized with it.

[0105] In a specific implementation, the device plays a video of a digital human, and the user can see the digital human's facial expressions, lip shape, and body language, and hear the digital human's voice at the same time. This combination of vision and hearing can make the user feel that the digital human is real, thus achieving a more natural human-computer interaction. The interaction is two-way, and the digital human can receive the user's input (such as voice, text, or action) and adjust the digital human's behavior accordingly.

[0106] In the related art, the text / voice input by the user is usually input into a large language model to obtain the corresponding text answer, and then the audio is generated from the text through the TTS technology, and the digital human is driven by the audio. However, this method has the problem that the digital human is difficult to drive in real time and the digital human video will freeze. Compared with the above, the present application responds to the user input instruction, determines the reply audio and the digital human image corresponding to the reply audio; based on the preset time frame sequence, each frame of reply audio is passed to the audio pipeline through the preset audio writing process, and each frame of digital human image is passed to the picture pipeline through the preset picture writing process; based on the time frame sequence and the preset video encoding process, each frame of reply audio in the audio pipeline and each frame of digital human image in the picture pipeline are video encoded to obtain the target digital human video, wherein the video encoding process, the audio writing process and the picture writing process are executed in parallel according to the time frame sequence; the target digital human video is played to realize the interaction between the digital human and the user. It is understandable that the present application proposes to execute the video encoding process, the audio writing process and the picture writing process in parallel according to the time frame sequence. On the one hand, the synchronization of the picture and the audio is maintained to avoid excessive backlog of single-modal data in the picture pipeline or the audio pipeline. On the other hand, the synchronous execution is performed according to the time frame sequence to realize real-time generation of video clips at a fast speed, thereby improving the smoothness of the digital human video, and further realizing the real-time driving of the digital human and the smoothness of the digital human video.

[0107] Based on the above first embodiment, the present application also proposes another embodiment. Referring to the figure, the 2D digital human real-time interaction method includes:

[0108] Step A100, during the interaction between the digital human and the user, determining a first video of the digital human in a first state and a second video of the digital human in a second state, wherein the first state refers to an initial silent state of the digital human, and the second state refers to a listening state of the digital human or a replying state of the digital human;

[0109] It should be noted that the first video of the digital human in the first state refers to the video of the digital human in the initial state, i.e., the "first state". In this state, the digital human is in silence, without any verbal communication, and may only display a static image or a simple animation, indicating that the digital human is waiting for the user's input or instructions. The second video of the digital human in the second state refers to the video of the digital human in the second state, i.e., the "second state". This state can be the listening state or the reply state of the digital human. In the listening state, the digital human may display some animations or expressions, indicating that it is listening to the user's speech attentively. In the reply state, the digital human shows that it is responding to the user's input, which may include the animation of speaking and the corresponding audio.

[0110] In the specific implementation, during the interaction, the device needs to determine which video should be played based on the user's input and the digital human's response. This means that the device needs to be able to recognize the user's instructions and select appropriate video content based on these instructions and the digital human's current state. The state transition of the digital human is based on the user's behavior and the device's feedback. For example, when the user starts talking, the digital human may switch from the initial silent state to the listening state; when the digital human needs to respond to the user, it may switch to the reply state. The entire interactive process is a dynamic and continuous interaction, and the state of the digital human and the corresponding video content will continue to change according to the user's behavior and the system's processing results.

[0111] In the specific implementation, the digital human states in this application mainly include three types: silent, speaking, and listening. The silent state means that the digital human's body and head are basically still, with only slight changes in breathing and natural blinking of eyes, corresponding to the digital human being in the initial state. The speaking state includes the regular speaking state and the action state: in the regular speaking state, the digital human is in a natural speaking state, with rhythmic head movements and natural facial expressions; in the action state, the digital human will complete the corresponding actions, usually while the digital human is making movements, he is also speaking (keeping the lip shape matching the audio), and when there is no voice in the audio, he can just make movements without speaking. The listening state is the state in which the digital human listens to the user's speech, with slight rhythmic head movements, a smile on the face, and eyes looking at the camera, which is used for the stage of waiting for user input.

[0112] In the specific implementation, the above description emphasizes that at different stages of the interaction between the digital human and the user, different video content needs to be prepared and used to reflect the different states of the digital human, thereby providing a richer and more natural interactive experience. This design enables the digital human to communicate with the user in a more humane way, enhancing the realism and fluency of the interaction.

[0113] Step A200, determining a corresponding initial state frame based on the first video;

[0114] In a specific implementation, the device determines a specific frame based on the first video (i.e., the initial silent state video of the digital human), called the "initial state frame". This frame represents the initial state of the digital human, usually the first frame at the beginning of the video, showing the digital human in a silent waiting state.

[0115] Step A300, adding the initial state frame to the front of the first frame and the back of the last frame of each digital human video to obtain a modified digital human video, wherein the digital human video includes the first video and the second video;

[0116] In the specific implementation, the determined initial state frames are inserted before the first frame and after the last frame of each digital human video sequence. This means that the initial state screen of the digital human is added at the beginning and end of the video sequence. The digital human video includes the first video (initial silent state) and the second video (listening state or reply state).

[0117] In a specific implementation, the device adds the initial state frame before the first frame and after the last frame of each digital human video to obtain the modified digital human video, including:

[0118] Based on the initial state frame, a corresponding intermediate frame is generated through a frame insertion algorithm; the initial state frame is added before the first frame and after the last frame of each digital human video, and the intermediate frame is added between the first frame and the initial state frame of each digital human video, and between the last frame and the initial state frame, to obtain a modified digital human video.

[0119] In a specific implementation, the device generates a series of intermediate frames based on the initial frame through an interpolation algorithm. The purpose of the interpolation algorithm is to create new frames between two key frames to increase the frame rate of the video and make the action look smoother.

[0120] In a specific implementation, the device inserts the initial state frame before the first frame and after the last frame of each digital human video sequence. This can provide a consistent start and end screen for the video sequence. Secondly, the device adds intermediate frames to the digital human video, that is, the generated intermediate frames are inserted between the first frame and the initial state frame of the digital human video, and between the last frame and the initial state frame. The purpose of this is to increase the smoothness of the video, especially when the digital human state changes, such as from a silent state to a listening or answering state.

[0121] In a specific implementation, the present application extracts corresponding silence, speech, and listening segments from the original video obtained by shooting, and processes the original video as follows: Fig. 9 First, the device adds an initial frame before the first frame and after the last frame of each original video, and adds several intermediate frames between the first and last frames of the original video and the initial frame through the PerVFI-based interpolation algorithm, so as to achieve a smooth transition of the video screen. Fig.10 As shown in the figure, the first and last frames of different original videos that have not been smoothed are inconsistent, and direct splicing will cause picture jumps. After adding transition frames, the start and end frames of different digital human base map videos are exactly the same, ensuring that after splicing different video clips, the digital human posture and position can remain consistent, and the picture is natural without jumps.

[0122] Step A400, splicing the modified digital human videos to obtain a spliced ​​digital human video, and playing the spliced ​​digital human video to achieve digital human state switching.

[0123] In a specific implementation, after the above processing, the device obtains a modified digital human video, which adds initial frames at the beginning and end positions based on the original video sequence. Such processing can provide consistent starting and ending pictures for the video sequence, enhancing the integrity and coherence of the video.

[0124] In a specific implementation, if the digital human video sequence includes multiple state-changing videos (such as initial silence, listening, answering, etc.), then inserting the initial state frame before and after the beginning and end of each video can make the entire video sequence more visually unified, and the user can clearly identify the beginning and end of the video when watching. This processing is a common technology when making digital human videos or performing video post-editing, which helps to improve the overall quality of the video and the viewing experience.

[0125] In the specific implementation, Fig.11 As shown, the digital human will play a silent video in the initial state. In order to avoid the frequent repetition of the digital human body movements affecting the user experience, three different silent video clips will be played in a loop in the initial state. The starting and ending frames of different clips are consistent, and the pictures can transition naturally. The length of each video is 8 seconds. In the case of user audio input, when the user presses the record button, the digital human will switch to the listening state; in the case of user text input, the digital human will switch to the listening state when the front-end page detects that the user starts typing text. In order to maintain the continuity of the digital human body posture, the user end will switch to the listening video after the current silent video ends. The listening video includes three different video clips, each of which is 8 seconds long. The three video clips will be played in a loop when the digital human remains in the listening state.

[0126] Furthermore, when the user finishes voice recording or completes text input, the digital human remains in a listening state, but a pre-recorded video will be played. The pre-recorded video is a digital human video that has been produced in advance. Its main function is to inform the user that the message has been received, reduce the user's waiting time, and improve the user experience. There are three types of pre-recorded videos, and the speech content includes: "Please wait a moment, I will think about how to reply you", "I am using the large language model to generate the reply content, please wait patiently", "Let me think about how to reply you, please wait patiently".

[0127] In the specific implementation, after receiving the first reply video, the digital human will switch to the speaking state. In the speaking state, the digital human will continue to play the speaking video until the speech content is finished and then switch back to the listening state. The speaking video includes several different background videos. The background videos include two types: regular speaking videos and action videos, and the video length is 8 seconds. By default, the regular speaking video will be used, and the digital human is in a natural speaking state without obvious body movements; if the corresponding action is triggered in the instruction, the action video will be used, and the actions include "greeting", "like", "goodbye", etc.

[0128] In the specific implementation, if the normal reply video is played, the digital human will switch back to the listening state after the playback is completed. Fig.12 As shown in the figure, the digital human will switch to the listening state after being interrupted and wait for the user's input. To maintain the continuity of the picture, the listening video will be played after the current speaking video clip is played. After the digital human finishes speaking, it will remain in the listening state while waiting for the next user input until the user starts the next input. If the user does not receive the command to start input after more than 24 seconds, it will switch to the silent state and remain until the user starts inputting.

[0129] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the 2D digital human real-time interaction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0130] This application also provides a 2D digital human real-time interactive device, please refer to Fig.13 , the 2D digital human real-time interactive device comprises:

[0131] A determination module 10, configured to determine a reply audio and a digital human image corresponding to the reply audio in response to a user input instruction;

[0132] The writing module 20 is used to transfer each frame of the reply audio to the audio pipeline through a preset audio writing process based on a preset time frame sequence, and to transfer each frame of the digital human image to the image pipeline through a preset image writing process;

[0133] The encoding module 30 is used to perform video encoding on each frame of the reply audio in the audio pipeline and each frame of the digital human image in the picture pipeline based on the time frame sequence and a preset video encoding process to obtain a target digital human video, wherein the video encoding process, the audio writing process and the picture writing process are executed in parallel according to the time frame sequence;

[0134] The playing module 40 is used to play the target digital human video to realize the interaction between the digital human and the user.

[0135] Optionally, the determining module 10 includes:

[0136] An audio determination module, used to determine a reply audio based on the user input instruction;

[0137] An extraction module, used for extracting audio features corresponding to the reply audio of each frame;

[0138] A division module, used to divide the audio features according to a preset time length to obtain divided audio features, so that each frame of audio includes a preset number of continuous audio features;

[0139] A generation module is used to generate a corresponding digital human lip image based on the divided audio features, and to generate a digital human image corresponding to the reply audio based on a preset digital human base map and the digital human lip image.

[0140] Optionally, the division module includes:

[0141] A copy module, used to copy the first feature and the last feature in the audio feature respectively to obtain a corresponding number of first copy features and last copy features;

[0142] An expansion module, used to expand the first copy feature to before the first feature, and to expand the tail copy feature to after the tail feature, to obtain an audio feature after edge expansion;

[0143] The feature division module is used to divide the audio features after edge expansion according to a preset time length to obtain divided audio features.

[0144] Optionally, the generating module includes:

[0145] An acquisition module is used to acquire the lip position information of the digital human;

[0146] A replacement module is used to replace the lip shape corresponding area in the preset digital human base map with the digital human lip shape image based on the lip shape position information, so as to obtain the digital human image corresponding to the reply audio, wherein the digital human base map is preloaded into the memory.

[0147] Optionally, the 2D digital human real-time interaction device further includes:

[0148] A video determination module, used to determine a first video of the digital human in a first state and a second video of the digital human in a second state during the interaction between the digital human and the user, wherein the first state refers to an initial silent state of the digital human, and the second state refers to a listening state of the digital human or a replying state of the digital human;

[0149] An initial state frame determination module, used to determine a corresponding initial state frame based on the first video;

[0150] An adding module, used for adding the initial frame to the front of the first frame and the back of the last frame of each digital human video to obtain a modified digital human video, wherein the digital human video includes the first video and the second video;

[0151] The splicing module is used to splice the modified digital human videos to obtain a spliced ​​digital human video, and play the spliced ​​digital human video to achieve digital human state switching.

[0152] Optionally, the joining module includes:

[0153] An intermediate frame generation module, used to generate a corresponding intermediate frame based on the initial frame through a frame insertion algorithm;

[0154] The video frame adding module is used to add the initial frame before the first frame and after the last frame of each digital human video, and add the intermediate frame between the first frame and the initial frame, and between the last frame and the initial frame of each digital human video to obtain the modified digital human video.

[0155] The 2D digital human real-time interaction device provided by the present application adopts the 2D digital human real-time interaction method in the above embodiment, which can solve the technical problem of 2D digital human real-time interaction. Compared with the prior art, the beneficial effects of the 2D digital human real-time interaction device provided by the present application are the same as the beneficial effects of the 2D digital human real-time interaction method provided by the above embodiment, and the other technical features of the 2D digital human real-time interaction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0156] The present application provides a 2D digital human real-time interaction device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the 2D digital human real-time interaction method in the above-mentioned embodiment one.

[0157] Reference below Fig.14, which shows a schematic diagram of the structure of a 2D digital human real-time interactive device suitable for implementing the embodiment of the present application. The 2D digital human real-time interactive device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.14 The 2D digital human real-time interactive device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0158] like Fig.14 As shown, the 2D digital human real-time interactive device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in a read-only memory (ROM: Read Only Memory) 1002 or the program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the 2D digital human real-time interactive device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the 2D digital human real-time interactive device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a 2D digital human real-time interactive device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0159] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0160] The 2D digital human real-time interaction device provided by the present application adopts the 2D digital human real-time interaction method in the above embodiment, which can solve the technical problem of 2D digital human real-time interaction. Compared with the prior art, the beneficial effects of the 2D digital human real-time interaction device provided by the present application are the same as the beneficial effects of the 2D digital human real-time interaction method provided by the above embodiment, and the other technical features of the 2D digital human real-time interaction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0161] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0162] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0163] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the 2D digital human real-time interaction method in the above-mentioned embodiment.

[0164] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0165] The computer-readable storage medium may be included in the 2D digital human real-time interactive device; or may exist independently without being assembled into the 2D digital human real-time interactive device.

[0166] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the 2D digital human real-time interactive device, the 2D digital human real-time interactive device enables the 2D digital human to interact in real time.

[0167] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0170] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned 2D digital human real-time interaction method, and can solve the technical problem of 2D digital human real-time interaction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the 2D digital human real-time interaction method provided in the above-mentioned embodiment, and will not be elaborated here.

[0171] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned 2D digital human real-time interaction method when executed by a processor.

[0172] The computer program product provided by this application can solve the technical problem of real-time interaction of 2D digital humans. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the real-time interaction method of 2D digital humans provided by the above embodiment, which will not be repeated here.

[0173] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A 2D digital human real-time interaction method, characterized in that: The method for real-time interaction of 2D digital humans comprises: In response to a user input instruction, determining a reply audio and a digital human image corresponding to the reply audio; Based on a preset time frame sequence, each frame of the response audio is transmitted to the audio pipeline through a preset audio writing process, and each frame of the digital human image is transmitted to the image pipeline through a preset image writing process; Based on the time frame sequence and the preset video encoding process, each frame of the reply audio in the audio pipeline and each frame of the digital human image in the picture pipeline are video encoded to obtain the target digital human video, wherein the video encoding process, the audio writing process and the picture writing process are executed in parallel according to the time frame sequence; The target digital human video is played to realize the interaction between the digital human and the user.

2. The 2D digital human real-time interaction method according to claim 1, characterized in that: The step of determining the reply audio and the digital human image corresponding to the reply audio comprises: Determining a reply audio based on the user input instruction; Extracting audio features corresponding to the reply audio of each frame; Dividing the audio features according to a preset time length to obtain divided audio features, so that each frame of audio includes a preset number of continuous audio features; Based on the divided audio features, a corresponding digital human lip image is generated, and based on a preset digital human base map and the digital human lip image, a digital human image corresponding to the reply audio is generated.

3. The 2D digital human real-time interaction method according to claim 2, characterized in that: The step of dividing the audio features according to the preset time length to obtain the divided audio features includes: Respectively copying the first feature and the last feature in the audio feature to obtain a corresponding number of first copy features and last copy features; Expanding the first copy feature to before the first feature, and expanding the last copy feature to after the last feature, to obtain an audio feature after edge expansion; The audio features after edge expansion are divided according to a preset time length to obtain divided audio features.

4. The 2D digital human real-time interaction method according to claim 2, characterized in that: The step of generating the digital human image corresponding to the reply audio based on the preset digital human base map and the digital human lip shape image comprises: Obtain the lip position information of the digital person; Based on the lip position information, the lip-shaped corresponding area in the preset digital human base map is replaced with the digital human lip image to obtain the digital human image corresponding to the reply audio, wherein the digital human base map is preloaded into the memory.

5. The 2D digital human real-time interaction method according to claim 1, characterized in that: The method for real-time interaction of 2D digital humans comprises: During the interaction between the digital human and the user, a first video of the digital human in a first state and a second video of the digital human in a second state are determined, wherein the first state refers to the initial silent state of the digital human, and the second state refers to the listening state of the digital human or the answering state of the digital human; Based on the first video, determining a corresponding initial state frame; Adding the initial state frame to the front of the first frame and the back of the last frame of each digital human video to obtain a modified digital human video, wherein the digital human video includes the first video and the second video; The modified digital human videos are spliced ​​to obtain a spliced ​​digital human video, and the spliced ​​digital human video is played to achieve digital human state switching.

6. The 2D digital human real-time interaction method according to claim 5, characterized in that: The step of adding the initial state frame to the front of the first frame and the back of the last frame of each digital human video to obtain a modified digital human video comprises: Based on the initial frame, generating a corresponding intermediate frame through a frame insertion algorithm; The initial state frame is added before the first frame and after the last frame of each digital human video, and the intermediate frame is added between the first frame and the initial state frame, and between the last frame and the initial state frame of each digital human video to obtain a modified digital human video.

7. A 2D digital human real-time interactive device, characterized in that: The device comprises: A determination module, configured to determine a reply audio and a digital human image corresponding to the reply audio in response to a user input instruction; A writing module, used for transferring each frame of reply audio to an audio pipeline through a preset audio writing process based on a preset time frame sequence, and transferring each frame of digital human image to an image pipeline through a preset image writing process; An encoding module, used for performing video encoding on each frame of reply audio in the audio pipeline and each frame of digital human image in the picture pipeline based on the time frame sequence and a preset video encoding process, so as to obtain a target digital human video, wherein the video encoding process, the audio writing process and the picture writing process are executed in parallel according to the time frame sequence; The playing module is used to play the target digital human video to realize the interaction between the digital human and the user.

8. A 2D digital human real-time interactive device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the 2D digital human real-time interaction method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the 2D digital human real-time interaction method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the 2D digital human real-time interaction method according to any one of claims 1 to 6 are implemented.

Images