Digital human interaction method, device, electronic device and storage medium

Abstract

The present disclosure provides a digital human interaction method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology. The digital human interaction method includes: initializing a first task branch, a second task branch and a third task branch; under the first task branch, generating a video background of an interactive video frame based on a preset video, wherein the interactive video frame is a video frame in which the digital human interacts with the user; under the second task branch, extracting facial feature information of the digital human from the preset video; under the third task branch, generating target audio data for the digital human to respond to the interactive question based on the interactive question input by the user; generating the interactive video frame based on the video background, the facial feature information and the target audio data, so that the digital human and the user can interact with each other through video. The present disclosure generates interactive video frames in a parallel processing manner, thereby improving efficiency.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a digital human interaction method, device, electronic device and storage medium. Background Art

[0002] Digital human refers to a virtual character image generated by computer graphics, artificial intelligence, natural language processing, deep learning and other technologies, which can imitate the appearance, voice, expression and body movements of real humans and interact with users. With the development of technologies such as the Internet, digital humans have evolved from simple static characters to virtual characters with real-time interactive capabilities. Real-time interaction means that digital humans can interact with users seamlessly and naturally, and can provide corresponding feedback in real time based on users' voice, text and other signals.

[0003] In related technologies, the real-time interaction of digital human systems relies on the computing power of remote servers for image rendering and voice generation, and is usually executed based on serial logic. The utilization rate of system resources and execution efficiency are low, and it is easy to cause network delays, thus affecting the smoothness of interaction and user immersion. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure propose a digital human interaction method, device, electronic device and storage medium to solve the problems existing in the related technology that the utilization rate and execution efficiency of system resources are low, and network delays are easily caused, thereby affecting the interaction smoothness and user immersion.

[0005] According to a first aspect of the present disclosure, a digital human interaction method is provided, comprising: initializing a first task branch, a second task branch, and a third task branch; under the first task branch, generating a video background of an interactive video frame based on a preset video, wherein the interactive video frame is a video frame in which the digital human interacts with a user; under the second task branch, extracting facial feature information of the digital human from the preset video; under the third task branch, generating target audio data for the digital human to respond to the interactive question based on the interactive question input by the user; and generating the interactive video frame based on the video background, the facial feature information, and the target audio data, so that the digital human can interact with the user through video.

[0006] According to a second aspect of the present disclosure, a digital human interaction device is provided, comprising: an initialization module, used to initialize a first task branch, a second task branch and a third task branch; a processing module, used to generate a video background of an interactive video frame based on a preset video under the first task branch, wherein the interactive video frame is a video frame in which the digital human interacts with a user; the processing module is also used to extract facial feature information of the digital human from the preset video under the second task branch; the processing module is also used to generate target audio data for the digital human to respond to the interactive question based on the interactive question input by the user under the third task branch; an interaction module is used to generate an interactive video frame of the digital human based on the video background, the facial feature information and the target audio data, so that the digital human can interact with the user through video.

[0007] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to execute the above-mentioned digital human interaction method.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned digital human interaction method.

[0009] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0010] In the digital human interaction method provided by the exemplary implementation of the present disclosure, the first task branch, the second task branch and the third task branch are initialized; under the first task branch, the video background of the interactive video frame is generated based on the preset video, wherein the interactive video frame is a video frame in which the digital human interacts with the user; under the second task branch, the facial feature information of the digital human is extracted from the preset video; under the third task branch, the target audio data for the digital human to respond to the interactive question is generated based on the interactive question input by the user; the interactive video frame of the digital human is correspondingly generated based on the video background, the facial feature information and the target audio data, so that the digital human and the user can interact with each other through video. The embodiment of the present disclosure adopts a multi-process processing method, initializes the three task branches of the first task branch, the second task branch and the third task branch, and performs video preprocessing based on the first task branch, performs preprocessing of the digital human based on the second task branch, and generates audio data based on the third task branch. By executing each task branch in parallel, the utilization rate and execution efficiency of system resources are improved. In addition, the embodiment of the present disclosure synthesizes the interactive video frame based on the processing results of the above task branches, so that the user can achieve real-time interaction with the digital human by obtaining the interactive video frame, thereby improving the interaction fluency and user immersion.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0013] Figure 1 A flow chart of a digital human interaction method according to an exemplary embodiment of the present disclosure is shown;

[0014] Figure 2 A flow chart showing another digital human interaction method according to an exemplary embodiment of the present disclosure is shown;

[0015] Figure 3 A schematic block diagram of a digital human interaction device according to an exemplary embodiment of the present disclosure is shown;

[0016] Figure 4 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0019] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The embodiments of the present disclosure provide a digital human interaction method, device, electronic device and storage medium. The embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0023] The present disclosure first provides a digital human interaction method. Figure 1 As shown, the digital human interaction method specifically includes the following steps:

[0024] S110: Initializing the first task branch, the second task branch, and the third task branch;

[0025] S120: Under the first task branch, a video background of an interactive video frame is generated based on a preset video, wherein the interactive video frame is a video frame in which the digital human interacts with the user;

[0026] S130: Under the second task branch, extracting facial feature information of the digital human from the preset video;

[0027] S140: Under the third task branch, based on the interactive question input by the user, generating target audio data for the digital human to respond to the interactive question;

[0028] S150: Generate interactive video frames of the digital human based on the video background, facial feature information and target audio data, so that the digital human and the user can interact through video.

[0029] The digital human interaction method provided in the embodiment of the present disclosure adopts a multi-process processing method, initializes three task branches, namely the first task branch, the second task branch and the third task branch, performs video preprocessing based on the first task branch, performs digital human preprocessing based on the second task branch, generates audio data based on the third task branch, and improves the utilization rate and execution efficiency of system resources by executing each task branch in parallel. In addition, the embodiment of the present disclosure synthesizes interactive video frames based on the processing results of the above-mentioned task branches, so that users can achieve real-time interaction with digital humans by obtaining the interactive video frames, thereby improving the interaction fluency and user immersion.

[0030] Next, in another embodiment, the above steps are described in more detail.

[0031] In step S110 , the first task branch, the second task branch, and the third task branch are initialized.

[0032] The digital human interaction method provided in the embodiment of the present application is used to realize the real-time interaction between the digital human and the user. For example, the user can input questions to the digital human interaction system, and the digital human interaction system displays the answers to the questions to the user in the form of digital human voice broadcasting video by executing the above digital human interaction method. Furthermore, the above digital human interaction system can also detect in real time whether the user inputs questions, and when it is detected that the user has input questions, repeat the above process to continuously output answers to the user in the form of oral video broadcasting, thereby realizing real-time interaction between the digital human and the user.

[0033] In the embodiments of the present application, the digital human refers to a virtual character image generated by computer graphics, artificial intelligence, natural language processing, deep learning and other technologies, which can imitate the appearance, voice, expression and body movements of real humans and interact with users. The digital human can be a simple static virtual image or a digital image with real-time interaction and high intelligence.

[0034] In the embodiment of the present application, the above-mentioned real-time interaction between the digital human and the user means that the digital human can interact with the user seamlessly and naturally, and can make corresponding feedback in real time according to the user's voice, text and other signals.

[0035] In an embodiment of the present application, the first task branch is used to generate a video background of an interactive video frame, wherein the interactive video frame is a video frame in which a digital human interacts with a user in real time, and the video background is the video background of the interactive video frame.

[0036] In the embodiment of the present application, the initialization process of the first task branch can be specifically implemented as follows: initializing a preset video, wherein the preset video is video data containing a digital human, and the initialization process is to load each frame of the preset video in advance. Exemplarily, the preset video can be obtained by the user by selecting from the existing templates in the system, and the embodiment of the present application does not specifically limit the method for obtaining the preset video.

[0037] In an embodiment of the present application, the second task branch is used to read the facial feature information of the digital person in the preset video to obtain a preprocessing file.

[0038] In an embodiment of the present application, exemplarily, the initialization process of the above-mentioned second task branch can be specifically implemented as follows: the face position of the digital human in the above-mentioned preset video can be determined by face detection technology; the face of the digital human is aligned, and the digital human face image is standardized through operations such as rotation, scaling and translation; the facial feature information in the digital human face image is read to obtain the above-mentioned preprocessing file.

[0039] In the embodiment of the present application, the third task branch is used to generate answers to interactive questions input by the user, wherein the interactive questions are questions that the user expects the digital human to answer.

[0040] In an embodiment of the present application, the initialization process of the above-mentioned third task branch can be specifically implemented as follows: through initialization, the above-mentioned digital human interaction system can continuously monitor whether the user has a request to access, that is, to detect in real time whether the user inputs an interactive question to the system, so that when a user request is detected, the digital human interaction method of the embodiment of the present application is executed to enable the digital human to answer the interactive question input by the user in the form of a spoken video.

[0041] In step S120, under the first task branch, a video background of an interactive video frame is generated based on a preset video, wherein the interactive video frame is a video frame in which a digital human interacts with a user.

[0042] In the embodiment of the present application, the interactive video frame is a video frame of the digital human interacting with the user in real time. By continuously generating interactive video frames, the digital human can eventually interact with the user in real time in the form of video.

[0043] In an embodiment of the present application, the above-mentioned generation of the video background of the interactive video frame based on the preset video can, exemplarily, be implemented as follows: after the preloading of the preset video is completed through the initialization process of the above-mentioned first task branch, the frame data corresponding to the current interactive video frame is extracted from the preset video, and the current frame data is standardized to obtain the video background of the above-mentioned interactive video frame.

[0044] In step S130, under the second task branch, facial feature information of the digital human is extracted from the preset video.

[0045] In the embodiment of the present application, the above-mentioned facial feature information is information used to identify and describe facial features. Exemplarily, the facial feature information may include data such as the position, size and shape of facial features such as face contour, eyes, nose, mouth, etc.

[0046] In an embodiment of the present application, the extraction of facial feature information of a digital human in a preset video can be implemented, exemplarily, as follows: extracting tensor data from the preprocessing file, and performing standardization processing on the extracted tensor data to obtain facial feature information of the digital human for use in the synthesis of subsequent interactive video frames.

[0047] In step S140, under the third task branch, based on the interactive questions input by the user, target audio data for the digital human to respond to the interactive questions is generated.

[0048] In the embodiment of the present application, the above interactive question is any question that the user inputs and expects the digital human to answer. For example, the interactive question may be "What will the weather be like tomorrow?" Furthermore, the user may continuously input multiple interactive questions to have a coherent conversation with the digital human.

[0049] In an embodiment of the present application, the user may input the above-mentioned interactive questions in text format or in audio format, and the embodiment of the present application does not specifically limit this.

[0050] In the embodiment of the present application, the above-mentioned target audio data refers to data in an audio format used by the digital human to answer the above-mentioned interactive questions.

[0051] In the embodiment of the present application, the digital human interaction system can generate text answers based on the interactive questions, and then convert the text answers into a target audio format to obtain the target audio data. The target audio format can be set based on actual needs, for example, the target audio format can be a PCM audio data block.

[0052] In an embodiment of the present application, the above-mentioned target audio data for the digital human to respond to the interactive questions based on the interactive questions input by the user can be generated, exemplarily, as follows: real-time detection and reception of the interactive questions input by the user; when the received interactive questions are audio information, converting the interactive questions into text information through voice recognition; retrieving candidate answers corresponding to the interactive questions in the target vector library based on the text information, and generating target audio data based on the candidate answers.

[0053] In the embodiment of the present application, the target vector library is a knowledge base for retrieving information related to the interactive question. For example, if the interactive question is a question in the financial field, the target vector library may be a financial knowledge base.

[0054] In an embodiment of the present application, when it is detected that the interactive question input by the user is in audio format, the interactive question is converted from audio format to text format through speech recognition technology, so that the subsequent retrieval of the text-based data in the above-mentioned target vector library is carried out; if it is detected that the interactive question input by the user is in text format, the retrieval is performed directly in the above-mentioned target vector library based on the text data input by the user.

[0055] In an embodiment of the present application, after receiving an interactive question in text format input by a user, or converting an interactive question in audio format input by a user into text format, the above-mentioned retrieval of candidate answers corresponding to the interactive question in a target vector library based on text information can be implemented, exemplarily, as follows: vectorizing the text information of the interactive question to obtain a vector representation of the text information; and determining the candidate answer based on the similarity between the vector representation of the text information and each vector in the target vector library.

[0056] Specifically, the above process can be implemented as follows: perform embedding (converting words, sentences or documents into vector form) processing on the interactive questions in the above text format to obtain the corresponding vector representation of the interactive questions; perform retrieval in the above target vector library based on the vector representation of the interactive questions, and calculate the cosine similarity between the vector representation of the interactive questions and each vector in the target vector library; determine the candidate answers to the interactive questions based on the sorting of cosine similarities, for example, use the retrieved vectors that rank in the top five in cosine similarity with the vector representation of the interactive questions as candidate answers.

[0057] In the embodiment of the present application, after obtaining the above candidate answers, the final answer to the interactive question can be further generated based on a large language model (LLM). The above LLM model refers to a deep learning model trained with a large amount of text data, which is designed to understand and generate human language.

[0058] In an embodiment of the present application, the process of generating target audio data based on candidate answers is implemented through the LLM model. Exemplarily, it can be implemented as follows: generate prompt information based on the candidate answers; input the prompt information into the large language model, and generate target text answers to interactive questions through the large language model; convert the target text answers into audio format to obtain target audio data.

[0059] In the embodiment of the present application, the above prompt information is a prompt word input into the large language model. By inputting the prompt word into the large language model, the accuracy of the target text answer generated by the model can be provided. Specifically, the candidate answer obtained above can be inserted into the Prompt template of the large language model to obtain the prompt information corresponding to the answer to the interactive question.

[0060] In an embodiment of the present application, the target text answer is a text-formatted answer to an interactive question. After obtaining the target text answer through a large language model, the embodiment of the present application can further convert the target text answer to obtain an answer in an audio format, that is, the target audio data. For example, it can be converted into a PCM format voice block and streamed. In addition, the embodiment of the present application can also convert the PCM streaming data into a byte stream, and the generated chunk-level byte stream is used as the target audio data for the subsequent synthesis of interactive video frames.

[0061] In step S150, interactive video frames of the digital human are correspondingly generated based on the video background, facial feature information and target audio data, so that the digital human and the user can interact via video.

[0062] In the embodiment of the present application, after obtaining the video background, the facial feature information of the digital human and the target audio data, the above-mentioned interactive video frame can be generated based on the above-mentioned information. Exemplarily, the process can be implemented as follows: lip shape synthesis processing is performed based on the video background, the facial feature information and the target audio data to generate the corresponding interactive video frame.

[0063] In an embodiment of the present application, the above-mentioned lip shape synthesis processing can be implemented as follows: extract audio features from the above-mentioned target audio data, and input the extracted voice feature data into a pre-trained audio-driven face model, so as to generate new facial features (i.e., lip shape movements) according to the voice feature data through the audio-driven face model, and obtain an interactive video frame in which the digital human outputs the above-mentioned target audio data with the generated lip shape movements. Among them, the above-mentioned audio-driven face model can be trained based on the original lip shape movements and facial feature information of the digital human. Exemplarily, the training process of the audio-driven face model can be implemented as follows: extract the original voice feature data of the digital human in the preset video, and the facial features in each frame of the preset video; align the original voice feature data with the corresponding facial feature data to form a training set, and input the training set into the pre-trained model for training to obtain the above-mentioned audio-driven face model.

[0064] In the embodiment of the present application, after the interactive video frame is generated based on the video background, facial feature information and target audio data, the interactive video frame can also be pushed to the target server in real time so that the user can read the interactive video frame through the target server. Specifically, the generated digital human interactive video frame can be pushed to the remote server through the RTC service for the user to watch. Among them, the real-time communication (RTC) service push usually refers to the process of the server pushing messages or data to the client during the real-time communication process.

[0065] In an embodiment of the present application, the process of the above-mentioned user reading the interactive video frame through the target server can be implemented as follows: the target server responds to the user's pull operation and sends the interactive video frame to the user, so that the digital human answers the interactive questions input by the user in the form of video broadcast.

[0066] The disclosed embodiment adopts a multi-process processing method to initialize three task branches, namely, the first task branch, the second task branch and the third task branch, and performs video preprocessing based on the first task branch, performs digital human preprocessing based on the second task branch, and generates audio data based on the third task branch. By executing each task branch in parallel, the utilization rate and execution efficiency of system resources are improved. In addition, the disclosed embodiment synthesizes interactive video frames based on the processing results of the above task branches, so that users can achieve real-time interaction with digital humans by obtaining the interactive video frames, thereby improving the interaction fluency and user immersion.

[0067] In a specific embodiment, the overall process of the digital human interaction method is described below. Figure 2 As shown, the digital human interaction method specifically includes the following steps:

[0068] Step S210: Start the thread.

[0069] In this step, when the digital human interaction time begins, the first thread (corresponding to the first task branch mentioned above), the second thread (corresponding to the second task branch mentioned above) and the third thread (corresponding to the third task branch mentioned above) are started and entered respectively.

[0070] Step S220: Initialize the third thread.

[0071] In this step, the third thread will perform initialization work. Specifically, the initialization work includes initialization of the basic application. Exemplarily, the initialization of the basic application can be API request service initialization, so that the entire service can monitor whether there is a request access (that is, whether there is a user input interaction problem).

[0072] Step S221: Initialize the WebRTC service.

[0073] In this step, the third thread initializes the WebRTC service to achieve ultra-low latency push and pull streaming. The WebRTC service is a real-time communication technology that allows network applications or sites to establish peer-to-peer connections between browsers without the help of an intermediary to achieve the transmission of video streams and / or audio streams or any other data.

[0074] Step S222: monitor in real time whether there is any dialogue information of the interactive digital human input by the user, and check the type of the dialogue information.

[0075] In this step, it is continuously monitored whether there is any dialogue information (i.e. the above-mentioned interactive question) of the interactive digital person sent by the user, and the type of the dialogue information is checked. If the type of the dialogue information is text type, step S223 is directly executed; if the type of the dialogue information is audio type, step S2221 is executed.

[0076] Step S2221: Convert the dialogue information from voice information to text information through the automatic speech recognition module.

[0077] Step S223: Connect the text information to the target vector library.

[0078] Step S224: vectorize the text information.

[0079] Step S225: Search in the target vector library based on the vectorized data.

[0080] In this step, the vector database is searched based on the vectorized data, and the top five vectors in the vector database are determined by similarity comparison as the search results (ie, the candidate answers mentioned above).

[0081] Step S226: Input the search results into the large language model.

[0082] In this step, the retrieval results obtained above are inserted into the Prompt template of the large language model.

[0083] Step S227: Obtain a text response to the dialogue message through the large language model, and establish a TTS WebSocket connection.

[0084] In this step, the large language model organizes and summarizes the retrieval results, obtains the text response of the dialogue information (that is, the target text answer mentioned above), and outputs it in the form of a stream.

[0085] In this step, establishing a TTS WebSocket connection refers to creating a WebSocket connection with a TTS (Text To Speech) interface, wherein WebSocket is a full-duplex communication protocol based on TCP, allowing two-way communication between a client and a server to achieve real-time data transmission.

[0086] Step S228: The stream output of the large language model is transmitted to TTS through the WebSocket interface, and response information in audio format is returned.

[0087] In this step, the streaming output of the large language model is transmitted to TTS through the WebSocket interface. TTS will convert the text response of the above dialogue information into the corresponding PCM format voice block and also stream it back to the local.

[0088] Step S229: Perform byte stream processing on the returned audio format response information.

[0089] In this step, the returned PCM streaming data is converted into a byte stream to generate a chunk-level byte stream, which is used as the audio data for synthesizing the digital human interactive video frame in the subsequent step S250.

[0090] Step S230: Initialize the second thread.

[0091] In this step, the second thread enters the initialization preprocessing stage, which mainly reads information such as the face matrix / key points of the digital person corresponding to the preset video to obtain a preprocessing file.

[0092] Step S231: extract tensor data from the preprocessing file.

[0093] Step S232: Standardize the extracted tensor data.

[0094] In this step, the extracted tensor data is standardized, and the standardized face information data is used as the face feature data of the digital human for synthesizing the digital human interactive video frame in the subsequent step S250. Exemplarily, the above-mentioned process of standardizing the tensor data may include basic affine transformations such as resizing and rotation, and color space changes such as brightness and contrast.

[0095] Step S240: Initialize the first thread.

[0096] In this step, the first thread enters the initialization phase, which mainly performs the initialization of the preset video data, that is, each frame of the video data of the digital human is prepared in advance.

[0097] Step S241: extracting current frame data from the preset video.

[0098] Step S242: performing standardization processing on the current frame data.

[0099] In this step, the current frame data is standardized, and the standardized data is used as the video background data for synthesizing the digital human interactive video frame in the subsequent step S250.

[0100] Step S250: synthesizing interactive video frames of the digital human.

[0101] In this step, lip shape synthesis is performed based on the audio data obtained in step S229, the face data of the digital human obtained in step S232, and the video background data obtained in step S242 to obtain the interactive video frame of the digital human.

[0102] Step S260: Push the synthesized interactive video frame to a remote server.

[0103] In this step, the generated lip-synthesized digital human interactive video frame is pushed to the remote server through the RTC service for users to stream and watch.

[0104] It should be noted that, after step S260, the embodiment of the present application can continue to obtain corresponding data from steps S228, S231 and S241 to continuously repeat the action of synthesizing the digital human lip shape until all the data are synthesized.

[0105] In addition, the modules corresponding to the large language model, embedding vectorization processing, speech recognition (ASR), text-to-speech processing (TTS), digital human synthesis and other functions used in the embodiments of the present application have been highly decoupled.

[0106] The above digital human interaction method has the following advantages:

[0107] (1) Users only need to use text or voice to ask a question, and the system can automatically complete automatic speech recognition, knowledge base retrieval, large language model call, text-to-speech, digital human lip shape synthesis, audio and video streaming, and other operations. No human intervention is required in the middle, and the end-to-end real-time interactive digital human generation can be completed;

[0108] (2) All modules are highly decoupled, which makes the dependencies of the overall solution clearer and simpler. At the same time, the large model module, embedding module, ASR module, TTS module, and digital human synthesis module can be updated and optimized at any time without affecting the stability of the overall solution, which is highly scalable.

[0109] (3) Multi-process processing is adopted to separate and parallelize the preprocessing of digital humans, video preprocessing, and audio processing, making greater use of the performance of the CPU and GPU and fully improving execution efficiency.

[0110] The above mainly introduces the solution provided by the embodiment of the present invention from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0111] The embodiment of the present invention can divide the digital human interaction device into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0112] Accordingly, the embodiment of the present disclosure also provides a digital human interaction device. Figure 3As shown, the digital human interaction device 300 may include an initialization module 310, a processing module 320 and an interaction module 330, wherein:

[0113] An initialization module 310 may be used to initialize a first task branch, a second task branch, and a third task branch;

[0114] The processing module 320 may be used to generate a video background of an interactive video frame based on a preset video under the first task branch, where the interactive video frame is a video frame in which the digital human interacts with the user;

[0115] The processing module 320 is further used to extract facial feature information of the digital person from the preset video under the second task branch;

[0116] The processing module 320 is further used to generate target audio data for the digital human to respond to the interactive question based on the interactive question input by the user under the third task branch;

[0117] The interactive module 330 can be used to generate interactive video frames of the digital human based on the video background, facial feature information and target audio data, so that the digital human can interact with the user through video.

[0118] In an embodiment of the present application, the above-mentioned processing module is specifically used to: detect and receive interactive questions input by users in real time; when the received interactive questions are audio information, convert the interactive questions into text information through speech recognition; retrieve candidate answers corresponding to the interactive questions in the target vector library based on the text information, and generate target audio data based on the candidate answers.

[0119] In an embodiment of the present application, the above-mentioned processing module is specifically used to: vectorize the text information of the interactive question to obtain a vector representation of the text information; and determine the candidate answers based on the similarity between the vector representation of the text information and each vector in the target vector library.

[0120] In an embodiment of the present application, the above-mentioned processing module is specifically used to: generate prompt information based on candidate answers; input the prompt information into a large language model, and generate a target text answer to the interactive question through the large language model; convert the target text answer into an audio format to obtain target audio data.

[0121] In an embodiment of the present application, the above-mentioned interactive module is specifically used to: perform lip shape synthesis processing based on video background, facial feature information and target audio data to generate corresponding interactive video frames.

[0122] In an embodiment of the present application, the above-mentioned interactive module is specifically used to: push the interactive video frame to the target server in real time, so that the user can read the interactive video frame through the target server.

[0123] In an embodiment of the present application, the above-mentioned interactive module is specifically used for: the target server sends an interactive video frame to the user in response to the user's pull operation, so that the digital human can answer the interactive questions input by the user in the form of video broadcast.

[0124] In addition, the specific implementation details of the above-mentioned digital human interaction device have been described in detail in the corresponding position of the digital human interaction method, so they will not be repeated here.

[0125] Figure 4 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 4 , which shows a structural schematic diagram of an electronic device 400 suitable for implementing the embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0126] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403 to implement the digital human interaction method of the embodiment described in the present disclosure. In RAM403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, ROM402, and RAM403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0127] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0128] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby realizing the digital human interaction method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0129] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0130] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0131] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0132] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0133] Initialize the first task branch, the second task branch and the third task branch;

[0134] Under the first task branch, a video background of an interactive video frame is generated based on a preset video, wherein the interactive video frame is a video frame in which a digital human interacts with a user;

[0135] Under the second task branch, facial feature information of the digital human is extracted from the preset video;

[0136] In the third task branch, based on the interactive questions input by the user, target audio data for the digital human to respond to the interactive questions is generated;

[0137] Based on the video background, facial feature information and target audio data, interactive video frames of the digital human are generated accordingly, so that the digital human can interact with the user through video.

[0138] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0139] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0140] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0142] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0143] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0145] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0146] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A digital human interaction method, characterized in that: include: Initialize the first task branch, the second task branch and the third task branch; Under the first task branch, a video background of an interactive video frame is generated based on a preset video, wherein the interactive video frame is a video frame in which a digital human interacts with a user; Under the second task branch, extracting facial feature information of the digital human from the preset video; Under the third task branch, based on the interactive question input by the user, target audio data for the digital human to respond to the interactive question is generated; The interactive video frame is correspondingly generated based on the video background, the facial feature information and the target audio data, so that the digital human and the user can interact via video.

2. The digital human interaction method according to claim 1, characterized in that: The step of generating target audio data for the digital human to respond to the interactive question based on the interactive question input by the user comprises: Detecting and receiving the interactive question input by the user in real time; When the received interactive question is audio information, converting the interactive question into text information through voice recognition; Based on the text information, candidate answers corresponding to the interactive question are retrieved from a target vector library, and the target audio data is generated based on the candidate answers.

3. The digital human interaction method according to claim 2, characterized in that: The step of retrieving a candidate answer corresponding to the interactive question in a target vector library based on the text information includes: Performing vectorization processing on the text information of the interactive question to obtain a vector representation of the text information; The candidate answer is determined based on the similarity between the vector representation of the text information and each vector in the target vector library.

4. The digital human interaction method according to claim 2, characterized in that: The generating the target audio data based on the candidate answers comprises: generating prompt information based on the candidate answers; Inputting the prompt information into a large language model, and generating a target text answer to the interactive question through the large language model; The target text answer is converted into an audio format to obtain the target audio data.

5. The digital human interaction method according to claim 1, characterized in that: The generating the interactive video frame based on the video background, the facial feature information and the target audio data includes: Lip synthesis processing is performed based on the video background, the facial feature information and the target audio data to generate the corresponding interactive video frame.

6. The digital human interaction method according to claim 1, characterized in that: After the interactive video frame is generated based on the video background, the facial feature information and the target audio data, the method further includes: The interactive video frame is pushed to a target server in real time, so that the user can read the interactive video frame through the target server.

7. The digital human interaction method according to claim 1, characterized in that: The user reads the interactive video frame through the target server, including: In response to the user's pull operation, the target server sends the interactive video frame to the user, so that the digital human answers the interactive question input by the user in a video broadcast manner.

8. A digital human interactive device, characterized in that: include: An initialization module, used to initialize the first task branch, the second task branch and the third task branch; A processing module, configured to generate a video background of an interactive video frame based on a preset video under the first task branch, wherein the interactive video frame is a video frame in which a digital human interacts with a user; The processing module is further used to extract facial feature information of the digital human from the preset video under the second task branch; The processing module is further used to generate target audio data for the digital human to respond to the interactive question based on the interactive question input by the user under the third task branch; The interactive module is used to generate interactive video frames of the digital human based on the video background, the facial feature information and the target audio data, so that the digital human can interact with the user through video.

9. An electronic device, comprising: processor; and a memory storing a program, wherein the program comprises instructions for causing the processor to execute the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images