Virtual digital human interaction method and device, electronic equipment and storage medium

Abstract

The invention relates to a virtual digital person interaction method and device, electronic equipment and a storage medium, and the method comprises the steps: displaying a virtual digital person in an idle state, and the idle state is a person state when the virtual digital person does not interact with a user; in response to an operation of triggering interaction with the virtual digital person, switching to display the virtual digital person in a listening state, the listening state representing a character state when the user speaks; and switching to display the virtual digital person who answers the question in response to receiving the question sent by the user within the specified duration of switching to display the virtual digital person in the listening state. Therefore, the virtual digital human can have a richer, more flexible and more vivid interaction form, and the immersive interaction experience between the user and the virtual digital human is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of virtual digital humans, and in particular to a virtual digital human interaction method and device, an electronic device, and a storage medium. Background Art

[0002] Digital humans can be understood as digitized or virtualized characters or individuals that exist in a virtual environment. They can be animated characters, virtual assistants, or characters generated by artificial intelligence. After the digital human is completed, it needs to be rendered and driven. The rendering of the digital human is to render the constructed digital human on the display. At this time, the rendered digital human is static, and the driving of the digital human is to move the rendered static digital human like a real human, so that the digital human has an extremely realistic and meticulous restoration of the hyper-realistic human image in the real world, including details such as facial expressions, skin texture, and eye luster, aiming to make the audience feel visually real.

[0003] The development of virtual digital humans has entered a rapid growth stage. They can be applied to a variety of different scenarios with digital human needs, such as the gaming industry, live broadcasting scenarios, intelligent question and answer, etc., to achieve intelligent human-computer interaction. However, in current virtual digital human products, only fixed postures are presented when questions need to be answered, making the experience of interacting with virtual digital humans more mechanical, and the interaction effect is not real and immersive enough, reducing the user's interactive experience. Summary of the invention

[0004] In view of this, the present disclosure proposes a virtual digital human interaction method and device, an electronic device and a storage medium, which can enable the virtual digital human to have a richer, more flexible and vivid interaction form, and improve the immersive interaction experience between the user and the virtual digital human.

[0005] According to one aspect of the present disclosure, a method for interacting with a virtual digital human is provided, comprising: displaying a virtual digital human in an idle state, wherein the idle state is the state of the human being when not interacting with a user; in response to triggering an interactive operation with the virtual digital human, switching to displaying a virtual digital human in a listening state, wherein the listening state is the state of the human being listening to a user speaking; in response to receiving a question from a user within a specified time period after switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human answering the question.

[0006] In one possible implementation, in response to receiving a question from a user within a specified time period of switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human answering the question includes: when the question is a preset question and is not a sensitive question, displaying a preset answer video corresponding to the preset question, the preset answer video is used to display the virtual digital human answering the preset question; when the question is not a preset question and is not a sensitive question, displaying a real-time answer video generated based on the question, the real-time answer video is used to display the virtual digital human expressing the answer text generated for the question.

[0007] In one possible implementation, in response to receiving a question from a user within a specified time period of switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human who answers the question also includes: when the question is a sensitive question, switching to displaying the virtual digital human in a fallback state, wherein the fallback state represents a character state in which the question cannot be answered; after displaying the virtual digital human in the fallback state, switching to displaying the virtual digital human in the listening state.

[0008] In one possible implementation, in response to receiving a question from a user within a specified time period when the display is switched to displaying the virtual digital human in a listening state, switching to displaying the virtual digital human who answers the question includes: in response to receiving a question from a user within a specified time period when the display is switched to displaying the virtual digital human in a listening state, obtaining a question text corresponding to the question; judging whether the question is a sensitive question based on the question text, and / or judging whether the question is a preset question; wherein, judging whether the question is a preset question includes: matching the question text with each preset question in a preset question library to obtain a matching result of the question text, and the matching result includes: whether it matches the preset questions in the preset question library, and, when matching the preset questions in the preset question library, the preset questions that match the question text.

[0009] In a possible implementation, the method further includes: in response to not receiving a question from the user within a specified time period after switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human in the idle state; or, in response to not receiving a question from the user within a specified time period after switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human in an interaction end state, wherein the interaction end state represents a character state in which interaction with the user has ended; and after displaying the virtual digital human in the interaction end state, switching to displaying the virtual digital human in the idle state.

[0010] In one possible implementation, in response to triggering an interactive operation with the virtual digital human, switching to displaying the virtual digital human in a listening state includes: in response to triggering an interactive operation with the virtual digital human, switching to displaying the virtual digital human in a welcoming state, wherein the welcoming state represents a character state that welcomes user interaction; after displaying the virtual digital human in the welcoming state, switching to displaying the virtual digital human in the listening state.

[0011] In one possible implementation, the method displays a virtual digital human on a holographic warehouse screen.

[0012] According to another aspect of the present disclosure, a virtual digital human interaction device is provided, comprising: an idle state display module, used to display a virtual digital human in an idle state, wherein the idle state is represented by the character state when not interacting with a user; a first switching module, used to switch to displaying a virtual digital human in a listening state in response to triggering an interactive operation with the virtual digital human, wherein the listening state is represented by the character state when listening to a user speaking; and a second switching module, used to switch to displaying a virtual digital human answering a question in response to receiving a question from a user within a specified time period of switching to displaying the virtual digital human in the listening state.

[0013] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0014] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0015] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0016] According to the interactive device of the embodiment of the present disclosure, by presetting a virtual digital human in an idle state, a virtual digital human that switches to a listening state when an interactive operation is triggered, and a virtual digital human that switches to an answering state after receiving a question from a user within a specified time period after switching to the listening state, it is possible to make the virtual digital human have a richer, more flexible and vivid expression form, thereby improving the interactive experience of the user and the virtual digital human.

[0017] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0019] Figure 1 A flowchart of a virtual digital human interaction method according to an embodiment of the present disclosure is shown.

[0020] Figure 2 A screenshot of a virtual digital human in an idle state according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A screenshot of a virtual digital human in a listening state according to an embodiment of the present disclosure is shown.

[0022] Figure 4 A screenshot of a virtual digital human in a welcoming state according to an embodiment of the present disclosure is shown.

[0023] Figure 5 A screenshot of a virtual digital human in an interaction ending state according to an embodiment of the present disclosure is shown.

[0024] Figure 6 A screenshot showing a virtual digital person answering a preset question according to an embodiment of the present disclosure is shown.

[0025] Figure 7 A screenshot showing a virtual digital person answering questions in real time according to an embodiment of the present disclosure.

[0026] Figure 8 A screenshot of a virtual digital human in a back-up state according to an embodiment of the present disclosure is shown.

[0027] Fig. 9 A flow chart of a state switching logic according to an embodiment of the present disclosure is shown.

[0028] Fig.10a and Fig.10b A schematic diagram of an interface for performing process orchestration using a process orchestrator according to an embodiment of the present disclosure is shown.

[0029] Fig.11 A block diagram of a virtual digital human interaction device according to an embodiment of the present disclosure is shown.

[0030] Fig.12 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0032] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0033] The term "and / or" herein is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent any one or more elements selected from the set consisting of A, B, and C. In the description of the present disclosure, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0034] It should be understood that the terms “include” and “comprising” used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0035] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0036] As described above, when there is no one to interact with the virtual digital human in the prior art, the digital human is in a standing static posture, and when there is someone to interact, the digital human is driven by dynamic body movements and mouth shapes. This will cause the interaction with the digital human to be more mechanical, and only when answering questions will a fixed posture be presented. The effect is not real and immersive enough. In view of this, the disclosed embodiment proposes a virtual digital human interaction method, which presets multiple digital human states and drives the virtual digital human to speak in real time in combination with the answer content generated by the artificial intelligence model. Among them, video materials can be preset for multiple states such as idle state, welcome state, listening state, bottom state, and end of interaction state, and answer videos can be preset for the questions that users interact most frequently (such as name, age, hobbies), and the use of artificial intelligence technology to drive the virtual digital human interaction in real time, which can extend the breadth of interactive content, that is, more questions can be responded to; among them, by controlling the display logic of various states of the virtual digital human, it can respond to questions for interaction, and can also give users the best immersive interactive experience with the most vivid effects and rich states.

[0037] The virtual digital human interaction method of the embodiment of the present disclosure can be deployed on various terminal devices through software or hardware modification. The terminal device involved in the embodiment of the present disclosure may refer to a device with a wireless connection function and / or a wired connection function. The wireless connection function refers to the ability to connect to other devices through wireless connection methods such as wifi and Bluetooth. The terminal device involved in the embodiment of the present disclosure may also communicate with other devices through a wired connection function. The terminal device involved in the embodiment of the present disclosure may be a touch screen or a non-touch screen. The touch screen can be controlled by clicking and sliding on the display screen with a finger or a stylus. The non-touch screen device can be connected to an input device such as a mouse, keyboard, touch panel, etc., and the terminal device can be controlled through the input device. For example, a device without a screen can be a Bluetooth speaker without a screen. The terminal device involved in the embodiment of the present disclosure may have a voice input function and a voice output function. The voice input function may refer to a device with a microphone or a speaker for collecting external audio, and the voice output function may refer to a device with a speaker or a speaker for playing audio externally. For example, the terminal device of the present application may include but is not limited to a holographic display device, a user equipment (UE), a mobile device, a user terminal, a terminal, a handheld device, a tablet computer, a laptop computer, a PDA, a computing device, a virtual reality device, an augmented reality device, etc.

[0038] Figure 1 A flowchart of a virtual digital human interaction method according to an embodiment of the present disclosure is shown. The method can be applied to the above-mentioned terminal device, such as Figure 1 As shown, the method includes: step S11 to step S13.

[0039] In step S11, the virtual digital human in an idle state is displayed, where the idle state refers to the state of the human being when not interacting with the user.

[0040] In practical applications, those skilled in the art can design the state of the character in the idle state in advance according to actual needs, and use the virtual digital human driving technology known in the art to drive the virtual digital human to display the designed idle state, and record the virtual digital human in the idle state to obtain and save the video material of the virtual digital human in the idle state. For example, Figure 2 A screenshot showing a virtual digital human in an idle state. For example, the idle state may be a state where the digital human is stationary, or a state where the digital human is performing activities on its own and not interacting with the user.

[0041] As described above, the virtual digital human interaction method of the disclosed embodiment can be deployed on various terminal devices through software or hardware modification. After the virtual digital human interaction function is started by the terminal device, the pre-recorded video material of the virtual digital human in an idle state can be played on the display screen of the terminal device (such as the holographic warehouse screen), thereby realizing the display of the virtual digital human in an idle state. It should be understood that the disclosed embodiment does not limit the display method of the virtual digital human during the interaction process. For example, the virtual digital human can be displayed on the holographic warehouse screen, and the holographic warehouse screen can be used to display the virtual digital human with a three-dimensional effect, which is conducive to improving the user's immersive interactive experience.

[0042] In actual applications, when the above-mentioned pre-recorded video material with the virtual digital human in the idle state begins to be played, the playback progress of the video material can be monitored in real time by real-time monitoring of the switching of video frames during the playback of the video material with the virtual digital human in the idle state, so as to respond in time according to the current state and user input. For example, if no user triggers the interactive operation with the virtual digital human when it is monitored that the video material has been played, the display screen can be controlled to play the video material with the virtual digital human in the idle state again, that is, the video material with the virtual digital human in the idle state can be played in a loop until a user triggers the interactive operation with the virtual digital human. When the user triggers the interactive operation with the virtual digital human, step S12 can be entered to switch to the virtual digital human in the listening state.

[0043] In step S12, in response to triggering an interactive operation with the virtual digital human, the virtual digital human is switched to display a listening state, where the listening state is represented by the state of the human body when listening to the user speaking.

[0044] Among them, the interactive operation with the virtual digital person can be triggered by, for example, a voice wake-up method, that is, the user can trigger the interactive operation with the virtual digital person by issuing a specified wake-up word (such as "Hello, Xiao A"), or wake up the virtual digital person for interaction. Among them, the above voice wake-up method can also be implemented by using an intelligent voice wake-up chip known in the art; or, a physical button for triggering the interactive operation can be provided on the terminal device, and the user can also trigger the interactive operation with the virtual digital person by clicking the physical button, or when the user performs a preset operation on the application (such as searching, sending bullet screens, starting customer service, etc.), the interactive operation with the virtual digital person can be triggered. The embodiments of the present disclosure do not limit the implementation method of triggering the interactive operation.

[0045] In practical applications, those skilled in the art can design the state of the character in the listening state in advance according to actual needs, and use the virtual digital human driving technology known in the art to drive the virtual digital human to display the designed listening state, and record the virtual digital human in the listening state to obtain and save the video material of the virtual digital human in the listening state. Furthermore, when the user triggers the interactive operation with the virtual digital human, the display screen of the terminal device can be controlled to smoothly switch from playing the video material of the virtual digital human in the idle state to playing the video material of the virtual digital human in the listening state, thereby switching to displaying the virtual digital human in the listening state. For example, Figure 3 A screenshot showing a virtual digital human in a listening state, such as Figure 3 As shown, the virtual digital person in the listening state can tilt one ear towards the user and make listening gestures or expressions. The screen can also display the prompt text "Please speak, I'm listening".

[0046] Considering that it may not be natural to switch directly from the idle state to the listening state, in a possible implementation, in response to triggering the interactive operation with the virtual digital human, switching to displaying the virtual digital human in the listening state may include:

[0047] In response to triggering an interactive operation with a virtual digital human, the virtual digital human is switched to display a welcoming state, where the welcoming state is a state of a character that welcomes user interaction;

[0048] After displaying the virtual digital human in a welcoming state, it switches to displaying the virtual digital human in a listening state.

[0049] In practical applications, those skilled in the art can design the state of the character in the welcome state in advance according to actual needs, and use the virtual digital human driving technology known in the art to drive the virtual digital human to display the designed welcome state, and record the virtual digital human in the welcome state to obtain and save the video material of the virtual digital human in the welcome state. Furthermore, when the user triggers the interactive operation with the virtual digital human, the display screen of the terminal device can be controlled to smoothly switch from playing the video material of the virtual digital human in the idle state to playing the video material of the virtual digital human in the welcome state, thereby switching to displaying the virtual digital human in the welcome state. For example, Figure 4 A screenshot of a virtual digital human in a welcoming state is shown, Figure 4 As shown, the welcome state can be expressed as a character state of greeting.

[0050] Then, after playing the video material of the virtual digital person in the welcoming state (i.e., displaying the virtual digital person in the welcoming state), the video material of the virtual digital person in the listening state can be switched to be played, thereby switching from the welcoming state to displaying the virtual digital person in the listening state. Among them, the playback progress of the video material of the virtual digital person in the welcoming state can be monitored in real time by real-time monitoring the switching of video frames during the playback of the video material, so as to switch to the video material of the virtual digital person in the listening state in time after playing the video material of the virtual digital person in the welcoming state. In this way, the interactive effect of the virtual digital person can be made more natural, smooth and realistic.

[0051] Among them, after switching to displaying the virtual digital human in a listening state (that is, switching to playing the video material with the virtual digital human in a listening state), the playback progress of the video material can be monitored in real time by real-time monitoring of the switching of video frames during the playback of the video material with the virtual digital human in a listening state, so as to respond in time according to the current state and user input. Among them, if no questions from the user are received within a specified time period after monitoring the switching to displaying the virtual digital human in a listening state (that is, the user does not ask any questions after waking up the virtual digital human), it is possible to switch to displaying the virtual digital human in an idle state. Therefore, in a possible implementation method, the method also includes: in response to not receiving any questions from the user within a specified time period after switching to displaying the virtual digital human in a listening state, switching to displaying the virtual digital human in an idle state.

[0052] It should be understood that if no questions from the user are received within the specified time period after the virtual digital human in the listening state is switched to be displayed, it can be considered that the user is no longer interacting with the virtual digital human. In this case, the virtual digital human can be returned to the idle state to wait for the next triggering of the interactive operation with the virtual digital human. The specified time period can be the material length of the video material of the virtual digital human in the listening state (such as 15 seconds), that is, when the video material of the virtual digital human in the listening state is played and the user does not ask any questions, the display can be switched to the idle state of the virtual digital human (that is, the video material of the virtual digital human in the idle state is played); or, the specified time period can also be N times the material length of the video material of the virtual digital human in the listening state, that is, when the video material of the virtual digital human in the listening state is played in a loop for N rounds and the user still does not ask any questions, the display can be switched to the idle state of the virtual digital human. Of course, the specified time period can also be other customized time periods, which are not limited in the embodiments of the present disclosure.

[0053] Taking into account that switching directly from the listening state to the idle state may not be natural enough, optionally, the method may also include: in response to not receiving a question from the user within a specified time period after switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human in the interaction end state, where the interaction end state is manifested as the character state of ending the interaction with the user; and after displaying the virtual digital human in the interaction end state, switching to displaying the virtual digital human in the idle state.

[0054] In actual applications, those skilled in the art can design the character status in the interaction end state in advance according to actual needs, and use the virtual digital human driving technology known in the art to drive the virtual digital human to display the designed interaction end state, and record the virtual digital human in the interaction end state to obtain and save the video material of the virtual digital human in the interaction end state. Furthermore, if no questions from the user are received within the specified time period when switching to displaying the virtual digital human in the listening state, the display screen of the terminal device can be controlled to smoothly switch from playing the video material of the virtual digital human in the listening state to playing the video material of the virtual digital human in the interaction end state, thereby achieving the switch to displaying the virtual digital human in the interaction end state. For example, Figure 5 A screenshot of a virtual digital human in an interaction-ending state is shown. Figure 5 As shown, the end state of interaction can be expressed as a character saying goodbye.

[0055] Then, after playing the video material of the virtual digital person in the interaction-ended state (i.e., displaying the virtual digital person in the interaction-ended state), the video material of the virtual digital person in the idle state can be switched to be played, thereby switching from the interaction-ended state to displaying the virtual digital person in the idle state. Among them, the playback progress of the video material of the virtual digital person in the interaction-ended state can be monitored in real time by real-time monitoring the switching of video frames during the playback of the video material, so as to switch to playing the video material of the virtual digital person in the idle state in time after playing the video material of the virtual digital person in the interaction-ended state. In this way, the interaction effect of the virtual digital person can be made more natural, smooth and realistic.

[0056] As described above, after switching to displaying a virtual digital human in a listening state (i.e., switching to playing video material with a virtual digital human in a listening state), the playback progress of the video material can be monitored in real time by monitoring the switching of video frames during the playback of the video material with the virtual digital human in a listening state, so as to respond in time according to the current state and user input. If a question from the user is received within the specified time period of monitoring the switching to displaying the virtual digital human in a listening state, step S13 can be entered to display the virtual digital human who answers the question.

[0057] In step S13, in response to receiving a question from a user within a specified time period after the virtual digital human in the listening state is switched to be displayed, the virtual digital human answering the question is switched to be displayed.

[0058] Users can ask questions in the form of voice, text, or by selecting from preset questions. Taking voice questions as an example, the terminal device can be equipped with an audio collection device, such as a microphone, etc. The audio collection device of the terminal device can be used to collect the voice of the question issued by the user, or the user's speaking audio, and then artificial intelligence technology can be used to analyze the question expressed in the voice of the question, and drive the virtual digital person to answer the question, thereby realizing the question-and-answer interaction with the user.

[0059] In the disclosed embodiment, by presetting a virtual digital human in an idle state, a virtual digital human that switches to a listening state when an interactive operation is triggered, and a virtual digital human that switches to an answering state after receiving a question from a user within a specified time period after switching to the listening state, it is possible to make the virtual digital human have a richer, more flexible and vivid expression form, thereby improving the interactive experience between the user and the virtual digital human.

[0060] Considering that some questions are frequently asked by users (such as name, age, hobbies, etc.), in order to shorten the response time required for the virtual digital human to answer questions and improve the interactive experience, answer videos can be pre-made for these questions frequently asked by users, so that the corresponding answer videos can be played directly after the user asks these questions, while some questions need to generate answer videos in real time. Some questions may involve sensitive questions that cannot be answered, so the state of the virtual digital human can be switched to respond. Therefore, in a possible implementation, the above step S13, in response to receiving a question sent by the user within the specified time length of the virtual digital human switched to display the listening state, switching to display the virtual digital human answering the question, can include:

[0061] When the question is a preset question and is not a sensitive question, a preset answer video corresponding to the preset question is displayed, and the preset answer video is used to display a virtual digital person who answers the preset question; when the question is not a preset question and is not a sensitive question, a real-time answer video generated based on the question is displayed, and the real-time answer video is used to display a virtual digital person who expresses the answer text generated for the question.

[0062] The step S13 may further include: when the question is a sensitive question, switching to displaying a virtual digital human in a fail-safe state, wherein the fail-safe state represents a character state that cannot answer the question; after displaying the virtual digital human in the fail-safe state, switching to displaying the virtual digital human in the listening state.

[0063] Therefore, different responses can be made for different types of problems.

[0064] In a possible implementation, in response to receiving a question from a user within a specified time period when the virtual digital human is switched to display a listening state, a question text corresponding to the question is obtained;

[0065] Determining whether the question is a sensitive question based on the question text, and / or determining whether the question is a preset question;

[0066] Among them, judging whether the question is a preset question may include: matching the question text with each preset question in the preset question library to obtain a matching result of the question text, the matching result including: whether it matches the preset questions in the preset question library, and, when matching with the preset questions in the preset question library, the preset questions that match the question text.

[0067] For example, when a user asks a question in voice, those skilled in the art may use any known speech recognition technology in the art to implement speech recognition of the question voice and obtain a question text corresponding to the question voice, and the embodiments of the present disclosure are not limited to this.

[0068] Among them, the preset questions in the preset question library can be questions that are frequently asked by some users. It should be understood that technical personnel in this field can set various preset questions according to actual conditions, such as, "What is your name", "How old are you", "What are your hobbies", etc., and the present embodiment of the present disclosure is not limited to this; among them, various preset questions can be saved in a database to obtain a preset question library. The preset question library can be saved locally on the terminal device to facilitate timely matching with the question text converted by the question. Of course, it can also be saved in a cloud server, and the present embodiment of the present disclosure is not limited to this.

[0069] In practical applications, the preset question library can use a vector database, so that all preset questions can be text-vectorized and stored in the vector database. When the user starts to interact, the question voice issued by the user can be converted into question text through real-time speech recognition technology, and then the question text can be text-vectorized in real time to obtain the question text vector of the question text. Then, the vector retrieval technology can be used to search in the vector database whether there are text vectors of semantically similar preset questions to obtain the matching results of the question text, so as to decide whether to use the preset answer or the real-time synthesized answer based on the matching results. Specifically, each preset question in the above preset question library is represented as a text vector, wherein the question text is matched with each preset question in the preset question library to obtain the matching results of the question text, which may include:

[0070] Convert the question text into a question text vector, and calculate the distance between the question text vector and the text vectors of each preset question in the preset question library;

[0071] In the case where there is a text vector in the preset question library whose distance to the question text vector is less than a specified threshold, the preset question corresponding to the text vector whose distance is less than the specified threshold is determined as the preset question matching the question text;

[0072] When there is no text vector in the preset question library whose distance to the question text vector is less than a specified threshold, it is determined that there is no preset question matching the question text in the preset question library.

[0073] Among them, converting the question text into the question text vector is also called text vectorization of the question text, and text vectorization is the process of converting the text into vector form. Vector is a quantity in n-dimensional space, composed of multiple numerical values, and is often used to represent the characteristics of information such as images, texts, and sounds, so as to facilitate computer processing and analysis. Especially in the fields of machine learning and deep learning, text vectorization helps to improve the training and prediction capabilities of the model; vector database can be understood as a database management system for storing and retrieving high-dimensional vector data. This type of vector database supports fast similarity search and efficient vector clustering, and is often used in fields such as machine learning, image and video processing. Vector databases can use special index structures to quickly find similar vectors. Vector retrieval technology refers to the process of finding other vectors similar to a given vector in a vector database. It relies on distance metrics (such as Euclidean distance or cosine similarity) to identify which vectors are relatively close in the feature space.

[0074] Therefore, those skilled in the art can use vector conversion technology known in the art to realize the conversion of question text into question text vector, which is not limited by the embodiment of the present disclosure. Then, for example, a distance calculation method such as Euclidean distance or cosine similarity can be used to calculate the distance between the question text vector and the text vectors of each preset question in the preset question library, which is not limited by the embodiment of the present disclosure. Among them, the specified threshold can be set according to actual needs, which is not limited by the embodiment of the present disclosure.

[0075] It should be understood that if the distance between two vectors is less than a specified threshold, the two vectors can be considered to be the most similar. Therefore, the preset question corresponding to the text vector with a distance less than the specified threshold can be determined as the preset question matching the question text. If there is no text vector with a distance less than the specified threshold between the question text vector and the preset question library, the question text is considered not to be a preset question. Among them, because text vectorization can capture the semantic relationship and contextual information between words, it can understand the true intention behind the query, rather than relying solely on literal matching. This is very effective for processing fuzzy queries, synonym replacement and other scenarios, and improves the relevance and accuracy of search results. By converting text into points in a high-dimensional vector space for vector retrieval, the distance (i.e., similarity) between question texts or preset questions can be quickly calculated, and efficient approximate search can be achieved, which greatly improves the retrieval speed and is conducive to improving the response speed of virtual digital human interaction.

[0076] It should be understood that the above-mentioned use of vector retrieval technology to match the question text with the preset questions is a possible implementation method provided by the embodiment of the present disclosure. In fact, those skilled in the art can use any text matching technology known in the art to achieve the matching of the question text with each preset question in the preset question library. For example, the question text obtained by speech recognition can be denoised in combination with the noise reduction algorithm known in the art, and the denoised question text can be calibrated. The calibrated question text can then be semantically matched with the preset questions using the trained semantic matching model to obtain the matching result of the question text. The embodiment of the present disclosure does not limit this.

[0077] In actual applications, those skilled in the art can set various preset questions and preset answers corresponding to various preset questions according to actual conditions (for example, for the preset question "How old are you this year", the corresponding preset answer can be "I am 20 years old this year"). Then, artificial intelligence technology can be used to generate answer audio with emotional tone corresponding to the preset answers based on the preset answers of various preset questions, and then the virtual digital human can be driven to be displayed as a virtual digital human expressing the preset answers, wherein the virtual digital human expressing the preset answers has dynamic mouth movements and dynamic body movements that match the preset answers and synthesizes answer audio with emotional tone corresponding to the preset answers. Then, the virtual digital human expressing the preset answers can be recorded to obtain the preset answer video corresponding to the preset question, so that when the question text matches the preset question, the preset answer video corresponding to the matched preset question can be directly played, thereby realizing the display of the virtual digital human answering the question; wherein, the preset answer videos corresponding to the various preset questions can be stored locally on the terminal device, or can also be stored in a cloud server, etc., which is not limited by the embodiments of the present disclosure.

[0078] Thus, when the matching result indicates that the question text matches the preset question in the preset question library, the preset answer video corresponding to the preset question matching the question text can be obtained and played, wherein the preset answer video is used to display the virtual digital person who answers the preset question matching the question text. In other words, when matching with the preset question in the preset question library, the preset answer video corresponding to the preset question matching the question text can be obtained and played on the display screen, thereby realizing the display of the virtual digital person who answers the question. For example, Figure 6 A screenshot of a virtual digital person answering a preset question, that is, a screenshot of a preset answer video, is shown.

[0079] In one possible implementation, if the matching result indicates that the question text does not match any of the preset questions in the preset question library, this means that the question raised by the user is not a preset question. In this case, an artificial intelligence model (such as a large language model) can be used to generate an answer text based on the question text, and based on the answer text, a real-time answer video can be generated; and the display screen can be controlled to play the real-time answer video; wherein the real-time answer video is used to display a virtual digital person expressing the answer text, and the virtual digital person displayed in the real-time answer video has dynamic mouth movements and dynamic body movements corresponding to the answer text.

[0080] It should be understood that those skilled in the art can use the virtual digital human driving technology known in the art to achieve driving the virtual digital human with dynamic mouth movements and dynamic body movements that match the answer text, and can use the artificial intelligence technology known in the art to convert the answer text into an answer audio with an emotional tone, and then use the answer audio with an emotional tone to drive the virtual digital human to produce mouth movements and synthesize the answer audio. It is also possible to set the loop logic of the virtual digital human's body movements or use artificial intelligence technology to synthesize the virtual digital human's body movements to obtain a virtual digital human with dynamic mouth movements and dynamic body movements that match the answer text, that is, to obtain a real-time answer video of the virtual digital human expressing the answer text. The answer video can be generated on the server side, and the real-time streaming technology known in the art, such as WebRTC (Web Real-Time Communications) technology, can be used to control the display screen to play the real-time answer video of the virtual digital human expressing the answer text in the form of a video stream, that is, to realize the display of the virtual digital human answering the question. For example, Figure 7 A screenshot of a virtual digital person answering a question in real time is shown, that is, a video screenshot of a real-time answering video.

[0081] Considering that the questions raised by users may be sensitive questions that cannot be answered, for example, questions about personal privacy information, company business information, etc., in order to improve the security and reliability of the interaction process, in a possible implementation, when the matching result indicates that the question text does not match the preset questions in the preset question library, it can be determined whether the question text involves sensitive questions, and sensitive questions are questions that cannot be answered;

[0082] When the question text does not involve sensitive issues, an artificial intelligence model is used to generate an answer text based on the question text, and based on the answer text, a real-time answer video is generated; the real-time answer video is played; wherein, the real-time answer video is used to display a virtual digital person expressing the answer text, and the virtual digital person displayed in the real-time answer video has dynamic mouth movements and dynamic body movements corresponding to the answer text.

[0083] In actual applications, technicians in this field can pre-set a sensitive question library according to actual needs. The sensitive question library may contain various unanswerable sensitive questions. The question text can be matched with the sensitive questions in the sensitive question library by referring to the implementation method of matching the above-mentioned question text with the preset questions in the preset question library. If the question text matches the sensitive questions in the sensitive question library, it is considered that the question text involves sensitive issues. Otherwise, it is considered that the question text does not involve sensitive issues. Alternatively, an artificial intelligence model can be used to perform semantic analysis on the question text to determine whether the question text involves sensitive issues. The embodiment of the present disclosure does not limit the method for determining whether the question text involves sensitive issues.

[0084] It should be understood that those skilled in the art can use artificial intelligence models known in the art, such as large language models, to realize the generation of answer texts according to question texts, and the embodiments of the present disclosure are not limited to this. Further, as described above, the answer text can be converted into an answer audio with an emotional tone using artificial intelligence technology known in the art, and then the answer audio with an emotional tone can be used to drive the virtual digital person to produce mouth movements and synthesize the answer audio, and the loop logic of the body movements of the virtual digital person can be set or the real-time body movements of the virtual digital person can be synthesized using artificial intelligence technology to obtain a virtual digital person with dynamic mouth movements and dynamic body movements matching the answer text, that is, to obtain a real-time answer video of the virtual digital person expressing the answer text, and the real-time push streaming technology known in the art can be used to control the display screen to play the real-time answer video of the virtual digital person expressing the answer text in the form of a video stream, that is, to realize the display of the virtual digital person answering the question.

[0085] In a possible implementation, it is also possible to first determine whether the question text involves sensitive issues. If so, the virtual digital human in the backup state is displayed, and after the virtual digital human in the backup state is displayed, the virtual digital human in the listening state is displayed. If it does not involve sensitive issues, it is then determined whether the question text matches the preset questions in the preset question library. If so, the preset answer video is displayed, and if not, the real-time answer video is displayed.

[0086] In actual applications, after playing the target answer video (i.e., the preset answer video or the real-time answer video), you can switch to displaying a virtual digital person in a listening state (i.e., switch to displaying video material of a virtual digital person in a listening state) so as to continue to receive questions input by the user and respond to the questions.

[0087] If the question text involves sensitive issues, it can be switched to display a virtual digital person in a backup state. Therefore, in a possible implementation, the method further includes:

[0088] When the question text involves sensitive issues, the virtual digital person will be switched to display the backup state, which means the person cannot answer the question.

[0089] After displaying the virtual digital human in the support state, switch to displaying the virtual digital human in the listening state.

[0090] In practical applications, those skilled in the art can design the state of the character in the backup state in advance according to actual needs, and use the virtual digital human driving technology known in the art to drive the virtual digital human to display the designed backup state, and record the virtual digital human in the backup state to obtain and save the video material of the virtual digital human in the backup state. Furthermore, when the question text involves sensitive issues, the display screen of the terminal device can be controlled to smoothly switch from playing the video material of the virtual digital human in the listening state to playing the video material of the virtual digital human in the listening state, thereby switching to displaying the virtual digital human in the listening state. For example, Figure 8 A screenshot of a virtual digital human in a state of being unable to answer is shown. The virtual digital human in the state of being unable to answer may present a language expression or body state that indicates embarrassment or refusal to answer.

[0091] Then, after playing the video material of the virtual digital person in the bottom-covering state (that is, after displaying the virtual digital person in the bottom-covering state), you can switch to playing the video material of the virtual digital person in the listening state, so as to switch from the bottom-covering state to displaying the virtual digital person in the listening state. Among them, the playback progress of the video material of the virtual digital person in the bottom-covering state can be monitored in real time by real-time monitoring the switching of video frames during the playback of the video material, so as to switch to playing the video material of the virtual digital person in the listening state in time after playing the video material of the virtual digital person in the bottom-covering state. In this way, the interaction effect of the virtual digital person can be made more natural, smooth and real, and the safety of the interaction process can be guaranteed.

[0092] To facilitate understanding of the virtual digital human interaction method of the present disclosure, the present disclosure also provides Fig. 9 A flow chart showing a state switching logic is shown, such as Fig. 9As shown, the virtual digital human is displayed in an idle state and determines whether to trigger an interactive operation; if no interactive operation is triggered, it remains displayed in an idle state; if an interactive operation is triggered, it switches to a listening state; after switching to the listening state, it determines whether there is voice interaction. If there is no voice interaction after switching to the listening state (that is, the question voice issued by the user is not obtained within the specified time), it returns to the idle state; if there is voice interaction after switching to the listening state (that is, the question voice issued by the user is obtained within the specified time), it determines whether it matches the preset question; if the preset question is matched, the preset answer video is played, and after playing the preset answer video, it returns to the listening state; if the preset question is not matched, it determines whether to trigger security interception (that is, whether to match sensitive questions). If security interception is triggered (that is, sensitive questions are matched), it switches to a fallback state, and returns to the listening state after the fallback state is displayed; if security interception is not triggered (that is, sensitive questions are not matched), the real-time answer video is played, and after playing the real-time answer video, it returns to the listening state. By using artificial intelligence technology to synthesize and drive virtual digital human interactions in real time, the breadth of interactive content can be extended, that is, more questions can be responded to.

[0093] It should be understood that the video material of the various states of the virtual digital human (idle state, welcoming state, listening state, back-up state, and end of interaction state) may only contain the virtual digital human or text, and may be without audio or with audio, depending on the character state designed by the user and actual needs, and the embodiments of the present disclosure do not limit this. Pre-recorded videos of virtual digital humans in states such as idle state, welcoming state, listening state, back-up state, and end of interaction state can be saved on the server or local terminal device and played on the terminal device at an appropriate time.

[0094] It should be understood that those skilled in the art can adopt artificial intelligence technology known in the art to develop the desired virtual digital human, customize the image, personality, etc. of the virtual digital human, and adjust the image, personality and other characteristics of the virtual digital human according to different usage scenarios, or manage the virtual digital human according to the usage scenario, and the embodiments of the present disclosure are not limited to this.

[0095] In practical applications, in order to improve the response speed of the virtual digital human, the question text can also be semantically detected through a semantic detection model (such as a binary classification model) to detect question texts that have no practical meaning and filter out those question texts that have no practical meaning; and the question texts raised by the user can also be field-classified through a pre-classifier. For example, the question text may be a casual chat question or a professional field question, etc., so that the field classification result of the question text can be input into the artificial intelligence model as auxiliary information, so that the artificial intelligence model can generate a more professional answer text based on the corresponding basic knowledge base or professional knowledge base.

[0096] It should be understood that those skilled in the art can use software development technologies known in the art, such as the Electron framework (a desktop application framework), to develop applications for implementing the virtual digital human method, and the embodiments of the present disclosure are not limited to this. In actual applications, users can deploy applications on terminal devices to implement the virtual digital human interactive function; wherein the terminal device is provided with supporting hardware devices for implementing the virtual digital human interactive method, such as a display screen, a processor, an intelligent voice wake-up chip (such as an ASRPRO chip), a microphone, a speaker, etc.; of course, other hardware devices, such as a camera, etc., can also be provided, and the embodiments of the present disclosure are not limited to this.

[0097] In practical applications, those skilled in the art can pre-arrange the switching logic of various states of the virtual digital human in the above-mentioned interactive process, or pre-arrange the workflow of different states, and can arrange the switching logic of various states by providing a process arranger to the user, for example, Fig.10a and Fig.10b A schematic diagram of the interface for process orchestration using a process orchestrator is shown, such as Fig.10a and Fig.10b As shown, you can set the start time and end time of each state, the state type of the next step (such as Fig.10a The "3" indicated by the "Next" in Fig.10b The current status type (such as Fig.10a The "3" indicated by the "Type" in Fig.10b The "2" indicated by the "Type" in the Fig.10a The "idle state 1 applause" indicated by the "description" in Fig.10b ) and audio plugins (such as Fig.10a If "plugin" is empty, it means there is no audio in this state. Fig.10b The "plugin" indicates "audioPlugin" which means that the state uses the audio plug-in identified as "audioPlugin", that is, the state has audio); it should be understood that the process arrangement interface of the process arranger can also provide other required functional controls, such as "move up (for arranging the state forward), move down (for arranging the state backward), delete (for deleting the state), copy (for copying the state)", etc., which are not limited in the embodiments of the present disclosure. In this way, the user can use the process arranger to arrange the start time, end time, corresponding audio plug-in, and next state of the current state, etc. of the current state, and customize the above process.

[0098] It should be understood that by defining a clear workflow and each link in the workflow, the above can be achieved, for example Fig. 9 The state switching logic shown in the figure can control the character state displayed by the virtual digital human according to the current state and user input. Process arrangement can effectively standardize the execution logic of each link in the workflow, ensure that the interaction of the virtual digital human is carried out according to the predetermined path, and flexibly respond to different user inputs and scenarios. After the process arrangement is completed, the arrangement content can be converted into a json data structure so that it can be processed by the application program that specifically implements the interactive method of the embodiment of the present disclosure.

[0099] In addition, a state manager can be set up in the application to monitor and manage the various states of the virtual digital human to ensure smooth switching between different states of the virtual digital human; the state manager can track and manage the state of components or applications in the software application to ensure that the application can maintain consistency and predictability during user interaction, data update or other operations. Specifically, the state manager can achieve real-time response function by real-time monitoring of video frame switching during the playback of video materials of virtual digital humans in various states, that is, real-time monitoring of the playback progress of video materials, so as to respond in time according to the current state and user input. By controlling the state of the virtual digital human through process orchestration and state manager, users can be given the best immersive interactive experience with the most vivid effects and rich states.

[0100] According to the interactive method of the disclosed embodiment, the virtual digital human can be displayed on the holographic warehouse screen. In order to achieve a more realistic and flexible interactive effect, a variety of digital human states are preset, such as idle state, listening state, and backup state. The digital human can also be driven to speak in real time with the answers to the questions returned by the large model. Through the management and display of process orchestration and state manager, a better immersive interactive experience of the digital human can be achieved.

[0101] According to the interactive method of the embodiment of the present disclosure, a hyper-realistic virtual digital human interactive method connected to a large language model can be implemented, which can not only answer more interactive questions, but also switch between various states such as idle state, backup state, listening state, etc., and provide preset answer videos for preset questions and answers of commonly used preset questions, showing users the extremely flexible and immersive interactive dialogue experience of the digital human.

[0102] According to the interactive method of the embodiment of the present disclosure, through the process arrangement and state manager of the virtual digital human, preset answer videos are prepared in advance for common preset questions, and artificial intelligence technology is used to drive the virtual digital human to synthesize answer text in real time. It can be combined with the most vivid effects and rich states and switched flexibly, which can ensure the breadth of answering questions and show the vivid performance of the digital human, giving users the best immersive interactive experience.

[0103] Fig.11A block diagram of a virtual digital human interaction device according to an embodiment of the present disclosure is shown as follows: Fig.11 As shown, the device comprises:

[0104] An idle state display module 101 is used to display a virtual digital human in an idle state, where the idle state refers to the state of the human being when not interacting with the user;

[0105] A first switching module 102 is used to switch to displaying a virtual digital human in a listening state in response to triggering an interactive operation with the virtual digital human, wherein the listening state is a state of the human being listening to a user speaking;

[0106] The second switching module 103 is used to switch to displaying the virtual digital human answering the question in response to receiving a question from the user within a specified time period of switching to displaying the virtual digital human in the listening state.

[0107] In one possible implementation, in response to receiving a question from a user within a specified time period of switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human answering the question includes: when the question is a preset question and is not a sensitive question, displaying a preset answer video corresponding to the preset question, the preset answer video is used to display the virtual digital human answering the preset question; when the question is not a preset question and is not a sensitive question, displaying a real-time answer video generated based on the question, the real-time answer video is used to display the virtual digital human expressing the answer text generated for the question.

[0108] In one possible implementation, in response to receiving a question from a user within a specified time period of switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human who answers the question also includes: when the question is a sensitive question, switching to displaying the virtual digital human in a fallback state, wherein the fallback state represents a character state in which the question cannot be answered; after displaying the virtual digital human in the fallback state, switching to displaying the virtual digital human in the listening state.

[0109] In one possible implementation, in response to receiving a question from a user within a specified time period when the display is switched to displaying the virtual digital human in a listening state, switching to displaying the virtual digital human who answers the question includes: in response to receiving a question from a user within a specified time period when the display is switched to displaying the virtual digital human in a listening state, obtaining a question text corresponding to the question; judging whether the question is a sensitive question based on the question text, and / or judging whether the question is a preset question; wherein, judging whether the question is a preset question includes: matching the question text with each preset question in a preset question library to obtain a matching result of the question text, and the matching result includes: whether it matches the preset questions in the preset question library, and, when matching the preset questions in the preset question library, the preset questions that match the question text.

[0110] In one possible implementation, the device also includes: a third switching module, which is used to switch to displaying the virtual digital human in the idle state in response to not receiving any questions from the user within a specified time period after switching to displaying the virtual digital human in the listening state; or, a fourth switching module, which is used to switch to displaying the virtual digital human in the interaction end state in response to not receiving any questions from the user within a specified time period after switching to displaying the virtual digital human in the listening state, wherein the interaction end state is a state of the character ending the interaction with the user; and after displaying the virtual digital human in the interaction end state, switch to displaying the virtual digital human in the idle state.

[0111] In one possible implementation, in response to triggering an interactive operation with the virtual digital human, switching to displaying the virtual digital human in a listening state includes: in response to triggering an interactive operation with the virtual digital human, switching to displaying the virtual digital human in a welcoming state, wherein the welcoming state represents a character state that welcomes user interaction; after displaying the virtual digital human in the welcoming state, switching to displaying the virtual digital human in the listening state.

[0112] In one possible implementation, the device displays a virtual digital human on a holographic warehouse screen.

[0113] According to the interactive device of the embodiment of the present disclosure, by presetting a virtual digital human in an idle state, a virtual digital human that switches to a listening state when an interactive operation is triggered, and a virtual digital human that switches to an answering state after receiving a question from a user within a specified time period after switching to the listening state, it is possible to make the virtual digital human have a richer, more flexible and vivid expression form, thereby improving the interactive experience of the user and the virtual digital human.

[0114] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation thereof can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0115] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0116] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0117] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0118] Fig.12 1 is a block diagram of an electronic device 800 according to an embodiment of the present disclosure. The electronic device 800 may be provided as the above-mentioned terminal device. Fig.12 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output interface 812 (I / O interface), a sensor component 814 , and a communication component 816 .

[0119] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0120] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0121] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0122] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0123] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0124] The input / output interface 812 provides an interface between the processing component 802 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0125] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0126] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0127] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0128] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0129] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0130] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0131] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0132] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0133] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0134] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0135] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0136] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0137] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A virtual digital human interaction method, characterized in that: include: Displaying a virtual digital human in an idle state, wherein the idle state is a state of the human being when not interacting with the user; In response to triggering an interactive operation with the virtual digital human, the virtual digital human is switched to display a listening state, wherein the listening state is the state of the human being listening to the user's speech; In response to receiving a question from a user within a specified time period after the virtual digital human in the listening state is switched to be displayed, the virtual digital human answering the question is switched to be displayed.

2. The method according to claim 1, characterized in that: In response to receiving a question from a user within a specified time period of switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human answering the question comprises: When the question is a preset question and is not a sensitive question, displaying a preset answer video corresponding to the preset question, wherein the preset answer video is used to display a virtual digital person answering the preset question; When the question is neither a preset question nor a sensitive question, a real-time answer video generated based on the question is displayed, wherein the real-time answer video is used to display a virtual digital person expressing an answer text generated for the question.

3. The method according to claim 2, characterized in that In response to receiving a question from a user within a specified time period of switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human answering the question also includes: When the question is a sensitive one, the virtual digital person is switched to a state of being unable to answer the question; After the virtual digital human in the support state is displayed, the display is switched to the virtual digital human in the listening state.

4. The method according to claim 2 or 3, characterized in that: In response to receiving a question from a user within a specified time period of switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human answering the question comprises: In response to receiving a question from a user within a specified time period when the virtual digital human in the listening state is switched to display, obtaining a question text corresponding to the question; Determining whether the question is a sensitive question based on the question text, and / or determining whether the question is a preset question; Wherein, determining whether the question is a preset question includes: The question text is matched with each preset question in the preset question library to obtain a matching result of the question text, wherein the matching result includes: whether it matches the preset questions in the preset question library, and, when matching with the preset questions in the preset question library, the preset questions that match the question text.

5. The method according to claim 1 or 4, characterized in that: The method further comprises: In response to not receiving a question from the user within a specified time period after switching to displaying the virtual digital human in the listening state, switching to displaying the virtual digital human in the idle state; or, In response to not receiving a question from the user within a specified time period after switching to displaying the virtual digital human in the listening state, the display is switched to displaying the virtual digital human in the interaction end state, where the interaction end state represents the character state of ending interaction with the user; and after displaying the virtual digital human in the interaction end state, the display is switched to displaying the virtual digital human in the idle state.

6. The method according to claim 1, characterized in that The method of switching to displaying the virtual digital human in a listening state in response to triggering an interactive operation with the virtual digital human includes: In response to triggering an interactive operation with a virtual digital human, switching to displaying a virtual digital human in a welcoming state, wherein the welcoming state is a state of a character welcoming user interaction; After displaying the virtual digital human in the welcoming state, the display switches to displaying the virtual digital human in the listening state.

7. The method according to any one of claims 1 to 6, characterized in that: The method displays a virtual digital human on a holographic warehouse screen.

8. A virtual digital human interactive device, characterized in that: include: An idle state display module, used to display a virtual digital human in an idle state, wherein the idle state refers to the state of the human being when not interacting with the user; A first switching module is used to switch to displaying the virtual digital human in a listening state in response to triggering an interactive operation with the virtual digital human, wherein the listening state is a state of the human being listening to a user speaking; The second switching module is used to switch to displaying the virtual digital human answering the question in response to receiving a question from the user within a specified time period of switching to displaying the virtual digital human in the listening state.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images