Voice interaction method and device, equipment, cluster, storage medium and program product

Abstract

The invention discloses a voice interaction method and device, equipment, a cluster, a storage medium and a program product, and relates to the technical field of artificial intelligence interaction, and the method comprises the steps: determining an interaction object; acquiring facial expression data and audio data of the interaction object in real time, wherein the facial expression data at least comprises mouth shape data; based on the audio data and the mouth shape data, determining the integrity of the question proposed by the interaction object; at a first moment, determining that the question is complete based on the audio data and the mouth shape data, and determining question information of the interaction object ending the first moment; determining a response corresponding to the question, and generating driving information based on the response; and outputting media data corresponding to the response based on the driving information. According to the method provided by the embodiment of the invention, the question answering accuracy of the digital person can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence interaction technology, and in particular to a voice interaction method, device, equipment, cluster, storage medium and program product. Background Technology

[0002] With the development of artificial intelligence (AI) technology, AI digital humans are gradually becoming an important part of people's daily lives. Interactive AI digital human screens, as a brand-new interactive method, have been widely used in various settings, such as exhibition halls, conference centers, businesses, and schools. These digital humans interact with the audience in real time through holographic projection technology, providing a variety of services including information dissemination, entertainment, and education.

[0003] Typically, AI digital human interaction systems consist of audio acquisition devices, such as microphones on a large screen or separate Bluetooth microphones, large-screen devices equipped with cameras, cloud-based digital human generation services, speech recognition services, speech generation services, and language model services. Users input a wake-up word through the audio acquisition device to activate the digital human, then ask questions, and wait for the digital human to analyze and understand the questions, providing audio answers and corresponding actions, achieving an experience similar to interacting with a real person.

[0004] However, in high-traffic areas such as exhibition halls and conference centers, environmental noise levels are often high. Therefore, to improve speech recognition accuracy when communicating with the digital human, it is recommended to use a dedicated microphone with strong noise reduction capabilities. Furthermore, because everyone speaks differently, including in terms of speech rate, volume, and pauses, these factors increase the difficulty for the device to correctly segment user input and filter noise in practical applications. This can lead to errors in the digital human's understanding of user questions, thus affecting the accuracy of its responses. Summary of the Invention

[0005] To address the aforementioned problems, this application provides a voice interaction method, apparatus, device, cluster, storage medium, and program product to solve the problem that electronic devices cannot accurately answer user questions due to their inability to correctly segment sentences and filter noise.

[0006] In a first aspect, this application provides a voice interaction method, comprising: determining an interaction object; acquiring facial expression data and audio data of the interaction object, wherein the facial expression data includes at least lip-shape data; determining the completeness of a question raised by the interaction object based on the audio data and lip-shape data; if the question is determined to be complete based on the audio data and lip-shape data, determining a response corresponding to the question, and generating driving information based on the response; and outputting media data corresponding to the response, such as audio and / or driving a digital human to display an animation corresponding to the response, based on the driving information.

[0007] The voice interaction method according to the embodiments of this application combines audio data and user facial expression data, which is more conducive to accurately analyzing the user's input questions and intentions. This allows for better determination of whether the question asked by the interactive object is complete or incomplete, thereby improving the experience of digital human large-screen interaction. Furthermore, the interaction process no longer relies on a dedicated microphone for input, avoiding incomplete sentences due to improper operation and reducing hardware costs.

[0008] As an embodiment of the first aspect of this application, determining that a problem is complete based on audio data and lip-sync data includes: determining that the lip-sync data of the interactive object is less than or equal to a lip-sync speech threshold, and that the signal value of the audio data is greater than an audio silence threshold, thus determining that the problem is complete. Combining the lip-sync speech threshold with the audio data can improve the accuracy of problem completeness judgment and can better identify noise.

[0009] As an embodiment of the first aspect of this application, the problem determination based on audio data and lip-shape data is complete, and further includes: noise reduction processing of the audio data to ensure that the identified problem is more accurate.

[0010] As an embodiment of the first aspect of this application, determining whether a problem is complete based on audio data and lip-sync data includes: when it is determined that the lip-sync data of the interactive object is less than or equal to a lip-sync speaking threshold, and the signal value of the audio data decreases to less than or equal to an audio silence threshold, then it is determined that the problem is complete. This determination method can not only accurately identify the completeness of the problem, but also analyze whether there is noise at the scene and make timely decisions on the handling method.

[0011] As an embodiment of the first aspect of this application, determining the completeness of a question posed by an interactive object based on audio data and lip-sync data includes: if the question is determined to be incomplete, and no response corresponding to the incomplete question is output within a preset time period, maintaining the data acquisition state. This avoids inaccurate answers to incomplete questions.

[0012] As an embodiment of the first aspect of this application, maintaining the data acquisition state includes: outputting a prompt message indicating that the interactive object is acquiring data or maintaining the state of the digital human listening, and continuing to acquire the audio data and facial expression data of the interactive object. Users can understand how to operate based on the state of the digital human.

[0013] As an embodiment of the first aspect of this application, determining that the problem is incomplete includes: determining that the mouth shape data of the interactive object is greater than the mouth shape speaking threshold, and the signal value of the audio data is reduced to less than or equal to the audio mute threshold; or, determining that the mouth shape data of the interactive object is greater than the mouth shape speaking threshold, and the signal value of the audio data is greater than the audio mute threshold.

[0014] As an embodiment of the first aspect of this application, determining that the problem is incomplete also includes prompting the interactive object to adjust the volume or move closer to the pickup device to ensure that the complete sentence is obtained.

[0015] As an embodiment of the first aspect of this application, determining the response corresponding to a question includes: acquiring complete audio data of the question; and obtaining the response corresponding to the question based on the audio data of the question and a large language model. Using a large language model or knowledge base can yield more accurate answers.

[0016] As an embodiment of the first aspect of this application, obtaining a response corresponding to a question based on audio data of the question and a language big model includes: converting the audio data of the question into text; inputting the text into the language big model, obtaining a response text corresponding to the text based on the language big model; and converting the response text into an audio response.

[0017] As an embodiment of the first aspect of this application, obtaining a response corresponding to a question based on the audio data and a large language model further includes: outputting a prompt indicating whether the response matches the intent of the interaction; if a first operation matching the intent of the interaction object is received, outputting audio corresponding to the response and / or driving a digital human to display an animation corresponding to the response; if a second operation not matching the intent of the interaction object is received, adjusting the lip-sync threshold and the audio mute threshold, and prompting the interaction object to ask the question again, until the response matches the intent of the interaction object. By confirming the accuracy of the question through user interaction, the audio mute threshold and the lip-sync threshold can be dynamically adjusted to the threshold that best matches the user, further improving the accuracy of answering questions.

[0018] As an embodiment of the first aspect of this application, the first operation includes: pressing a specific function key, assuming a specific posture, or inputting a keyword that matches the intent; the second operation includes: pressing a specific function key, assuming a specific posture, or inputting a keyword that does not match the intent. These operations can identify the user's intent.

[0019] As an embodiment of the first aspect of this application, determining the interaction object includes: receiving a wake-up word input by a user and acquiring the user's lip-sync data; and determining the interaction object from the user whose wake-up word and lip-sync data match based on the wake-up word and lip-sync data. Using a combination of wake-up word and lip-sync data to confirm the interaction object not only allows for accurate identification of the interaction object but also eliminates the need for dedicated voice acquisition equipment, reducing hardware costs and improving the user's interactive experience.

[0020] As an embodiment of the first aspect of this application, the method further includes: extracting and storing voiceprint features from the audio data of the interactive object; when it is determined that the interactive object is not asking a question for the first time, directly obtaining the lip-sync threshold and audio silence threshold corresponding to the interactive object based on the voiceprint features. This approach facilitates the rapid acquisition of the lip-sync threshold and audio silence threshold matching the interactive object when the interactive object asks a question again, thereby quickly obtaining an accurate answer.

[0021] As an embodiment of the first aspect of this application, the facial expression data also includes one or more of eyebrow movement data, eye movement data, and nose movement data.

[0022] As an embodiment of the first aspect of this application, the media data includes animation data of a digital human or an object of a specific shape on the display screen, as well as output audio data.

[0023] Secondly, this application also provides a voice interaction device, comprising:

[0024] The object confirmation module is used to determine the interaction object;

[0025] The acquisition module is used to acquire facial expression data and audio data of the interactive object. The facial expression data includes at least mouth shape data.

[0026] The integrity verification module is used to determine the integrity of the questions raised by the interactive object based on audio data and lip-sync data.

[0027] The generation module, if the question is determined to be complete based on audio data and lip-sync data, is used to determine the response corresponding to the question and generate driving information based on the response;

[0028] The output module is used to output media data corresponding to the response based on the driver information.

[0029] Thirdly, this application also provides an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor for executing the voice interaction method explained in the first aspect.

[0030] Fourthly, this application also provides a computing device cluster, including at least one computing device, each computing device including a processor and memory;

[0031] A processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device to cause a cluster of computing devices to perform the voice interaction method explained in the first aspect.

[0032] Fifthly, this application also provides a computer-readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the voice interaction method explained in the first aspect.

[0033] Sixthly, this application also provides a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform the voice interaction method explained in the first aspect. Attached Figure Description

[0034] Figure 1 This is a scene diagram illustrating user interaction with a digital human according to an embodiment of this application;

[0035] Figure 2 This is a data flow diagram of user interaction with the digital human in some embodiments;

[0036] Figure 3 This is a flowchart illustrating the interaction between a user and a digital human large-screen device in some embodiments;

[0037] Figure 4 This is a flowchart of a voice interaction method based on digital humans, as described in an embodiment of this application.

[0038] Figure 5 This is a schematic diagram of the structure of the electronic device of this application;

[0039] Figure 6 This is a flowchart of a voice interaction method according to another embodiment of this application;

[0040] Figure 7 This is a schematic diagram of the software module structure of the large-screen device and the cloud device in an embodiment of this application;

[0041] Figure 8 This is an interactive flowchart of the software modules in the embodiments of this application;

[0042] Figure 9 This is a flowchart illustrating the adjustment of preset values ​​based on detected lip shape and audio in an embodiment of this application.

[0043] Figure 10 This is a schematic diagram of the structure of the voice interaction device according to an embodiment of this application;

[0044] Figure 11 This is a schematic diagram of the structure of a computing device cluster according to an embodiment of this application;

[0045] Figure 12 This is an interaction diagram between computing device clusters according to embodiments of this application;

[0046] Figure 13 This is a schematic diagram of an on-chip system according to an embodiment of this application. Detailed Implementation

[0047] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0048] The illustrative embodiments of this application include, but are not limited to, a voice interaction method, apparatus, device, cluster, storage medium, and program product based on digital humans.

[0049] To facilitate understanding of the technical solution of this application, the technical problem to be solved by this application will be explained first.

[0050] refer to Figure 1 , Figure 1 This illustration shows a scenario of user interaction with a digital human, based on an embodiment of this application. Figure 1 As shown, the scene includes a holographic projection screen device 02 and a cloud device 01 connected to it. The screen displays a digital human 03 (media data), who is conversing with a user 04. Digital human 03 is a digital virtual avatar, a software system created using artificial intelligence technology. This digitally similar avatar can perform specific actions, such as mimicking human speech. When user 04 interacts with digital human 03, the hardware of the screen device 02 captures the user's audio, processes it through audio processing software, converting it into machine-readable text information. Based on this text information, it provides responses to user 04's interactions, such as answers to user questions. The screen device 02 generates corresponding voice responses and outputs the digital human's corresponding actions based on this output. Through the expressive output of digital human 03, users can experience an effect close to conversing with a real person. For example, the large-screen device 02 may include audio acquisition devices, such as microphones, and image capture cameras, as well as software services: Artificial Intelligence Generated Content (AIGC), i.e., digital human generation services, Automatic Speech Recognition (ASR) services, and Text-to-Speech (TTS) services. The cloud device 01 may include services such as a large language model service and a knowledge base system. Users can wake up the digital human through the audio acquisition device and ask questions. The large-screen device 02 uploads the questions to the cloud device 01, where the software service system analyzes and understands the questions, outputting audio and actions, thus achieving an interactive experience with a real person.

[0051] In the above embodiments, the description uses the deployment of AIGC digital human generation service, ASR speech recognition service, and TTS speech generation service on a large-screen device as an example. In other embodiments, these software modules can also be deployed on edge computing nodes, such as servers or gateways close to the large-screen device, to reduce network transmission latency. Furthermore, these software or some services can also be deployed in the cloud; this application does not limit this to any particular deployment.

[0052] In other embodiments, the language big model service and knowledge base system can also be deployed on a large-screen device on the edge.

[0053] It should be noted that in this application's embodiments, digital humans are used as media data for description. In some embodiments, media data can also be other objects, such as animals, or robots or objects of a specific shape, etc., and this application is not limited in this respect.

[0054] refer to Figure 2 , Figure 2 Data flow diagrams illustrating user interaction with the digital human are shown in some embodiments. For example... Figure 2 As shown, the interaction process between the questioner and the digital human includes the following steps: ① Real person asks a question. The user can first wake up the digital human and then use a dedicated microphone with noise reduction to ask the question. In conference centers or exhibition halls, it is usually necessary to have someone guide the user to avoid multiple people asking questions. ② The digital human large-screen device collects audio through a dedicated microphone. ③ ASR speech recognition. The ASR automatic speech recognition module processes the audio stream, that is, the ASR module processes the input speech into text, and detects whether the input speech is a complete sentence according to a fixed silence detection duration (e.g., 500ms). When it is recognized as a complete sentence, the text is input into a language model, such as the Pangu model or a knowledge base question-and-answer system. ④ The language model (or large language model) or knowledge base question-and-answer system generates an answer (response) and outputs the answer to the text-to-speech module. ⑤ TTS speech synthesis. TTS converts the text of the answer into the corresponding speech and sends it to the digital human generation service. ⑥ The digital human generation service generates the corresponding driving parameters and sends the driving parameters and the response corresponding to the question to the processor of the large-screen device. The processor of the large-screen device drives the digital human to make movements based on the driving parameters and controls the speakers to play audio. Users can listen to audio and watch digital human animations, realizing an interactive experience with a real person.

[0055] In the above description, during the interaction between the user and the digital human, the questioner needs to hold a microphone and also needs to manually intervene to avoid interference from those around them, resulting in a poor user experience. Furthermore, in determining whether the user's input speech is a complete sentence, relying solely on silence detection to judge whether the question is finished is easily influenced by the user's speaking habits. For example, for people who speak slowly or frequently pause, silence detection may exceed the default threshold, leading to incomplete questions and potentially unintended answers. In other embodiments, to address the low accuracy issue mentioned above in judging the completeness of the questioner's input, a step using a large language model is added. For example, after the ASR module performs speech recognition as described in ③ASR speech recognition, the recognized text is first sent to the large language model, which then performs a completeness judgment on the question. If the large language model has not yet recognized the completeness of the question, the ASR module continues processing. However, performing completeness judgment through preprocessing by the large language model every time increases latency, typically greater than 1 second, impacting the user's interactive experience.

[0056] The following section further explains the user-digital human interaction process in the embodiments described above, taking into account the hardware and software structure of the large-screen device.

[0057] refer to Figure 3 , Figure 3 Flowcharts illustrating user interaction with the digital human large-screen device in some embodiments are shown. For example... Figure 3 As shown, this includes large-screen devices and cloud devices. The large-screen devices include hardware such as dedicated microphones, and software modules such as ASR modules and digital human / TTS modules. The cloud devices include large language models (such as the Pangu large model / knowledge base question-answering system). Figure 3 The interactive process shown is Figure 2 This corresponds to steps ①-⑥ of the interactive process. This interactive process mainly includes two parts: the first part is to wake up the digital human, and the second part is to identify whether the user's question has been completed and provide an answer.

[0058] The first part, the process of waking up the digital human, includes S10-S13, specifically: S10, local sound pickup and noise reduction. Sound pickup involves acquiring audio data, for example, the user inputting a wake-up word through a dedicated microphone. Noise reduction is performed by the dedicated microphone itself to reduce audio noise, ensuring the accuracy of the wake-up word. S11, wake-up detection. This determines the accuracy of the wake-up word during multi-person conversations, which can be done through text detection. S12, the audio module corresponding to the microphone sends a wake-up command to the digital human / TTS. For example, after determining the wake-up word, the audio module sends it to the digital human / TTS to trigger the digital human's action. S13, the digital human / TTS returns a welcome message, preparing to answer user questions.

[0059] The second part identifies whether the user's question is complete, including two implementation examples. The difference between the two examples lies in the process of determining the completeness of the user's question after the user inputs the question via voice. For example... Figure 3 As shown, the first embodiment (Scheme 1) includes steps S14-S18, and the second embodiment (Scheme 2) includes steps S14, S19-S25.

[0060] The interaction process of the two embodiments described above is described below.

[0061] First, let's describe Option 1, such as... Figure 3 As shown, in S14, the audio module corresponding to the microphone sends the voice question to the ASR. After the digital human is awakened, the user asks a question to the digital human through a dedicated microphone, inputting audio data. In S15, the ASR module performs silence detection according to the configured threshold. The ASR module obtains the audio data (question) input by the user, recognizes the speech and converts it to text, and recognizes the audio signal. If the audio signal value in the audio is less than or equal to the threshold, such as 500ms (threshold silence detection), then it is determined that the question has been completed (the sentence is complete). In S16, the ASR module notifies the language model that the question has been completed. The language model searches based on the question and finds the answer corresponding to the question. In S17, the language model returns the answer to the digital human / TTS module. The TTS module converts the text into speech, and the digital human service module generates the corresponding driving parameters. In S18, the digital human responds. The large-screen device can drive the digital human animation display through the driving parameters, and at the same time output the response audio corresponding to the question through the speaker. During this interaction, the questioner needs to hold the microphone, and there is also the need for human intervention from people around, resulting in a poor user experience. Furthermore, the ASR module's use of threshold silence detection to judge the completeness of sentences or the accuracy of question completion is relatively low, resulting in low accuracy of the answers.

[0062] Option 2, such as Figure 3As shown, S14 involves voice questioning. The user asks a question to the digital human through a dedicated microphone. After the user asks the question, the large-screen device sends the question content to the ASR module. Unlike Scheme 1, in S19, the ASR module sends the voice to the language big model, which determines whether the question is complete. S20 involves the language big model predicting the completion of the question. That is, the language big model can determine the probability that the user may have completed the question based on the input voice. When the probability exceeds a set value, such as 90%, the question is considered complete. S21 involves the language big model sending the recognition result to the ASR module. S22 involves the ASR module continuing detection or returning text. The ASR module can further detect silence according to a configured threshold. After determining that the question is complete, it converts the audio to text and executes S23, sending the result of recognizing the completion of the question to the language big model / knowledge base system. S24 involves the language big model / knowledge base system returning the answer to the digital human / TTS module. The TTS module converts the text into speech, and the digital human service module generates the corresponding driving parameters. This step is the same as S17. S25 involves the digital human / TT driving the digital human to respond. Large-screen devices can drive digital human animation displays through driving parameters, while simultaneously outputting audio responses to questions via speakers—the same step as S18. While this interaction improves the completeness of recognized sentences or the accuracy of question completion, each ASR recognition result requires a large model to determine whether the question has been completed. This process is time-consuming, significantly increasing latency and reducing the interactive experience.

[0063] To address the aforementioned problems, this application provides a voice interaction method that, in addition to silence detection, adds lip-shape detection. Through an adaptive algorithm, it achieves optimal matching of parameters between the two methods, eliminating the need for manual parameter adjustments and enabling more accurate judgment of question completion, ultimately improving the experience of interacting with the digital human. Furthermore, the method in this application's embodiments can accurately identify the questioner (interaction object), reducing reliance on the sampling microphone, lowering the possibility of inaccurate recognition due to improper operation, and reducing hardware costs.

[0064] The voice interaction method of this application embodiment will be described below with reference to the accompanying drawings.

[0065] refer to Figure 4 , Figure 4 A flowchart of a voice interaction method according to an embodiment of this application is shown. This method can be executed by a large-screen device. (In conjunction with...) Figure 4 As shown, the interaction method includes S410-S460.

[0066] S410, Determine the interaction target. The interaction target is the questioner or person asking the question, identified from the users in front of the large screen based on audio and facial expression data.

[0067] For example, during the confirmation process of the interaction object, the large-screen device can compare the user's input voiceprint with the stored voiceprint features. When it is determined that the user's voiceprint already exists, the current interaction object can be directly identified based on the voiceprint features. In addition, it can also determine whether it is the person currently speaking by combining the keywords in the voice with the mouth shape of the facial expression.

[0068] In some embodiments, other methods may be used to determine the interaction object. For example, a specific action performed by the user, such as a specific gesture, can be used to determine the interaction object. Alternatively, a dedicated hardware device can be used to determine the interaction object, such as a microphone paired with a large screen; the user picking up the microphone indicates a confirmed interaction object. This application does not limit the scope of these embodiments.

[0069] S420: Obtain facial expression data and audio data of the interactive object.

[0070] Once the interaction target is identified, facial expression data can be directly acquired using an image acquisition device. For example, a camera on a large-screen device can capture the user's facial image, including key facial points such as the eyes, nose, and area around the mouth. Changes in the position of a single key point or a combination of changes in multiple key points can reflect changes in expression. For instance, a continuously opening and closing mouth indicates that the user is speaking. Changes in key points or combinations of key points further highlight the state of speech. Audio data can capture the voice signal of the interaction target, which, combined with facial expression data, makes it easier to accurately determine whether the interaction target has completed asking the question.

[0071] S430: Determine if the question is complete. If, at time T1 (the first time), the question is complete, proceed to S440. Otherwise, return to S420 and continue acquiring facial expression data and audio data corresponding to the interaction until the question is confirmed to be complete.

[0072] S440, confirm the corresponding problem information based on the information received up to time T1.

[0073] For example, the moment when the interactive object inputs audio, i.e., the moment the question is asked, is denoted as T0. If it is determined at time T1 that the question asked by the interactive object is complete, then the audio input by the interactive object during the time period from T0 to T1 is obtained as the audio data of the complete question. Further, based on the recognition results of the audio information, the response (answer) to the question is determined.

[0074] S450 determines the corresponding answer based on the question information and generates the driving information.

[0075] In some embodiments, the question information is sent to the cloud, and relevant responses are obtained based on a language big data model or a knowledge base system, or a combination of both. The specific process and algorithms for obtaining the answers can be found in existing technologies and are not described in detail in this embodiment.

[0076] After determining the answer, the large-screen device can generate driving information corresponding to the answer, such as determining the sequence of actions the digital human should perform, such as nodding, waving, opening and closing the mouth, and audio playback instructions.

[0077] The S460 outputs audio and displays digital human animations based on driver information.

[0078] For example, based on the driving parameters of the hand gesture, the large-screen device controls the digital human on the large screen to output the hand gesture, and outputs the corresponding audio response to the question based on the playback command.

[0079] The voice interaction method according to the embodiments of this application combines audio data and user facial expression data, which is more conducive to accurately analyzing the user's input questions and intentions. This allows for better determination of whether the interaction object's question is complete, thereby improving the experience of digital human large-screen interaction. Furthermore, this interaction process no longer relies on a dedicated microphone for input, avoiding incomplete sentences due to improper operation and reducing hardware costs.

[0080] The voice interaction method of this application embodiment will be described below in conjunction with the specific structure of the electronic device.

[0081] refer to Figure 5 , Figure 5 A schematic diagram of the electronic device 100 of this application is shown. The structure of the electronic device can be as follows: Figure 1 The schematic diagram of the master device can also be a schematic diagram of other slave devices.

[0082] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) connector 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0083] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0084] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0085] The processor 110 can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.

[0086] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0087] In some embodiments, the processor 110 receives the user's audio and identifies the interaction object, for example, by using voiceprint features, image features, or sensor data to identify the user. It acquires the interaction object's facial expression data and audio data in real time, where the facial expression data may include lip-sync data. Based on the audio data and the lip-sync data, it determines the completeness of the question posed by the interaction object. For example, it uses an adaptive threshold algorithm for audio silence detection and lip-sync silence detection, employing existing lip-sync speaking thresholds and audio silence thresholds, and compares them with the audio data and facial expression data input by the interaction object to determine whether the interaction object's current question is complete or whether the question is comprehensive. When at time T1 (the first time), if it is determined that the question is complete, it obtains the question information from the start of the user's question to time T1. The processor 110 sends the question information to the cloud server, which determines the response corresponding to the question based on the question information and generates driving information based on the response. The processor 110 receives the driving information and controls the speaker 170A to output audio corresponding to the response, and / or controls the digital human display on the screen 194 to display an animation corresponding to the response.

[0088] In some embodiments, after obtaining the question and answer corresponding to the question, the processor 110 further outputs confirmation information, for example, by asking the interactive object via voice whether the response is the desired answer. Alternatively, the response can be displayed on the display screen 194 for the interactive object to confirm whether the response is accurate. If the interactive object confirms that it is correct, the processor 110 stores the interactive object's voiceprint information, as well as the corresponding audio mute threshold and lip-sync threshold. If the interactive object confirms that it is incorrect, the processor 110 controls the audio module 170 to output a re-entry question, or controls the display screen 194 to provide a text prompt for re-entry. The processor 110 dynamically adjusts the audio mute threshold upon re-obtaining the audio data and lip-sync data input by the interactive object. Adjusting the audio mute threshold refers to the threshold for the duration of silence when the user finishes speaking, and adjusting the lip-sync threshold refers to the threshold for when the user's lip movements remain still after speaking, until the interactive object confirms that the response is accurate. The adjusted audio mute threshold and lip-sync threshold are stored along with the interactive object's voiceprint or facial features, so that when the interactive object asks a question again, the corresponding threshold can be directly obtained, improving the response speed. In addition, by dynamically adjusting the audio mute threshold and lip-sync threshold, personalized and precise matching can be achieved for each user, improving the accuracy of responses and enhancing the interactive experience.

[0089] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0090] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0091] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0092] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0093] In some embodiments, both the mobile communication module 150 and the wireless communication module 160 can enable the electronic device 100 to communicate with the cloud. The processor 110 can control the mobile communication module to send questions posed by the interactive object to the cloud, where the cloud analyzes and understands the questions and returns a response.

[0094] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0095] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0096] In some embodiments, the display screen 194 may display a digital human, along with information related to responses or questions. The processor 110 may control the digital human on the display screen 194 to perform corresponding actions. For example, when the digital human greets the user, it may wave its hand. Or, when the electronic device outputs an answer to a question from the interactive object, the digital human's mouth may open and close, as if in a real conversation with the user.

[0097] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0098] In some embodiments, the camera 193 can acquire facial expression data and body movements of the interactive object. It can also acquire video data of the entire exhibition hall environment and people.

[0099] Electronic device 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor. For example, it can acquire audio data input by the user through the microphone and output the corresponding response audio through speaker 170A.

[0100] The voice interaction method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. This method can be applied to devices such as... Figure 5 In electronic devices with the hardware structure shown, or in electronic devices with more or fewer components than shown, or in electronic devices with similar hardware and software structures, or in electronic devices with different component arrangements, or in electronic devices with different component arrangements.

[0101] The voice interaction method provided in this application can be applied to electronic devices. For example, the electronic device can be a large-screen device, a smart TV, an in-vehicle device, or a mobile phone, tablet computer, wearable device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), artificial intelligence (AI) device, or any other device with a display screen and audio player. This application does not limit the specific type of electronic device. In the following embodiments, a large-screen electronic device is used as an example for explanation.

[0102] refer to Figure 6 , Figure 6 A flowchart of a voice interaction method according to another embodiment of this application is shown. The method is executed by a large-screen device and includes steps S601-S608.

[0103] S601, local wake-up, which means acquiring the user's wake-up voice and lip-reading data to wake up the digital human.

[0104] The wake-up voice can be acquired through the microphone of the large-screen device (also known as local wake-up). In some embodiments, the wake-up voice can be a specific wake-up word, such as calling the name of the digital human.

[0105] Lip shape data can be captured by the camera of a large-screen device. For example, if multiple people are standing in front of a large-screen device, their facial expression data will be captured by the camera.

[0106] S602 matches lip movements and wake words, and identifies the speaker.

[0107] Once the wake word is confirmed to be accurate, the digital human can be activated, i.e., the digital human is woken up. The wake word corresponds to a threshold for mouth opening or closing, such as the distance between the upper and lower lips. This threshold is compared with the mouth shape changes of multiple users. The user whose mouth shape changes match the threshold is the interaction target (the current speaker), thus completing speaker recognition. In this process, combining the wake word with mouth shape allows for accurate identification of the interaction target. Compared to existing technologies that only use keywords for audio recognition of interaction targets, this method can more accurately identify the interaction target. Furthermore, this method eliminates the need for a dedicated microphone for questioning, reducing reliance on hardware devices and lowering hardware costs.

[0108] S603, obtain audio data and lip-sync data input by the interactive object.

[0109] For example, when the interactive object asks a question to the digital human, the microphone of the large-screen device picks up the audio data. At the same time, the camera captures lip movement data as the interactive object speaks, such as images of the state of its lips opening and closing.

[0110] S604 determines the relationship between the currently input audio data and lip-sync data and the threshold. In the following text, audio data is referred to as audio.

[0111] For example, audio can be compared with an audio mute threshold, and lip movements can be compared with a lip-reading threshold. Based on the comparison results, it can be determined whether the interactive object's question is completed or whether the question is complete.

[0112] The comparison between audio and the audio silence threshold can be achieved using Voice Activity Detection (VAD). This detection method is used in speech processing to detect the presence of a speech signal, and in speech recognition, it can detect whether a person's voice is silent. For example, when silence is detected, it can be preliminarily determined that the user's input question has been completed.

[0113] The audio mute threshold in this application embodiment can be a threshold level, such as -60dBFS (Full Scale), below which audio is considered mute. Furthermore, in some embodiments, energy threshold, zero-crossing rate, short-time average magnitude, and short-time average absolute value can also be used as the audio mute threshold; this application embodiment does not limit the specific method used.

[0114] The process of comparing lip movements with lip-reading thresholds employs lip-reading detection. Lip-reading detection can identify faces in images, determine if the mouth is moving, and extract lip-reading change features. These features can be input into lip-reading models to recognize speech pronunciation. This allows for the identification of the lip shape associated with wake words, thus determining the interactive target. Furthermore, combining this with audio silence detection can accurately identify the interactive target and the question posed by that target. The following explains several results for comparing the current input audio and lip movements with thresholds.

[0115] The lip-sync threshold can be measured using the mouth opening degree, such as the distance between the upper and lower lips, which is 1 cm. When the mouth opening degree exceeds 1 cm, the system considers the user to be speaking; otherwise, it considers the user not speaking. In some embodiments, the mouth shape change rate can also be used; for example, if the mouth shape change rate is less than 0.3 times per second, the system considers the user not speaking. In other embodiments, mouth feature vectors, mouth shape durations, and mouth shape similarities can also be used as lip-sync thresholds. These thresholds can be adjusted according to actual conditions to achieve the best recognition results.

[0116] In some embodiments, in addition to the lip-shape data described above, audio data can also be combined with other facial expression data. For example, based on lip opening and closing (lip-shape data), it can be combined with eyebrow movements to improve the accuracy of question identification. For instance, raised eyebrows indicate surprise, curiosity, or inquiry. Frowning expresses confusion, disagreement, or doubt. Furthermore, in other embodiments, facial expression data may also include cheek movements. That is, audio data is combined with lip-shape data and cheek movements to improve the accuracy of question identification. For example, puffed-out cheeks may indicate dissatisfaction or anger, while relaxed cheeks indicate relaxation or happiness. Using this combination of facial expression data allows for more accurate identification of the user's intent.

[0117] like Figure 6 As shown, step S604 will perform subsequent processing based on four different judgment results. The four judgment results are as follows:

[0118] The first scenario involves audio frequencies exceeding the audio mute threshold, while lip movements are less than or equal to the lip-reading threshold, indicating that the interactive object's question has been completed. In this case, since the lip movements indicate a stopped speaking state, but the audio volume remains high, interference noise is detected, and the audio data undergoes noise reduction processing. Specific noise reduction methods can be found in existing audio noise reduction techniques and will not be detailed here. After noise reduction, the ASR module recognizes the audio data and converts it into text.

[0119] The second scenario involves audio values ​​less than or equal to the audio mute threshold, and lip movements greater than the lip-reading threshold. This indicates that the user is still speaking, and the question is incomplete. The user is then prompted to adjust the volume of the microphone, or to move closer to the digital human or increase the volume, in order to clearly capture the user's audio data and continue capturing audio until the question is confirmed to be complete.

[0120] The third type is when the audio is less than or equal to the audio mute threshold and the lip movement is less than or equal to the lip speech threshold, indicating that the interactive object has stopped speaking and the interactive object's question has been completed. The ASR directly recognizes the audio data and converts the audio into text.

[0121] The fourth type is when the audio is greater than the audio mute threshold and the lip movement is greater than the lip speech threshold, indicating that the interactive object is still speaking. In this case, the sound pickup state is maintained until it is determined that the interactive object has completed its question.

[0122] S605, the language big model performs semantic understanding of text and confirms questions and intentions through digital humans and interactive objects.

[0123] For example, when the interaction object has completed asking a question, the large-screen device sends the text corresponding to the question to a language model in the cloud. The language model understands and analyzes the text and provides a corresponding response. After the response returns to the large-screen device, it is determined whether the response accurately answers the interaction object's question by asking the user. If the user confirms that the response is accurate, it means that the response expresses their intention. If the interaction object confirms that the response is inaccurate, it means that the response does not express the interaction object's intention, and the question needs to be asked again. Specific methods for obtaining the answer to the question based on the language model or knowledge base can refer to existing technologies, which will not be described in detail in this application.

[0124] S606, decides whether to adjust the lip-sync threshold and audio mute threshold.

[0125] When the large screen device receives a negative instruction from the interactive object, meaning that the interactive object indicates that the response provided by the digital human is inaccurate, it determines that the audio mute threshold and lip-sync threshold need to be adjusted, and executes S607.

[0126] S607, adjust the audio mute threshold and lip-sync threshold.

[0127] For example, when the interactive object interacts with the digital human for the first time, the large-screen device uses pre-stored audio silence thresholds and lip-reading thresholds as initial values ​​for judgment. At this point, the thresholds have not been verified and are not personalized relative to the interactive object. Therefore, when answering the interactive object's question, the accuracy of the answer can be further determined. Furthermore, if the interactive object's response is inaccurate, it performs a corresponding negative action (second operation). For example, the interactive object might verbally reply with words like "inaccurate" or "wrong," or click a feature function key, such as clicking the "set no" option on the screen, or make a specific gesture, such as waving or shaking its head, to confirm the inaccuracy of the answer. This allows for adjustment of the initial thresholds based on the interactive object's speaking habits. For example, if the question is incomplete, the audio silence threshold and lip-reading threshold are increased. If the sentence contains the question but other text, the thresholds are decreased, and the user is asked again to confirm the effect until the given answer is deemed accurate.

[0128] When the large-screen device receives an accurate response from the interactive object, that is, the interactive object performs the correct corresponding action (first operation), for example, the interactive object replies with affirmative words such as "correct" or "yes" via voice, or clicks a special function key, such as the interactive object clicking the correct function key option set on the screen, or a specific gesture, such as making an OK gesture, nodding, etc., then the answer is confirmed to be accurate, and the audio mute threshold and lip-sync threshold are not adjusted, and S608 is executed.

[0129] The S608 records and stores the voiceprint information of the interacting object, as well as the audio silence threshold and lip-sync threshold. When the interacting object interacts with the digital human again, the large-screen device can directly obtain the corresponding audio silence threshold and lip-sync threshold based on the voiceprint information and quickly obtain the question data, improving interaction efficiency and experience.

[0130] According to the interaction method of this application embodiment, by waking up and capturing the speaker's lip movements with a camera, the actual interaction object can be accurately identified, without the need for a dedicated microphone. When the interaction object asks a question, its audio data and lip movement data are acquired in real time, and each data is compared with a threshold value. The results of these comparisons are then used to comprehensively determine whether the question is complete, allowing for accurate assessment of its completeness. Furthermore, potential adverse factors, such as low microphone volume, ambient noise, or pauses in speech, can be identified and mitigated. In addition, the response is confirmed with the interaction object, and the lip movement threshold and audio silence threshold are dynamically adjusted. Through multiple rounds of correction, a lip movement threshold and audio silence threshold matching the interaction object are obtained, achieving a personalized communication experience. The interaction object's voiceprint and the matching lip movement threshold and audio silence threshold are stored and saved so that when the interaction object interacts with the digital human in subsequent interactions, the large-screen device can directly obtain the threshold value based on the voiceprint information, improving interaction efficiency and experience.

[0131] The voice interaction method of this application embodiment will be described below in conjunction with the software architecture of large-screen devices and cloud devices.

[0132] refer to Figure 7 , Figure 7 A schematic diagram of the software module structure of the large-screen device and cloud device according to an embodiment of this application is shown. Figure 7 As shown, the large-screen device includes: a voice wake-up module, a lip-sync detection module, a voiceprint module, a silence detection and lip-sync detection threshold adjustment module, ASR, TTS, a digital human management module, a digital human voice driving module, and audio and video transmission modules. The cloud-based device includes a large language model / knowledge base service.

[0133] These modules can be configured as follows: Figure 5The software system of the electronic device 100 shown can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment takes a layered Android system as an example. A layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. These modules can be located in the application layer, application framework layer, and / or system libraries, etc.

[0134] The voice wake-up module can launch a device or application by recognizing a specific wake-up word or command. For example, it can detect whether the user has entered the corresponding voice for a preset wake-up word; a successful match indicates successful wake-up and notifies the user that the digital human can be activated. This voice wake-up module can perform... Figure 6 The S602 step involves determining whether the wake word is accurate; if accurate, the digital human is activated.

[0135] The lip-sync module acquires lip-sync data captured by the camera and checks if it matches the lip-sync of the wake word. It identifies the best match from multiple lip-sync samples to determine the questioner (interaction target). This lip-sync module can perform... Figure 6 The steps for identifying the speaker in S602 are shown.

[0136] The silence detection and lip-sync detection threshold adjustment module is used to dynamically adjust the thresholds based on an initial threshold and actual input to achieve the optimal matching value. This module is used for execution. Figure 6 S607 in the middle.

[0137] The ASR module is used for converting input audio data into text. In S604, this ASR module can perform the text processing of audio data as described in the first and third schemes.

[0138] The digital human management module is responsible for controlling the overall interaction process of the digital human. For example, it sends the text of the question posed by the interaction object to the language model or knowledge base, or sends the response returned by the language model or knowledge base to the digital human generation module.

[0139] The digital human generation module is used to generate driving parameters corresponding to the response, drive the digital human to speak through the driving parameters, including mouth shape, facial expressions and body movements, and send the response text to TTS.

[0140] A TTS module is used for text-to-speech conversion. In the embodiments of this application, the speech can be used by the digital human to answer questions. The TTS module can be used in situations such as... Figure 6 After S605, as shown, once the response to the question is determined, the TTS module converts the text of the response into audio so that the large-screen device can output the audio, achieving a realistic dialogue with the digital human.

[0141] The audio and video transmission module sends the image frames generated by the digital human inference to the display screen of the large screen device for display, and sends the audio data to the speaker of the large screen device for audio output.

[0142] The language model / knowledge base service analyzes and understands the question, and determines the content of the digital human's response. A combination of a knowledge base and a language model is typically used to provide the response, which can improve the accuracy of the response. The language model / knowledge base service corresponds to the execution... Figure 6 S605 in the middle.

[0143] The voiceprint module is used to extract features from the face and voice of the interacting object after identifying the object or determining that the response to the interacting object meets the intent. The extracted data, along with the corresponding audio silence threshold and lip-sync threshold, is stored in the system database to prepare for rapid data matching in subsequent questions. The voiceprint module executes accordingly. Figure 6 S608 in the middle.

[0144] The following is combined with Figure 6 The interaction process of each module in the application is described in further detail, illustrating the voice interaction method of this embodiment.

[0145] refer to Figure 8 , Figure 8 The interactive flowchart of the software module in an embodiment of this application is shown. Figure 8 As shown, the large-screen device includes: a voice wake-up module, a lip-sync detection module, a voiceprint module, a silence detection and lip-sync detection threshold adjustment module, an ASR module, a TTS module, a digital human management module, a digital human voice-driven module, and audio and video transmission modules. The cloud-based device includes a language large model / knowledge base service module.

[0146] like Figure 8 As shown, the interaction process between software modules includes two interaction scenarios.

[0147] Scenario 1, Initial Q&A. In this scenario, such as... Figure 8 As shown, the interaction process includes S801-S816.

[0148] S801, the voice wake-up module detects voice in real time.

[0149] The voice wake-up module features real-time voice detection, continuously monitoring surrounding sounds and responding upon hearing a specific wake-up word or command. The wake-up word can be set to multiple languages, such as Chinese, English, and Japanese, allowing users in different language environments to use the device.

[0150] In some embodiments, the wake word can be customized according to the user's personal preferences.

[0151] S802, the voice wake-up module notifies the lip-sync detection module that it has received the wake-up word.

[0152] For example, when the voice wake-up module receives a wake-up word, it triggers the lip-sync detection module to start, thereby monitoring the user's lip shape in real time. In some embodiments, the lip-sync detection module may not require notification from the voice wake-up module to start, but may instead remain in detection mode while the large-screen device is powered on.

[0153] S803, the lip shape detection module detects the questioner's lip shape data in real time.

[0154] The lip-shape detection module can capture facial data of people in front of the screen in real time using a camera, locate the face, and then locate the lip area to observe lip movements in more detail. By continuously tracking changes in lip movements, it can more accurately capture subtle changes in the questioner's speech.

[0155] S804, the voice wake-up module confirms successful wake-up and notifies the lip-sync detection module.

[0156] The voice wake-up module also has a wake-up word detection function, which can detect keywords. For example, if the pre-set wake-up word for the digital human is "Hello, Lily", the voice wake-up module will capture the wake-up word and determine that the questioner's intention is to wake up the digital human, and then trigger the next action.

[0157] S805, the lip shape detection module detects the questioner's lip shape and captures and determines the lip shape.

[0158] The lip-sync detection module analyzes lip movement sequences to identify specific lip shapes or pronunciations. It compares the identified lip shapes with wake words to determine the interaction target. Furthermore, based on the interaction target's lip shape, it can determine the corresponding audio silence threshold and lip-sync speaking threshold.

[0159] S806, the silence detection and lip-sync detection dynamic adjustment module adjusts preset values ​​based on the detected lip shape and audio.

[0160] The preset values ​​refer to the pre-set silence detection threshold and lip-sync threshold. This process can include two scenarios. First, during the initial dialogue, the dynamic adjustment module for silence and lip-sync selects an initial value matching the interactive object from pre-stored audio silence and lip-sync thresholds. However, this initial value is not verified and may be inaccurate. Therefore, in the second scenario, after answering the interactive object's question for the first time, the interactive object can be asked if the answer is accurate. If the answer is accurate, there is no need to adjust the initial values ​​of the audio silence and lip-sync thresholds. If the interactive object's answer is inaccurate, the audio silence and lip-sync thresholds are adjusted based on the interactive object's audio signal value and lip movements until the interactive object confirms accuracy. This achieves dynamic threshold adjustment, allowing for more accurate judgment of the completeness of the interactive object's input, thus improving the interactive experience.

[0161] Furthermore, in some embodiments, the large-screen device may not need to ask whether the answer is accurate after providing it; instead, it may directly ask the user whether the question itself is accurate. For example, a voice prompt might ask, "Is your question: 'Would you like to ask a route map for shopping mall A?'" If the user confirms yes, the answer is output directly. If the answer is no, the audio mute threshold and lip-sync threshold are adjusted, and the user is prompted to "repeat your question" until a positive answer is received. The threshold adjustment then ends and the settings are saved. This method can improve the speed of threshold adjustment.

[0162] The following description, in conjunction with the accompanying drawings, explains the process by which the S806 adjusts preset values ​​based on the detected lip shape and audio.

[0163] refer to Figure 9 , Figure 9 A flowchart illustrating the adjustment of preset values ​​based on detected lip movements and audio is shown in an embodiment of this application. The steps in this flowchart are executed by the silence detection and lip movement detection threshold adjustment module, and this flowchart corresponds to… Figure 6 S604 in the middle.

[0164] like Figure 9 As shown, the process may include S910 and S920. S902 further includes S921-S927.

[0165] S910, the silence detection and lip-sync detection threshold adjustment module obtains the audio data and lip-sync data input by the interactive object. The audio data refers to the audio of the interactive object asking a question. The lip-sync data is extracted based on facial expression data, including dynamic lip-sync data and the degree of lip opening / closing.

[0166] S920 determines the currently input audio and lip movements.

[0167] When confirming that the user is asking a question for the first time, the silence detection and lip-sync detection threshold adjustment module directly uses the initial values ​​of the audio silence threshold and lip-sync speech threshold stored in the system. Then, it compares the obtained audio data and lip-sync data with the initial values.

[0168] When the audio value exceeds the audio mute threshold and the lip movement is less than or equal to the lip-reading threshold, it is determined that the interactive object has completed asking a question. S921 determines that there is interfering noise and executes S922 to perform noise reduction processing on the audio data.

[0169] If the audio is less than or equal to the audio mute threshold, and the lip movement is greater than the lip speech threshold, it indicates that the interactive object is still speaking. Therefore, it is determined that the interactive object's question is incomplete. The system then prompts the interactive object to adjust the volume of the microphone, and executes step S924, prompting the user to move closer to the digital figure or increase the volume. Step S925 continues microphone pickup, acquiring new audio and lip movement data for further evaluation until a complete question is determined.

[0170] When the audio is less than or equal to the audio silence threshold and the lip movement is less than or equal to the lip speech threshold, it indicates that the interactive object has stopped speaking and the interactive object's question has been completed. Then, S926 is executed to send the audio data to the ASR for recognition and convert the audio into text.

[0171] If the audio value is greater than the audio mute threshold and the lip movement is greater than the lip speech threshold, it indicates that the interactive object is still speaking. In this case, S927 is executed to continue picking up sound and to reacquire new audio and lip movement data for judgment until it is determined that the interactive object has completed asking the question.

[0172] The four results described above are all for determining whether the current user's question is complete. When the question is determined to be complete, it means that the sentence is initially judged to be complete. However, to further verify the accuracy of the judgment, the following can be executed: Figure 8 S809-S814 in the document describes a further adjustment of the audio silence threshold and the lip-sync threshold. It also serves as a process to re-determine whether the interactive object's question has been completed. Through multiple verifications, the optimal threshold is dynamically obtained, thereby improving the accuracy of determining whether the question has been completed.

[0173] The S807 silence detection and lip-sync detection dynamic adjustment module sends the problematic voice data to the ASR and notifies the speech-to-text conversion function.

[0174] In the S808, the ASR module converts speech to text and notifies the digital human management module.

[0175] In some embodiments, after the ASR module obtains the audio, if it determines that there is noise, it will also perform preprocessing on the audio, such as noise reduction and gain adjustment, to improve the quality of subsequent audio processing.

[0176] S809, the digital human management module sends the text corresponding to the question to the language big model / knowledge base, and notifies the language big model / knowledge base that this is the first interaction and that the question confirmation stage has begun.

[0177] S810, Language Big Model / Knowledge Base Start-up Question Correction Question Answering Mode.

[0178] For example, large-screen devices can have two modes: a corrective question-and-answer mode and a regular mode. In corrective question-and-answer mode, the user's question needs to be confirmed. In regular mode, the user doesn't need to confirm the question; the answer is output directly. Regular mode typically corresponds to scenarios where the user asks a question again, i.e., scenario two, subsequent question-and-answer sessions. Corrective question-and-answer mode corresponds to scenario one, i.e., the initial question-and-answer session.

[0179] S811, the language big model / knowledge base obtains the corresponding answer to the question and notifies the digital human generation module to drive the digital human to confirm the accuracy of the question with the questioner.

[0180] In this process, the digital human generation module can generate driving parameters for the digital human based on the response, such as audio, "Are you satisfied with XX's answer?", and motion parameters, such as mouth opening data, the digital human's posture and movements.

[0181] S812, the digital human generation module notifies the TTS module to generate speech from the response text, and controls the audio and video sending modules to output the audio corresponding to the response, as well as display the animation of the digital human.

[0182] After the TTS module generates speech (audio) from the response text, the audio and video sending module sends the corresponding audio to the speaker and outputs the audio. After receiving the instruction to drive the digital human's movements, the audio and video sending module drives the digital human's movements according to the driving parameters, so as to achieve synchronization between the digital human's movements and audio, thereby achieving the effect of imitating real human dialogue.

[0183] In addition, the digital human management module can issue confirmation commands, which can be output as audio through a speaker. For example, after the digital human outputs a response audio, it can ask the questioner if they are satisfied with the answer. Alternatively, it can display text on a screen, asking the questioner if they are satisfied with the current answer. Furthermore, after receiving the result from the questioner, it will provide feedback on the result.

[0184] S813, the language big model / knowledge base module obtains the questioner's confirmation result and adjusts the corresponding threshold according to the confirmation result.

[0185] If the language model / knowledge base module confirms that the results are inaccurate, then the thresholds are adjusted. For example, if the question is incomplete, it indicates that the threshold setting is too high, so the audio silence threshold and lip-sync threshold are increased. If other text is present in the question, it indicates that the threshold setting is too low, so the thresholds are decreased. After adjustment, the user is asked the question again to confirm the effect. Through multiple rounds of correction, the optimal audio silence threshold and lip-sync threshold, and other feature data for the questioner are obtained, i.e., S814 is executed to complete the proofreading.

[0186] Once the result is confirmed to be accurate, or accurate after multiple rounds of verification, the language big model / knowledge base can execute S815.

[0187] S815, the Language Big Model / Knowledge Base module notifies the Digital Human Management module that proofreading is complete.

[0188] S816, the digital human management module notifies the voiceprint module to obtain voiceprint information and maintain the threshold.

[0189] The voiceprint module obtains voiceprint information from the acquired audio data and stores the voiceprint information along with the audio silence threshold and lip-sync threshold. This allows the system to quickly obtain the corresponding audio silence threshold and lip-sync threshold based on the voiceprint information when the questioner asks a question again, thereby improving interaction efficiency.

[0190] Scenario 2, Follow-up Q&A (asking the question again). In Scenario 2, the interaction process includes S817-S819.

[0191] S817, the voice wake-up module detects voice information and notifies the silence detection and lip-sync detection dynamic adjustment module to be in a non-wake-up state.

[0192] For example, the voice wake-up module obtains the audio data of the user's question and identifies the user's identity based on the voiceprint information in the audio data.

[0193] The non-wake-up case indicates that this is not the first time the user has asked a question.

[0194] S818, the silence detection and lip-sync detection dynamic adjustment module notifies the voiceprint module and triggers the voiceprint module to recognize voiceprint information and obtain the previously saved audio silence threshold and lip-sync speaking threshold. The silence detection and lip-sync detection dynamic adjustment module then updates the system-set audio silence threshold and lip-sync speaking threshold to the threshold corresponding to the voiceprint information.

[0195] Subsequently, when the same questioner asks a question, the large-screen device can directly convert the voice data and lip-sync data into text. The language model / knowledge base module then provides an answer to the question based on the text and lip-sync data, and outputs the corresponding audio and animation. During this process, no threshold correction confirmation is required (S819), confirming the normal procedure. The silence detection and lip-sync detection dynamic adjustment modules do not need further correction confirmation. The normal procedure can be the standard mode, i.e., a process without correction confirmation.

[0196] In some embodiments, in subsequent interaction scenarios, processing of the accuracy of question-answer matching can also be added. For example, if the internal matching answer is not ideal, with a matching rate below 70%, it is determined that the identified question cannot find an answer. The threshold adjustment process in steps S809-S814 above can be repeated, and the interactive object can be asked to re-enter the question. The digital human management module can also execute S820 to start question-answering stability detection. If the question cannot be matched multiple times, the threshold adjustment is restarted to improve the overall digital human interaction experience. The adjustment process can refer to steps S809-S814 above.

[0197] Furthermore, considering changes in environmental or personal factors, such as long intervals between initial and subsequent questions (e.g., six months or a year), a user's volume or speaking speed may alter. Therefore, when a user asks a question for the first time, a time limit can be set for storing the audio mute threshold and lip-sync threshold for previously asked questions, such as one year. After one year, automatic threshold adjustment needs to be restarted. Additionally, user physical changes can be considered when determining whether to adjust thresholds. For example, if a user has just returned from a run, their speaking speed and volume may differ from their normal state. Therefore, using the thresholds determined in the initial question for subsequent questions could affect the accuracy of the judgment. In this case, factors such as heart rate can be incorporated into the judgment. If the heart rate is within the normal range, the thresholds from the initial question are used; if the heart rate exceeds the normal range in a subsequent question, threshold adjustment is restarted. Combining these factors can further improve the accuracy of responses, thereby enhancing the interactive experience.

[0198] It should be noted that the above embodiments all use mouth shape data as an example for illustrating facial expression data. In some embodiments, facial expression data may also include one or more of eyebrow movement data, eye movement data, and nose movement data. This facial expression data can be recognized by a small on-device model. By recognizing the expressions of the interactive object, adding emotional parameters, and combining them with the above-mentioned judgment on whether the user's question is completed, the user's intention can be judged more accurately, and whether the user's question is incomplete can be judged more accurately. This allows the digital human to make more accurate question answers based on understanding text and combining emotional data, gradually approaching the experience of communicating with a real person.

[0199] In addition, facial expressions can be added to the driving parameters of the digital human to make the entire interaction process more vivid and realistic.

[0200] In summary, based on the above solutions, the voice interaction method of this application accurately determines the interaction object by combining lip-sync detection and wake-word recognition. Then, by combining voice silence detection and lip-sync detection, and leveraging the continuous interaction between the digital human and the user, it gradually learns and adjusts the silence and lip-sync detection parameters most suitable for the user to meet personalized needs. Furthermore, voiceprint technology is used to store personal interaction data, allowing this data to be directly accessed in future interactions, thereby improving the overall experience of AI digital human interaction.

[0201] This application also provides a voice interaction device 1000, including:

[0202] Object confirmation module 1010 is used to determine the interaction object;

[0203] The acquisition module 1020 is used to acquire facial expression data and audio data of the interactive object. The facial expression data includes at least mouth shape data.

[0204] The integrity verification module 1030 is used to determine the integrity of the question raised by the interactive object based on audio data and lip-sync data.

[0205] The generation module 1040 is used to determine the response corresponding to the question when the question is determined to be complete based on audio data and lip-sync data, and to generate driving information based on the response;

[0206] The output module 1050 is used to output media data corresponding to the response based on the drive information.

[0207] The object verification module 1010, acquisition module 1020, integrity verification module 1030, generation module 1040, and output module 1050 can all be implemented in software or hardware. For example, the implementation of the object verification module 1010 will be described below. Similarly, the implementation of the acquisition module 1020, integrity verification module 1030, generation module 1040, and output module 1050 can refer to the implementation of the object verification module 1010.

[0208] As an example of a software functional unit, the object verification module 1010 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the object verification module 1010 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0209] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0210] As an example of a hardware functional unit, the object verification module 1010 may include at least one computing device, such as a server. Alternatively, the object verification module 1010 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0211] The object verification module 1010 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the object verification module 1010 includes multiple computing devices that can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the object verification module 1010 includes multiple computing devices that can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0212] It should be noted that, in other embodiments, the object confirmation module 1010 can be used to execute any step in the voice interaction method, the acquisition module 1020 can be used to execute any step in the voice interaction method, the integrity confirmation module 1030 can be used to execute any step in the voice interaction method, the generation module 1040 can be used to execute any step in the voice interaction method, and the output module 1050 can be used to execute any step in the voice interaction method. The steps implemented by the object confirmation module 1010, the acquisition module 1020, the integrity confirmation module 1030, the generation module 1040, and the output module 1050 can be specified as needed. The voice interaction device 1000 can achieve all its functions by implementing different steps in the voice interaction method through the object confirmation module 1010, the acquisition module 1020, the integrity confirmation module 1030, the generation module 1040, and the output module 1050 respectively.

[0213] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0214] like Figure 11 As shown, the computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster may store the same memory for executing the above embodiments. Figures 4-9 The instructions for the explained voice interaction method.

[0215] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store memory for executing the above embodiments. Figures 4-9 The explained voice interaction method includes some instructions. In other words, a combination of one or more computing devices 1100 can jointly execute instructions for performing the above-described embodiments. Figures 4-9 The instructions for the explained voice interaction method.

[0216] It should be noted that the memory 1106 in different computing devices 1100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the voice interaction device. That is, the instructions stored in the memory 1106 of different computing devices 1100 can implement the functions of one or more modules among the object verification module, acquisition module, integrity verification module, generation module, and output module.

[0217] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 12 One possible implementation is shown. For example... Figure 12 As shown, the two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this possible implementation, the memory 1106 in computing device 1100A stores instructions for performing the functions of the integrity verification module. Meanwhile, the memory 1106 in computing device 1100B stores instructions for performing the functions of the object verification module, the acquisition module, the generation module, and the output module.

[0218] Figure 12The connection method between the computing device clusters shown can be based on the fact that the voice interaction method provided in this application needs to obtain a large amount of audio data and facial expression data of the interactive object (e.g., a large amount of stored data) and to judge and recognize the completeness of the user's sentences through a large language model. Therefore, considering the processing speed and storage capacity, the function implemented by the integrity confirmation module is handed over to the computing device 1100A for execution.

[0219] It should be understood that Figure 12 The functions of computing device 1100B shown can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100A can also be performed by multiple computing devices 1100.

[0220] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 11 and Figure 12 The connection method of the computing device cluster. The difference is that the memory 1106 in one or more computing devices 1100 within this computing device cluster can store the same memory used to execute the above embodiments. Figures 4-9 The instructions for the explained voice interaction method.

[0221] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store memory for executing the above embodiments. Figures 4-9 The explained voice interaction method includes some instructions. In other words, a combination of one or more computing devices 1100 can jointly execute instructions for performing the above-described embodiments. Figures 4-9 The instructions for the explained voice interaction method. This application also provides an electronic device, including:

[0222] Memory, used to store instructions executed by one or more processors of the device, and

[0223] Processor, used to execute the above embodiments combined with Figures 4 to 9 The method explained.

[0224] This application also provides a computer-readable storage medium, which can be any usable medium that a computing device can store, or a data storage device such as a data center containing one or more usable media. The usable medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the actions described in the above embodiments. Figures 4 to 9 The explained method, or instructions to the computing device to perform the above embodiments. Figures 4 to 9 The method explained.

[0225] This application also provides a computer program product containing instructions. The computer program product may be software or a program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the above-described embodiments. Figures 4 to 9 The method explained.

[0226] Now for reference Figure 13 The diagram shown is a block diagram of a SoC (System on Chip) 1400 according to an embodiment of this application. Figure 13 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 13 In this SoC 1400, the following are included: an interconnect unit 1450 coupled to an application processor 1410; a system proxy unit 1480; a bus controller unit 1490; an integrated memory controller unit 1440; a group or one or more coprocessors 1420, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 1430; and a direct memory access (DMA) unit 1460. In one embodiment, the coprocessor 1420 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a general-purpose computing on GPU (GPGPU), a high-throughput many integrated core (MIC) processor, or an embedded processor.

[0227] The static random access memory (SRAM) cell 1430 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of those instructions. These instructions may include, when executed by at least one unit in the processor, causing the SoC 1400 to perform the processing method according to the above embodiments, as detailed in the above embodiments. Figure 4 and Figure 9 The voice interaction methods explained will not be elaborated upon here.

[0228] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0229] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0230] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0231] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0232] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0233] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A voice interaction method, characterized in that, include: Identify the interaction target; Acquire facial expression data and audio data of the interactive object, wherein the facial expression data includes at least mouth shape data; Based on the audio data and the lip-sync data, determine the completeness of the question posed by the interactive object; If the question is determined to be complete based on the audio data and the lip-sync data, a response corresponding to the question is determined, and driving information is generated based on the response; Based on the driving information, media data corresponding to the response is output.

2. The method according to claim 1, characterized in that, The determination of the problem's completeness based on the audio data and the lip-sync data includes: If the lip-sync data of the interactive object is less than or equal to the lip-sync speaking threshold, and the signal value of the audio data is greater than the audio mute threshold, then the question is considered complete.

3. The method according to claim 2, characterized in that, The step of determining the completeness of the problem based on the audio data and the lip-sync data also includes: noise reduction processing of the audio data.

4. The method according to claim 1, characterized in that, The determination of the problem's completeness based on the audio data and the lip-sync data includes: When it is determined that the lip-sync data of the interactive object is less than or equal to the lip-sync speaking threshold, and the signal value of the audio data decreases to less than or equal to the audio mute threshold, then the question is determined to be complete.

5. The method according to claim 1, characterized in that, The step of determining the completeness of the question raised by the interactive object based on the audio data and the lip-sync data includes: If the question is determined to be incomplete, and no response corresponding to the incomplete question is output within a preset time period, the data acquisition state is maintained.

6. The method according to claim 5, characterized in that, The state of maintaining the picked-up data includes: Output a prompt message indicating that the interactive object is picking up data or maintain the state of the digital human listening, and continue to pick up the audio data and facial expression data of the interactive object.

7. The method according to claim 5 or 6, characterized in that, The determination that the problem is incomplete includes: Determine that the lip-sync data of the interactive object is greater than the lip-sync speaking threshold, and the signal value of the audio data decreases to less than or equal to the audio mute threshold, or... It is determined that the mouth shape data of the interactive object is greater than the mouth shape speaking threshold, and the signal value of the audio data is greater than the audio mute threshold.

8. The method according to claim 7, characterized in that, The determination that the problem is incomplete also includes: The interactive object is prompted to adjust the volume or move closer to the microphone.

9. The method according to claim 1, characterized in that, Determining the response corresponding to the question includes: Obtain the complete audio data for the problem; The response corresponding to the question is obtained based on the audio data and language model of the question.

10. The method according to claim 9, characterized in that, Obtaining the response corresponding to the question based on the audio data and language big data model of the question includes: Convert the audio data of the problem into text; The text is input into the language model, and the corresponding response text is obtained based on the language model. The response text is converted into audio.

11. The method according to claim 9, characterized in that, The process of obtaining the response corresponding to the question based on the audio data and language big model of the question further includes: Output a prompt indicating whether the response matches the intent corresponding to the interaction; If a first operation that matches the intent of the interaction object is received, output the media data corresponding to the response; If a second operation is received that does not conform to the intention of the interactive object, the lip-sync threshold and the audio mute threshold are adjusted, and the interactive object is prompted to ask the question again until the response conforms to the intention of the interactive object.

12. The method according to claim 11, characterized in that, The first operation includes: pressing a specific function key, posing in a specific posture, or inputting keywords that represent the intended action; The second operation includes: pressing a specific function key, posing in a specific posture, or inputting keywords that do not conform to the intended meaning.

13. The method according to claim 1, characterized in that, The determination of the interaction object includes: Receive the wake word input by the user and obtain the user's lip shape data; Based on the wake word and the lip-sync data, the interaction object that matches the wake word and lip-sync data is determined from the user.

14. The method according to claim 12, characterized in that, Also includes: Extract the voiceprint features from the audio data of the interactive object and store them; When it is determined that the interactive object is not asking a question for the first time, the lip-sync threshold and audio mute threshold corresponding to the interactive object are directly obtained based on the voiceprint features.

15. The method according to claim 1, characterized in that, The facial expression data also includes one or more of the following: eyebrow movement data, eye movement data, and nose movement data.

16. The method according to claim 1, characterized in that, The media data includes animation data of digital humans or objects of a specific shape displayed on the screen, as well as output audio data.

17. A voice interaction device, characterized in that, include: The object confirmation module is used to determine the interaction object; The acquisition module is used to acquire facial expression data and audio data of the interactive object, wherein the facial expression data includes at least mouth shape data; The integrity verification module is used to determine the integrity of the question raised by the interactive object based on the audio data and the lip-sync data. The generation module is used to determine the completeness of the question based on the audio data and the lip-sync data, determine the response corresponding to the question, and generate driving information based on the response; The output module is used to output media data corresponding to the response based on the driving information.

18. An electronic device, characterized in that, include: A memory for storing instructions executed by one or more processors of an electronic device, and a processor for executing the voice interaction method according to any one of claims 1 to 16.

19. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the voice interaction method as described in any one of claims 1 to 16.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the voice interaction method according to any one of claims 1 to 16.

21. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, the electronic device performs the voice interaction method according to any one of claims 1 to 16.

Images