Human-computer interaction method and apparatus, computer readable storage medium and terminal device

Abstract

A human-computer interaction method, the method comprising: acquiring a first human-computer interaction voice and a first human-computer interaction image (S301); determining a reply text corresponding to the first human-computer interaction voice (S302); performing emotional analysis on the first human-computer interaction image, so as to obtain emotional information corresponding to the first human-computer interaction image (S303); performing voice synthesis on the reply text according to the emotional information, so as to obtain a second human-computer interaction voice (S304); and performing human-computer interaction according to the second human-computer interaction voice (S305).

Description

Human-computer interaction method, device, computer-readable storage medium, and terminal device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 31, 2023, with application number 202311135695.X and invention name “Human-computer interaction method, device, computer-readable storage medium and terminal device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the field of artificial intelligence technology, and in particular relates to a human-computer interaction method, apparatus, computer-readable storage medium, and terminal device. Background Art

[0003] With the development of intelligent technology, various intelligent terminal devices capable of human-computer interaction are gradually becoming part of human life. For example, smart speakers such as Amazon Echo, Apple Siri, and Microsoft XiaoIce can engage in multi-round conversations with people, helping people solve problems through human-computer interaction. However, existing human-computer interaction methods lack emotional compatibility and struggle to resonate with humans, resulting in an unnatural human-computer interaction process and a poor user experience. Technical issues

[0004] In view of this, the embodiments of the present application provide a human-computer interaction method, apparatus, computer-readable storage medium and terminal device to solve the problem that the existing human-computer interaction methods have poor matching at the emotional level, are difficult to resonate with humans, resulting in a relatively unnatural human-computer interaction process and affecting the user experience. Technical Solutions

[0005] A first aspect of an embodiment of the present application provides a human-computer interaction method, which may include:

[0006] Acquiring a first human-computer interaction voice and a first human-computer interaction image;

[0007] Determining a reply text corresponding to the first human-computer interaction voice;

[0008] Performing emotion analysis on the first human-computer interaction image to obtain emotion information corresponding to the first human-computer interaction image;

[0009] Performing speech synthesis on the reply text according to the emotional information to obtain a second human-computer interaction speech;

[0010] Perform human-computer interaction according to the second human-computer interaction voice.

[0011] In a specific implementation of the first aspect, performing emotion analysis on the first human-computer interaction image to obtain emotion information corresponding to the first human-computer interaction image may include:

[0012] Extracting key point features from the first human-computer interaction image to obtain key point features of the first human-computer interaction image;

[0013] Determining expression information, action information, and posture information corresponding to the first human-computer interaction image based on the key point features;

[0014] Emotional information corresponding to the first human-computer interaction image is determined based on the expression information, action information, and posture information.

[0015] In a specific implementation of the first aspect, performing speech synthesis on the reply text according to the emotion information to obtain the second human-computer interaction speech may include:

[0016] determining a response tone and response intonation corresponding to the emotional information;

[0017] The reply text is speech-synthesized according to the reply tone and the reply intonation to obtain the second human-computer interaction voice.

[0018] In a specific implementation of the first aspect, performing human-computer interaction according to the second human-computer interaction voice may include:

[0019] Performing virtual talking head synthesis according to the second human-computer interaction voice to obtain a second human-computer interaction image corresponding to the second human-computer interaction voice;

[0020] Performing audio and video mixed encoding on the second human-computer interaction voice and the second human-computer interaction image to obtain human-computer interaction audio and video;

[0021] Human-computer interaction is performed according to the human-computer interaction audio and video.

[0022] In a specific implementation of the first aspect, obtaining the first human-computer interaction voice may include:

[0023] Obtain the original human-computer interaction voice collected by the preset microphone array;

[0024] Performing sound source localization according to the original human-computer interaction speech to obtain a sound source direction corresponding to the original human-computer interaction speech;

[0025] The original human-computer interaction speech is filtered according to the sound source direction to obtain the first human-computer interaction speech.

[0026] In a specific implementation of the first aspect, filtering the original human-computer interaction speech according to the sound source direction to obtain the first human-computer interaction speech may include:

[0027] The speech corresponding to the sound source direction in the original human-computer interaction speech is retained, and the speech corresponding to other directions in the original human-computer interaction speech is filtered out to obtain the first human-computer interaction speech.

[0028] In a specific implementation of the first aspect, determining a reply text corresponding to the first human-computer interaction voice may include:

[0029] Performing speech recognition on the first human-computer interaction speech to obtain a speech recognition text corresponding to the first human-computer interaction speech;

[0030] Using a preset speech model to perform spoken semantic understanding on the speech recognition text to obtain a standard written text;

[0031] A response text corresponding to the standard written text is determined.

[0032] A second aspect of the embodiments of the present application provides a human-computer interaction device, which may include:

[0033] An information acquisition module, configured to acquire a first human-computer interaction voice and a first human-computer interaction image;

[0034] A text determination module, configured to determine a reply text corresponding to the first human-computer interaction voice;

[0035] an emotion analysis module, configured to perform emotion analysis on the first human-computer interaction image to obtain emotion information corresponding to the first human-computer interaction image;

[0036] A speech synthesis module, configured to perform speech synthesis on the reply text according to the emotional information to obtain a second human-computer interaction speech;

[0037] A human-computer interaction module is used to perform human-computer interaction according to the second human-computer interaction voice.

[0038] In a specific implementation of the second aspect, the sentiment analysis module may include:

[0039] A key point feature extraction submodule, configured to extract key point features from the first human-computer interaction image to obtain key point features of the first human-computer interaction image;

[0040] An image information determination submodule, configured to determine expression information, action information, and posture information corresponding to the first human-computer interaction image based on the key point features;

[0041] The emotion information determination submodule is used to determine the emotion information corresponding to the first human-computer interaction image based on the expression information, action information and posture information.

[0042] In a specific implementation of the second aspect, the speech synthesis module may include:

[0043] A tone and intonation determination submodule, configured to determine a response tone and intonation corresponding to the emotional information;

[0044] The speech synthesis submodule is used to perform speech synthesis on the reply text according to the reply tone and the reply intonation to obtain the second human-computer interaction speech.

[0045] In a specific implementation of the second aspect, the human-computer interaction module may include:

[0046] a talking head synthesis submodule, configured to synthesize a virtual talking head according to the second human-computer interaction voice to obtain a second human-computer interaction image corresponding to the second human-computer interaction voice;

[0047] A hybrid coding submodule, configured to perform audio and video hybrid coding on the second human-computer interaction voice and the second human-computer interaction image to obtain human-computer interaction audio and video;

[0048] The human-computer interaction submodule is used to perform human-computer interaction based on the human-computer interaction audio and video.

[0049] In a specific implementation of the second aspect, the information acquisition module may include:

[0050] The voice collection submodule is used to obtain the original human-computer interaction voice collected by the preset microphone array;

[0051] A sound source localization submodule, configured to perform sound source localization based on the original human-computer interaction speech to obtain a sound source direction corresponding to the original human-computer interaction speech;

[0052] The speech filtering submodule is used to filter the original human-computer interaction speech according to the sound source direction to obtain the first human-computer interaction speech.

[0053] In a specific implementation of the second aspect, the speech filtering submodule may include:

[0054] The speech filtering unit is used to retain the speech corresponding to the sound source direction in the original human-computer interaction speech and filter out the speech corresponding to other directions in the original human-computer interaction speech to obtain the first human-computer interaction speech.

[0055] In a specific implementation of the second aspect, the text determination module may include:

[0056] a speech recognition submodule, configured to perform speech recognition on the first human-computer interaction speech to obtain a speech recognition text corresponding to the first human-computer interaction speech;

[0057] A semantic understanding submodule, configured to use a preset speech model to perform spoken semantic understanding on the speech recognition text to obtain a standard written text;

[0058] The text determination submodule is used to determine the reply text corresponding to the standard written text.

[0059] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned human-computer interaction methods are implemented.

[0060] The fourth aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned human-computer interaction methods when executing the computer program.

[0061] A fifth aspect of the embodiments of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the steps of any one of the above-mentioned human-computer interaction methods. Beneficial effects

[0062] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain a first human-computer interaction voice and a first human-computer interaction image; determine a reply text corresponding to the first human-computer interaction voice; perform sentiment analysis on the first human-computer interaction image to obtain sentiment information corresponding to the first human-computer interaction image; perform speech synthesis on the reply text based on the sentiment information to obtain a second human-computer interaction voice; and perform human-computer interaction based on the second human-computer interaction voice. Through the embodiments of the present application, sentiment analysis can be performed on the first human-computer interaction image to obtain corresponding sentiment information, based on which a voice for human-computer interaction can be generated, thereby improving the emotional matching of the human-computer interaction method, making the human-computer interaction process more natural, and helping to improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0064] FIG1 is a schematic diagram of a half-duplex human-computer dialogue system;

[0065] Figure 2 is a schematic diagram of a full-duplex human-computer dialogue system;

[0066] FIG3 is a flow chart of an embodiment of a human-computer interaction method in an embodiment of the present application;

[0067] FIG4 is a schematic diagram of a full-duplex human-computer dialogue system using the human-computer interaction method of the present application;

[0068] FIG5 is a structural diagram of an embodiment of a human-computer interaction device according to an embodiment of the present application;

[0069] FIG6 is a schematic block diagram of a terminal device in an embodiment of the present application. Modes for Carrying Out the Invention

[0070] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0071] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0072] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0073] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0074] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0075] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0076] With the development of intelligent technology, various intelligent terminal devices capable of human-computer interaction are gradually entering all aspects of human life. For example, smart speakers such as Amazon Echo, Apple Siri, and Microsoft XiaoIce can engage in multi-round conversations with people, helping people solve problems through human-computer interaction. In a typical human-computer interaction process, a preset microphone or other sound pickup device can capture the user's voice (the user's speaking sound), and a corresponding reply text can be generated based on the user's voice. Afterwards, the reply audio corresponding to the reply text can be played through a preset speaker or other sound-speaking device.

[0077] Specifically, a half-duplex human-computer dialogue system can be shown in Figure 1. The user's voice can be recorded via a microphone or other sound pickup device. To improve the voice quality, the voice can be processed through automatic gain control (AGC) and adaptive noise suppression (ANS). Afterwards, if the terminal device has been awakened, it can directly enter the automatic speech recognition (ASR) mode. If the terminal device has not been awakened, the voice wake-up algorithm can be used to match the voice in real time to see if there is a preset wake-up word. If the preset wake-up word is matched, the voice recognition mode can be entered to convert the voice into text in real time. When the user finishes speaking, the voice recognition result can be obtained and recorded as a question. Afterwards, the reply text corresponding to the question can be generated through natural language understanding (NLU), dialogue management (DM), and natural language generation (NLG). Through speech synthesis, the reply text can be converted into a reply voice and broadcast by a speaker or other sound device.

[0078] It's important to understand that the above process is a simple human-computer interaction process, a full-link voice cascade, meaning that recording and broadcasting occur on the same thread. However, the broadcasting process of this half-duplex human-computer dialogue system cannot be interrupted. The user can only proceed to the next round of human-computer dialogue after the reply voice broadcast has finished. If the reply voice is long, the user may wait for a long time, affecting the user experience.

[0079] In order to solve the above problems, recording and broadcasting can be assigned to different threads and carried out simultaneously, so that a full-duplex human-computer dialogue system as shown in Figure 2 can be obtained. In a full-duplex human-computer dialogue system, users can interrupt when the reply voice is being broadcast, without having to wait for the completion of the broadcast, thus forming a "listening while talking" effect. In this process, since the speaker device is broadcasting the reply voice, the reply voice will be recorded by the pickup device, and the propagation path of the reply voice will form a loop, or even produce howling, resulting in an erroneous response. Therefore, it is necessary to add an acoustic echo cancellation module on the basis of the half-duplex human-computer dialogue system to eliminate or weaken the reply voice recorded by the pickup device. Specifically, the reply voice can be obtained by recording in the system, and used as an echo reference signal to transmit to the acoustic echo cancellation module. The acoustic echo cancellation module can eliminate the reply voice recorded by the pickup device.

[0080] However, the human-computer interaction methods used in existing human-computer dialogue systems have poor matching at the emotional level and are difficult to resonate with humans, resulting in a relatively unnatural human-computer interaction process and affecting the user experience.

[0081] In view of this, the embodiments of the present application provide a human-computer interaction method, apparatus, computer-readable storage medium and terminal device to solve the problem that the existing human-computer interaction methods have poor matching at the emotional level, are difficult to resonate with humans, resulting in a relatively unnatural human-computer interaction process and affecting the user experience.

[0082] It should be noted that the execution subject of the method of the present application is a terminal device, specifically, it may include but is not limited to desktop computers, notebooks, PDAs, smart interactive screens, robots and other common computing devices in the prior art.

[0083] Referring to FIG3 , an embodiment of a human-computer interaction method in an embodiment of the present application may include:

[0084] Step S301: Acquire a first human-computer interaction voice and a first human-computer interaction image.

[0085] In an embodiment of the present application, the user's voice can be recorded by a preset sound pickup device to obtain a first human-computer interaction voice, and an image containing the user can be captured by a preset camera capture device to obtain a first human-computer interaction image. Specifically, a preset camera can be used to record the video, and the first human-computer interaction image can be obtained from the video recording results.

[0086] In the embodiment of the present application, obtaining the first human-computer interaction voice may specifically include the following process:

[0087] Step S3011: Obtain original human-computer interaction speech collected by a preset microphone array.

[0088] In the embodiment of the present application, the preset sound pickup device can be a microphone array composed of multiple microphones. By using the microphone array to collect voice, the original human-computer interaction voice can be obtained.

[0089] The microphone array may be any common microphone array in the prior art, including but not limited to a cross array, a circular array, a rectangular array, a spiral array and other common microphone arrays in the prior art.

[0090] Step S3012: Perform sound source localization based on the original human-computer interaction speech to obtain the sound source direction corresponding to the original human-computer interaction speech.

[0091] It should be understood that the original human-computer interaction voice may contain the voices of people other than the user, which may cause the terminal device to be unable to accurately recognize the user's voice, thereby affecting the normal conversation with the user.

[0092] Therefore, in order to preserve the user's voice to a greater extent, the embodiment of the present application can perform sound source positioning based on the original human-computer interaction voice to obtain the sound source direction corresponding to the original human-computer interaction voice.

[0093] Typically, a microphone array may include at least three microphones, each of which may be arranged at a different position in space. The position of the sound source may be inferred by utilizing the propagation differences of the user's voice between different microphones.

[0094] Specifically, any common sound source localization method in the prior art can be used to localize the sound source according to actual needs, including but not limited to a positioning method based on controllable beamforming with maximum output power, a positioning method based on high-resolution spectrum estimation, and a positioning method based on arrival time delay difference estimation.

[0095] It should be noted that, in one specific implementation of the embodiments of the present application, before performing sound source localization based on the original human-computer interaction speech, the original human-computer interaction speech may be subjected to a preset audio signal processing to obtain a signal-processed output speech, and subsequent operations may be performed on the signal-processed output speech. For example, the original human-computer interaction speech may be automatically amplified to maintain the amplitude of the original human-computer interaction speech within an appropriate range as much as possible; thereafter, noise suppression may be performed on the automatically amplified output speech to obtain a signal-processed output speech, thereby reducing the impact of background noise or interfering sounds and improving the clarity and audibility of the original human-computer interaction speech.

[0096] Step S3013: Filter the original human-computer interaction speech according to the sound source direction to obtain the first human-computer interaction speech.

[0097] Among them, the first human-computer interaction voice is the voice corresponding to the direction of the sound source.

[0098] In an embodiment of the present application, the original human-computer interaction voice can be filtered according to the direction of the sound source so that its sound gain in the direction of the sound source is maximized to obtain the first human-computer interaction voice.

[0099] Specifically, any common directional filtering algorithm in the prior art can be used as needed to retain the speech corresponding to the sound source direction in the original human-computer interaction speech and filter out speech corresponding to other directions in the original human-computer interaction speech to obtain the first human-computer interaction speech. For example, minimum variance distortionless response (MVDR) estimation can be used to minimize the variance of the recorded signal by adjusting the weight coefficients of the signals of each microphone in the microphone array, thereby achieving the purpose of suppressing interference.

[0100] Step S302: Determine a reply text corresponding to the first human-computer interaction voice.

[0101] In the embodiment of the present application, step S302 may specifically include the following process:

[0102] Step S3021: Perform voice recognition on the first human-computer interaction voice to obtain a voice recognition text corresponding to the first human-computer interaction voice.

[0103] In an embodiment of the present application, speech recognition may be performed on the first human-computer interaction speech, and the first human-computer interaction speech may be converted into text to obtain speech recognition text.

[0104] In a specific implementation method of an embodiment of the present application, feature extraction can be performed on the first human-computer interaction speech to obtain speech features, such as extracting mel frequency or mel frequency cepstral coefficients, etc.; thereafter, the extracted speech features can be recognized through a preset recognition model, such as a hidden Markov model, a recurrent neural network, a convolutional neural network, or any other artificial intelligence model, and converted into speech recognition text.

[0105] It should be noted that the human-computer dialogue system may be in an unaware state, such as in a sleep state, in which case normal human-computer interaction will not be possible. Therefore, before formal human-computer interaction, the user can use a preset keyword to wake up the terminal device through voice. After the sound pickup device captures the original human-computer interaction voice, it can use keyword matching to detect whether the original human-computer interaction voice contains the preset keyword. If the preset keyword is matched in the original human-computer interaction voice, the terminal device can be woken up to enable normal voice conversation. If the preset keyword is not matched in the original human-computer interaction voice, the terminal device can be kept in its original state, such as continuing to sleep.

[0106] Step S3022: Use a preset speech model to perform spoken semantic understanding on the speech recognition text to obtain a standard written text.

[0107] It should be understood that since different users may have significant differences in speaking habits and expressions, the speech recognition text is not easy to understand. Therefore, in an embodiment of the present application, a preset speech model can be used to perform spoken language understanding (SLU) on the speech recognition text to obtain a standard written text.

[0108] Specifically, the speech model can be an N-Gram model. Based on the probability of each word in the speech recognition text appearing in a standard corpus, the N-Gram model can calculate the most probable parsed sentence corresponding to each word in the standard corpus, and use this parsed sentence as the standard written text. The standard corpus can be a pre-built corpus consisting of various types of standard written language. The N-Gram model can continuously modify the speech recognition text based on the context, generating a standard written text corresponding to the speech recognition text through error correction, supplementation, word order adjustment, and written language rewriting.

[0109] For example, if the speech recognition text is "How to prevent the epidemic in real time", the N-Gram model can divide the speech recognition text into four words: "how", "real time", "epidemic", and "protection". According to the probability of these four words appearing in the standard corpus, the chain rule is used to calculate the parsing sentences corresponding to these four words, and the parsing sentence with the maximum probability corresponding to these four words in the standard corpus is obtained, which is "How to implement epidemic prevention". "How to implement epidemic prevention" is used as the standard written text corresponding to "How to prevent the epidemic in real time", thereby achieving the purpose of error correction.

[0110] For example, if the speech recognition text is "What is the new coronavirus", the N-Gram model can divide the speech recognition text into four words: "what", "is", "new coronavirus", and "virus". According to the probability of these four words appearing in the standard corpus, the chain rule is used to calculate the parsing sentences corresponding to these four words, and the parsing sentence with the maximum probability corresponding to these four words in the standard corpus is obtained, which is "What is the new coronavirus". "What is the new coronavirus" is used as the standard written text corresponding to "What is the new coronavirus", so as to supplement the speech recognition text and obtain a more complete text.

[0111] For example, the speech recognition text is "Should I wear a mask before going out?" The N-Gram model can divide the speech recognition text into six words: "go out", "whether", "want", "wear", "mask", and "first". According to the probability of these six words appearing in the standard corpus, the chain rule is used to calculate the parsing sentences corresponding to these six words, and the parsing sentence with the maximum probability corresponding to these six words in the standard corpus is obtained, which is "Should I wear a mask before going out?" "Should I wear a mask before going out" is used as the standard written text corresponding to "Should I wear a mask before going out?", so that the word order of the speech recognition text can be adjusted to obtain a text that is easier to understand.

[0112] For example, if the speech recognition text is "What is contact transmission", the N-Gram model can divide the speech recognition text into four words: "what", "called", "contact", and "transmission". According to the probability of these four words appearing in the standard corpus, the chain rule is used to calculate the parsing sentences corresponding to these four words, and the parsing sentence with the maximum probability corresponding to these four words in the standard corpus is obtained, which is "What is contact transmission". "What is contact transmission" is used as the standard written text corresponding to "What is contact transmission", so that the speech recognition text can be rewritten in written language to obtain a more standard text.

[0113] In a specific implementation method of an embodiment of the present application, the speech recognition text can also be mapped to the corresponding standard written text through a preset artificial intelligence model, wherein the artificial intelligence model can include but is not limited to any common deep learning model in the existing technology such as long short-term memory network, Transformer model, generative adversarial network, etc.

[0114] Step S3023: Determine the reply text corresponding to the standard written text.

[0115] In an embodiment of the present application, a standard written text may be recorded as a question, and a reply text corresponding to the standard written text may be determined through natural language understanding, dialogue management, and natural language generation.

[0116] Natural language understanding can be used to convert questions into data structures that can be understood by the terminal device, helping the terminal device understand the user's intent and extract key information. Specifically, existing natural language understanding methods can be used to understand the questions, including but not limited to using lexical analysis, syntactic analysis, named entity recognition, and other technologies to extract the user's intent.

[0117] Afterward, dialogue management, control of the dialogue flow, and decision-making can be performed based on the results of natural language understanding. A comprehensive analysis of the natural language understanding results, historical dialogue information, and context can be performed to determine the appropriate actions to take. These actions can include follow-up questions, clarifications, and confirmations. Specifically, state tracking can be performed using any of the existing state tracking methods, such as those based on artificial rules, generative models, or discriminative pattern models. The dialogue strategy can then be generated based on the state tracking results.

[0118] Natural language generation (NLG) can convert the abstract expressions output by dialogue management into syntactically valid and semantically accurate natural language sentences, achieving contextual coherence, precise responses, and good readability. Specifically, NLG can be performed using any of the common NLG methods currently available, such as those based on rule templates / sentence rules, language models, or deep learning.

[0119] Step S303: Perform sentiment analysis on the first human-computer interaction image to obtain sentiment information corresponding to the first human-computer interaction image.

[0120] It should be noted that if speech synthesis is performed directly based on the reply text to obtain a reply voice, the reply voice broadcast will have a poor emotional match and a rather stiff tone and intonation, which may make the user feel that they are not communicating with a real person, affecting the user experience. Therefore, in an embodiment of the present application, an emotional analysis can be performed on the first human-computer interaction image to obtain emotional information corresponding to the first human-computer interaction image, and then the emotional adjustment of the reply voice can be performed based on the emotional information.

[0121] Specifically, step S303 may include the following process:

[0122] Step S3031: extract key point features from the first human-computer interaction image to obtain key point features of the first human-computer interaction image.

[0123] In the embodiments of the present application, key point features of the first human-computer interaction image can be extracted using any image feature extraction method known in the art to obtain key point features of the first human-computer interaction image. The feature extraction method may include, but is not limited to, NARF key point extraction, SIFT key point extraction, Harris key point extraction, and other common image feature extraction methods known in the art.

[0124] Step S3032: Determine expression information, action information, and posture information corresponding to the first human-computer interaction image based on key point features.

[0125] It should be noted that images containing various emotions can be collected in advance, and key point features can be extracted from these images. Based on these key point features, the expression information, action information and posture information of these images can be labeled, so as to construct a database containing the mapping relationship between key point features and expression information, action information and posture information.

[0126] In an embodiment of the present application, the key point features of the first human-computer interaction image can be feature matched with the key point features pre-stored in the database to obtain the pre-stored key point features with the highest matching degree, and the expression information having a mapping relationship with the pre-stored key point features can be determined as the expression information of the first human-computer interaction image, the action information having a mapping relationship with the pre-stored key point features can be determined as the action information of the first human-computer interaction image, and the posture information having a mapping relationship with the pre-stored key point features can be determined as the posture information of the first human-computer interaction image.

[0127] Step S3033: Determine the emotional information corresponding to the first human-computer interaction image based on the expression information, action information, and posture information.

[0128] It should be understood that the emotional state of a person in an image can be determined through their expressions, movements, and postures. For example, if a person in an image is smiling, it can be assumed that the person is in a happy emotional state. For another example, if the person's movements are lively and energetic, it can be assumed that the person is in a relatively positive emotional state; while if the person's movements are stiff and depressed, it can be assumed that the person is in a relatively negative emotional state. For another example, if the person's posture is upright and their limbs are relaxed, it can be assumed that the person is in a relatively relaxed emotional state; while if the person in an image is dejected and tense, it can be assumed that the person is in a relatively tense emotional state.

[0129] Therefore, in an embodiment of the present application, the facial expression information, action information and posture information of the first human-computer interaction image can be comprehensively considered, and the preset emotion classification model can be used to predict the emotion information corresponding to the first human-computer interaction image, such as happiness, surprise, sadness, anger, disgust and fear; wherein the emotion classification model is a pre-trained artificial intelligence model for image emotion classification, which may include but is not limited to common classification models in the existing technology such as support vector machines or random forests.

[0130] Step S304: Perform speech synthesis on the reply text according to the emotional information to obtain a second human-computer interaction speech.

[0131] In an embodiment of the present application, the tone and intonation of the reply text can be adjusted accordingly based on the emotional information to provide a reply that is more in line with the user's emotions.

[0132] Specifically, the reply tone and reply intonation corresponding to the emotional information may be determined.

[0133] For example, if the emotional information of the character in the first human-computer interaction image is anger or disgust, a relatively calm and rational reply tone can be used; if the emotional information of the character in the first human-computer interaction image is fear, a relatively concerned reply tone can be used; if the emotional information of the character in the first human-computer interaction image is sadness, a relatively gentle reply tone can be used; if the emotional information of the character in the first human-computer interaction image is happiness or surprise, a relatively positive and pleasant reply tone can be used.

[0134] For example, if the emotional information of the character in the first human-computer interaction image is sadness, anger, disgust and fear, a relatively smooth reply tone can be used; if the emotional information of the character in the first human-computer interaction image is happiness or surprise, a relatively bright reply tone can be used.

[0135] Afterwards, the reply text can be synthesized into text to speech (TTS) based on the reply tone and intonation to obtain the second human-computer interaction voice. Specifically, the reply text can be processed, including language analysis, word boundary determination, sentence segmentation, and other operations to improve the quality and fluency of speech synthesis; then, the processed text content can be used to generate a speech signal based on a preset speech synthesis model; wherein, the speech synthesis model can be any common artificial intelligence model for speech synthesis in the prior art, including but not limited to recurrent neural networks, long short-term memory networks, variational autoencoders, and generative adversarial networks.

[0136] Step S305: Perform human-computer interaction according to the second human-computer interaction voice.

[0137] It's important to understand that simply broadcasting the second human-computer interaction voice without visually giving the user the feeling of interacting with a human can result in a lack of realism. Therefore, to enhance the realism of the interaction, a virtual talking head (VTH) can be generated. During the human-computer interaction, the VTH can animate in response to the changes in the interactive voice, making the interaction more rich and realistic.

[0138] In an embodiment of the present application, a virtual talking head may be synthesized based on the second human-computer interaction voice to obtain a second human-computer interaction image corresponding to the second human-computer interaction voice; wherein the second human-computer interaction image is an image including the virtual talking head.

[0139] In a specific implementation of an embodiment of the present application, a virtual talking head synthesis can be performed using a preset synthesis model for performing virtual talking head synthesis. Specifically, an image of a real person's head while speaking can be collected in advance, the head features can be extracted, and the audio features of the corresponding speaking voice can be extracted; the audio features are used as the input of a preset initial model, and the head features are used as the expected output of the initial model, and the initial model is trained to obtain a synthesis model. The synthesis model can be a common artificial intelligence model in the prior art, including but not limited to a hidden Markov model, a back-propagation neural network, and a mixed Gaussian model. Afterwards, the audio features of the second human-computer interaction voice can be extracted, and the audio features can be used as the input of the synthesis model to obtain a second human-computer interaction image corresponding to the second human-computer interaction voice.

[0140] After obtaining the second human-computer interaction image, the second human-computer interaction voice and the second human-computer interaction image may be subjected to audio and video mixed encoding to obtain human-computer interaction audio and video; human-computer interaction may be performed based on the human-computer interaction audio and video.

[0141] In a specific implementation method of the embodiment of the present application, it is necessary not only to ensure the synchronous alignment and streaming playback of the second human-computer interaction voice and the second human-computer interaction image, but also to ensure that they can be played directly on the web page. Here, FFmpeg audio and video hybrid encoding and Real Time Messaging Protocol (RTMP) can be used for pushing and pulling streams. Specifically, an Nginx-RTMP service can be set up on the Nginx server, wherein a client can be installed on the terminal device, and the Nginx-RTMP service is an intermediate bridge connecting the Nginx server and the client; after that, the second human-computer interaction voice and the second human-computer interaction image can be obtained frame by frame, and the second human-computer interaction voice and the second human-computer interaction image can be synthesized into a data stream in the FLV streaming media format through audio and video encoding, and the data stream is pushed to the Nginx-RTMP service. The client only needs to pull the data stream through the specified RTMP address and play it in a streaming manner.

[0142] In another specific implementation of the embodiment of the present application, in order to improve the effect of human-computer interaction, the human-computer interaction audio and video can be displayed through a preset speaker device and a preset smart large screen.

[0143] For ease of understanding, the human-computer interaction method of the present application will be introduced below with reference to the accompanying drawings.

[0144] Figure 4 shows a schematic diagram of a human-computer dialogue system to which the human-computer interaction method of the present application is applied. Specifically, the human-computer interaction method of the present application can be applied to a half-duplex human-computer interaction system and a full-duplex human-computer interaction system. The original human-computer interaction voice (the user's speaking voice) can be collected by a microphone array or other sound pickup device. After that, the original human-computer interaction voice can be automatically amplified and noise suppressed to improve the quality of the voice; in particular, in a full-duplex human-computer dialogue system, the original human-computer interaction voice also needs to be acoustically echo-cancelled to eliminate or reduce the broadcast sound in the original human-computer interaction voice; and in order to make the original human-computer interaction voice clearer, the original human-computer interaction voice can be localized to obtain the sound source direction, and then the voices corresponding to other directions in the original human-computer interaction voice can be filtered out to obtain a clearer first human-computer interaction voice. If the terminal device is in an unaware state (such as a sleeping state), the user can wake up the terminal device through a preset keyword. If the preset keyword is matched, the first human-computer interaction voice can be recognized to obtain a voice recognition text. At this stage, the voice recognition text can also be converted into a more understandable standard written text through spoken semantic understanding. Afterwards, the reply text can be obtained through three technologies: natural language understanding, dialogue management, and natural language generation. In order to improve the user experience, the camera can be used to capture the first human-computer interaction image (including the user's image), and the key points of the first human-computer interaction image can be extracted to obtain the first human-computer interaction text. Based on the user's facial expression information, action information and posture information in the interactive image, emotional analysis can be performed to obtain emotional information corresponding to the first human-computer interaction image, and the reply text can be voice-synthesized according to the emotional information to obtain a second human-computer interaction voice; afterwards, in order to enhance the realism of human-computer interaction, a virtual talking head can be synthesized according to the second human-computer interaction voice to obtain a second human-computer interaction image corresponding to the second human-computer interaction voice; the second human-computer interaction voice and the second human-computer interaction image can be mixedly encoded with audio and video to obtain human-computer interaction audio and video, and the human-computer interaction audio and video can be displayed through a preset speaker device and a preset screen to conduct human-computer interaction.

[0145] In summary, the embodiment of the present application obtains a first human-computer interaction voice and a first human-computer interaction image; determines a reply text corresponding to the first human-computer interaction voice; performs sentiment analysis on the first human-computer interaction image to obtain sentiment information corresponding to the first human-computer interaction image; performs speech synthesis on the reply text based on the sentiment information to obtain a second human-computer interaction voice; and performs human-computer interaction based on the second human-computer interaction voice. Through the embodiment of the present application, sentiment analysis can be performed on the first human-computer interaction image to obtain corresponding sentiment information, based on which a voice for human-computer interaction can be generated, thereby improving the emotional matching of the human-computer interaction method, making the human-computer interaction process more natural, and helping to improve the user experience.

[0146] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0147] Corresponding to the human-computer interaction method described in the above embodiment, FIG5 shows a structural diagram of an embodiment of a human-computer interaction device provided in an embodiment of the present application.

[0148] In this embodiment, a human-computer interaction device may include:

[0149] An information acquisition module 501 is configured to acquire a first human-computer interaction voice and a first human-computer interaction image;

[0150] A text determination module 502 is configured to determine a reply text corresponding to the first human-computer interaction speech;

[0151] The emotion analysis module 503 is configured to perform emotion analysis on the first human-computer interaction image to obtain emotion information corresponding to the first human-computer interaction image;

[0152] The speech synthesis module 504 is used to perform speech synthesis on the reply text according to the emotional information to obtain a second human-computer interaction speech;

[0153] The human-computer interaction module 505 is configured to perform human-computer interaction according to the second human-computer interaction voice.

[0154] In a specific implementation of the embodiment of the present application, the sentiment analysis module may include:

[0155] A key point feature extraction submodule, configured to extract key point features from the first human-computer interaction image to obtain key point features of the first human-computer interaction image;

[0156] An image information determination submodule, configured to determine expression information, action information, and posture information corresponding to the first human-computer interaction image based on the key point features;

[0157] The emotion information determination submodule is used to determine the emotion information corresponding to the first human-computer interaction image based on the expression information, action information and posture information.

[0158] In a specific implementation of the embodiment of the present application, the speech synthesis module may include:

[0159] A tone and intonation determination submodule, configured to determine a response tone and intonation corresponding to the emotional information;

[0160] The speech synthesis submodule is used to perform speech synthesis on the reply text according to the reply tone and the reply intonation to obtain the second human-computer interaction speech.

[0161] In a specific implementation of the embodiment of the present application, the human-computer interaction module may include:

[0162] a talking head synthesis submodule, configured to synthesize a virtual talking head according to the second human-computer interaction voice to obtain a second human-computer interaction image corresponding to the second human-computer interaction voice;

[0163] A hybrid coding submodule, configured to perform audio and video hybrid coding on the second human-computer interaction voice and the second human-computer interaction image to obtain human-computer interaction audio and video;

[0164] The human-computer interaction submodule is used to perform human-computer interaction based on the human-computer interaction audio and video.

[0165] In a specific implementation of the embodiment of the present application, the information acquisition module may include:

[0166] The voice collection submodule is used to obtain the original human-computer interaction voice collected by the preset microphone array;

[0167] A sound source localization submodule, configured to perform sound source localization based on the original human-computer interaction speech to obtain a sound source direction corresponding to the original human-computer interaction speech;

[0168] The speech filtering submodule is used to filter the original human-computer interaction speech according to the sound source direction to obtain the first human-computer interaction speech.

[0169] In a specific implementation of the embodiment of the present application, the speech filtering submodule may include:

[0170] The speech filtering unit is used to retain the speech corresponding to the sound source direction in the original human-computer interaction speech and filter out the speech corresponding to other directions in the original human-computer interaction speech to obtain the first human-computer interaction speech.

[0171] In a specific implementation of the embodiment of the present application, the text determination module may include:

[0172] a speech recognition submodule, configured to perform speech recognition on the first human-computer interaction speech to obtain a speech recognition text corresponding to the first human-computer interaction speech;

[0173] A semantic understanding submodule, configured to use a preset speech model to perform spoken semantic understanding on the speech recognition text to obtain a standard written text;

[0174] The text determination submodule is used to determine the reply text corresponding to the standard written text.

[0175] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0176] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0177] Figure 6 shows a schematic block diagram of a terminal device provided in an embodiment of the present application. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0178] As shown in FIG6 , the terminal device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the aforementioned human-computer interaction method embodiments are implemented, such as steps S301 to S305 shown in FIG3 . Alternatively, when the processor 60 executes the computer program 62, the functions of the modules / units in the aforementioned apparatus embodiments are implemented, such as the functions of modules 501 to 505 shown in FIG5 .

[0179] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 62 in the terminal device 6.

[0180] Those skilled in the art will understand that Figure 6 is merely an example of the terminal device 6 and does not constitute a limitation on the terminal device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 6 may also include input and output devices, network access devices, buses, etc.

[0181] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0182] The memory 61 can be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the terminal device 6. Furthermore, the memory 61 can include both an internal storage unit of the terminal device 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the terminal device 6. The memory 61 can also be used to temporarily store data that has been output or is about to be output.

[0183] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0184] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0185] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0186] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0188] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0189] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0190] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A human-computer interaction method, characterized in that: include: Acquire a first human-computer interaction voice and a first human-computer interaction image; Determining a reply text corresponding to the first human-computer interaction voice; Performing emotion analysis on the first human-computer interaction image to obtain emotion information corresponding to the first human-computer interaction image; Performing speech synthesis on the reply text according to the emotion information to obtain a second human-computer interaction speech; Perform human-computer interaction according to the second human-computer interaction voice.

2. The human-computer interaction method according to claim 1, characterized in that: The performing emotion analysis on the first human-computer interaction image to obtain emotion information corresponding to the first human-computer interaction image includes: Extracting key point features from the first human-computer interaction image to obtain key point features of the first human-computer interaction image; Determine expression information, action information and posture information corresponding to the first human-computer interaction image according to the key point features; Emotional information corresponding to the first human-computer interaction image is determined according to the expression information, action information and posture information.

3. The human-computer interaction method according to claim 1, characterized in that: The performing speech synthesis on the reply text according to the emotion information to obtain a second human-computer interaction speech includes: Determining a response tone and response intonation corresponding to the emotional information; The reply text is speech-synthesized according to the reply tone and the reply intonation to obtain the second human-computer interaction speech.

4. The human-computer interaction method according to claim 1, characterized in that: The performing human-computer interaction according to the second human-computer interaction voice includes: Performing virtual talking head synthesis according to the second human-computer interaction voice to obtain a second human-computer interaction image corresponding to the second human-computer interaction voice; Performing audio and video mixed encoding on the second human-computer interaction voice and the second human-computer interaction image to obtain human-computer interaction audio and video; Human-computer interaction is performed according to the human-computer interaction audio and video.

5. The human-computer interaction method according to claim 1, characterized in that: The obtaining of the first human-computer interaction voice comprises: Obtaining the original human-computer interaction voice collected by the preset microphone array; Performing sound source localization according to the original human-computer interaction speech to obtain a sound source direction corresponding to the original human-computer interaction speech; The original human-computer interaction speech is filtered according to the sound source direction to obtain the first human-computer interaction speech.

6. The human-computer interaction method according to claim 5, characterized in that: The filtering the original human-computer interaction speech according to the sound source direction to obtain the first human-computer interaction speech includes: The speech corresponding to the sound source direction in the original human-computer interaction speech is retained, and the speech corresponding to other directions in the original human-computer interaction speech is filtered out to obtain the first human-computer interaction speech.

7. The human-computer interaction method according to any one of claims 1 to 6, characterized in that: The determining of the reply text corresponding to the first human-computer interaction voice includes: Performing speech recognition on the first human-computer interaction speech to obtain a speech recognition text corresponding to the first human-computer interaction speech; Using a preset speech model to perform spoken semantic understanding on the speech recognition text to obtain a standard written text; A response text corresponding to the standard written text is determined.

8. A human-computer interaction device, characterized in that: include: An information acquisition module, used to acquire a first human-computer interaction voice and a first human-computer interaction image; A text determination module, used to determine a reply text corresponding to the first human-computer interaction voice; An emotion analysis module, used to perform emotion analysis on the first human-computer interaction image to obtain emotion information corresponding to the first human-computer interaction image; A speech synthesis module, used for performing speech synthesis on the reply text according to the emotion information to obtain a second human-computer interaction speech; A human-computer interaction module is used to perform human-computer interaction according to the second human-computer interaction voice.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the human-computer interaction method according to any one of claims 1 to 7 are implemented.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the human-computer interaction method according to any one of claims 1 to 7 are implemented.

Images