Human-computer interaction method, device, system, electronic device, computer-readable medium, and program

The method analyzes user modalities to recognize intentions and mood, generating responsive animated character videos, addressing the limitations of traditional systems by enhancing emotional interaction.

JP7592170B2Active Publication Date: 2024-11-29BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023535742
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-02-09
Filing Date
2021-12-15
Publication Date
2024-11-29
Estimated Expiration
2041-12-15

AI Technical Summary

Technical Problem

Traditional virtual digital human customer service systems lack the ability to respond appropriately to users in various moods, resulting in a poor interaction experience due to their emotionless nature and limited capability in complex human-computer interactions.

Method used

A human-computer interaction method that analyzes user information from multiple modalities (image and audio data) to recognize user intentions and mood characteristics, generating an announcement video with an animated character that reflects the user's mood, using trained models for facial and voice analysis to provide responsive feedback.

Benefits of technology

Enhances human-computer interaction by providing effective feedback tailored to users' moods, ensuring emotional communication and improving the interaction experience through animated character responses.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007592170000001
    Figure 0007592170000001
  • Figure 0007592170000002
    Figure 0007592170000002
  • Figure 0007592170000003
    Figure 0007592170000003
Patent Text Reader

Abstract

The present disclosure discloses a human-computer interaction method and apparatus, which relates to the artificial intelligence technical field, particularly to technical fields such as computer vision and deep learning. One specific embodiment of the method includes receiving information of at least one modality of a user, recognizing, based on the information of the at least one modality, user intention information and a user mood characteristic corresponding to the intention information, determining response information to the user based on the intention information, selecting, based on the user mood characteristic, a character mood characteristic to provide feedback to the user, and generating, based on the character mood characteristic and the response information, an announcement video with an animated character image corresponding to the character mood characteristic.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] <CROSS REFERENCE TO RELATED APPLICATIONS> This disclosure claims priority to a Chinese patent application filed on February 9, 2021, bearing application number 202110174149.1 and entitled "Human-Computer Interaction Method, Apparatus, System, Electronic Device, and Computer Media," the entire text of which is incorporated herein by reference.

[0002] The present disclosure relates to the field of artificial intelligence technology, particularly to technical fields such as computer vision and deep learning, and in particular to human-computer interaction methods, devices, system, Electronic device, computer readable medium and computer program To Regarding. [Background technology]

[0003] Conventional virtual digital human customer service systems can only complete simple human-computer interaction and may be perceived as emotionless robots. They only perform simple voice recognition and semantic understanding. In complex counter service systems, simple voice recognition and semantic understanding alone cannot respond appropriately to users with different moods, resulting in a poor user interaction experience. Summary of the Invention

[0004] The embodiments of the present disclosure include a human-computer interaction method, an apparatus, system, Electronic device, computer readable medium and computer program M provide.

[0005] In a first aspect, an embodiment of the present disclosure provides a human-computer interaction method including the steps of receiving information of at least one modality of a user; recognizing, based on the information of the at least one modality, intention information of the user and a mood characteristic of the user corresponding to the intention information; determining answer information to the user based on the intention information; selecting, based on the user's mood characteristic, a mood characteristic of a character to be fed back to the user; and generating, based on the character's mood characteristic and the answer information, an announcement video with an animated character image corresponding to the character's mood characteristic.

[0006] In some embodiments, the information of the at least one modality includes image data and voice data of a user, and the step of recognizing the user's intention information and the user's mood features corresponding to the intention information based on the information of the at least one modality includes the steps of recognizing the user's facial expression features based on the user's image data, acquiring text information from the voice data, extracting the user's intention information based on the text information, and acquiring the user's mood features corresponding to the intention information based on the voice data and the facial expression features.

[0007] In some embodiments, the step of recognizing the user's intention information and the user's mood characteristic corresponding to the intention information based on the information of the at least one modality further includes the step of obtaining the user's mood characteristic from text information.

[0008] In some embodiments, the step of acquiring a user's mood feature corresponding to the intention information based on the voice data and the facial expression features includes the steps of inputting the voice data into a trained voice mood recognition model and acquiring a voice mood feature output by the voice mood recognition model, inputting the facial expression features into a trained facial expression mood recognition model and acquiring a facial expression mood feature output by the facial expression mood recognition model, and weighted-summing the voice mood feature and the facial expression mood feature to acquire a user's mood feature corresponding to the intention information.

[0009] In some embodiments, the information of the at least one modality includes user image data and text data, and the step of recognizing the user's intention information and the user's mood features corresponding to the intention information based on the information of the at least one modality includes the steps of recognizing the user's facial expression features based on the user's image data, extracting the user's intention information based on the text data, and obtaining the user's mood features corresponding to the intention information based on the text data and the facial expression features.

[0010] In some embodiments, the step of generating an announcement video with an animated character image corresponding to the character's mood characteristics based on the character's mood characteristics and answer information includes the steps of generating an answer voice based on the answer information and the character's mood characteristics, and obtaining an announcement video with an animated character image corresponding to the character's mood characteristics based on the answer voice, the character's mood characteristics, and a pre-created model of the animated character image.

[0011] In some embodiments, the step of obtaining an announcement video with an animated character image according to the character's mood features based on the answer voice, the character's mood features, and a pre-created model of the animated character image includes the steps of inputting the answer voice and the character's mood features into a trained mouth-shape-driven model and obtaining mouth shape data output by the mouth-shape-driven model, inputting the answer voice and the character's mood features into a trained facial expression-driven model and obtaining facial expression data output by the facial expression-driven model, driving the model of the animated character image based on the mouth shape data and the facial expression data to obtain a movement sequence of the three-dimensional model, rendering the movement sequence of the three-dimensional model to obtain a picture sequence of video frames, and synthesizing the picture sequence of the video frames to obtain an announcement video with an animated character image according to the character's mood features, where the mouth-shape-driven model and the facial expression-driven model are trained based on a pre-marked voice of the same character and voice mood information obtained from the voice.

[0012] In a second aspect, an embodiment of the present disclosure provides a human-computer interaction device comprising: a receiving unit configured to receive information of at least one modality of a user; a recognition unit configured to recognize intention information of the user and a mood feature of the user corresponding to the intention information based on the information of the at least one modality; a determination unit configured to determine answer information to the user based on the intention information; a selection unit configured to select a mood feature of a character to be fed back to the user based on the mood feature of the user; and an announcement unit configured to generate an announcement video with an animated character image corresponding to the mood feature of the character based on the mood feature of the character and the answer information.

[0013] In some embodiments, the information of the at least one modality includes image data and voice data of a user, and the recognition unit comprises a recognition subunit configured to recognize facial expression features of a user based on the image data of the user, a text acquisition subunit configured to acquire text information from the voice data, an extraction subunit configured to extract intention information of the user based on the text information, and a feature acquisition subunit configured to acquire mood features of the user corresponding to the intention information based on the voice data and the facial expression features.

[0014] In some embodiments, the mood characteristics of the user of the recognition unit are further obtained from text information.

[0015] In some embodiments, the feature acquisition subunit comprises: an audio acquisition module configured to input audio data into a trained audio mood recognition model and acquire audio mood features output by the audio mood recognition model; an facial expression acquisition module configured to input facial expression features into the trained facial expression mood recognition model and acquire facial expression mood features output by the facial expression mood recognition model; and an addition module configured to weightedly sum the audio mood features and the facial expression mood features to acquire a user mood feature corresponding to the intention information.

[0016] In some embodiments, the information of the at least one modality includes image data and text data of a user, and the recognition unit comprises: a recognition module configured to recognize facial expression features of a user based on the image data of the user; an extraction module configured to extract intention information of the user based on the text data; and a feature acquisition module configured to acquire mood features of the user corresponding to the intention information based on the text data and the facial expression features.

[0017] In some embodiments, the announcement unit comprises: Generate a response voice based on the response information and the person mood characteristicsand a video acquisition subunit configured to acquire an announcement video with an animated character image according to the character's mood characteristics based on the answer voice, the character's mood characteristics, and a model of the animated character image created in advance.

[0018] In some embodiments, the video capture subunit includes a mouth shape driving module configured to input the answer voice and the character's mood features into a trained mouth shape driving model and obtain mouth shape data output by the mouth shape driving model, an expression driving module configured to input the answer voice and the character's mood features into a trained expression driving model and obtain expression data output by the expression driving model, a model driving module configured to drive a model of an animated character image based on the mouth shape data and the expression data and obtain a motion sequence of a three-dimensional model, a picture capturing module configured to render the motion sequence of the three-dimensional model to obtain a picture sequence of video frames, and a video capturing module configured to synthesize the picture sequence of the video frames to obtain an announcement video with an animated character image according to the character's mood features, where the mouth shape driving model and the expression driving model are trained based on the voice of the same character that has been previously marked and the voice mood information obtained from the voice.

[0019] In a third aspect, an embodiment of the present disclosure provides a human-computer interaction system, the system comprising: a collection device, a display device, and an interaction platform respectively connected to the collection device and the display device, the collection device is configured to collect information of at least one modality of a user, the interaction platform is configured to receive the information of the at least one modality of the user, and based on the information of the at least one modality, recognize the user's intention information and the user's mood characteristic corresponding to the intention information, determine answer information to the user based on the intention information, select a character mood characteristic to be fed back to the user based on the user's mood characteristic, and generate an announcement video with an animated character image corresponding to the character's mood characteristic based on the character mood characteristic and the answer information, and the display device is configured to receive and play the announcement video.

[0020] In a fourth aspect, an embodiment of the present disclosure provides an electronic device comprising one or more processors and a storage device having one or more programs stored therein, the one or more programs, when executed by the one or more processors, causing the one or more processors to realize a human-computer interaction method according to any of the embodiments of the first aspect.

[0021] In a fifth aspect, an embodiment of the present disclosure provides a computer readable medium having a computer program stored thereon, the computer program being configured to, when executed by a processor, implement the human-computer interaction method according to any of the embodiments of the first aspect.

[0022] In a sixth aspect, an embodiment of the present disclosure provides a computer program that, when executed by a processor, executes the human-computer interaction method according to any of the embodiments of the first aspect. M provide.

[0023] The human-computer interaction method and apparatus provided by the embodiment of the present disclosure first receive information of at least one modality of a user. Then, based on the information of the at least one modality, recognize the user's intention information and the user's mood characteristic corresponding to the intention information. Then, based on the intention information, determine answer information to the user. Then, based on the user's mood characteristic, select a mood characteristic of a character to be fed back to the user. Finally, based on the character's mood characteristic and the answer information, generate an announcement video with an animated character image according to the character's mood characteristic. Thus, by analyzing the information of at least one modality of a user and determining the mood characteristic of a character to be fed back to the user, effective feedback according to the mood is provided to users in different moods, and communication of emotions during the human-computer interaction process is ensured. [Brief description of the drawings]

[0024] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of the non-limiting embodiments which is given with reference to the following drawings. [Figure 1] FIG. 1 illustrates an exemplary system architecture to which an embodiment of the present disclosure can be applied. [Diagram 2] 1 is a flowchart of one embodiment of a human-computer interaction method according to the present disclosure. [Diagram 3] 1 is a flow chart of one embodiment for recognizing user intent information and user mood characteristics according to the present disclosure. [Figure 4] FIG. 1 is a structural schematic diagram illustrating an embodiment of a human-computer interaction device according to the present disclosure. [Diagram 5] FIG. 1 is a structural schematic diagram illustrating an embodiment of a human-computer interaction system according to the present disclosure. [Figure 6]FIG. 1 is a structural schematic diagram of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The present disclosure will be described in more detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of illustrating the related invention, and are not intended to limit the invention. For the sake of convenience, only parts related to the invention are shown in the drawings.

[0026] In addition, the embodiments and features of the embodiments of the present disclosure can be combined with each other as long as no contradiction occurs. Hereinafter, the present disclosure will be described in detail with reference to the drawings and embodiments.

[0027] FIG. 1 illustrates an exemplary system architecture 100 to which the human-computer interaction method according to the present disclosure can be applied.

[0028] 1, system architecture 100 may include terminal devices 101, 102, a self-service terminal 103, a network 104, and a server 105. Network 104 is used as a medium to provide communication links among terminal devices 101, 102, self-service terminal 103, and server 105. Network 104 may include various types of connections, typically including wireless communication links.

[0029] To send and receive messages, the terminals 101, 102 and the self-service terminal 103 communicate with a server 105 via a network 104. Various communication client applications may be installed on the terminals 101, 102 and the self-service terminal 103, such as, for example, instant communication tools, mail clients, etc.

[0030] The terminal devices 101, 102 may be hardware or software. If the terminal devices 101, 102 are hardware, they may be user devices having communication and control functions capable of communicating with the server 105. If the terminal devices 101, 102 are software, they may be installed in the user devices. The terminal devices 101, 102 may be realized as multiple software or software modules (e.g., software or software modules for providing distributed services), or may be realized as a single software or software module. There is no particular limitation here.

[0031] The server 105 may be a server providing various services, for example, a back-end server supporting a client question and answer system in the terminal devices 101, 102 and the self-service terminal 103. The back-end server may analyze and process information of at least one modality of related users collected in the terminal devices 101, 102 and the self-service terminal 103, and feed back the processing result (for example, announcement video with animated character images) to the terminal devices or the self-service terminal.

[0032] It should be noted that the server may be hardware or software. If the server is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. If the server is software, it may be implemented as multiple software programs or software modules (e.g., software programs or software modules for providing distributed services), or as a single software program or software module. There are no particular limitations here.

[0033] It should be noted that the human-computer interaction method provided by the embodiments of the present disclosure is generally performed by the server 105 .

[0034] It should be understood that the numbers of terminal devices, networks and servers in Figure 1 are merely exemplary, and the numbers of terminal devices, networks and servers may be arbitrarily increased or decreased according to the needs of implementation.

[0035] As shown in FIG. 2, a flow 200 of an embodiment of a human-computer interaction method according to the present disclosure is shown, which includes the following steps:

[0036] In step 201, information of at least one modality of a user is received.

[0037] In this embodiment, the executing entity in which the human-computer interaction method is executed may receive information from different sources of the user during the same period. The information from different sources is information of different modalities, and when there is information from different sources, it is also referred to as information of at least one modality. Specifically, the information of at least one modality may include one or more of image data, voice data, and text data.

[0038] In this embodiment, the information of at least one modality of the user is information sent by the user or / and information related to the user. For example, the image data is image data of the user's face, the user's body, the user's hair, etc., the voice data is voice data of the user's voice, and the text data is data of characters, symbols, numbers, etc. input by the user to the execution subject. The information of at least one modality of the user can analyze the user's intention, and identify the user's question and purpose, as well as the user's mood state when the user asks a question or inputs information.

[0039] In fact, the information of different modalities may be descriptive information of the same thing collected by different sensors. For example, in the case of video search, the information of different modalities includes voice data and image data of the same user collected in the same period, and the voice data and image data correspond to each other at the same time. Furthermore, there is a task-based interactive communication process in which a user transmits image data, text data, etc. of the same user in the same period to an executing subject via a user terminal.

[0040] In this embodiment, the executing entity of the human-computer interaction method (e.g., the server 105 shown in FIG. 1) may receive information of at least one modality of the user through multiple methods. For example, a data set to be processed may be collected in real time from a user terminal (e.g., the terminal devices 101, 102, and the self-service terminal 103 shown in FIG. 1), and the information of at least one modality may be extracted from the data set to be processed. Alternatively, a data set to be processed including information of multiple modalities may be obtained from a local memory, and the information of at least one modality may be extracted from the data set to be processed. Alternatively, the information of at least one modality may be information transmitted in real time by the terminal.

[0041] In step 202, a user's intention information and a mood feature of the user corresponding to the intention information are recognized based on information of at least one modality.

[0042] In this embodiment, the user's intention information is information that represents the contents of the user's question, purpose, greeting, etc. After acquiring the user's intention information, the execution subject may provide different feedback according to differences in the contents of the intention information.

[0043] The user's mood characteristic is an individual's mood state when the user sends or presents information in different modalities, and specifically, the mood states include anger, sadness, joy, rage, disgust, and so on.

[0044] Furthermore, it may have different methods for recognizing the user's intention information and the user's mood characteristics based on the information of different modalities of the user.

[0045] In some optional embodiments of the present disclosure, the information of at least one modality includes image data and voice data of a user, and the step of recognizing the user's intention information and the user's mood features corresponding to the intention information based on the information of the at least one modality includes the steps of recognizing the user's facial expression features based on the user's image data, obtaining text information from the voice data, extracting the user's intention information based on the text information, and obtaining the user's mood features corresponding to the intention information based on the voice data and the facial expression features.

[0046] In this optional embodiment, when the information of at least one modality of the user includes image data and voice data of the user, the method includes the steps of recognizing the user's facial expression features based on the user's image data, acquiring text information based on the voice data, extracting intention information based on the text information, and acquiring the user's mood features based on the voice data and the facial expression features. This allows the user's mood to be comprehensively judged based on the mood features included in both the user's facial expression (facial expression features) and voice (voice data), thereby improving the reliability of analyzing the user's mood features to a certain extent.

[0047] In some optional embodiments of the present disclosure, the information of at least one modality includes user image data and text data, and a method for recognizing user intention information and user mood features based on the information of the at least one modality includes the steps of recognizing user facial expression features based on the user image data, extracting user intention information based on the text data, and obtaining user mood features corresponding to the intention information based on the text data and the facial expression features.

[0048] In the method for recognizing user intention information and user mood characteristics provided by this optional embodiment, when the information of the user's modality includes image data and text data, the user's facial expression characteristics are recognized based on the image data, intention information is extracted based on the text data, and the user's mood characteristics are obtained based on the text data and the facial expression characteristics. In this way, by comprehensively judging the user's mood based on the mood included in both the user's facial expression (facial expression characteristics) and language (text information), a reliable mood analysis method is provided for extracting the intention information and mood of a deaf-mute person.

[0049] Optionally, the information of at least one modality includes image data, text data and voice data of a user. The method for recognizing the user's intention information and the user's mood feature based on the information of the at least one modality includes: recognizing the user's facial expression feature based on the user's image data, extracting the user's intention information based on the text data and the voice data, and acquiring the user's mood feature corresponding to the intention information based on the text data, the facial expression feature and the voice data.

[0050] In this optional embodiment, when the information of at least one modality includes all three of the user's image data, text data, and voice data, the user's mood can be judged comprehensively based on the mood information contained in the three aspects of the user's facial expression (facial expression features), voice (voice data), and language (text information), thereby improving the reliability of the user's mood analysis.

[0051] In this embodiment, text information and text data are both different representation forms of text, and the representations of text information and text data are used only to distinguish the source or processing manner of text.

[0052] In addition, since the user's language, characters and facial expressions can all reflect the user's mood, a user's mood feature can be obtained. In some optional embodiments of this embodiment, the step of obtaining the user's mood feature corresponding to the intention information based on the voice data and facial expression features includes: inputting the voice data into a trained voice mood recognition model to obtain the voice mood feature output by the voice mood recognition model, inputting the facial expression features into a trained facial expression mood recognition model to obtain the facial expression mood feature output by the facial expression mood recognition model, and weighted-summing the voice mood feature and the facial expression mood feature to obtain the user's mood feature corresponding to the intention information.

[0053] In this optional embodiment, a trained facial mood recognition model and a vocal mood recognition model are used to recognize facial mood features and vocal mood features, respectively, thereby enabling the real-time mood state of a user to be quickly obtained from information of at least one modality of the user, providing a reliable and robust foundation for realizing animated character images capable of expressing emotions.

[0054] Optionally, the step of obtaining a user's mood feature corresponding to the intention information based on the text data, facial expression features and voice data may further include the steps of inputting the text data into a trained text mood recognition model and obtaining a text mood feature output by the text mood recognition model, inputting the voice data into a trained voice mood recognition model and obtaining a voice mood feature output by the voice mood recognition model, inputting the facial expression features into the trained voice mood recognition model and obtaining a facial expression mood feature output by the facial expression mood recognition model, and weighted-summing the text mood feature, the voice mood feature and the facial expression mood feature to obtain a user's mood feature corresponding to the intention information.

[0055] In this embodiment, the voice mood recognition model is configured to recognize mood features in a user's voice data and determine the mood state of the user at the time of speaking, the facial expression mood recognition model is configured to recognize facial expression features related to mood among the user's facial expression features and determine the mood state of the user when expressing a certain facial expression, and the text mood recognition model is configured to recognize mood features in the user's text data and determine the mood state expressed by text output by the user.

[0056] The facial mood recognition model, the voice mood recognition model and the text mood recognition model may be models trained on a large amount of marked text data, facial features and voice data about the same user, and the resulting voice mood features, facial mood features and text mood features are all used to represent the user's mood state (joy, anger, sadness, fear). Note that the voice mood recognition model and the facial mood recognition model in this optional embodiment can also be applied to other embodiments.

[0057] In step 203, answer information to the user is determined based on the intention information.

[0058] In this embodiment, the answer information to the user is information corresponding to the user's intention information, and the answer information is also audio content that requires announcement by an animated character image. For example, the user's intention information is a question, "How tall is Mr. Li Si?" The answer information is a response, "Mr. Li Si's height is 1.8 meters."

[0059] After obtaining the user's intent information, the actor can identify answer information in various ways, such as querying a knowledge base, searching a knowledge graph, and so on.

[0060] In step 204, a character mood characteristic for providing feedback to the user is selected based on the user's mood characteristic.

[0061] In this embodiment, the character mood feature represents a mood state of the animated character image, which may be the same as or different from the mood state expressed by the user's mood feature. For example, if the user's mood feature is angry, the character's mood feature may be expressed as calm. If the user's mood feature is happy, the character's mood feature may be expressed as happy as well.

[0062] After obtaining the mood feature of the user, the executing subject of the human-computer interaction method can select one or more mood features from a preset mood feature library as the mood feature of the character based on the mood feature of the user, and the mood feature of the character is applied to the animated character image to realize the expression of the mood feature of the animated character image.

[0063] In step 205, an announcement video is generated using an animated character image according to the character's mood characteristics, based on the character's mood characteristics and the answer information.

[0064] In this embodiment, the announcement video using the animated character image is a video in which a virtual animated character image announces information as an announcer, and both the character's mood characteristics and answer information are information to be expressed by the animated character image. In order to express the answer information vividly and intuitively, the answer information may be converted into an answer voice. The answer voice to be announced is expressed by a virtual mouth opening movement of the animated character image in the announcement video using the animated character image. The character's mood characteristics are expressed by a virtual change in facial expression of the animated character image.

[0065] In the communication between the animated character image and the user, the synthesized voice of the animated character image may include soothing character mood information based on the character's mood characteristics. At the same time, a facial expression corresponding to the character's mood characteristics may be selected and expressed on the face of the animated character image, improving the richness of the facial expressions of the animated character image.

[0066] In order to make the answer voice more vivid, in some optional implementations of this embodiment, the step of generating an announcement video with an animated character image corresponding to the character's mood characteristics based on the character's mood characteristics and the answer information includes the steps of generating an answer voice based on the answer information and the character's mood characteristics, and obtaining an announcement video with an animated character image corresponding to the character's mood characteristics based on the answer voice, the character's mood characteristics, and a pre-created model of the animated character image.

[0067] In this optional embodiment, the model of the animated character image may be a three-dimensional model obtained by modeling a three-dimensional image. Here, modeling a three-dimensional image is a process of virtually constructing a three-dimensional space using three-dimensional image production software to create a model having three-dimensional data. Furthermore, each part of the animated character image may be modeled (e.g., face contour modeling, independent mouth modeling, independent hair modeling, independent body modeling, independent bone structure modeling, facial expression modeling, etc.), and the model of the animated character image may be obtained by combining the models of each selected part.

[0068] In this optional embodiment, the answer voice generated based on the answer information and the character's mood characteristics includes the pre-analyzed character's mood elements, so that the voice in the acquired announcement video with the animated character image is emotional and the user is easily immersed in the mood. The action of the animated character image in the announcement video with the animated character image acquired based on the character's mood characteristics is emotional and has an infectious effect.

[0069] In some optional embodiments of this embodiment, the step of obtaining an announcement video with an animated character image according to the character's mood characteristics based on the answer voice, the character's mood characteristics, and a model of the animated character image created in advance includes the steps of inputting the answer voice and the character's mood characteristics into a trained mouth shape driven model and obtaining mouth shape data output by the mouth shape driven model, inputting the answer voice and the character's mood characteristics into a trained facial expression driven model and obtaining facial expression data output by the facial expression driven model, driving the model of the animated character image based on the mouth shape data and the facial expression data to obtain a motion sequence of the three-dimensional model, rendering the motion sequence of the three-dimensional model to obtain a picture sequence of video frames, and synthesizing the picture sequence of the video frames to obtain an announcement video with an animated character image according to the character's mood characteristics, where the mouth shape driven model and the facial expression driven model are trained based on the voice of the same character marked in advance and the voice mood information obtained from the voice.

[0070] In this optional embodiment, the mouth shape driven model is a model for recognizing the movement trajectory of the lips of an animated character image in three-dimensional space, and the mouth shape driven model can also be combined with a mouth shape library to obtain mouth shape data at different times of an animated character image, and the mouth shape data is also data of mouth shape changes of an animated character image.

[0071] In this optional embodiment, the expression-driven model is a model for recognizing the trajectory of the movement of facial feature points of an animated character image in three-dimensional space, and the expression-driven model can also be combined with an expression library to obtain facial expression data of the animated character image at different times, and the facial expression data is also data of facial expression changes of the animated character image.

[0072] In this optional embodiment, the mouth-shape-driven model and the facial expression-driven model can be trained based on the pre-marked voice of the same character and the voice mood information obtained from the voice, so that the mouth shape and voice of the obtained animated character image can be more fitted and matched seamlessly, making the animated character image in the announcement video more lively and vibrant.

[0073] Alternatively, a speech-to-animation (STA) model may be used to directly realize announcement videos using animated character images according to the character's mood. The speech-to-animation model may be obtained by uniformly training a number of different types of models (such as avatar models and voice synthesis models), and by combining artificial intelligence and computer graphics, the pronunciation mouth shape corresponding to the voice can be calculated in real time, and the facial expressions of the animated character images can be minutely moved to realize synchronous expression of the voice and image of the animation.

[0074] The data involved in training the audio-video synthesis model mainly includes image data, audio data and text data. The three types of data have a certain degree of intersection, such as the audio in the video data for image training, the audio data for speech recognition training, and the audio data for speech synthesis training are consistent. The text data corresponding to the audio data for speech recognition training is consistent with the text data corresponding to the audio data for image training. These consistency is to improve the accuracy during training of the audio-video synthesis model, and in addition to this, we also need artificially marked data, i.e., facial expression, mood features of the image.

[0075] The audio / video synthesis model includes an avatar model and a voice synthesis model. The modeling of the avatar model includes a basic static model of the character's basic face and facial contour, the five senses, and the body, as well as a dynamic model of the character's mouth shape, facial expression, and movement. The voice synthesis model incorporates the character's mood characteristics in addition to the most basic tone model.

[0076] The human-computer interaction method provided by the embodiment of the present disclosure first receives information of at least one modality of a user. Then, based on the information of the at least one modality, recognizes the user's intention information and the user's mood characteristic corresponding to the intention information. Then, based on the intention information, determines answer information to the user. Then, based on the mood characteristic of the user, selects a mood characteristic of a character to be fed back to the user. Finally, based on the mood characteristic of the character and the answer information, generates an announcement video with an animated character image according to the mood characteristic of the character. Thus, by analyzing the information of at least one modality of a user and determining the mood characteristic of the animated character image, effective feedback according to the mood is provided to users in different moods, and communication of emotions during the human-computer interaction process is ensured.

[0077] In another embodiment of the present disclosure, the at least one modality of information includes image data and audio data of the user. As shown in FIG. 3, a flow chart 300 of an embodiment for recognizing user intent information and user mood characteristics according to the present disclosure is shown. The method includes the following steps:

[0078] In step 301, facial features of a user are recognized based on image data of the user.

[0079] In this embodiment, the recognition of facial expression features refers to locating and extracting facial organ features, texture regions and predefined feature points. The recognition of facial expression features is the core step in facial expression recognition and is also the key to face recognition, which determines the final face recognition result and directly affects the high or low recognition rate.

[0080] In this optional embodiment, facial expressions also belong to body language and can reflect the mood of a user, and each user's mood profile has a corresponding facial expression.

[0081] The image data of the user includes facial image data, and the facial image data is analyzed to identify the user's facial features.

[0082] Alternatively, the image data of the user may include body image data of the user. By analysing the body image data, the facial features of the user may be more clearly defined.

[0083] In step 302, text information is obtained from the audio data.

[0084] In this embodiment, text information can be obtained by a mature voice recognition model. For example, an ASR (Automatic Speech Recognition) model that can convert voice into text may be used. By inputting voice data into the ASR model, characters output from the ASR model can be obtained, making it possible to recognize text information.

[0085] In step 303, user intention information is extracted based on the text information.

[0086] In this optional embodiment, the text information is information obtained by converting the user's voice data into text. The intent information is obtained by a mature intent recognition model, and the intent information is subjected to processing such as sentence detection, word segmentation, part-of-speech tagging, syntactic analysis, text classification / clustering, and information extraction using an NLU (Natural Language Understanding) model to perform semantic analysis on the text information and identify the user's intent information.

[0087] In step 304, a mood feature of the user corresponding to the intention information is obtained based on the voice data, the text information and the facial expression feature.

[0088] In this optional embodiment, when determining the user's mood feature, the text information recognized by the voice model can be combined with the user's voice data (speaking manner) and the user's facial expression features to jointly determine the user's mood feature, which is more accurate than determining the user's facial expression based only on the user's facial expression or only on the user's voice information, thereby selecting more appropriate answer information and character mood features to apply to the animated character image, and making it convenient to communicate with the user through the animated character image.

[0089] In the method for recognizing user's intention information and user's mood characteristics provided by the present embodiment, when the information of the user's modality includes image data and voice data, the method includes the steps of recognizing the user's facial expression characteristics based on the image data, acquiring text information based on the voice data, extracting intention information based on the text information, and acquiring the user's mood characteristics based on the voice data, the text information, and the facial expression characteristics. This makes it possible to comprehensively judge the user's mood based on the mood included in the three aspects of the user's facial expression (facial expression characteristics), voice (voice data), and language (text information), and to increase the reliability of analyzing the user's mood characteristics.

[0090] Further referring to FIG. 4, as an implementation of the method shown in each of the above figures, the present disclosure provides an embodiment of a human-computer interaction apparatus, which embodiment corresponds to the embodiment of the method shown in FIG. 2, and which can be specifically applied to various electronic devices.

[0091] As shown in FIG. 4, a human-computer interaction apparatus 400 according to an embodiment of the present disclosure is provided, and the apparatus 400 includes a receiving unit 401, a recognition unit 402, a determination unit 403, a selection unit 404, and an announcement unit 405. Here, the receiving unit 401 may be configured to receive information of at least one modality of a user. The recognition unit 402 may be configured to recognize, based on the information of the at least one modality, the intention information of the user and the mood feature of the user corresponding to the intention information. The determination unit 403 may be configured to determine answer information to the user based on the intention information. The selection unit 404 may be configured to select, based on the mood feature of the user, a mood feature of a character to be fed back to the user. The announcement unit 405 may be configured to generate an announcement video with an animated character image according to the mood feature of the character based on the mood feature of the character and the answer information.

[0092] In this embodiment, in the human-computer interaction device 400, the specific processing of the receiving unit 401, the recognition unit 402, the determination unit 403, the selection unit 404, and the announcement unit 405 and the technical effects achieved thereby can refer to the relevant descriptions of step 201, step 202, step 203, step 204, and step 205 in the corresponding embodiment of FIG. 2 respectively.

[0093] In some embodiments, the information of at least one modality includes image data and voice data of a user. The recognition unit 402 includes an identification subunit (not shown), a text acquisition subunit (not shown), an extraction subunit (not shown), and a feature acquisition subunit (not shown). Here, the recognition subunit may be configured to recognize a facial expression feature of a user based on the image data of the user. The text acquisition subunit may be configured to acquire text information from the voice data. The extraction subunit may be configured to extract an intention information of the user based on the text information. The feature acquisition subunit may be configured to acquire a mood feature of the user corresponding to the intention information based on the voice data and the facial expression feature.

[0094] In some embodiments, the mood characteristics of the user of the recognition unit are further obtained from text information.

[0095] In some embodiments, the feature acquisition subunit includes a voice acquisition module (not shown), a facial expression acquisition module (not shown), and an addition module (not shown). Here, the voice acquisition module may be configured to input the voice data into a trained voice mood recognition model and acquire a voice mood feature output by the voice mood recognition model. The facial expression acquisition module may be configured to input the facial expression feature into the trained facial mood recognition model and acquire a facial mood feature output by the facial mood recognition model. The addition module may be configured to weighted-sum the voice mood feature and the facial mood feature to acquire a user's mood feature corresponding to the intention information.

[0096] In some embodiments, the information of at least one modality includes image data and text data of a user, and the recognition unit 402 includes a recognition module (not shown), an extraction module (not shown), and a feature acquisition module (not shown). Here, the recognition module may be configured to recognize facial expression features of a user based on the image data of the user. The extraction module may be configured to extract intention information of the user based on the text data. The feature acquisition module may be configured to acquire mood features of the user corresponding to the intention information based on the text data and the facial expression features.

[0097] In some embodiments, the announcement unit 405 comprises a generating subunit (not shown) and a video acquisition subunit (not shown), where the generating subunit: Generate a response voice based on the response information and the person mood characteristics The video acquisition subunit may be configured to acquire an announcement video with an animated character image according to the character's mood characteristics, based on the answer voice, the character's mood characteristics, and a model of the animated character image created in advance.

[0098] In some embodiments, the video acquisition subunit includes a mouth shape driving module (not shown), a facial expression driving module (not shown), a model driving module (not shown), a picture acquisition module (not shown), and a video acquisition module (not shown). The video acquisition subunit includes a mouth shape driving module configured to input the answer voice and the character's mood features into a trained mouth shape driving model and acquire mouth shape data output by the mouth shape driving model, an facial expression driving module configured to input the answer voice and the character's mood features into a trained facial expression driving model and acquire facial expression data output by the facial expression driving model, a model driving module configured to drive a model of an animated character image based on the mouth shape data and the facial expression data and acquire a motion sequence of a three-dimensional model, a picture acquisition module configured to render the motion sequence of the three-dimensional model to acquire a picture sequence of video frames, and a video acquisition module configured to synthesize the picture sequence of the video frames to acquire an announcement video with an animated character image according to the character's mood features. Here, the mouth shape driven model and the facial expression driven model are trained based on the voice of the same character that has been previously marked and the voice mood information obtained from the voice.

[0099] In the human-computer interaction device provided by the embodiment of the present disclosure, first, the receiving unit 401 receives information of at least one modality of the user. Then, the recognition unit 402 recognizes the intention information of the user and the mood feature of the user corresponding to the intention information based on the information of at least one modality. Then, the determination unit 403 determines answer information to the user based on the intention information. Then, the selection unit 404 selects the mood feature of the character to be fed back to the user based on the mood feature of the user. Finally, the announcement unit 405 generates an announcement video by an animated character image according to the mood feature of the character based on the mood feature of the character and the answer information. Thus, by analyzing the information of at least one modality of the user and determining the mood feature of the character of the animated character image, effective feedback according to the mood is provided to users in different moods, and the communication of emotions during the human-computer interaction process is ensured.

[0100] Further referring to FIG. 5, as an implementation of the method shown in each of the above figures, the present disclosure provides an embodiment of a human-computer interaction system, which embodiment of the system corresponds to the embodiment of the method shown in FIG.

[0101] As shown in FIG. 5, an embodiment of the present disclosure provides a human-computer interaction system 500, which includes a collection device 501, a display device 502, and an interaction platform 503 connected to the collection device 501 and the display device 502, respectively. The collection device 501 is configured to collect information of at least one modality of a user. The interaction platform 503 is configured to receive information of at least one modality of a user, recognize the intention information of the user and the mood characteristic of the user corresponding to the intention information based on the information of the at least one modality, determine answer information to the user based on the intention information, select a mood characteristic of a character to be fed back to the user based on the mood characteristic of the user, and generate an announcement video with an animated character image according to the mood characteristic of the character based on the mood characteristic of the character and the answer information. The display device 502 is configured to receive and play the announcement video.

[0102] In this embodiment, the collecting device is a device for collecting at least one modality of information of a user, and different types of the collecting device are different according to the information of different modalities. For example, the information of at least one modality includes image data and voice data of a user, and accordingly, the collecting device may include a camera and a speaker. In addition, the information of at least one modality includes text data of a user, and the collecting device may further include an input device such as a keyboard and a mouse.

[0103] In this embodiment, the collection device 501, the display device 502 and the interaction platform 503 may be installed separately or integrated to form an integrated device (e.g., the self-service terminal or terminal device in FIG. 1).

[0104] Please refer to FIG. 6, which is a structural schematic diagram showing an electronic device 600 applied to realize an embodiment of the present disclosure.

[0105] 6, the electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 601 capable of performing various suitable operations and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. The RAM 603 further stores various programs and data necessary for the operation of the electronic device 600. The processing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0106] Typically, input devices 606 including, for example, a touch screen, touch pad, keyboard, and mouse, output devices 607 including, for example, a liquid crystal display (LCD), a speaker, and a vibrator, storage devices 608 including, for example, a magnetic tape and a hard disk, and a communication device 609 may be connected to the I / O interface 605. The communication device 609 allows the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 6 illustrates the electronic device 600 having various devices, it should be understood that it is not required to implement or include all of the devices illustrated. Optionally, more or less devices may be implemented. Each block illustrated in FIG. 6 may represent one device or multiple devices as needed.

[0107] In particular, according to embodiments of the present disclosure, the processes described with reference to the flowcharts above may be implemented as a computer software program. For example, embodiments of the present disclosure may be implemented as a computer program embodied in a computer readable medium. MThe computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication device 609, or can be installed from the storage device 608 or the ROM 602. When the computer program is executed by the processing device 601, it performs the above-mentioned functions defined in the method of the embodiment of the present disclosure.

[0108] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of the computer-readable storage medium may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by, or that can be embedded in, an instruction execution system, apparatus, or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal in baseband or propagated as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal may take various forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium. The computer-readable signal medium may transmit, propagate, or transmit a program used by or embedded in an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium may be transmitted by any suitable medium, including, but not limited to, an electrical wire, an optical cable, RF (Radio Frequency), or the like, or any suitable combination thereof.

[0109] The computer-readable medium may be included in the server, or may exist separately from the server. The computer-readable medium stores one or more programs, and when the one or more programs are executed by the processor, causes the server to receive information of at least one modality of a user, recognize the user's intention information and the user's mood characteristic corresponding to the intention information based on the information of the at least one modality, determine answer information to the user based on the intention information, select a character's mood characteristic to be fed back to the user based on the user's mood characteristic, and generate an announcement video with an animated character image according to the character's mood characteristic based on the character's mood characteristic and the answer information.

[0110] Computer program code for carrying out the operations of the embodiments of the present disclosure can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and traditional procedural programming languages ​​such as "C" or similar programming languages. The program code can run entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via an Internet service provided by an Internet service provider).

[0111] The flowcharts and block diagrams in the drawings illustrate systems, methods and computer programs according to various embodiments of the present disclosure. To The diagrams illustrate the architecture, functions and operations that can be realized by the above-mentioned embodiment. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a portion of code. The module, program segment, or portion of code includes one or more executable instructions for implementing a certain logic function. It should be noted that in some alternative embodiments, the functions shown in the blocks may be executed in a different order than that shown in the drawings. For example, two blocks shown in succession may actually be executed substantially in parallel in response to such functions, or may sometimes be executed in the reverse order. It should be further noted that all blocks in the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, may be implemented in a dedicated hardware-based system that executes a certain function or operation, or may be implemented in a combination of dedicated hardware and computer instructions.

[0112] The units described in the embodiments of the present disclosure may be implemented in software or hardware. The described units may be located in a processor, and may be described as "a processor including a receiving unit, a recognizing unit, a determining unit, a selecting unit, and an announcing unit." Here, the names of these units do not limit the units themselves in some cases, and for example, the receiving unit may be described as a unit "configured to receive information of at least one modality of a user."

[0113] The above description is merely a description of the preferred embodiments of the present disclosure and the technical principles applied thereto. Those skilled in the art should understand that the scope of the present disclosure is not limited to the technical solution consisting of a specific combination of the above-mentioned technical features, but should also include other technical solutions consisting of any combination of the above-mentioned technical features or their equivalent features within the scope of the above-mentioned disclosure. For example, the above features may be substituted with technical features having similar functions (but are not limited to these) disclosed in the embodiments of the present disclosure.

Claims

1. 1. A human-computer interaction method comprising: receiving at least one modality of information of a user, the at least one modality of information including image data and audio data of the user; recognizing the user's intention information based on the information of the at least one modality; inputting the voice data into a trained voice mood recognition model and obtaining voice mood features output by the voice mood recognition model; Recognizing facial expression features of the user based on image data of the user, inputting the facial expression features into a trained facial expression mood recognition model, and obtaining facial expression mood features output by the facial expression mood recognition model; obtaining a user mood feature corresponding to the intention information by performing a weighted sum of the voice mood feature and the facial expression mood feature; determining answer information for the user based on the intention information; selecting a mood characteristic of a character to be fed back to the user based on the mood characteristic of the user; generating an announcement video using an animated character image according to the mood characteristics of the character and the answer information; A human-computer interaction method comprising:

2. The step of recognizing the user's intention information based on the information of the at least one modality, obtaining text information from the audio data; extracting the user's intention information based on the text information acquired from the voice data; 2. The human-computer interaction method of claim 1, comprising:

3. The at least one modality of information further includes text data of the user; In the step of recognizing the user's intention information based on the information of at least one type of modality, the user's intention information is extracted based on text data of the user.

2. The human-computer interaction method of claim 1.

4. The method further includes a step of obtaining a text mood feature of the user from the text data of the user, In the step of acquiring a user mood feature corresponding to the intention information, the text mood feature, the voice mood feature, and the facial expression mood feature are weighted and summed to acquire a user mood feature corresponding to the intention information. A human-computer interaction method according to claim 3.

5. generating an announcement video using an animated character image according to the mood characteristic of the character based on the mood characteristic of the character and the answer information, generating an answer voice based on the answer information and a mood characteristic of the character; obtaining an announcement video with an animated character image according to the mood characteristics of the character based on the answer voice, the mood characteristics of the character, and a model of a pre-created animated character image; 5. A human-computer interaction method according to claim 1, comprising:

6. The step of obtaining an announcement video with an animated character image according to the mood characteristics of the character based on the answer voice, the mood characteristics of the character, and a model of a pre-created animated character image includes: inputting the answer voice and the mood feature of the character into a trained mouth shape driven model, and obtaining mouth shape data output by the mouth shape driven model; inputting the answer voice and the mood feature of the character into a trained facial expression driven model, and obtaining facial expression data output by the facial expression driven model; driving a model of the animated character image based on the mouth shape data and the facial expression data to obtain a motion sequence of a three-dimensional model; rendering a motion sequence of the three-dimensional model to obtain a picture sequence of video frames; synthesizing the picture sequence of the video frames to obtain an announcement video with animated character images according to the mood characteristics of the characters; Here, the mouth shape driving model and the facial expression driving model are trained based on the voice of the same character that has been previously marked and the voice mood information obtained from the voice. A human-computer interaction method according to claim 5.

7. A human-computer interaction device, comprising: a receiving unit configured to receive at least one modality of information of a user, the at least one modality of information including image data and voice data of the user; a recognition unit configured to recognize intention information of the user based on information of the at least one modality, input the voice data into a trained voice mood recognition model, obtain a voice mood feature output by the voice mood recognition model, recognize facial expression features of the user based on image data of the user, input the facial expression features into a trained facial mood recognition model, obtain a facial mood feature output by the facial mood recognition model, and weighted-sum the voice mood feature and the facial mood feature to obtain a user mood feature corresponding to the intention information; A determination unit configured to determine answer information to the user based on the intention information; A selection unit configured to select a mood characteristic of a character to provide feedback to the user based on the mood characteristic of the user; an announcing unit configured to generate an announcing video with an animated character image according to the mood characteristic of the character based on the mood characteristic of the character and the answer information; A human-computer interaction device comprising:

8. 1. A human-computer interaction system comprising: The system includes a collection device, a display device, and an interaction platform connected to the collection device and the display device, respectively; The collection device is configured to collect at least one modality of information of a user, the at least one modality of information including image data and voice data of the user; the interaction platform is configured to receive information of at least one modality of the user, recognize intention information of the user based on the information of the at least one modality, input the voice data into a trained voice mood recognition model, obtain a voice mood feature output by the voice mood recognition model, recognize facial expression features of the user based on image data of the user, input the facial expression features into a trained facial mood recognition model, obtain a facial mood feature output by the facial mood recognition model, weighted-sum the voice mood feature and the facial mood feature to obtain a user mood feature corresponding to the intention information, determine answer information to the user based on the intention information, select a character mood feature to be fed back to the user based on the user mood feature, and generate an announcement video with an animated character image according to the character mood feature and the answer information; The display device is configured to receive and play the announcement video. Human-computer interaction systems.

9. An electronic device comprising one or more processors and a storage device storing one or more programs, An electronic device which, when the one or more programs are executed by the one or more processors, causes the one or more processors to execute the human-computer interaction method according to any one of claims 1 to 4.

10. A computer-readable medium on which a computer program is stored, A computer readable medium which, when executed by a processor, performs the human-computer interaction method according to any one of claims 1 to 4.

11. A computer program which, when executed by a processor, performs the human-computer interaction method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Information publication device

    JP1997081632A

  • Conversation control apparatus and conversation control method

    JP2005157494A