System and method for intelligent initiation of human-computer dialogue based on multimodal sensor input

Abstract

The present teachings relate to a method, system, medium and implementation for enabling communication with a user. Information characterizing the surroundings of a user who is about to participate in a new conversation is received via a communication platform, wherein the information is obtained from a scene in which the user is located and captures characteristics of the user and the scene. Relevant features are extracted from the information. The state of the user is inferred based on the relevant features, and the conversation context of the scene is determined based on the relevant features. A topic for a new conversation is determined based on the user, and feedback is generated to initiate a new conversation with the user based on the topic, the user state, and the conversation context.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 62 / 612,163, filed on December 29, 2017, the contents of which are incorporated herein by reference in their entirety.

[0003] This application is a subsidiary of U.S. Patent Application _________ (Attorney Docket No. 047437-0502424), filed on December 27, 2018, International Application _________ (Attorney Docket No. 047437-0461769), filed on December 27, 2018, U.S. Patent Application _________ (Attorney Docket No. 047437-0502426), filed on December 27, 2018, International Application _________ (Attorney Docket No. 047437-0461770), filed on December 27, 2018, U.S. Patent Application _________ (Attorney Docket No. 047437-0461771), filed on December 27, 2018 No. 047437-0502427), International Application _________ filed on December 27, 2018 (Attorney Docket No. 047437-0461772), U.S. Patent Application _________ filed on December 27, 2018 (Attorney Docket No. 047437-0502428), U.S. Patent Application _________ filed on December 27, 2018 (Attorney Docket No. 047437-0502429), International Application _________ filed on December 27, 2018 (Attorney Docket No. 047437-0461774), U.S. Patent Application ___ filed on December 27, 2018 ______(Attorney Docket No. 047437-0502430), International Application _________ filed on December 27, 2018 (Attorney Docket No. 047437-0461776), U.S. Patent Application _________ filed on December 27, 2018 (Attorney Docket No. 047437-0502431), International Application _________ filed on December 27, 2018 (Attorney Docket No. 047437-0461777), U.S. Patent Application _________ filed on December 27, 2018 (Attorney Docket No. 047437-0502432), International Application No. _________ filed on December 27, 2018 (Attorney Docket No. 047437-0461778), U.S. Patent Application No. _________ filed on December 27, 2018 (Attorney Docket No. 047437-0502547), International Application No. _________ filed on December 27, 2018 (Attorney Docket No. 047437-0461815), U.S. Patent Application No. _________ filed on December 27, 2018 (Attorney Docket No. 047437-0502549), International Application No. _________ filed on December 27, 2018 (Attorney Docket No. 047437-0461817),Related to U.S. Patent Application ________ (Attorney Docket No. 047437-0502551), filed on December 27, 2018, and International Application _________ (Attorney Docket No. 047437-0461818), filed on December 27, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present teachings generally relate to human-machine communication. Specifically, the present teachings relate to adaptive human-machine communication. Background Art

[0005] Computer-assisted conversational systems are becoming more common, thanks to the ubiquity of Internet connectivity, advances in artificial intelligence, and the boom in e-commerce. For example, more and more call centers are equipped with automated conversational robots to handle user calls. Hotels are beginning to install a variety of kiosks that can answer questions from travelers or guests. Online reservations (whether for travel accommodation or theater tickets) are also increasingly being made using chatbots. In recent years, automated human-computer communication in other areas has also become more common.

[0006] Such conventional computer-assisted dialogue systems are usually pre-programmed with specific questions and answers based on well-known conversation patterns in different fields. Unfortunately, human interlocutors may be unpredictable and sometimes do not follow the pre-planned conversation pattern. In addition, in some cases, the human interlocutor may go off topic during the process and continue the fixed conversation pattern, which may be annoying or lose interest. When this happens, such machine conventional dialogue systems often cannot continue to keep the human interlocutor engaged, so the human-machine dialogue is either interrupted and the task is handed over to the human operator, or the human interlocutor directly leaves the dialogue, which is undesirable.

[0007] In addition, traditional machine-based dialogue systems are often not designed to handle human emotional factors, let alone take into account how to handle such emotional factors when talking to humans. For example, traditional machine dialogue systems often do not initiate a conversation unless a person starts the system or asks certain questions. Even if a traditional dialogue system does not initiate a conversation, it has a fixed way to start a conversation, which does not vary from person to person or adjust based on observation. Therefore, although they are programmed to faithfully follow the pre-designed dialogue mode, they are usually unable to act and adapt to the dynamic development of the conversation so that the conversation can be conducted in a way that attracts people to participate. In many cases, when the person who is set to engage in the conversation is obviously upset or frustrated, the traditional machine dialogue system is unaware of it and continues the conversation in the same way that upsets that person. This not only makes the conversation end unpleasantly (the machine is still unaware of this), but also makes that person unwilling to have a conversation with any machine-based dialogue system in the future.

[0008] In some applications, it is critical to conduct a human-computer conversation thread based on what is observed from the person in order to determine how to proceed effectively. An example is an education-related conversation. When a chatbot is used to teach a child to read, it is necessary to monitor whether the child is perceptive in the way being taught and continue to process in order to proceed effectively. Another limitation of traditional dialogue systems is that they are unaware of context. For example, traditional dialogue systems do not have the ability to observe the context of the conversation and improvise dialogue strategies to attract user participation and improve user experience.

[0009] Therefore, there exists a need for methods and systems that address these limitations. Summary of the invention

[0010] The teachings disclosed herein relate to methods, systems and programming for human-computer interaction. Specifically, the teachings relate to adaptive human-computer interaction.

[0011] In one example, a method implemented on a machine for enabling communication with a user is disclosed, the machine having at least one processor, a memory, and a communication platform capable of connecting to a network. Information characterizing the surroundings of a user who is about to participate in a new conversation is received via the communication platform, wherein the information is obtained from a scene in which the user exists and captures characteristics of the user and the scene. Relevant features are extracted from the information. The state of the user is inferred based on the relevant features, and the conversation context around the scene is determined based on the relevant features. The topic of the new conversation is determined based on the user, and feedback is generated to initiate a new conversation with the user based on the topic, the user state, and the conversation context.

[0012] In different instances, a system for enabling communication with a user is provided. The system includes a multimodal data analysis unit, which is configured to receive information characterizing the surrounding situation of a user who is about to participate in a new conversation, wherein the information is obtained from a scene in which the user exists and captures the characteristics of the user and the scene, and extracts relevant features from the information. The system includes a user state estimator and a conversation context information determiner, the user state estimator is configured to estimate the user state based on relevant features, and the conversation context information determiner is configured to determine the conversation context of the scene surrounding based on relevant features. The system also includes a conversation controller, which is configured to determine the topic of the new conversation based on the user, and generate feedback based on the topic, the user state, and the conversation context to initiate a new conversation with the user.

[0013] Other concepts relate to software that implements this teaching. A software product according to this concept includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters associated with the executable program code, and / or information related to a user, request, content, or other additional information.

[0014] In one example, a machine-readable non-transitory medium is provided, on which is encoded information for enabling communication with a user, wherein the information, once read by a machine, causes the machine to perform a series of steps. Information characterizing the surroundings of a user who is about to participate in a new conversation is received via a communication platform, wherein the information is obtained from a scene in which the user exists and captures characteristics of the user and the scene. Relevant features are extracted from the information. The state of the user is inferred based on the relevant features, and the conversation context around the scene is determined based on the relevant features. A topic of the new conversation is determined based on the user, and feedback is generated based on the topic, the user state, and the conversation context to initiate a new conversation with the user.

[0015] Other advantages and novel features will be given in part in the following description, and in part will be apparent to those skilled in the art upon examination of the following description and drawings or learned by making or operating the examples. The advantages of the present teachings can be realized and obtained by practicing and applying the various embodiments of the methods, apparatus, and combinations given in the detailed examples discussed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The methods, systems and / or programming described herein are further described in the form of exemplary embodiments. These exemplary embodiments are described in detail with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, wherein similar reference numerals represent similar structures throughout the several drawings, wherein:

[0017] Figure 1 shows the connections between a user device, an agent device, and a user interaction system during a conversation according to an embodiment of the present teachings;

[0018] Figure 2 An exemplary high-level system diagram for an automated conversation companion having multi-layer processing capabilities according to an embodiment of the present teachings is shown;

[0019] Figure 3 The present invention shows a dialogue process of observing background information and using it to infer the context to design an adaptive dialogue strategy according to an embodiment of the present teaching;

[0020] Figure 4 shows an exemplary multi-layer processing and communication between different processing layers of an automated conversation companion according to an embodiment of the present teachings;

[0021] Figure 5 An exemplary high-level system diagram of a human-computer dialogue framework according to an embodiment of the present teaching is shown;

[0022] Figure 6 is a flow chart of an exemplary process of a human-computer dialogue framework according to an embodiment of the present teaching;

[0023] Fig. 7A shows exemplary types of multimodal data that can be acquired during a conversation according to an embodiment of the present teachings;

[0024] Figure 7B shows exemplary types of user status information according to an embodiment of the present teaching;

[0025] Figure 7C An exemplary type of conversation environment that can be observed and used as background information according to an embodiment of the present teachings is shown;

[0026] Figure 8 An exemplary high-level system diagram of a multimodal data analyzer according to an embodiment of the present teachings is shown;

[0027] Fig. 9 is a flow chart of an exemplary process of a multimodal data analyzer according to an embodiment of the present teachings;

[0028] Fig.10 An exemplary high-level system diagram of a dialog controller according to an embodiment of the present teachings is shown;

[0029] Fig.11 is a flow chart of an exemplary process of a dialog controller according to an embodiment of the present teaching;

[0030] Fig.12 shows the architecture of a mobile device that can be used to implement a particular system of the present teachings;

[0031] Fig.13 The architecture of a computer that can be used to implement a particular system of the present teachings. DETAILED DESCRIPTION

[0032] In the following detailed description, various specific details are given by way of example to provide a detailed understanding of the relevant teachings. However, it should be clear to those skilled in the art that the present teachings can be implemented without these details. In other examples, well-known methods, processes, components and / or circuits are introduced at a relatively high level without details to avoid unnecessarily obscuring the implementation of the present teachings.

[0033] This teaching generally relates to systems, methods, media and other implementations of the embodiments shown in this teaching, and related concepts are given in the human-computer dialogue that can configure this teaching. Specifically, this teaching relates to the concept of adaptively initiating a human-computer dialogue based on observed surrounding conditions (including the status of the person who will participate in the dialogue, the setting of the environment, etc.). In addition, during the dialogue process, the settings and people of the dialogue are continuously observed and analyzed, and used to adaptively conduct the dialogue accordingly in order to enhance the user experience and improve user engagement. Although certain exemplary features and disclosures are given to describe the concepts associated with this teaching, it should be understood that this teaching can be applied to any setting different from that given here without limitation.

[0034] Figure 1 A general framework of a human-computer dialogue with information flows between different participants according to an embodiment of the present teaching is shown. As shown, the framework includes a user device 110, an agent device 160, and a user interaction system 170 connected together via a network connection during the dialogue process. The user device may have multiple sensors of more than one modality, such as a sensor 140 for acquiring visual data (images or videos), a sensor 130 for obtaining audio data containing the user's speech or sounds from the dialogue environment, a sensor for acquiring text information (what the user wrote on the display), or a tactile sensor (not shown) for capturing the user's touch, movement, etc. Such sensor data can be captured during the dialogue process and used to promote understanding of the user (expression, emotion, intention) and the user's surroundings. Such sensor data provides background information and can be explored so that the response is customized accordingly.

[0035] In a variant embodiment, a human interlocutor may communicate directly with the user interaction system 170 without the agent device 160. When the agent device is present, it may correspond to a robot that can be controlled by the user interaction system 170. In some embodiments, the user device 110 may communicate directly with the user interaction system 170 without the agent device. In such an embodiment, the functions that would be performed by the agent device (such as speaking a word or sentence, sending a text message, or expressing a specific emotion, such as frowning, smiling, or sadness) may be performed by the user interaction system 170. Therefore, although the following disclosure may discuss specific concepts about an agent device or a user interaction system, the concepts disclosed herein may be applied to a combination of the two or any one of them. In addition, in the embodiments described herein, reference to the machine side may mean an agent device or a user interaction system, or a combination thereof.

[0036] As shown, Figure 1The connection between any two participants in the communication can be bidirectional. The proxy device 160 can interact with the user via the user device 110 to achieve a dialogue in a bidirectional manner. In operation, the proxy device 160 can be controlled by the user interaction engine 170 to, for example, speak a response to the user operating the user device 110. According to the present teaching, input from the user side (including, for example, user speech or action, user appearance, and information about the user's surroundings) is provided to the proxy device 160 or the user interaction system 170 via a network connection.

[0037] In some embodiments, the proxy device 160 or the user interaction system 170 may be configured to process such inputs and use the relevant information identified from such inputs to dynamically and adaptively generate responses to the user. Such responses include, for example, utterances that transmit language responses, or changes to be made in the proxy device to present different scenes that will be displayed to the user. For example, the proxy device or the user interaction system 170 may observe from the input from the user device that the user is wearing a yellow T-shirt and looks tired. An adaptive response from the proxy device or the user interaction system may be generated that comments on how the user looks. Such a response can redirect the user's attention to the conversation. As another example, assume that the response from the machine (or the proxy device, or the user interaction system) is to present an object, such as a tree, on the user device. Based on the sensor input from the user device, it can be determined that the user's surrounding conditions indicate that it is winter now. Knowing this, the machine can adaptively instruct the user device to present a tree without leaves (consistent with the user's surrounding conditions).

[0038] As another example, if the machine response is to present a duck on the user device, for example, information about the user may be retrieved from the user information database 130 regarding the user's color preferences, and the presentation instructions may be provided to the user device by customizing the duck with the user's preferred color (e.g., yellow or brown). Such customization is performed adaptively with respect to the user and the current conversation environment based on immediate observation of the user or his surroundings or the user's known preferences.

[0039] In this way, the machine side (agent device or user interaction system or a combination thereof) can make decisions on how to respond to the user based on what is observed during the conversation. Based on input from the user device, the machine side can determine the state of the conversation and the user's expression / emotion / mentality, and generate a response based on the specific situation of the conversation and the intended intent of the conversation (e.g., for teaching children English vocabulary) given the observed situation. In some cases, if the information received from the user device indicates that the user looks bored and impatient, the machine side can change the state of the conversation related to one topic (e.g., geometry in a conversation related to math education) to a different topic that the user may be interested in, such as basketball. Such a topic transition can be determined based on observing that the user is staring at the basketball in the room. The topic transition thus continues to keep the user engaged in the conversation.

[0040] In some embodiments, the user device may be configured to process raw sensor data acquired from different modalities and send the processed information (e.g., relevant features of the raw input) to the proxy device or user interaction engine for further processing. This will reduce the amount of data transmitted over the network and increase communication efficiency.

[0041] As shown, during a conversation between a user and a machine, the user device 110 may continuously collect multimodal sensor data about the user and his or her surroundings, which may be analyzed to detect any information related to the conversation and used to intelligently control the conversation in an adaptive manner. This may enhance the user experience and / or engagement. The sensor data provides context around the conversation and enables the machine to understand the context so that the conversation can be managed more effectively.

[0042] Figure 2 An exemplary high-level system diagram for an automated conversation companion with multi-layer processing capabilities according to an embodiment of the present teachings is shown. In the embodiment shown here, the overall system may include components / functional modules residing in the user device 110, the agent device 160, and the user interaction engine 170. The overall system described herein includes multiple processing layers and hierarchical structures, which together perform human-computer interaction in an intelligent manner. In the embodiment shown, there are 5 layers, including layer 1 for front-end applications and front-end multimodal data processing, layer 2 for characterizing the conversation setting, layer 3 where the conversation management module resides, layer 4 for the inferred mentality of different participants (people, agents, devices, etc.), and layer 5 for so-called utilities. Different layers may correspond to different levels of processing, from raw data collection and processing at layer 1 to processing of changing the utility of the conversation participants at layer 5.

[0043] The term "utility" is thus defined as the preferences of participants based on detected state identification, which are associated with the conversation history. Utilities can be associated with participants in a conversation, whether the participant is a person, an automatic companion or other intelligent device. Utilities for specific participants can characterize different states of the world, whether physical, virtual or even mental. For example, a state can be characterized as a specific path for a conversation to travel along a complex map of the world. In different instances, the current state evolves to the next state based on the interaction between multiple participants. The state can also be participant-related, that is, when different participants participate in the interaction, the state brought about by such an interaction may change. The utilities associated with the participants can be organized into a hierarchy of preferences, and such a preference hierarchy can evolve over time based on the participant choices made during the conversation and the preferences expressed. Such preferences (which can be characterized as an ordered sequence of choices made from different options) are called utilities. This teaching discloses such a method and system: through this method and system, an intelligent automatic companion can learn the user's utility through a conversation with a human conversationalist.

[0044] exist Figure 2 In the complete system shown, the front-end application in layer 1 and the front-end multimodal data processing may reside in the user device 110 and / or the proxy device 160. For example, the camera, microphone, keyboard, display, renderer, speaker, chat bubble, user interface elements may be components or functional modules of the user device. For example, there may be an application or client running on the user device, which may include Figure 2 The functions before the external application program interface (API) shown. In some embodiments, the functions beyond the external API can be considered as a back-end system, or reside in the user interaction engine 170. The application running on the user device can obtain multimodal data (audio, image, video, text) from sensors or circuits of the user device. The multimodal data is processed to generate text or other types of signals (such as objects such as detected user faces, speech understanding results, etc.) that characterize the characteristics of the original multimodal data and send them to layer 2 of the system.

[0045] In layer 1, multimodal data may be acquired via sensors such as cameras, microphones, keyboards, displays, speakers, chat bubbles, presenters, or other user interface elements. Such multimodal data may be analyzed to infer or deduce a variety of features that can be used to infer higher-level characteristics (e.g., expressions, characters, gestures, emotions, actions, attention, intent, etc.). Such high-level characteristics may be obtained by processing units on layer 2, and then processed via Figure 2The internal API shown is used by higher-level components to, for example, intelligently infer or infer additional information related to the conversation at a higher conceptual level. For example, the inferred emotions, attention, or other characteristics of the conversation participants obtained at layer 2 can be used to infer the mentality of the participants. In some embodiments, this mentality can also be inferred at layer 4 based on additional information, such as the recorded surrounding environment or other additional information in this surrounding environment, such as sound.

[0046] The inferred mindset of the participants, whether with a human or an automated companion (machine), can be relied upon by the dialogue management of layer 3 to determine, for example, how to conduct a conversation with a human interlocutor. How each conversation progresses often characterizes the preferences of the human user. Such preferences can be captured dynamically during the conversation at the utility layer (i.e., layer 5). Figure 2 As shown, the utilities on layer 5 represent the evolving states, which represent the evolving preferences of the participants, and they can also be used by the dialogue management on layer 3 to decide appropriate or intelligent ways to interact.

[0047] Information sharing between different layers can be achieved through API. Figure 2 In some embodiments shown in , information sharing between layer 1 and other layers is via an external API, while information sharing between layers 2-5 is via an internal API. It can be understood that this is merely a design choice, and other implementations can also implement the teachings given here. In some embodiments, through the internal API, different layers (2-5) can access information generated or stored on other layers to support processing. Such information may include general configurations applied to the dialogue (e.g., the role of the agent device is an avatar, a preferred voice or a virtual environment to be generated for the dialogue, etc.), the current state of the dialogue, the current dialogue history, known user preferences, inferred user intent / emotion / mentality, etc. In some embodiments, certain information that can be shared via an external API can be accessed from an external database. For example, a specific configuration related to a desired role of the agent device (e.g., a duck) can be accessed from, for example, an open source database, which provides parameters (e.g., parameters for visually presenting a duck, and / or parameters required for presenting speech from a duck).

[0048] Figure 3 The dialog process according to an embodiment of the present teaching is shown, wherein background information is observed and used to infer the context in order to design an adaptive dialog strategy. Figure 3It can be seen that operations on different layers can be performed, which together promote intelligent dialogue in a collaborative manner. In the example shown, when the user is observed to be in a normal state, the 160 agent device may first ask the user at 302 "How are you today?" to initiate a dialogue. In other embodiments disclosed herein, if the user looks out of shape, the machine (agent device 160 or user interaction system 170) may initiate a dialogue using different sentences, such as "Are you okay?" (not shown). In response to the speech at 302, the user may respond with the speech "OK" at 304. In order to manage the dialogue, the automatic dialogue machine may actuate different sensors to capture dynamic scenarios during the dialogue, thereby making it possible to observe the user and the surrounding situation. For example, the automatic dialogue machine may actuate multimodal sensors to obtain multimodal data. Such multimodal data may include audio, visual or text data. For example, visual data may capture the user's facial expressions. Visual data may also reveal background information around the conversation scene. The image of the scene may reveal the presence of a basketball, a table, and a chair, which provides information about the environment and can be used in dialogue management to enhance the user's participation. The audio data may capture not only the user's voice response, but also other surrounding information, such as the pitch of the response, the way the user speaks the response, or the user's accent.

[0049] Based on the acquired multimodal data, analysis may be performed by the automatic conversation machine (e.g., through a front-end user device or through a back-end user interaction engine 140) to assess the user's attitude, emotions, mentality, and utility. For example, based on the visual data analysis, the automatic conversation machine may detect that the user is showing sadness, no smile, the user's tone is low, and the speech is slow. Characterizing the user's state in the conversation may be performed on layer 2 based on the multimodal data acquired on layer 1. Based on such detected observations, the automatic companion may infer (at 306) that the user is not so interested in the current topic and is not highly engaged. For example, such an inference of the user's emotions or mental state may be performed on layer 4 based on the characteristics of the multimodal data associated with the user.

[0050] In order to respond to the user's current state (which is characterized by low participation), the automatic dialogue machine may determine to make the user active, thereby making the user better involved. In the example shown, the automatic dialogue machine can take advantage of what is available in the dialogue environment by asking the user a question at 308, "Do you want to play a game?" For example, using a customized voice personalized for the user, by converting text to speech, this question can be sent as a voice in an audio manner. In this case, the user can respond by saying "OK." at 310. Based on the continuously acquired multimodal data related to the user, for example, via the processing at layer 2, it can be observed that in response to the invitation to play the game, the user's eyes show a look around, in particular, the user's eyes may stare at the place where the basketball is. At the same time, the automatic companion may also observe that once the suggestion to play the game is heard, the user's facial expression changes from "sad" to "smile". Based on the user characteristics observed in this way, the automatic companion can infer at 312 that the user is interested in basketball.

[0051] Based on the new information acquired and the inferences based thereon, the automated companion may decide to utilize the basketball available in the environment to make the conversation more engaging for the user while still achieving the educational goal for the user. In this case, the conversation management at layer 3 may adapt the conversation to talk about the game and utilize the observation that the user is staring at the basketball in the room to make the conversation more interesting for the user while still achieving the goal of, for example, educating the user. In an exemplary embodiment, the automated companion generates a response suggesting that the user play a spelling game (at 314) and asking the user to spell the word "basketball."

[0052] Based on observations of the user and the environment, given the automated companion's adaptive conversation strategy, the user may respond by providing the spelling of the word "basketball" (at 316). Observations may be made continuously on how enthusiastic the user is in answering the spelling question. If the user appears to be responding quickly with a more cheerful attitude, the automated companion may infer at 318 that the user is now more engaged, based on, for example, multimodal data acquired when the user answers the spelling question. To further encourage the user to actively participate in the conversation, the automated companion may then generate a positive response "Good job!" and instruct that this response be delivered to the user in a cheerful, encouraging, positive voice.

[0053] Figure 4 An exemplary multi-layer process and communication between multiple processing layers of an automated dialog system according to an embodiment of the present teachings is shown. Figure 4The dialogue manager 410 in corresponds to the functional component of the dialogue management in layer 3. The dialogue manager is an important part of the automatic companion and manages the dialogue. Traditionally, the dialogue manager uses the user's words as input and determines how to respond to the user. This is done without considering the user's preferences, the user's mentality / emotions / intentions, or the surrounding environment of the dialogue, that is, without giving any weight to the different available states of the relevant world. The lack of understanding of the surrounding world often limits the engagement or perceived authenticity of the conversation between the human user and the intelligent agent.

[0054] In certain embodiments of the present teachings, the utility of the conversation participants related to the ongoing conversation is fully utilized to allow for a more personalized, flexible and engaging conversation. This promotes the agent to play different roles so as to be more effective in different tasks, such as arranging appointments, booking travel, ordering equipment and supplies, and researching multiple topics online. When the intelligent agent recognizes the user's dynamic mindset, emotions, intentions and / or utilities, this enables the agent to engage the human conversationalist in the conversation in a more targeted and effective manner. For example, when an educational agent teaches a child, the child's preferences (such as his favorite color), observed emotions (such as, sometimes the child does not want to continue the course), and intentions (such as, the child reaches for the ball on the floor instead of focusing on the course) can allow the educational agent to flexibly adjust the subject of concern to the toy, and may adjust the way to continue the conversation with the child in order to give the child a break, thereby achieving the overall goal of educating the child.

[0055] As another example, the present teachings can be used to enhance the services of a user service agent by asking more appropriate questions given what is observed from the user in real time, and thus achieve an improved user experience. By developing methods and means to learn and adapt to the preferences or mindsets of participants in a conversation, which is rooted in the essential aspects of the present teachings as disclosed herein, the conversation can be conducted in a more engaging manner.

[0056] The Dialogue Manager (DM) 410 is the core component of the automatic companion. Figure 4 As shown, DM 410 (layer 3) takes inputs from different layers, including inputs from layer 2 and inputs from higher abstraction layers, such as inferred mindset from layer 4 and utility / preference from layer 5. As shown, on layer 1, multimodal information is acquired from sensors of different modalities, which are processed to obtain features that characterize the data, for example. This may include signal processing of visual, audio, and textual modalities.

[0057] Features obtained from processing multimodal data can be further processed on layer 2 to achieve language understanding and / or multimodal data understanding, including vision, text, and any combination thereof. Some of this understanding may be for a single modality, such as speech understanding, and some may be for understanding the surroundings of users participating in the conversation based on integrated information. Such understanding can be physical (e.g., identifying specific objects in the scene), cognitive (e.g., recognizing what the user said, or a loud sound, etc.) or mental (e.g., specific emotions, such as the user's stress inferred based on the tone of voice, facial expressions, or user gestures).

[0058] The modality-data understanding generated on layer 2 can be used by DM410 to determine how to respond. In order to enhance participation and user experience, DM410 can also determine the response based on the inferred user mentality from layer 4 and the utility of the users participating in the conversation from layer 5. The output of DM410 corresponds to the response to the user determined accordingly. In order to transmit the response to the user, DM410 can also plan the way to transmit the response. The form of transmitting the response can be determined based on information from multiple sources, such as the user's emotions (for example, if the user is an unhappy child, the response can be presented in a gentle voice), the user's utility (for example, the user may prefer a voice similar to a specific accent of his parents), or the surrounding situation in which the user is (for example, a noisy space, so that the response needs to be given at a high volume). DM410 can output the determined response together with such transmission parameters.

[0059] In some embodiments, the transmission of the responses so determined is achieved by generating a transmittable form of each response based on a variety of parameters associated with the responses. In general, the responses are transmitted in the form of speech in some natural language. The responses can also be transmitted in the form of speech coupled with a specific non-verbal expression: the expression is part of the transmitted response, such as a nod, a shake of the head, a wink, or a shrug. There may be other forms of transmittable responses that are audio but not language, such as a whistle.

[0060] To deliver a response, a deliverable form of the response may be generated via, for example, language response generation and / or behavioral response generation, such as Figure 4As shown. Such a response for its determined transmittable form can then be used by the renderer to actually present the response in the expected form. For the transmittable form of natural language, the text of the response can be used to synthesize a speech signal via, for example, text-to-speech technology, according to transmission parameters (e.g., volume, accent, style, etc.). For any response or part thereof (e.g., a specific expression) to be transmitted in a non-verbal form, the expected non-verbal expression can be translated into a control signal via, for example, animation, and the control signal can be used to control a specific part of the agent device (the tangible presentation of the automatic companion) to perform a specific mechanical movement, thereby transmitting the non-verbal expression of the response, such as nodding, shrugging or whistling. In some embodiments, in order to transmit the response, a specific software component can be called to present different facial expressions of the agent device. Such a rendition of the response can also be performed by the agent at the same time (e.g., speaking the response in a joking voice, and the agent's face is hung with a big smile).

[0061] Figure 5 An exemplary high-level system diagram of a dialog manager 410 for personalized dialog management according to an embodiment of the present teaching is shown. In the illustrated embodiment, the dialog manager 410 includes a multimodal data analysis unit 510, a user state estimator 520, a dialog context information determiner 530, a dialog controller 540, a feedback indication generator 550, and a dialog response output unit 560. The dialog manager 410 shown here controls how to intelligently initiate a dialog with a user based on the observed content, and how to respond to the user during the dialog based on the observed content.

[0062] The multimodal data to be analyzed by the multimodal data analysis unit 510 may include: auditory signals, which may correspond to ambient sounds or speech; visual signals, which may include video and picture images; textual information; or information from a tactile sensor, which depicts, for example, the movement of a finger, hand, head, or other part of the user's body. Fig. 7A Shown in.

[0063] Based on the processed multimodal data from the multimodal data analysis unit 510, the user state estimator 520 and the conversation context information determiner 530 estimate the state of the user and the context information around the conversation scene, respectively. Figure 7BExemplary types of user status information according to an embodiment of the present teaching are shown. The user status may include the user's tangible appearance, including skin color, hair color / hair style, skin tone, eye color, clothing (color, style), etc. The user status may also involve the user's expression, such as smiling, frowning, blinking, confusion, the loudness of the voice, etc. The user status may also be related to what the user says and how he speaks. Based on the audio signal, speech recognition can be performed based on the audio signal to determine the text of the user's speech. The pitch and tone of the voice can also be recognized, which will be used together with the visual clues from the visual data to infer the user's expression or emotion. The user's expression can be used to infer his / her emotions, such as happiness, sadness, distress, boredom, ... and so on. In some cases, expression / emotion can also be used to infer intent. For example, if the user's expression indicates that he / she is bored but his / her eyes are fixed on a nearby ball, it can be inferred that the user is interested in talking about topics related to the ball related to the intention (such as a basketball game).

[0064] In order to facilitate the user state estimator 520, a suitable model may be provided at 515. User state estimation may be performed based on different models. In the illustrated embodiment, the user state detection model may include models for detecting features of different aspects related to the user, including but not limited to appearance (the color of the clothes the user wears, skin color / tone, gender, hair color, eye color, etc.), facial expressions (smiling, frowning, crying, sad... etc.), emotions (sad, happy, excited, angry,... etc.), moods (angry, indifferent, interested, absent-minded, etc.). Based on these models, different information or features may be extracted and used to make judgments about the user state.

[0065] In order to infer the background information around the dialogue scene, based on the environment detection model 525, the dialogue background information determiner 530 detects various objects existing in the scene captured by the multimodal data and extracts relevant features thereof. In order to facilitate the determination of the background information in the dialogue environment in which the user is located, the environment detection model 525 may be appropriately provided to detect different types of scenes, different types of objects, different characteristics of the environment, and the like. Figure 7CAn exemplary type of background information and / or environment detection model according to an embodiment of the present teaching is shown. As can be seen from the figure, the conversation environment may imply a hierarchical structure of concepts at different conceptual levels, from lower-level concepts (such as objects present in the conversation environment (computers, desks, dining tables, balls, chairs, cabinets, trees, grass, sky, lakes, children, swings, slides, mountains, etc.), places (offices, parks, beaches, wilderness, shops, playgrounds, etc.)) to the nature of the place (such as vacation places, workplaces, transportation places, etc.). Using such a model, the conversation context information determiner 530 can detect not only what is present in the scene around the user, but also the characteristics of the place and the nature of the place for the user.

[0066] Back to Figure 5 , the inferred user state (from the user state inferer 520) and the determined context information of the underlying conversation (from the conversation context information determiner 530) can then be used by the conversation controller 540 to adaptively initiate a conversation with the user or determine how to respond to the last response from the user. The determination of feedback (what to say to the user to initiate a conversation, or a response to the last utterance from the user) is based on the project 535 currently applied to the conversation (e.g., a math education thread), the conversation tree 545 (which provides the general flow of the conversation), and user information (e.g., preferences from the user profile 555). The feedback generated in this way is then processed on the feedback indication generator 550 to generate an indication for feedback (e.g., a presentation indication to be sent to the user device in order to present the machine's response), which is then sent to the user device via the conversation response output unit 560.

[0067] Figure 6 Flowchart of an exemplary process of the dialog manager 410 according to an embodiment of the present teaching. When the multimodal data analysis unit 510 receives multimodal sensor data from the user device at 610, it analyzes the multimodal sensor data at 620. Based on the analyzed multimodal sensor data, the user state estimator 520 infers the user state based on the user state detection model 515 at 630. As discussed herein, the user's state includes characteristics of the user at different conceptual levels, some at the appearance level, some at the emotion level, and so on.

[0068] Based on the analyzed multimodal sensor information, the conversation context information determiner 530 determines the context information of the conversation at 640. In order to determine feedback to the user (either the initial speech to the user or the response to the user's last speech), the conversation controller 540 determines the items / topics related to the current conversation at 650. From the items so determined, the conversation controller 540 can access the appropriate conversation tree corresponding to the item and identify the node in the conversation tree corresponding to the current conversation. Based on the conversation tree, user status and context information, the conversation controller 540 determines the progress of the conversation at 660. There are two possible conversation scenarios. One is when the conversation thread is at the beginning, for example, the user has just appeared, and the conversation controller 540 needs to initiate a conversation based on the observed content. The other scenario is that the conversation is ongoing, so the conversation controller 540 will determine how to respond to the user.

[0069] For a new session in which the dialogue controller 540 is about to initiate a session, if the user is newly present, the dialogue controller 540 is about to initiate a session. In this case, the identified dialogue tree is at its initial node, and the dialogue controller 540 will determine whether the first node of the dialogue tree is appropriate given the presumed user state. For example, if the first utterance of the machine according to the dialogue tree is "Are you ready for today's project?" but the presumed user state may indicate that the user is actually currently upset, the dialogue controller 540 may determine not to take the indication in the dialogue tree, but to generate different initiation sentences based on, for example, user preferences from the user profile 555 and / or detected in the dialogue scene (e.g., user appearance or certain objects in the user's environment). For example, it may be known that the user loves to play basketball (by the user profile 555), and the presence of a ball is observed in the scene (background information from the dialogue background information determiner 530), and the dialogue controller 540 may determine to initiate a dialogue by saying "Do you want to play ball for a while?" to attract the user's participation.

[0070] If the conversation is an ongoing thread, the conversation controller 540 may determine a response based on, for example, whether the user state is appropriate for following the conversation tree of the current conversation. If the presumed user state indicates that it is not appropriate to follow the conversation tree, for example, the last utterance from the user does not correspond to any predicted answer in the conversation tree and the user looks at a loss, the conversation controller 540 may determine not to follow the conversation tree, but to identify some alternative response. Such an alternative response can be adaptively determined based on the user state and background information. For example, the user state may show that the user is wearing a swimsuit, and the background information shows that the user is near a swimming pool. In order to attract users to participate in the English education conversation thread, the conversation controller 540 may determine that the user is attracted to the project by responding "Can you tell me how to spell "swimming" and "pond"? ". By adapting the conversation according to the user state (lost emotions, the appearance of wearing a swimsuit) and the context of the conversation (there is a swimming pool in the environment), thereby identifying an alternative response to the user, the user's attention can be redirected, and the participation and user experience can be improved.

[0071] Based on the feedback determined by the dialogue controller 540 (how to initiate a dialogue, or how to respond to a user in an ongoing dialogue), the feedback indication generator 550 generates an indication for the user device at 670 that can be used to present feedback to the user, and sends the indication to the dialogue response output unit 560, which then sends the indication to the user device at 680.

[0072] Figure 8 An exemplary high-level system diagram of a multimodal data analysis unit 510 according to an embodiment of the present teaching is shown. As discussed herein, the multimodal data analysis unit 510 is used to analyze multimodal sensor data, extract relevant information, and identify useful clues that can be used to adapt the dialogue. Such clues include user-related features and environment-related features. Based on such multimodal features (visual, auditory, textual or tactile), the multimodal data analysis unit 510 can also infer higher-level concepts such as expressions, emotions, intentions, etc., for example based on models. In the illustrated embodiment, the multimodal data analysis unit 510 includes a raw data processing portion, a feature extraction portion, and an inference portion.

[0073] The raw data processing section includes an audio information processor 800, a visual information processor 805, and a text information processor 810. Additional raw sensor data processing units may also be included, such as a processor for processing tactile information (not shown). The processor for processing the raw data may be a processor for low-level signal processing, such as for removing noise, enhancing image quality, reducing subsampling of redundant data, and the like.

[0074] The feature extraction part can use the output from the original processing unit as input to extract features (face, eyes, skin, sky, tree, words from speech or text, speech-related metrics (such as speed, clarity, loudness, pitch, etc.) related to certain characteristics of interest (such as expressions such as smiles, appearances such as eye color, skin tone / color, surrounding objects such as tables, tree colors, swimming pools, etc.). This part may include an auditory sound analyzer 815 (for analyzing environmental sounds), a speech recognition unit 820 (for recognizing the user's and / or other people's words), a user facial feature detector 825 (for detecting faces, facial features and facial expressions), a user appearance analyzer 830 (such as clothing, skin color, etc.) and a surrounding situation feature detector 835 (for detecting, for example, different types of objects appearing in a conversation environment, etc.). These components detect features of the corresponding modalities, and the features detected in this way are sent to the inference part to infer higher-level concepts such as emotions, intentions, etc.

[0075] The inference portion in this illustrated embodiment includes: an emotion estimator 840, which can infer the user's emotions based on the visual and auditory features from components 815, 820, 825, and 830; and an intention estimator 850, which can infer the intention based not only on the visual and auditory features from these four components, but also on the inferred emotions from the emotion estimator 840. For example, the user facial feature detector 825 may have detected that the user is looking in the direction of a ball, and the emotion estimator 840 can infer that the user is confused about the subject of the conversation. Based on these visual clues, the intention estimator 850 can infer that the user's attention is not on the conversation and wants (intends) to play with the ball. Such inferred emotions and intentions can be used by the conversation controller 540 to determine how to continue the conversation and redirect the user's attention, as discussed herein. In some embodiments, higher-level inferences, such as inferring emotions or intentions, can be implemented in the user state estimator 520.

[0076] The multimodal data analysis unit 510 may also include a relevant clue identifier 860 that takes input from all components of the feature extraction part and the inference part (i.e., 815, 820, 825, 830, 835, 840, and 850) and identifies clues related to the conversation based on, for example, the conversation model / tree 837. For example, if the conversation is about travel plans, certain features related to the user (e.g., eye color) or present in the conversation scene (e.g., a toy duck on the table) may not be relevant to the conversation management. In this case, the relevant clue identifier 860 may filter out these features and not send them to the user state estimator 520 or the conversation context information determiner 530 (see Figure 5 ).

[0077] Fig. 9 9 is a flow chart of an exemplary process of the multimodal data analysis unit 510 according to an embodiment of the present teachings. Upon receiving the multimodal data at 910, components in the multimodal data analysis unit 510 process different information to extract and identify relevant information respectively. For example, by Figure 8 It can be seen that in order to identify useful clues in the audio domain, the audio information processor 800, the auditory sound analyzer 815, and the speech recognition unit 820 process the audio signal at 940 to identify environmental sounds and / or the user's voice. In order to identify useful clues in the visual domain, the visual information processor 805, the user facial feature detector 825, and the user appearance analyzer 830 process the visual data at 930 to identify the user's visual features (e.g., eyes, nose, skin color, etc.), facial expressions, and user appearance (e.g., hair, clothing, etc.). In order to identify useful clues in the user's surrounding environment, the surrounding feature detector 835 processes data from multiple domains at 920 to extract relevant features.

[0078] Based on the detected audio and visual features, the emotion estimator 840 infers the user's emotion at 950. Based on the audio / visual features and / or the inferred user emotions, the intent estimator 850 infers the user's intent at 960. The relevant clue identifier 860 then filters the features and inferences to identify relevant visual / audio / environmental clues at 970 and sends such relevant clues to different components in the dialog manager 410 at 980 (see Figure 5 ).

[0079] Fig.10 An exemplary high-level system diagram of a dialogue controller 540 according to an embodiment of the present teaching is shown. In the embodiment shown here, the dialogue controller 540 includes a user information processor 1000, a background information processor 1010, a dialogue progress determiner 1040, a dialogue initiation determiner 1030, a dialogue response determiner 1020, and a dialogue feedback generator 1050. As discussed herein, the dialogue controller 540 determines whether to initiate a dialogue or generate a response to the user in an ongoing dialogue thread. In each case, the dialogue controller 540 will generate feedback, so that the feedback is an initial greeting sent to the user in the case of starting a new dialogue, or a generated response to the user in an ongoing dialogue. In each case, the feedback is adaptively and intelligently generated based on the user state and the background information from the scene.

[0080] Fig.11 FIG. 1 is a flow chart of an exemplary process of a dialog controller according to an embodiment of the present teaching. In operation, the user information processor 1000 receives user status information (from Figure 5 ) and user utilities / preferences from layer 5 (see Figure 2 and 4 ). Based on the received information, the user information processor 1000 analyzes such information at 1110 to identify different types of relevant information, which may include, for example, the user's current mental state (bored), the user's utility / preference (likes to mix fun things with learning to be more effective), the user's appearance (wearing a yellow swimsuit and a red swim cap), the item scheduled to be learned is English, and / or the current topic of the item scheduled by the user is to spell simple English words (if it is an ongoing conversation), etc. Such relevant information about the user from different sources can be further used to determine a conversation initiation / response strategy.

[0081] In addition, the context information processor 1010 receives context information from the conversation context information determiner 530 at 1120 (see Figure 5 ), and analyzes the received context information at 1130 to identify relevant context information. For example, the context information processor 1010 may identify certain objects in the scene (e.g., tables, chairs, computers, ..., bookcases) based on the received context information, and conclude that the user is in an office. In other cases, if the received context information reports other types of objects, such as trees, blue sky, benches, swings, etc., the context information processor 1010 may infer that the user is in a park. Such conclusions based on context information may also be used to determine a dialog initiation / response strategy.

[0082] To proceed to determine how the conversation progresses, depending on whether it is a new conversation or an ongoing conversation, relevant information (e.g., the state of the conversation) is extracted by the conversation progress determiner 1040 from, for example, the user state information at 1140 and used to determine the mode of operation accordingly at 1050. If it is a new conversation, the conversation progress determiner 1040 calls the conversation initiation determiner 1030 to determine the initiation strategy. Otherwise, the conversation mode determiner 1040 calls the conversation response determiner 1020 to generate a response strategy.

[0083] When called, the conversation initiation determiner 1030 analyzes the processing results from the user information processor 1000 and the background information processor 1010 to understand the user and the user's surroundings. For example, the user may be a young boy with brown eyes, dark skin, curly hair, wearing a yellow T-shirt (by the user status information), and is known to be a slow person (i.e., shy person) when meeting and talking to strangers (by the user utility). The processing results from the background information processor 1010 may indicate that the user is currently in an office with a table, a chair, a computer on the table, and a box labeled Lego sets on the table.

[0084] Based on what is known about the current user (a slow starter), the conversation initiation determiner 1030 may decide, based on, for example, a machine learning model, that it is more appropriate to start a conversation by first making the user feel comfortable by talking about things that a young boy might like, before engaging the user in an intended conversation (e.g., a project to learn math that the user has logged into). Given contextual information about the user's surroundings, in order to appropriately initiate with respect to the current user, the conversation initiation determiner 1030 may generate a personalized conversation initiation strategy at 1160 that utilizes what is observed in the scene and determines how to approach the user at the beginning. For example, the conversation initiation determiner 1030 may utilize the Lego set observed on the table and decide to start a conversation by asking, "Do you like Lego?" Alternatively, the conversation initiation determiner 1030 may utilize the observed appearance of the user and decide to start a conversation by saying, "You look very happy today!" Starting a conversation in this manner is based on a personalized strategy about the user, which enables a better user experience and enhanced engagement. In this way, the conversation initiation strategy for each user may be different, but is intelligently adaptive for each user.

[0085] In terms of determining a response to a user in an ongoing conversation, under normal circumstances, the dialog response determiner 1020 can determine a response based on, for example, what is indicated in the relevant dialog tree, as a traditional dialog system would do. However, a conversation will not always go as planned, and in some cases, the machine agent in the conversation needs to be more adaptable and creative to continue to engage the user without unknowingly angering the user. When such a situation is detected (e.g., by understanding the situation based on multimodal sensor data and analysis thereof), the dialog response determiner 1020 can perform functions similar to the dialog initiation determiner 1030 by utilizing relevant information about the user's state and the context of the user's surrounding environment.

[0086] When determining a response to an ongoing conversation, additional operational parameters need to be considered compared to the judgment about how to initiate the conversation. For example, since it is an ongoing conversation, there is a conversation history, which may affect the response to be decided. In addition, the current user state associated with the current state of the corresponding dialogue tree (e.g., the user's emotions) may also affect the decision for the appropriate response, either along the same dialogue tree, or outside the dialogue tree. For example, if the user is in a conversation about geometry with a machine agent, and the user gives several answers incorrectly (as shown in the dialogue history). The machine agent may also have observed that the user seems frustrated. In this case (which can be observed from the dialogue history and the multimodal data obtained from the user), the dialogue response determiner 1020 may determine that the user needs to deviate a little from the current topic based on a specific machine learning model (which may indicate that when the user is frustrated due to an incorrect answer, it may be appropriate to deviate from the topic so as not to lose the user).

[0087] Regarding what the machine agent should say to implement the strategy of switching the user's attention to something else, the dialogue response determiner 1020 may utilize what is observed about the user in the scene or the scene context. In some cases, the dialogue response determiner 1020 may utilize the detected appearance of the user and design certain distracting responses. For example, if the user is observed wearing a T-shirt with the words "Red Sox" printed on it, the dialogue response determiner 1020 may decode to ask the user "Are you a fan of the Red Sox?" In some cases, the dialogue response determiner 1020 may utilize what is observed in the scene (background) to design a necessary and relevant response. For example, if the context information indicates that the user is currently in an office, and the office scene contains Lego on a table, the dialogue response determiner 1020 may decide to ask the user "Do you like Lego?" with the intention that the Lego in the scene will be used later to continue the discussion about geometry (e.g., "What shape is this piece of Lego?") Based on such intelligent and dynamic processing, the dialogue response determiner 1020 may determine the appropriate response at 1150 taking into account the actual dialogue context observed. Such responses generated to an ongoing conversation thread are adapted accordingly to each individual user and the various contexts observed at that moment.

[0088] The feedback (either what was originally said to the user generated by the dialog initiation determiner 1030, or a response to the user generated by the dialog response determiner 1020 in an ongoing dialog) is then sent to the dialog feedback generator 1050, which generates the feedback to be provided to the user at 1180 and sends the feedback so generated to the feedback indication generator 550 at 1190 (see Figure 5 ), where appropriate presentation instructions will be generated and provided to the user device to present the feedback to the user.

[0089] Fig.12 The architecture of a mobile device is shown, which can be used to partially or completely implement a particular system of the present teachings. In this example, the user device on which content is presented and interacted with is a mobile device 1100, which includes but is not limited to a smart phone, a tablet computer, a music player, a handheld game console, a global positioning system (GPS) receiver, and a wearable computing device (such as glasses, a wristwatch, etc.) or other form factors. The mobile device 1200 in this example includes one or more central processing units (CPUs) 1240, one or more graphics processing units (GPUs) 1230, a display 1220, a memory 1260, a communication platform 1210 such as a wireless communication module, a storage 1290, and one or more input / output (I / O) devices 1250. Any other suitable components, including but not limited to a system bus or a controller (not shown) may also be included in the mobile device 1200. As shown in FIG. Fig.12 As shown, a mobile operating system 1270 (e.g., iOS, Android, Windows Phone, etc.) and one or more applications 1280 may be loaded from storage 1290 into memory 1260 for execution by CPU 1240. Application 1280 may include a browser or any other suitable mobile app for receiving and presenting content streams on mobile device 1200. Communication with mobile device 1200 may be achieved via I / O device 1250.

[0090] In order to implement the various modules, units and functions thereof introduced in the present disclosure, a computer hardware platform may be used as a hardware platform for one or more of the elements introduced herein. The hardware elements, operating systems, and programming languages ​​of such computers are conventional in nature, and it is assumed that those skilled in the art are sufficiently familiar with them in order to adapt these technologies to the advertisement matching inquiries disclosed herein. A computer with a user interface element may be used to implement a personal computer (PC) or other type of workstation or terminal device, but the computer may also operate as a server if programmed appropriately. It is believed that those skilled in the art are familiar with the structure, programming, and general operation of such computer equipment, and therefore, the accompanying drawings may be self-explanatory.

[0091] Fig.13The architecture of a computing device that can be used to implement a specific system for implementing this teaching is shown. This specific system for implementing this teaching has a functional block diagram of a hardware platform, which includes a user interface element. The computer can be a general-purpose computer or a special-purpose computer. Both can be used to implement a specific system for this teaching. This computer 1300 can be used to implement any component of the teaching as introduced here. For example, a dialogue manager, a dialogue controller, etc. can be implemented on a computer such as computer 1300, via its hardware, software program, firmware or a combination thereof. Although only one such computer is shown for convenience, the computer functions related to this teaching introduced here can be implemented in a distributed manner on several similar platforms, thereby distributing the processing load.

[0092] For example, the computer 1300 includes a COM port 1350 connected to a network connected thereto to facilitate data communications. The computer 1300 also includes a central processing unit (CPU) 1320, which takes the form of one or more processors for executing program instructions. The exemplary computer platform includes: an internal communication bus 1310; various forms of program memory and data storage, such as disks 1370, read-only memory (ROM) 1330 or random access memory (RAM) 1340, for various data files to be processed and / or communicated by the computer and possible program instructions to be executed by the CPU. The computer 1300 also includes an I / O component 1360, which supports input / output flows between the computer and other components therein (e.g., interface elements 1380). The computer 1300 may also receive programming and data via network communications.

[0093] Thus, embodiments of the methods of enhancing advertising services and / or other processing as outlined above may be implemented in a program. The program aspects of the present technology may be viewed as a "product" or "article of manufacture" typically in the form of executable code and / or related data carried on or implemented in a machine-readable medium. Tangible, non-transitory "memory" type media include any or all memory or other storage for a computer, processor, etc., or its associated modules, such as a variety of semiconductor memories, tape drives, disk drives, etc., which may provide storage for software programming at any time.

[0094] All or part of the software may sometimes be transmitted over a network (e.g., the Internet or a variety of other telecommunications networks). For example, such transmission may enable the loading of software from one computer or processor to another (e.g., from a host or other system of a management server or search engine operator to a hardware platform of a computing environment or other systems implementing a computing environment or similar functions associated with query / advertisement matching). Therefore, another type of medium that can carry software elements includes optical, electrical, and electromagnetic waves, such as through physical interfaces between local devices, through wired and optical fixed networks, and through a variety of air links. Physical elements that carry such waves (e.g., wired or wireless links, optical links, etc.) are also considered to be media that carry software. As used herein, in addition to being limited to tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0095] Therefore, machine-readable media can take a variety of forms, including but not limited to tangible storage media, carrier media, or physical transmission media. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer, etc., which can be used to implement the system shown in the drawings or any of its components. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include: coaxial cables, copper wires and optical fibers, including wires that constitute the bus within the computer system. Carrier transmission media can take the form of electrical or electromagnetic signals or sound or light waves (such as those generated during radio frequency (RF) and infrared (IR) data communications). General forms of computer-readable media therefore include, for example, floppy disks, foldable disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card paper tapes, any other physical storage media with patterns of holes, RAMs, PROMs and EPROMs, flash EPROMs, any other memory chips or cartridges, carriers that transmit data or instructions, links or cables that transmit such carriers, or any other media from which a computer can read programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution.

[0096] Those skilled in the art will appreciate that the present teachings are applicable to a variety of modifications and / or enhancements. For example, although the implementation of the various components described above can be implemented in a hardware device, it can also be implemented as a software-only solution, such as installed on an existing server. In addition, the enhanced advertising service based on user-organized local advertising disclosed herein is also implemented as firmware, a firmware / software combination, a firmware / hardware combination, or a hardware / firmware / software combination.

[0097] Although the present teaching and / or other examples have been described above, it will be apparent that various modifications may be made thereto, and the subject matter disclosed herein may be implemented in various forms and examples, and the present teaching may be applied in various applications, only some of which are described herein. The appended claims are intended to claim any and all applications, modifications and variations that fall within the true scope of the present teaching.

Claims

1. A system for enabling communication with a user, comprising: A multimodal data analysis unit configured to: Receiving information characterizing the surroundings of a user who is about to participate in a new conversation, wherein: This information is obtained from the context in which the user exists and captures the characteristics of the user and the context, and extracting relevant features from the information; A user state estimator configured to estimate the state of the user based on the relevant features; The user state describes at least one of the following: the appearance of the user observed in the scene, the expression of the user inferred based on the information obtained from the scene, one or more user emotions inferred based on the user expression, and the user intention inferred based on at least one of the expression and the one or more emotions; the appearance includes skin color, hair color, hairstyle, skin tone, eye color, clothing color, clothing style; A conversation context information determiner, configured to determine a conversation context around the scene based on relevant features; Based on the environment detection model, the conversation context information determiner detects various objects existing in the scene captured by multimodal data and extracts their relevant features. The conversation environment implies the hierarchical structure of concepts at different conceptual levels, from objects and places at the lower level to the properties of the place. A dialog controller, which is configured to: Based on the user, determine the topic of the new conversation, and Generate feedback to initiate a new conversation with the user based on the topic, user status, and conversation context; the topic is one of the following: Topics related to a program that the user has previously registered for; topics dynamically determined based on the user's status and / or the context of the conversation related to the scenario; and combinations thereof; Further comprising a feedback indication generator configured to generate one or more indications for presenting feedback to a user; Feedback is communicated to the user to initiate a new conversation via at least one of voice and visual means; The multimodal data analysis unit includes a raw data processing part, a feature extraction part and an inference part; the raw data processing part includes an audio information processor, a visual information processor and a text information processor; the feature extraction part includes an auditory sound analyzer, a speech recognition unit, a user facial feature detector, a user appearance analyzer and a surrounding situation feature detector, which detects features of the corresponding modality, and the detected features are sent to the inference part to infer higher-level concepts; the inference part includes: an emotion inferer, which infers the user's emotions based on visual and auditory features; and, an intention inferer, which infers the intention based on visual and auditory features and the inferred emotions; the inferred emotions and intentions are used by the dialogue controller to determine how to continue the dialogue and redirect the user's attention; The multimodal data analysis unit further includes a relevant clue identifier, which obtains input from all components of the feature extraction part and the inference part, and identifies clues related to the dialogue based on the dialogue model / tree, and the relevant clue identifier filters out features that are not related to dialogue management from specific features related to the user or existing in the dialogue scene; The dialogue controller includes a user information processor, a background information processor, a dialogue progress determiner, a dialogue initiation determiner, a dialogue response determiner and a dialogue feedback generator; The user information processor receives the user state information from the user state estimator and the user utility / preference from the utility layer to identify different types of relevant information, and further uses the relevant information about the user from different sources to determine the dialogue initiation / response strategy; The context information processor receives the context information from the dialog context information determiner, analyzes the received context information to identify relevant context information, and determines a dialog initiation / response strategy based on the context information; The dialog progress determiner extracts relevant information from the user state information and determines the operation mode accordingly. If it is a new dialog, the dialog progress determiner calls the dialog initiation determiner to determine the initiation strategy. Otherwise, the dialog mode determiner calls the dialog response determiner to generate a response strategy. When called, the dialog initiation determiner analyzes processing results from the user information processor and the background information processor to understand the user and the user's surroundings; Based on what is known about the current user, the conversation initiation determiner can decide based on a machine learning model to start a conversation by first making the user feel comfortable by talking about something the user might like before engaging the user in the intended conversation; given contextual information about the user's surroundings, the conversation initiation determiner generates a personalized conversation initiation strategy; Determine a response for the user in an ongoing conversation, and the conversation response determiner determines the response based on the content indicated in the relevant conversation tree; by utilizing relevant information related to the user status and the background of the user's surrounding environment, the conversation response determiner can generate a personalized response; the conversation response determiner can determine that the user needs to deviate a little from the current topic based on the machine learning model; the conversation response determiner uses what the user observes in the scene or the scene background to implement a strategy to switch the user's attention to other things; feedback is sent to the conversation feedback generator, the conversation feedback generator generates feedback to be provided to the user, and sends the generated feedback to the feedback indication generator, and a suitable presentation indication will be generated and provided to the user device to present the feedback to the user.

2. The system of claim 1, wherein: The information includes multimodal sensor data in at least one of audio, visual, textual, and tactile modalities, wherein the audio sensor data records auditory sounds from a scene and / or speech from a user.

3. The system of claim 1, wherein: The conversation context includes at least one of the following: At least one object present in the scene and its characteristics; Classification of putative scenarios; The characteristics of the scene; and Sounds heard from the environment of the scene.

4. A method for enabling communication with a user, implemented on a machine, based on a system for enabling communication with a user according to any one of claims 1 to 3, the machine having at least one processor, a memory and a communication platform, the method comprising: Receiving, via the communication platform, information representing the surroundings of a user who is about to participate in the new conversation, wherein: This information is obtained from the context in which the user exists and captures the characteristics of the user and the context; extracting relevant features from the information; Based on the relevant features, the user's state is inferred; the user state describes at least one of the following: the user's appearance observed in the scene, the user's expression inferred based on information obtained from the scene, one or more user emotions inferred based on the user's expression, and the user's intention inferred based on at least one of the expression and the one or more emotions; the appearance includes skin color, hair color, hairstyle, skin tone, eye color, clothing color, clothing style; Determine the conversation context around the scene based on relevant features; Based on the user, determine the topic of the new conversation; the topic is one of the following: Topics related to a program that the user previously registered for; A topic dynamically determined based on the user's state and / or the context of the conversation relevant to the scenario; and their combination; Generate feedback to initiate a new conversation with the user based on the topic, user status, and conversation context; generating one or more instructions for presenting feedback to a user; Feedback is communicated to the user to initiate a new conversation via at least one of voice and visual means.

5. The method of claim 4, wherein The information includes multimodal sensor data in at least one of audio, visual, textual, and tactile modalities, wherein the audio sensor data records auditory sounds from a scene and / or speech from a user.

6. The method of claim 4, wherein The conversation context includes at least one of the following: At least one object present in the scene and its characteristics; Classification of putative scenarios; The characteristics of the scene; and Sounds heard from the environment of the scene.

7. A machine-readable non-transitory medium having encoded thereon information for enabling communication with a user, wherein: Once the information is read by a machine, the machine executes the method as described in any one of claims 4 to 6: Receiving information characterizing a surrounding situation of a user who is about to participate in the new conversation, wherein the information is obtained from a scene in which the user exists and captures characteristics of the user and the scene; extracting relevant features from the information; Based on the relevant features, the user's state is inferred; the user state describes at least one of the following: the user's appearance observed in the scene, the user's expression inferred based on information obtained from the scene, one or more user emotions inferred based on the user's expression, and the user's intention inferred based on at least one of the expression and the one or more emotions; the appearance includes skin color, hair color, hairstyle, skin tone, eye color, clothing color, clothing style; Determine the conversation context around the scene based on relevant features; Based on the user, determine the topic of the new conversation; the topic is one of the following: Topics related to a program that the user previously registered for; A topic dynamically determined based on the user's state and / or the context of the conversation relevant to the scenario; and their combination; Generate feedback to initiate a new conversation with the user based on the topic, user status, and conversation context; generating one or more instructions for presenting feedback to a user; Feedback is communicated to the user to initiate a new conversation via at least one of voice and visual means.

8. The medium of claim 7, wherein The information includes multimodal sensor data in at least one of audio, visual, textual, and tactile modalities, wherein the audio sensor data records auditory sounds from a scene and / or speech from a user.

9. The medium of claim 7, wherein The conversation context includes at least one of the following: at least one object present in the scene and its characteristics; Classification of putative scenarios; The characteristics of the scene; and Sounds heard from the environment of the scene.

Images