System and method for adaptive dialogue via scenario modeling using combined neural networks

Abstract

The present teachings relate to methods, systems, media, and implementations for adaptive dialogue. One or more images of a dialogue scene are received, and the images capture the surroundings of a dialogue between a user and a machine based on a dialogue strategy. One or more objects present in the scene are detected from the image based on a first artificial neural network. Spatial relationships between the one or more objects are detected from the image based on a second artificial neural network. Scene modeling information of the scene is then generated based on the one or more objects and the spatial relationship, and used for adaptive dialogue.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 885,526, filed on August 12, 2019, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present teachings generally relate to computers. More specifically, the present teachings relate to computerized intelligent agents. Background Art

[0004] Computer-assisted dialogue systems have become increasingly popular with advances in artificial intelligence technology and the explosion of internet-based communications due to the ubiquity of internet connectivity. For example, more and more call centers deploy automated dialogue robots to handle customer calls. Hotels have installed various kiosks that can answer questions from tourists or guests. Online reservations (whether for tourist accommodation or theater tickets, etc.) are also more frequently completed by chatbots. Automated human-computer communication in other areas has also become increasingly popular in recent years.

[0005] Such traditional computer-assisted dialogue systems are usually pre-programmed with certain questions and answers based on well-known conversation patterns in a particular related field. Unfortunately, human interlocutors can be unpredictable and sometimes do not follow pre-planned conversation patterns. In addition, in some cases, human interlocutors may go off topic during the process, and continuing with a fixed conversation pattern may cause irritation or loss of interest. When this happens, such traditional machine dialogue systems often will not be able to continue to engage the human interlocutor, so either the human-machine dialogue must be terminated to hand over the task to the human operator, or the human interlocutor simply leaves the dialogue, which is undesirable.

[0006] Furthermore, conventional machine-based dialogue systems are often not designed to address human emotional factors, let alone consider how to address the emotional aspects of conversation when talking to humans. For example, conventional machine dialogue systems typically do not initiate a conversation unless someone activates the system or asks some questions. Even if a conventional dialogue system does initiate a conversation, it has a fixed way of talking and does not vary from person to person or adjust based on observation. Therefore, since they are programmed to faithfully follow the pre-designed dialogue patterns, they are often unable to react to the unplanned dynamics of the conversation and adjust in order to keep the conversation going in a way that continues to engage people. For example, when the person participating in the conversation is clearly annoyed or frustrated, the conventional machine dialogue system is completely unaware and will continue to talk in an irritating way. This not only makes the conversation unpleasant (conventional machine dialogue systems are still unaware of this), but also keeps people away from any conversations with machine-based dialogue systems in the future.

[0007] In some applications, conducting a human-computer dialogue session based on what is observed from humans is crucial to determine how to proceed effectively. An example is a dialogue related to education. When a chatbot is used to teach a child to read, it must be continuously monitored and addressed whether the child is perceptive to his / her teaching style in order to make it effective. Another limitation of traditional dialogue systems is their context unawareness. For example, traditional dialogue systems do not have the ability to observe the context of the conversation and improvise dialogue strategies in order to engage the user in a context-related manner to improve the user experience.

[0008] Therefore, there is a need for methods and systems that address these limitations. Summary of the invention

[0009] The teachings disclosed herein relate to methods, systems and programming for data processing. More specifically, the teachings relate to methods, systems and programming related to modeling a scene to generate scene modeling information and its utilization.

[0010] In one example, a method implemented on a machine having at least one processor, a storage device, and a communication platform capable of connecting to a network for an adaptive dialogue. One or more images of a dialogue scene are received, the images capturing the surroundings of a dialogue between a user and a machine based on a dialogue strategy. One or more objects present in the scene are detected from the image based on a first artificial neural network. A spatial relationship between the one or more objects is detected from the image based on a second artificial neural network. Then, scene modeling information of the scene is generated based on the one or more objects and the spatial relationship, and the scene modeling information is used for the adaptive dialogue.

[0011] In different examples, the present invention discloses a system for adaptive dialogue, which includes a dynamic dialogue scene modeling unit, an object / feature detection unit, an object spatial relationship detector, and a scene model generation unit. The dynamic dialogue scene modeling unit is configured to receive one or more images capturing the surroundings of a dialogue scene, in which a user and a machine participate in a dialogue based on a dialogue strategy. The object / feature detection unit is configured to detect one or more objects present in the dialogue scene from the one or more images via a first type of artificial neural network. The object spatial relationship detector is configured to detect the spatial relationship between one or more objects based on the one or more images via a second type of artificial neural network. The scene model generation unit is configured to generate scene modeling information representing the dialogue scene based on the one or more objects and the spatial relationship, wherein the scene modeling information is used for adaptive dialogue.

[0012] Other concepts relate to software for implementing the present teachings. A software product according to this concept includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters associated with the executable program code, and / or information related to a user, request, content, or other additional information.

[0013] In one example, a machine-readable, non-transitory, and tangible medium having data for an adaptive dialogue recorded thereon, wherein the medium, when read by a machine, causes the machine to perform a series of steps. One or more images of a dialogue scene are received, the images capturing the surroundings of a dialogue between a user and a machine based on a dialogue strategy. One or more objects present in the scene are detected from the image based on a first artificial neural network. A spatial relationship between the one or more objects is detected from the image based on a second artificial neural network. Scene modeling information of the scene is then generated based on the one or more objects and the spatial relationship, and the scene modeling information is used for the adaptive dialogue.

[0014] Additional advantages and novel features will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The advantages of the present teachings may be realized and obtained by practice or use of the various aspects of the methods, tools, and combinations set forth in the detailed examples discussed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The methods, systems and / or programs described herein are further described according to exemplary embodiments. These exemplary embodiments are described in detail with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, in which the same reference numerals represent similar structures in the several views of the drawings, and in which:

[0016] Figure 1Depicted is a network environment for facilitating a conversation between a user operating a user device and an agent device incorporating a user interaction engine according to an embodiment of the present teachings;

[0017] Figure 2A-2B depicts connections between a user device, an agent device, and a user interaction engine during a conversation according to an embodiment of the present teachings;

[0018] Figure 3A illustrates an exemplary structure of an exemplary type of proxy device having a proxy body according to an embodiment of the present teachings;

[0019] Figure 3B An exemplary proxy device according to an embodiment of the present teachings is illustrated;

[0020] Figure 4A An exemplary dialogue scenario is illustrated;

[0021] Figure 4B illustrates exemplary aspects of operations for implementing an adaptive dialogue strategy according to an embodiment of the present teachings;

[0022] Figure 5A depicts an exemplary high-level system diagram of a system for learning dialog environment modeling via a combined neural network according to an embodiment of the present teachings;

[0023] Figure 5B is a flow chart of an exemplary process for a system for learning dialog environment modeling via a combined neural network according to an embodiment of the present teachings;

[0024] Fig. 6A An exemplary high-level system diagram of a dynamic dialogue scene modeling unit according to an embodiment of the present teachings is depicted;

[0025] Figure 6B is a flow chart of an exemplary process of a dynamic dialogue scene modeling unit according to an embodiment of the present teaching;

[0026] Fig. 7A An exemplary high-level system diagram depicting a dialog system in an augmented reality scenario constructed via dialog scenario modeling according to an embodiment of the present teachings;

[0027] Figure 7B is a flowchart of an exemplary process of a dialogue system in an augmented reality scene constructed via dialogue scene modeling according to an embodiment of the present teachings;

[0028] Fig. 8A An exemplary high-level system diagram depicting a dialogue system in a virtual reality scene constructed via dialogue scene modeling according to an embodiment of the present teachings;

[0029] Figure 8Bis a flow chart of an exemplary process of a dialogue system in a virtual reality scene constructed via dialogue scene modeling according to an embodiment of the present teachings;

[0030] Fig. 9 is an illustrative diagram of an exemplary mobile device architecture that can be used to implement a dedicated system for implementing the present teachings in accordance with various embodiments; and

[0031] Fig.10 is an illustrative diagram of an exemplary computing device architecture that can be used to implement a special purpose system implementing the present teachings, according to various embodiments. DETAILED DESCRIPTION

[0032] In the following detailed description, many specific details are set forth by way of example to facilitate a thorough understanding of the relevant teachings. However, it is clear to those skilled in the art that the present teachings may be practiced without these details. In other cases, well-known methods, processes, components and / or circuit systems have been described at a relatively high level without detailed description to avoid unnecessarily obscuring various aspects of the present teachings.

[0033] This teaching aims to address the deficiencies of traditional human-computer dialogue systems and provide methods and systems that enable more effective and realistic human-computer dialogue. This teaching combines artificial intelligence in an automated companion with an agent device (which is combined with backbone support from a user interaction engine) so that the automated companion can conduct a dialogue based on continuously monitored multimodal data (which indicates the surrounding situation of the dialogue), thereby adaptively estimating the mentality / emotion / intention of the participants in the dialogue, and adaptively adjusting the conversation strategy based on dynamically changing information / estimates / context information.

[0034] The automated companion according to the present teaching can personalize the conversation by adjusting in multiple aspects, including but not limited to the topic of the conversation, the hardware / components used to perform the conversation, and the expressions / behaviors / gestures used to deliver responses to the human interlocutor. The adaptive control strategy flexibly changes the dialogue strategy based on observations about the human interlocutor's acceptance of the dialogue and the dialogue context, thereby making the conversation more real and productive. The dialogue system according to the present teaching can be configured to implement a goal-driven strategy, including dynamically configuring the hardware / software components that are considered to be most suitable for the current user to achieve the expected goal. Such optimization is based on learning execution, including learning from previous conversations and ongoing conversations by continuously evaluating the behavior / reactions of the human interlocutor about the expected goal during the conversation. The path for implementing the goal-driven strategy can be determined to keep the human interlocutor engaged in the conversation, even in some cases, the path may appear to deviate from the expected goal at certain moments.

[0035] More specifically, the present teachings disclose a user interaction engine that provides backbone support to an agent device to facilitate more realistic and engaging conversations with a human interlocutor. Figure 1 A networked environment 100 for facilitating a conversation between a user operating a user device and an agent device incorporating a user interaction engine is depicted in accordance with an embodiment of the present teachings. Figure 1 In the exemplary networked environment 100, one or more user devices 110 (such as user devices 110-a, 110-b, 110-c, and 110-d), one or more proxy devices 160 (such as proxy devices 160-a, ..., 160-b), a user interaction engine 140, and a user information database 130, each of which can communicate with each other via a network 120. In some embodiments, the network 120 can correspond to a single network or a combination of different networks. For example, the network 120 can be a local area network ("LAN"), a wide area network ("WAN"), a public network, a private network, a private network, a public switched telephone network ("PSTN"), the Internet, an intranet, a Bluetooth network, a wireless network, a virtual network, and / or any combination thereof. In one embodiment, the network 120 can also include various network access points. For example, the environment 100 can include wired or wireless access points, such as, but not limited to, base stations or Internet exchange points 120-a, ..., 120-b. Base stations 120 - a and 120 - b may facilitate communications to / from user equipment 110 and / or proxy device 160 utilizing one or more other components in networking framework 100 , for example, across different types of networks.

[0036] The user device (e.g., 110-a) may be of different types to facilitate the user to operate the user device to connect to the network 120 and transmit / receive signals. Such user device 110 may correspond to any suitable type of electronic / computing device, including but not limited to a desktop computer (110-d), a mobile device (110-a), a device incorporated into a transportation vehicle (110-b), ..., a mobile computer (110-c), or a fixed device / computer (110-d). Mobile devices may include but are not limited to mobile phones, smart phones, personal display devices, personal digital assistants ("PDAs"), game consoles / devices, wearable devices (such as watches, Fitbits, pins / broaches, headphones, etc.). The transportation vehicle in which the device is embedded may include a car, a truck, a motorcycle, a boat, a ship, a train, or an airplane. The mobile computer may include a laptop computer, an ultrabook device, a handheld device, etc. The fixed device / computer may include a television, a set-top box, a smart home device (e.g., a refrigerator, a microwave oven, a washing machine or dryer, an electronic assistant, etc.) and / or a smart accessory (e.g., a light bulb, a light switch, an electronic photo frame, etc.).

[0037] The agent device (e.g., any of 160-a, ..., 160-b) can correspond to one of the different types of devices capable of communicating with a user device and / or the user interaction engine 140. As described in more detail below, for example, each agent device can be viewed as an automated companion device that interacts with a user using backbone support from the user interaction engine 140. The agent device as described herein can correspond to a robot, which can be a gaming device, a toy device, a designated agent device (such as a travel agent or a weather agent). The agent device as disclosed herein can facilitate and / or assist in user interaction with an operating user device. In doing so, the agent device can be configured as a robot capable of controlling some parts of it via backend support from the application server 130, for example, making certain physical movements (such as the head), showing certain facial expressions (such as a crooked smile), or speaking in a certain voice or tone (such as an excited tone) to display certain emotions.

[0038] When a user device (e.g., user device 110-a) is connected to an agent device (e.g., 160-a) (e.g., via a contact connection or a contactless connection), a client running on the user device (e.g., 110-a) can communicate with an automated companion (agent device or user interaction engine, or both) to enable an interactive dialogue between a user operating the user device and the agent device. The client can act independently in some tasks, or can be remotely controlled by the agent device or user interaction engine 140. For example, in response to a question from a user, the agent device or user interaction engine 140 can control the client running on the user device to render a responsive speech to the user. During a conversation, the agent device may include one or more input mechanisms (e.g., a camera, a microphone, a touch screen, buttons, etc.) that allow the agent device to capture input related to the user or to the local environment associated with the conversation. Such input can assist the automated companion in understanding the atmosphere of the conversation's surroundings (e.g., the user's movements, ambient sounds) and the human interlocutor's state of mind (e.g., the user picking up a ball can indicate that the user is bored) so that the automated companion can respond accordingly and conduct the conversation in a way that keeps the user interested and engaged.

[0039] In the illustrated embodiment, user interaction engine 140 can be a back-end server, which can be centralized or distributed. It is connected to a proxy device and / or a user device. It can be configured to provide backbone support to proxy device 160, and guide the proxy device to talk in a personalized and customized manner. In some embodiments, user interaction engine 140 can receive information from a connected device (proxy device or user device), analyze this information, and control the flow of conversation by sending instructions to proxy device and / or user device. In some embodiments, user interaction engine 140 can also communicate directly with user equipment, for example, to provide dynamic data (for example, a control signal for a client running on a user device to render certain responses).

[0040] In general, the user interaction engine 140 can control the state and flow of the conversation between the user and the proxy device. The flow of each conversation can be controlled based on different types of information associated with the conversation, for example, information about the users participating in the conversation (e.g., from the user information database 130), the conversation history, the surrounding information of the conversation, and / or real-time user feedback. In some embodiments, the user interaction engine 140 can be configured to obtain various sensory inputs (such as but not limited to audio input, image input, tactile input and / or contextual input), process these inputs, form an understanding of the human interlocutor, generate a response based on this understanding accordingly, and control the proxy device and / or the user device to perform the conversation based on the response. As an illustrative example, the user interaction engine 140 can receive audio data representing a speech from a user operating a user device, and generate a response (e.g., text), which can then be delivered to the user in the form of a computer-generated speech as a response to the user. As another example, the user interaction engine 140 can also generate one or more instructions for controlling the proxy device to perform a specific action or set of actions in response to the speech.

[0041] As shown, during a human-computer dialogue, a user who is a human interlocutor in the dialogue can communicate with an agent device or a user interaction engine 140 across a network 120. This communication can involve data in multiple modes, such as audio, video, text, etc. Via a user device, a user can send data (e.g., a request, an audio signal representing the user's speech, or a video of a scene surrounding the user) and / or receive data (e.g., a text or audio response from an agent device). In some embodiments, multimodal user data can be analyzed to understand the voice or gestures of a human user after being received by an agent device or a user interaction engine 140, so that the user's emotions or intentions can be estimated and used to determine a response to the user.

[0042] Figure 2ADepicted is a specific connection between a user device 110-a, a proxy device 160-a, and a user interaction engine 140 during a conversation according to an embodiment of the present teaching. As can be seen, the connection between any two parties of these entities can be bidirectional, as discussed herein. The proxy device 160-a can interact with the user via the user device 110-a to conduct a conversation in a bidirectional communication. On the one hand, the proxy device 160-a can be controlled by the user interaction engine 140 to issue a response to the user operating the user device 110-a. On the other hand, input from the user site (including, for example, both the user's speech / action and information about the user's surroundings) is provided to the proxy device via these connections. The proxy device 160-a can be configured to process such input and dynamically adjust its response to the user. For example, the user interaction engine 140 can instruct the proxy device to render a tree on the user device. In the case of knowing that the user's surroundings (based on visual information from the user device) show green trees and lawns, the proxy device can customize the tree to render it as a lush green tree. If the scene from the user site shows that it is winter weather, the proxy device can be controlled to render trees on the user device with parameters for trees without leaves. As another example, if the proxy device is instructed to render a duck on the user device, the proxy device can retrieve information about color preferences from the user information database 130 and generate parameters for customizing the duck with the user's preferred colors before sending instructions for rendering to the user device.

[0043] In some embodiments, such input from the user site and the results of its processing may also be transmitted to the user interaction engine 140 to facilitate the user interaction engine 140 to better understand the specific circumstances associated with the conversation, so that the user interaction engine 140 can determine the state of the conversation, the user's emotions / mindset, and generate a response based on the specific circumstances of the conversation and the intended purpose of the conversation (e.g., teaching children English vocabulary). For example, if the information received from the user device indicates that the user seems bored and is becoming impatient, the user interaction engine 140 may determine to change the state of the conversation to a topic of interest to the user (e.g., based on information from the user information database 130) in order to continue to engage the user in the conversation.

[0044] In some embodiments, the client running on the user device can be configured to process raw input of different modes obtained from the user site, and send the processed information (e.g., relevant features of the raw input) to the proxy device or the user interaction engine for further processing. This will reduce the amount of data transmitted over the network and improve communication efficiency. Similarly, in some embodiments, the proxy device can also be configured to process information from the user device and extract useful information for, for example, customization purposes. Although the user interaction engine 140 can control the state and process control of the conversation, making the user interaction engine 140 lightweight will better improve the scale of the user interaction engine 140.

[0045] Figure 2B Depicted with Figure 2A , with additional details about user device 110-a. As shown, during a conversation between a user and agent 210, user device 110-a can continuously collect multimodal sensor data related to the user and its surroundings, which can be analyzed to detect any information related to the conversation and used to intelligently control the conversation in an adaptive manner. This can further enhance the user experience or engagement. Figure 2B Exemplary sensors are illustrated, such as video sensor 230, audio sensor 240, ... or tactile sensor 250. The user device may also send text data as part of the multimodal sensor data. Together, these sensors provide contextual information around the conversation and can be used to allow user interaction system 140 to understand the situation in order to manage the conversation. In some embodiments, the multimodal sensor data may be processed first on the user device, and important features in the different modalities may be extracted and sent to the user interaction system 140 so that the conversation can be controlled with an understanding of the context. In some embodiments, the raw multimodal sensor data may be sent directly to the user interaction system 140 for processing.

[0046] like Figure 2A-2B As shown in , the proxy device may correspond to a robot having different parts including its head 210 and its body 220. Figure 2A-2B The agent device shown in appears to be a humanoid robot, but it can also be constructed in other forms, such as a duck, a bear, a rabbit, etc. Figure 3AAn exemplary structure of an agent device with an example type of agent body according to an embodiment of the present teaching is illustrated. As presented, the agent device may include a head and a body, wherein the head is attached to the body. In some embodiments, the head of the agent device may have additional parts, such as a face, a nose, and a mouth, some of which may be controlled to, for example, make movements or expressions. In some embodiments, the face on the agent device may correspond to a display screen on which a face can be rendered, and the face may be a human or animal face. Such a displayed face may also be controlled to express emotions.

[0047] The body parts of the proxy device may also correspond to different forms, such as a duck, a bear, a rabbit, etc. The body of the proxy device may be stationary, movable, or semi-movable. A proxy device with a stationary body may correspond to a device that can sit on a surface (such as a table) to have a face-to-face conversation with a human user sitting next to the table. A proxy device with a movable body may correspond to a device that can move on a surface (such as a table surface or a floor). Such a movable body may include a part that can be kinematically controlled to move physically. For example, the proxy body may include a foot that can be controlled to move in space when needed. In some embodiments, the body of the proxy device may be semi-movable, that is, some parts are movable and some parts are not. For example, the tail on the body of an proxy device with a duck appearance may be movable, but the duck cannot move in space. A bear body proxy device may also have arms that can be movable, but the bear can only sit on the surface.

[0048] Figure 3B An exemplary proxy device or automated companion 160-a according to an embodiment of the present teachings is illustrated. The automated companion 160-a is a device that interacts with a person using voice and / or facial expressions or body gestures. For example, the automated companion 160-a corresponds to an animatronic peripheral having different parts, including a head portion 310, an eye portion (camera) 320, a mouth with a laser 325 and a microphone 330, a speaker 340, a neck portion 350 with a servo, one or more magnets or other components 360 that can be used for contactless detection of presence, and a body portion corresponding to, for example, a charging base 370. In operation, the automated companion 160-a can be connected to a user device via a network connection, which can include a mobile multifunction device (110-a). Once connected, the automated companion 160-a and the user device interact with each other via, for example, voice, motion, gestures, and / or via pointing with a laser pointer.

[0049] Other exemplary functions of the automated companion 160-a may include reactive expressions in response to a user's responses, for example, via an interactive video cartoon character (e.g., an avatar) displayed on, for example, a screen (as part of a face on the automated companion). The automated companion may use a camera (320) to observe the user's presence, facial expressions, gaze direction, surroundings, etc. The animatronic embodiment may "see" by pointing its head (310) containing the camera (320), "listen" using its microphone (340), and "point" by directing its head (310) which may be moved via a servo system (350). In some embodiments, the head of the proxy device may also be remotely controlled, for example, by the user interaction system 140 or by a client in the user device (110-a) via a laser (325). As Figure 3B The exemplary automated companion 160 - a shown in FIG. 1 may also be controlled to “speak” via a speaker ( 330 ).

[0050] Figure 4A An exemplary dialogue scene 400 according to an embodiment of the present teaching is illustrated, wherein a robot agent 410 interacts with a user in a dialogue in the dialogue scene. In this illustration, the dialogue scene 400 is a room with multiple objects (including tables, chairs, computers on the tables, walls, windows, and various paintings and / or boards on the walls). In addition to such fixtures, there are also robot agents 410 and users 420 on the tables. When interacting with the user 420, the robot agent 410 can observe the dialogue scene via scene modeling. This scene modeling can provide useful information for the robot agent 410 and help the robot agent 451 adjust its dialogue strategy when necessary. For example, if the robot agent 410 notices that the user 420 is not well involved in answering questions from the agent and always looking in a certain direction, then by using information from the scene modeling, the robot agent 410 can estimate the object that the user is looking at. In this case, the robot agent 410 can ask some questions related to the object in order to continue to involve the user in the ongoing dialogue.

[0051] In a dialog system, such an adaptive dialog strategy may be needed to enhance user experience and engagement. As described above, the adaptive dialog strategy may require not only understanding of the dialog itself, but also understanding of the surroundings of the dialog, such as the environment in which the dialog is conducted. Figure 4BDifferent exemplary aspects of the functionality supporting adaptive dialogue strategies are shown. For example, to adjust the dialogue strategy, environmental modeling can be performed to enable the robotic agent to understand its surroundings. Spontaneous dialogue is part of the adaptive dialogue strategy, which can require the robotic agent to dynamically control the conversation using the adaptive strategy. In some embodiments, in order to make adjustments, the robotic agent can also create some type of collaborative augmented or virtual reality environment based on observed needs, for example, by including an adaptively determined environment and / or some virtual agents to deliver adaptively determined content to the user in the augmented / virtual dialogue reality. The present teaching focuses on the environmental modeling aspect.

[0052] Obtaining an understanding of the dialogue environment can be the basis of an adaptive dialogue strategy. For example, during a planned conversation between a virtual agent and a user, the virtual agent can be present in a dialogue environment (real, virtual, or enhanced) to have a conversation, which can be based on, for example, objects present in the dialogue scene. Such objects can also be real or virtual. For example, if a robot agent is teaching a student user the concept of addition, then knowing that the student likes fruit, the robot agent can render multiple fruits in the dialogue scene and ask the student user to perform addition. In this case, these fruits can be rendered in the dialogue scene in a manner consistent with the existing real scene, for example, the fruit must be displayed in front of the user instead of behind the user, or rendered on the table instead of in the sky.

[0053] Figure 5A An exemplary high-level system diagram of a system 500 for modeling a dialog environment via a combined neural network according to an embodiment of the present teaching is depicted. In this illustrated embodiment, the system 500 includes a convolutional neural network 520 and a graph neural network 530, which are used to learn different aspects of objects present in the dialog scene. This combined use of different neural networks is to train each neural network to capture different characteristics of the environment, for example, learning a CNN-based object / feature detection model 540 via a convolutional neural network 520 and learning a GNN-based spatial relationship detection model 550 via a graph neural network 530. In some embodiments, such models are represented by parameters associated with their respective neural networks, including, for example, the number of layers, the number of nodes in each layer, the connections between different nodes, the weights associated with different connections, the parameters of the functions used by different nodes, etc. In such embodiments, using such parameters, each neural network can then be configured accordingly and used to perform the learned tasks.

[0054] In the system 500, the convolutional neural network 520 is used to learn a CNN-based object / feature detection model 540 based on supervised training data based on the training images 510. The training images 510 include images with labeled objects, and such objects may include human users present in the conversation scene and features associated therewith, as well as other types of non-human objects (such as trees, benches, the sky, pictures on the wall, tables, chairs, computers, or toys that may be observed in the conversation scene). For example, a user may be identified in the conversation scene, and features characterizing the user may include his / her posture, orientation, etc., and such a person and his / her features may be detected / extracted using the CNN-based object feature detection model 540.

[0055] Similarly, the graph neural network 530 is used to learn a GNN-based spatial relationship detection model 550 based on labeled image training data based on the training images 510. The labeled data used to train the GNN-based spatial relationship detection model 550 may include various spatial relationships observed in the scenes in the training images, such as the spatial relationships between captured objects may include the spatial relationships between the user and various objects present in the scene. For example, a user and a table may be identified in a conversation scene, and each may be associated with different characteristics, such as their respective physical positions, orientations, etc. The spatial relationship between the two can be determined based on their corresponding characteristics, or can be detected / extracted from the training images. Such identified spatial relationships between different objects in the training images can be used to construct a GNN-based object feature detection model 550.

[0056] To facilitate training, objects (including people) and their attributes in scene image 510-2 in training image 510 are identified, stored in object / attribute 510-1, and used for supervised learning by convolutional neural network 520 to obtain object / feature detection model 540. On the other hand, the spatial relationships between objects identified from the training images are also identified, labeled, and stored in object spatial relationship 510-3. Graph neural network 530 uses such identified spatial relationships between objects in different scene images for supervised training to obtain GNN-based spatial relationship detection model 550.

[0057] Figure 5Bis a flowchart of an exemplary process of a system 500 for learning dialog environment modeling via a combined neural network according to an embodiment of the present teaching. At 555, a training scene image 510-2 is accessed to obtain a labeled object and associated features (attributes) 510-1 at 560. Spatial relationships 510-3 between such identified objects are also obtained at 565. The convolutional neural network 520 is then trained using the scene image 510-2 and the labeled objects / attributes at 570 to obtain a CNN-based object / feature detection model 540 at 575. At the same time, the scene image 510-2 and the spatial relationships between objects in the extracted scene image are used to train a graph neural network 530 at 580 to obtain a GNN-based spatial relationship detection model 550 at 585.

[0058] The derived CNN object / feature detection model 540 can then be used to detect objects and associated attributes from the image of the conversation scene. Similarly, the derived GNN-based spatial relationship detection model 550 can then be used to detect the spatial relationship between objects detected from the image of the conversation scene. Fig. 6A An exemplary high-level system diagram of a dynamic dialogue scene modeling unit 600 according to an embodiment of the present teachings is depicted. In the illustrated embodiment, the dynamic dialogue scene modeling unit 600 includes an object / feature detection unit 610, a scene estimation unit 620, an object spatial relationship detector 630, and a scene model generation unit 640. In operation, the dynamic dialogue scene modeling unit 600 receives an image of a dialogue scene and derives a scene model representing the dialogue scene based on a CNN-based object / feature detection model 540 and a GNN-based spatial relationship detection model 550.

[0059] Figure 6B 6 is a flowchart of an exemplary process of a dynamic dialogue scene modeling unit 600 according to an embodiment of the present teaching. When an image of a dialogue scene is received at 655, an object / feature detection unit 610 detects an object and its associated features based on a CNN-based object / feature detection model 540 at 660. In some embodiments, the object / feature detection unit 610 is a trained convolutional neural network configured based on parameters specified in a CNN-based object / feature detection model 540. Based on the detected objects and features, the scene estimation unit 620 can then estimate the nature of the dialogue scene based on the detected scene objects / features at 665. Since each type of scene can have some characteristic objects, the type of dialogue scene can be estimated based on the objects detected from the scene. For example, an office scene typically has a table, some chairs, a bookcase, and / or a computer on the table. If some such objects are detected from the image of the scene, it can be inferred or estimated that the dialogue scene involves an office.

[0060] In order to facilitate the adaptive dialogue strategy, the spatial relationship between the detected scene objects can also be detected based on the detected objects and their features. At 670, the object spatial relationship detector 630 can identify the spatial relationship between the objects based on the received scene image and the GNN-based spatial relationship detection model 550. In some embodiments, the spatial relationship detector 630 is a trained graph neural network configured based on the parameters specified in the GNN-based spatial relationship detection model 550. In some embodiments, the object spatial relationship detector 630 can detect the spatial relationship between different objects directly from the scene image (for example, if it learns directly from the image when it is trained). In some embodiments, the object spatial relationship detector 630 can also receive information about the detected objects / features from the object / feature detection unit 610 and use this information to guide it to detect the spatial relationship between these detected objects. In some embodiments, the object spatial relationship detector 630 can also receive information about the estimated scene type from the scene estimation unit 620 and use this information to facilitate its determination of the spatial relationship between objects. For example, if the estimated scene type is an office scene, this knowledge can be used to determine that the computer is on the desktop (because in the office, the computer is usually on the desktop). Using the identified scene objects / features and their spatial relationships, the scene model generation unit 640 obtains a scene model at 675 and then stores a representation of the scene model at 680 .

[0061] This dynamically generated scenario model can be used to determine an adaptive dialogue strategy. In some embodiments, a scenario model created based on a dialogue scenario can be used to determine how to enhance a dialogue scenario for purposes such as better attracting users or performing certain specific tasks. Fig. 7A An exemplary high-level system diagram of a dialogue system 700 for dialogue in an augmented reality scene constructed via dialogue scene modeling according to an embodiment of the present teaching is depicted. In this illustrated example application, a user 705 participates in a dialogue with the adaptive dialogue system 700. The utterance of the user 705 is received by a spoken language understanding (SLU) unit 702. With the understanding of what the user said (which is determined by the SLU unit 702), the dialogue manager 710 determines a response based on a dialogue tree 725 and instructs a response generator 730 to generate a text response, which is then converted into a speech response by a text-to-speech (TTS) unit 735 for delivery to the user 705.

[0062] During the conversation, the dynamic conversation scene modeling unit 600 may also receive a scene image capturing the conversation environment, and accordingly at 650 (as described herein with reference to Figure 6A-6BThe dynamic dialog scene modeling unit 600 generates a scene model 705 representing the surrounding environment of the dialog with the user. Such scene models can be continuously created and adapted to changes in the scene. For example, the user moves around in the scene, so that the features associated with the user and his / her spatial relationship with other objects in the scene can change accordingly. The changing features of the objects in the dialog scene and the changing spatial relationships between these objects can be captured by the dynamic dialog scene modeling unit 600 via continuous scene modeling, and therefore, the scene model stored in the environment modeling database 650 is adaptively updated over time.

[0063] When the dialogue manager 710 decides to continue the dialogue using an enhanced dialogue scene, the scene model for the surrounding environment is used by the enhanced scene generation unit 720 to generate an enhanced scene, for example, by visualizing certain selected virtual objects in the dialogue scene to generate an enhanced scene. The dialogue manager 710 can also call the enhanced dialogue strategy generator 715 to create an enhanced dialogue strategy for an adapted dialogue in the enhanced dialogue scene. For example, if the dialogue system 700 participates in a dialogue with the user 705 to teach the user the concept of addition. During the dialogue, it can be identified that the user's response to the questions presented on the computer screen is poor and the user seems distracted. In this case, the dialogue manager 710 can decide to better attract the user by projecting colored virtual objects in the scene in order to continue the dialogue about addition. To facilitate this, the dialogue manager 710 can call the enhanced scene generation unit 720 to create virtual objects (e.g., balls of different colors), and call the enhanced dialogue strategy generator 715 to generate a dialogue strategy for governing the dialogue based on the enhanced scene.

[0064] The created virtual objects will be visualized or projected in the dialogue scene. In order to render virtual objects in the actual scene in a consistent manner (to form an augmented reality scene), various parameters need to be determined, such as the 3D position in the scene for visualizing virtual objects, the direction of projecting virtual objects (e.g., must be in the user's field of view), the size of the virtual objects after projection, etc. According to the present teaching, such parameters can be determined based on scene modeling that characterizes objects already present in the scene and their spatial relationships. For example, if the scene modeling indicates that the user is sitting in a chair near a table, facing a computer screen on the table, then visualization can be performed at a position that will not be blocked by the computer on the table. If the user is distracted and has stood up and walked around in the dialogue scene, then visualization needs to be provided in the user's field of view. To achieve this, the enhanced scene generation unit 720 can access the adaptively updated scene modeling information from the database 650 and determine the parameters required to generate the enhanced scene accordingly. Once the enhanced scene and enhanced dialogue strategy are generated accordingly, the dialogue manager 710 conducts the dialogue based on the enhanced dialogue strategy consistent with the generated enhanced scene.

[0065] Figure 7B 7 is a flowchart of an exemplary process of a dialogue system 700 for creating an adaptive dialogue strategy for an augmented reality scene by modeling a dialogue scene according to an embodiment of the present teaching. The dialogue system 700 receives various inputs at 740, including an image capturing a dialogue scene and additional information (such as audio data representing the user's speech). In order to model the surrounding situation of the dialogue, the dynamic dialogue scene modeling unit 600 detects objects and their features from the scene image at 745, and generates a scene model for the dialogue scene at 750. At the same time, based on the received audio information representing the speech of the user 705, the SLU unit 702 performs spoken language understanding at 755 and sends the SLU result to the dialogue manager 710.

[0066] Based on the current state of the dialogue and the surrounding information represented by the scene model, the dialogue manager 710 determines at 760 whether an enhanced dialogue scene is needed for enhancing the dialogue. If the enhanced dialogue is not needed, the dialogue manager 710 continues to determine the response to the user based on the SLU result from the SLU unit 702 and the dialogue tree 725 at 780. Then at 785, this determined response is delivered to the user via the TTS unit 735. If an enhanced dialogue is needed, the dialogue manager 710 calls the enhanced scene generation unit 720 to create an enhanced dialogue scene. This is achieved by identifying the virtual objects to be rendered in the dialogue scene at 765 and determining the parameters for rendering such virtual objects in the scene based on the dynamically updated scene model associated with the dialogue scene. In order to promote enhanced dialogue, the enhanced dialogue strategy generator 715 is called at 770 to determine the enhanced dialogue strategy corresponding to the enhanced dialogue. At 775, based on the virtual objects and rendering parameters, the augmented reality scene is rendered accordingly, and then the dialogue manager 710 performs an enhanced dialogue on the enhanced dialogue scene by determining the response according to the enhanced dialogue strategy at 780. The response is then delivered to the user at 785. The process then returns to 740 to receive the updated scene image and the next utterance from the user.

[0067] Continuing with the example of teaching the concept of digital addition, where an augmented dialogue is conducted in an augmented dialogue scene, where colored balls are rendered or thrown in the air to the user (to attract the user's attention to increase participation), an augmented dialogue strategy can be generated based on the content of the augmented scene. In some embodiments, the dialogue manager 710 may have conducted a dialogue in accordance with the original strategy to present different types of objects to the user on the computer screen and ask the user to add and provide an answer. When it is observed that the user is not well engaged (answering incorrectly or looking away) in the case of using such a communication means, the dialogue manager 710 may decide to use an augmented means by placing the tutorial content in a more interesting way to attract the user's attention. The dialogue manager 710 can provide the content in the original dialogue to the augmented scene generation unit 720 so that virtual and attractive objects (e.g., different animals of different colors) can be selected based on the original tutorial material. For example, if the original content is "There are 2 candies in one box and 5 candies in another box. What is the total number of candies?", then the augmented content generated accordingly can be 2 orange bears and 5 blue dolphins, and the question "How many animals are there?" The corresponding augmented dialogue strategy can be changed accordingly to adapt to the new augmented content.

[0068] In some embodiments, a scene model derived via dynamic dialogue scene modeling based on one or more scene images captured at one location can be used to construct a virtual scene accordingly in different locations to perform a conversation. Unlike an enhanced scene that can add virtual objects to a real scene to form an enhanced scene, a virtual scene created based on a scene modeling result can be a computer-created virtual scene created based on the content described in the scene modeling. For example, if an office scene is modeled based on one or more scene images (wherein a table, two chairs, a computer on the table, and a bookcase are physically arranged in a certain spatial relationship), then, once this scene is detected, it can be represented electronically by specifying the objects, features of the objects, and spatial arrangements that exist. This representation can then be used to create a virtual scene of the same object (but virtual), which has the specified features of these virtual objects, and these virtual objects are arranged in the virtual scene according to the spatial relationships in the original physical scene.

[0069] Fig. 8AAn exemplary high-level system diagram of a dialogue system 800 according to an embodiment of the present teaching is depicted, which operates to conduct a dialogue in a virtual reality scene constructed via scene modeling. In the illustrated example application of a user-machine dialogue in a virtual scene created via real scene modeling, a user 805 participates in a dialogue with the dialogue system 800. The utterance of the user 805 is received by a spoken language understanding (SLU) unit 802, which then analyzes the received audio data and generates a result representing an understanding of what the user said. The dialogue manager 830 determines a response based on a dialogue tree 835 and instructs the response generator 730 and the TTS unit 840 to generate an audio response via text-to-speech (TTS) conversion. In this example application, the dialogue is conducted in a virtual environment, which is created based on scene modeling information (which is based on scenes at different locations), and a dynamic dialogue scene modeling unit 600 is deployed at the location to receive one or more scene images, and modeling information is generated from these scene images to model the scenes at the different locations.

[0070] When generating scene modeling information, the dynamic dialogue scene modeling unit 600 may receive one or more images capturing the remote basic scene to be modeled, and accordingly generate scene modeling information according to the reference scene. Figure 6A-6B The disclosed method generates a scene model. This scene modeling information can be created continuously over time and adapt to changes that occur in the scene. For example, if the scene is an outdoor scene and objects in the scene can move around, such as people walking around in a park scene (e.g., people sitting on a bench and then leaving), clouds flowing in the sky, etc. The changing features of the objects in the scene can also cause changes in the spatial relationship between them, and such dynamics can be captured by the continuously generated scene modeling information. Based on this dynamically updated scene modeling information, the dialogue system 800 can use the information stored in the database 650 to create a virtual scene. In some embodiments, the dialogue system 800 can also use static scene modeling information and create a static virtual scene, where virtual objects are created based on the representation of corresponding base objects in a remote base scene, and are spatially configured in the virtual scene with such a 3D posture similar to that in the base scene.

[0071] In order to generate a virtual scene, the dialogue system 800 includes a virtual reality scene generation unit 800, a virtual reality renderer 810, and a dynamic content creation tool 820. The virtual reality scene generation unit 800 accesses the scene modeling information from the database 650, and generates a virtual scene accordingly according to the accessed scene modeling information. Since the database 650 stores scene models obtained from different remote scenes, the virtual reality scene generation unit 800 can select a scene model suitable for the ongoing dialogue. For example, if the ongoing dialogue requires an office scene with windows, tables, chairs, and bookcases, the virtual reality scene generation unit 800 can retrieve a scene model that meets this criterion. If the ongoing dialogue requires an outdoor scene with a bench and a lake in a park, the virtual reality scene generation unit 800 can select another scene model for this purpose. The need for the virtual scene to be created and the type of virtual scene can be determined based on, for example, the content required to advance the dialogue or the desire to continue to attract the user.

[0072] The ability to create virtual reality dialogue scenes promotes adaptive dialogue strategies. For example, during an ongoing dialogue process, the dialogue manager 830 can also decide to change the virtual scene based on the situation observed during the dialogue, such as to better attract the user 805 or provide an enhanced user experience. For example, if the user looks tired in the virtual office scene and the interaction with the robot agent is not very good, the dialogue manager 830 can call the virtual reality scene generation unit 800 to create an outdoor scene of a sunny park to cheer up the user. As another example, if the user appears bored in a tutorial session teaching addition and it is known that the user likes animals, different virtual scenes on the farm can be created, which have different animals on the farm. In this way, it can make the user more interested while completing the goal of the tutorial. In some embodiments, the different scene models stored in the environment modeling database 650 can provide a selection of many different types of scene models, which can be retrieved to render virtual scenes based on the basic scenes that existed elsewhere in the past.

[0073] In the case where a virtual scene is created, the dialogue manager 830 can also call the dynamic content creation tool 820 to create dialogue content that is consistent not only with the virtual scene but also with the original dialogue strategy indicated by the dialogue tree 835. For example, in a dialogue in a virtual office scene, the original dialogue content indicated by the dialogue tree may be to ask the user to add two numbers displayed on the computer screen. If the virtual office scene is not changed to a virtual outdoor scene with multiple red and green benches in the scene due to the need to better attract users, then the dialogue content needs to be changed accordingly to ask how many benches there are in total (add the red and green benches). The dynamic content creation tool 820 can receive the original dialogue content and the scene modeling information of the outdoor scene from the dialogue tree 835, and generate virtual reality dialogue content and store it in the storage device 825 so that the dialogue manager 830 can now use it.

[0074] Figure 8B 8 is a flow chart of an exemplary process of a dialogue system 800 according to an embodiment of the present teaching, the system operating to conduct a dialogue in a virtual reality scene generated via dialogue scene modeling. In a dialogue conducted in a virtual scene, a scene model is first selected at 850 from a database 650, which stores various scene models constructed by a dynamic dialogue scene modeling unit 600 based on images of different scenes at different locations. In some embodiments, this selection can be made based on a dialogue strategy to be applied to the dialogue (e.g., the planned dialogue content specified in, for example, a dialogue tree 835). In some embodiments, the virtual reality scene for conducting the dialogue can be selected based on the preferences of the users currently participating in the dialogue. In some embodiments, the virtual scene can be selected based on both the current dialogue strategy and the preferences of the users involved.

[0075] Using the selected scene model, the virtual reality scene generation unit 800 calls the virtual reality renderer 810, which renders the selected virtual reality scene according to the scene model at 855. In some embodiments, the rendering can be performed based on a certain scene rendering configuration 815 (which can specify different parameters that can be used to control the rendering). In some embodiments, the rendering can also be performed based on a certain preference of the user (not shown), for example, the user may prefer a darker environment. Based on the selected virtual reality scene, the dialogue content used by the dialogue manager 830 to control the dialogue can also be created accordingly with respect to the virtual reality scene. This is achieved by the dynamic content creation tool 820. Content creation can be done automatically. For example, if the planned dialogue content asks a question about adding two numbers (for example, represented by different objects), then a scene model for the virtual reality scene can be selected so that the rendered virtual scene has the same number of objects. In addition, it is necessary to develop dynamic dialogue content that is consistent with the dialogue strategy and the virtual scene content so that the dialogue strategy can be executed for the virtual scene. The dynamic content creation tool 820 dynamically generates dialogue content with respect to the virtual scene based on the planned dialogue content. It receives the dialogue content specified in the dialogue tree at 860 and generates dynamic dialogue content created according to the expected dialogue content and the rendered virtual scene at 865.

[0076] In this generated virtual scene, when the dialogue system 800 receives an utterance from a user at 870, the SLU unit 802 performs spoken language understanding at 875 to obtain a result representing what the user said. This result is provided to the dialogue manager 830, which then determines a response at 880 based on the planned dialogue content represented in the dialogue tree 835 or the dynamic dialogue content stored in 825. The determined response is then used by the TTS unit 840 at 885 to generate a voice response through a text-to-speech operation and delivered to the user at 890 as a response to the utterance. In some embodiments, the dialogue manager 830 may determine whether a virtual scene needs to be changed based on the current state of the dialogue at 895. If a change is required, the process proceeds to 850 to select / render a different virtual scene suitable for the current dialogue state, and dynamically updates the dialogue content accordingly based on the new virtual scene before the user responds to the delivered response at 870. If the virtual scene does not need to be changed, the dialogue system 800 returns directly to 870 to receive the next utterance from the user.

[0077] In some embodiments, once the dialogue system renders the virtual scene, it does not change the virtual scene until the dialogue is complete. In some embodiments, scene modeling can also be used in scenes that can combine virtual scenes and enhance dialogue scenes. For example, in a virtual scene created based on a selected scene model, additional virtual objects can be projected as needed, such as throwing 3 bears and 5 dolphins into the virtual dialogue scene and asking the user to perform the addition.

[0078] Fig. 9 is an illustrative diagram of an exemplary mobile device architecture that can be used to implement a dedicated system for implementing the present teachings according to various embodiments. In this example, the user device on which the present teachings are implemented corresponds to the mobile device 900, including but not limited to a smart phone, a tablet computer, a music player, a handheld game console, a global positioning system (GPS) receiver, and a wearable computing device (e.g., glasses, a wristwatch, etc.), or any other form of device. The mobile device 900 may include one or more central processing units ("CPUs") 940, one or more graphics processing units ("GPUs") 930, a display 920, a memory 960, a communication platform 910 (such as a wireless communication module), a storage device 990, and one or more input / output (I / O) devices 940. Any other suitable components, including but not limited to a system bus or a controller (not shown), may also be included in the mobile device 900. As shown in FIG. Fig. 9 As shown in , a mobile operating system 970 (e.g., iOS, Android, Windows Phone, etc.) and one or more applications 980 may be loaded from storage 990 into memory 960 for execution by CPU 940. Application 980 may include a browser or any other suitable mobile application for managing a conversation system on mobile device 900. User interaction may be implemented via I / O device 940 and provided to the automated conversation partner via network(s) 120.

[0079] In order to realize the various modules, units and their functions described in the present disclosure, a computer hardware platform can be used as one or more (one or more) hardware platforms for the elements described herein. The hardware components, operating systems and programming languages ​​of such computers are conventional in nature, and it is assumed that those skilled in the art are fully familiar with these technologies so that those technologies are adapted to the appropriate settings as described herein. A computer with a user interface element can be used to realize a personal computer (PC) or other types of workstations or terminal devices, but if properly programmed, a computer can also serve as a server. It is believed that those skilled in the art are familiar with the structure, programming and general operation of this computer equipment, so the accompanying drawings should be self-explanatory.

[0080] Fig.101000 is an illustrative diagram of an exemplary computing device architecture that can be used to implement a dedicated system that implements the present teachings according to various embodiments. Such a dedicated system incorporating the present teachings has a functional block diagram illustration of a hardware platform that includes a user interface element. The computer can be a general purpose computer or a special purpose computer. Both can be used to implement the dedicated system of the present teachings. This computer 1000 can be used to implement any component of a conversation or dialog management system, as described herein. For example, the conversation management system can be implemented on a computer such as computer 1000 via its hardware, software program, firmware, or a combination thereof. Although only one such computer is shown for convenience, the computer functions associated with the conversation management system described herein can be implemented in a distributed manner on multiple similar platforms to distribute the processing load.

[0081] For example, the computer 1000 includes a COM port 1050, which is connected to a network connected thereto to facilitate data communication. The computer 1000 also includes a central processing unit (CPU) 1020 in the form of one or more processors for executing program instructions. The exemplary computer platform includes an internal communication bus 1010, program storage devices and data storage devices in different forms (e.g., disks 1070, read-only memory (ROM) 1030 or random access memory (RAM) 1040), various data files for processing and / or transmission by the computer 1000, and program instructions that may be executed by the CPU 1020. The computer 1000 also includes an I / O component 1060, thereby supporting input / output flows between the computer and other components therein (such as user interface elements 1080). The computer 1000 can also receive programming and data via network communications.

[0082] Thus, as outlined above, aspects of the methods of dialog management and / or other processes may be implemented in programming. The programmatic aspects of the technology may be considered a "product" or "article of manufacture," typically in the form of executable code and / or associated data carried or implemented in a machine-readable medium. Tangible, non-transitory "storage device" type media include any or all memory or other storage devices for computers, processors, etc., or their associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which may provide storage at any time for software programming.

[0083] All or part of the software may sometimes be transmitted over a network such as the Internet or various other telecommunication networks. For example, such communication may enable software to be loaded from one computer or processor into another, for example, in connection with conversation management. Thus, another type of medium that may carry software elements includes optical, electrical, and electromagnetic waves, such as across physical interfaces between local devices, through wired and optical landline networks, and through various air links. Physical elements that carry such waves (such as wired or wireless links, optical links, etc.) may also be considered as media that carry software. As used herein, unless limited to tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0084] Therefore, machine-readable media can take a variety of forms, including but not limited to tangible storage media, carrier media, or physical transmission media. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in (one or more) any computer, etc., which can be used to implement the system or any component thereof, as shown in the figure. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that form buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transport data or instructions, cables or links that transport such carriers, or any other media from which a computer can read programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution.

[0085] Those skilled in the art will recognize that the present teachings are susceptible to various modifications and / or enhancements. For example, while the various component embodiments described above may be implemented in a hardware device, it may also be implemented as a pure software solution—for example, installed on an existing server. Furthermore, the fraudulent network detection techniques disclosed herein may be implemented as firmware, a firmware / software combination, a firmware / hardware combination, or a hardware / firmware / software combination.

[0086] Although the above has described what is considered to constitute the present teaching and / or other examples, it should be understood that various modifications may be made thereto and the subject matter disclosed herein may be implemented in various forms and examples, and that the teaching may be applied to many applications, only some of which are described herein. The following claims are intended to claim any and all applications, modifications, and variations that fall within the true scope of the present teaching.

Claims

1. A method implemented on at least one machine, characterized in that The at least one machine includes at least one processor, a memory, and a communication platform capable of connecting to a network for an adaptive conversation, the method comprising: Receiving one or more images, the one or more images capturing surroundings of a conversation scene in which a user and a machine are engaged in a conversation based on a conversation strategy; Detecting one or more objects present in the conversation scene from the one or more images via a first type of artificial neural network; detecting, via a second type of artificial neural network, a spatial relationship between the one or more objects based on the one or more images; and generating scene modeling information representing the dialogue scene based on the one or more objects and the spatial relationship, wherein the scene modeling information is used for adaptive dialogue; Also included is determining when to adjust the conversation by evaluating at least one of: whether the conversation needs to be adjusted; and whether the dialogue scene is to be enhanced to allow adjustment of the dialogue in the enhanced dialogue scene; The method further comprises: when the dialogue scene is to be enhanced, selecting one or more virtual objects to be rendered in the conversation scene; Based on the scene modeling information, determining parameters for rendering the one or more virtual objects; and projecting the one or more virtual objects in the conversation scene according to the parameters to create the enhanced conversation scene; It also includes: adjusting the dialogue in the enhanced dialogue scenario according to an enhanced dialogue strategy consistent with the enhanced dialogue scenario.

2. The method according to claim 1, characterized in that The first type of artificial neural network is a convolutional neural network (CNN), which is configured based on a CNN-based object / feature detection model, which is trained via supervised learning based on labeled objects detected from training images.

3. The method according to claim 1, characterized in that The second type of artificial neural network is a graph neural network GNN, which is configured based on a spatial relationship detection model of the GNN, and the GNN-based spatial relationship detection model is trained through supervised learning based on the spatial relationship of labels between objects detected from training images.

4. The method according to claim 1, characterized in that The step of adjusting the dialogue in the enhanced dialogue scenario comprises: generating the enhanced dialogue strategy based on the enhanced dialogue scenario and the dialogue strategy; and Based on the enhanced dialogue strategy, the dialogue is conducted in the enhanced dialogue scenario.

5. A machine-readable and non-transitory medium, characterized in that Information for adaptive dialogue is recorded thereon, wherein when the information is read by a machine, the machine executes the following steps: Receiving one or more images, the one or more images capturing surroundings of a conversation scene in which a user and a machine are engaged in a conversation based on a conversation strategy; Detecting one or more objects present in the conversation scene from the one or more images via a first type of artificial neural network; detecting, via a second type of artificial neural network, a spatial relationship between the one or more objects based on the one or more images; and generating scene modeling information representing the dialogue scene based on the one or more objects and the spatial relationship, wherein the scene modeling information is used for adaptive dialogue; The information, when read by a machine, also causes the machine to perform the following steps: Determining when to adjust the conversation is done by evaluating at least one of the following: whether the conversation needs to be adjusted; and whether the dialogue scene is to be enhanced to allow adjustment of the dialogue in the enhanced dialogue scene; When the information is read by a machine, the machine further executes the following steps: when the dialogue scene is to be enhanced, selecting one or more virtual objects to be rendered in the conversation scene; Based on the scene modeling information, determining parameters for rendering the one or more virtual objects; and projecting the one or more virtual objects in the conversation scene according to the parameters to create the enhanced conversation scene; When the information is read by a machine, it also causes the machine to perform the following steps: adjusting the dialogue in the enhanced dialogue scenario according to an enhanced dialogue strategy consistent with the enhanced dialogue scenario.

6. The medium according to claim 5, characterized in that The first type of artificial neural network is a convolutional neural network (CNN), which is configured based on a CNN-based object / feature detection model, which is trained via supervised learning based on labeled objects detected from training images.

7. The medium according to claim 5, characterized in that The second type of artificial neural network is a graph neural network GNN, which is configured based on a spatial relationship detection model of the GNN, and the GNN-based spatial relationship detection model is trained through supervised learning based on the spatial relationship of labels between objects detected from training images.

8. The medium according to claim 5, characterized in that The step of adjusting the dialogue in the enhanced dialogue scenario comprises: generating the enhanced dialogue strategy based on the enhanced dialogue scenario and the dialogue strategy; and Based on the enhanced dialogue strategy, the dialogue is conducted in the enhanced dialogue scenario.

9. A system for adaptive dialogue, characterized in that: include: a dynamic dialogue scene modeling unit configured to receive one or more images, the one or more images capturing surroundings of a dialogue scene in which a user and a machine participate in a dialogue based on a dialogue strategy; an object / feature detection unit configured to detect one or more objects present in the conversation scene from the one or more images via a first type of artificial neural network; an object spatial relationship detector configured to detect a spatial relationship between the one or more objects based on the one or more images via an artificial neural network of a second type; as well as a scene model generating unit, configured to generate scene modeling information representing the dialogue scene based on the one or more objects and the spatial relationship, wherein the scene modeling information is used for adaptive dialogue; Also included is a dialog manager configured to determine when to adjust the dialog by evaluating at least one of: whether the dialogue needs to be adjusted; as well as whether the dialogue scene is to be enhanced to allow adjustment of the dialogue in the enhanced dialogue scene; The device further comprises an enhanced scene generation unit, wherein the enhanced scene generation unit is configured to: when the dialogue scene is to be enhanced, selecting one or more virtual objects to be rendered in the conversation scene; Based on the scene modeling information, determining parameters for rendering the one or more virtual objects; as well as projecting the one or more virtual objects in the conversation scene according to the parameters to create the enhanced conversation scene; The dialog manager is further configured to adjust the dialog in the enhanced dialog context according to an enhanced dialog strategy consistent with the enhanced dialog context.

10. The system according to claim 9, characterized in that The first type of artificial neural network is a convolutional neural network (CNN), which is configured based on a CNN-based object / feature detection model, which is trained via supervised learning based on labeled objects detected from training images.

11. The system according to claim 9, characterized in that The second type of artificial neural network is a graph neural network GNN, which is configured based on a spatial relationship detection model of the GNN, and the GNN-based spatial relationship detection model is trained through supervised learning based on the spatial relationship of labels between objects detected from training images.

12. The system of claim 9, wherein: It also includes an enhanced dialogue strategy generator, which is configured to generate the enhanced dialogue strategy based on the enhanced dialogue scenario and the dialogue strategy, wherein the dialogue manager is further configured to conduct the dialogue in the enhanced dialogue scenario based on the enhanced dialogue strategy.

Images