Machine Interaction

By creating an autonomous agent with a virtual environment in human-computer interaction, the problem of difficult natural interaction methods is solved, and the computer vision and interaction modules are used to realize a natural user interface experience, reducing technical obstacles, and enhancing the interaction between users and computers Effect.

CN112424736BActive Publication Date: 2024-11-19SOUL MACHINES LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980048089.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-07-03
Filing Date
2019-07-19
Publication Date
2024-11-19
Estimated Expiration
2039-07-19

Smart Images

  • Figure CN112424736B_ABST
    Figure CN112424736B_ABST
Patent Text Reader

Abstract

Interaction with a computer is provided via an autonomous virtual embodied agent. The computer outputs digital content, which includes any content that exists in the form of digital data and can be presented to a user. A subset of the digital content or all of the digital content is configured as shared digital content that can be presented to both the user and the agent.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The use of computer science techniques to facilitate interaction between humans and machines, and more particularly, but not exclusively, the use of embodied agents to facilitate human-machine interaction. Background Art

[0002] Human-computer interaction (HCI) is a field of computer science that involves using techniques from computer graphics, operating systems, software programming, and cognitive science to improve the interaction between humans and computers. Autonomous agents can improve human-computer interaction by assisting human users in operating computers. Since autonomous agents are able to perform flexible and / or autonomous actions in an environment (which can be virtual or real), autonomous interface agents can be considered as robots whose sensors and effectors interact with the input and output capabilities of a computer interface. Autonomous interface agents can interact with a computer interface in parallel with a user, with or without user instructions or collaboration.

[0003] EP1444687B1 entitled "Method for managing hybrid active human-computer dialogue based on interactive speech" discloses a method for managing hybrid active human-computer dialogue based on speech interaction. EP2610724A1 entitled "System and method for online user assistance" discloses the display of virtual animation characters for online user assistance.

[0004] It is expected that: autonomous interface agents will use artificial intelligence techniques to intelligently process information and interact and present themselves in a more human-like manner. Human users may find it easier, faster, and / or more interactive with computers using human-like communication methods (including body language and voice intonation). Second, increasing the realism of agent actions and responses can reduce perceived technical barriers, such as the uncanny valley effect.

[0005] Purpose of the Invention

[0006] It is an object of the present invention to improve human-machine interaction (HMI) or at least to provide the public or industry with a useful choice. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 shows user-agent interaction in a shared interaction environment;

[0008] Figure 2 shows the user-agent interaction from the agent's perspective;

[0009] Figure 3 A schematic diagram of human-computer interaction is shown;

[0010] Figure 4illustrates the interaction of a user agent with digital content behind the agent;

[0011] Figure 5 A system diagram for human-computer interaction is shown;

[0012] Figure 6 A process diagram for human-computer interaction is shown;

[0013] Figure 7 a flow chart showing a feedback loop between an agent and an environment;

[0014] Figure 8 shows user interaction in a virtual reality context;

[0015] Fig. 9 A system diagram for human-computer interaction in a virtual reality context is shown;

[0016] Fig.10 shows user interaction in an augmented reality context;

[0017] Fig.11 A system diagram for human-computer interaction with a mobile application is shown;

[0018] Figures 12 to 16 Screen shots showing human-computer interaction via an agent;

[0019] Fig.17 A schematic diagram showing a computer vision system using an agent to interact with digital content;

[0020] Fig.18 Digital content associated with the embedded action and / or agent-perceptible locator is shown;

[0021] Fig.19 An example is shown of how a representation of a user captured by a camera may be displayed on an end-user display;

[0022] Fig. 20 An example of rotation is shown;

[0023] Fig.21 A user interface for setting a significance value is shown;

[0024] Fig. 22 An attention system for multiple visual feeds is shown;

[0025] Fig.23 A class diagram illustrating an example implementation of a user-agent interaction. DETAILED DESCRIPTION

[0026] Interaction with a computer is provided via an autonomous virtual embodied agent (hereinafter referred to as an "agent"). The computer outputs digital content, which includes any content that exists in the form of digital data and can be presented to a user. A subset of the digital content or all of the digital content is configured as shared digital content that can be presented to both the user and the agent. Figure 1 A user 3 and agent 1 are shown perceiving shared digital content 5. The shared digital content 5 can be manipulated and / or perceived (interacted with) by both the user 3 and the agent 1. The shared environment 6 includes all of the shared digital content 5 and is perceivable by both the agent 1 and the user 3. The agent 1 can "physically" interact with the shared digital content 5 within its agent virtual environment 8 (using an embodiment thereof).

[0027] The agent 1 is in and aware of its agent virtual environment 8. Shared digital content 5 is presented to the agent 1 within the agent virtual environment 8 (AYE). The plane 14 of the agent virtual environment 8 maps to an area on a computer screen (end-user display device 10). The agent virtual environment 8 may optionally include AVE objects 11, with which the agent 1 can interact. Unlike the shared digital content 5, the user 3 cannot interact directly with the AVE object, but the AVE object is presented to the user 3 as part of the agent 1's environment in a compressed view displayed to the user 3 as shown in 11RER. The agent 1 can move around its agent virtual environment 8 and interact with digital objects and AVE objects within the context of the physical laws of the agent virtual environment 8. For example, the agent 1 can pull down the shared digital content 5 and walk to and sit on the AVE object 11 (virtual stool). The agent 1 can also see a presentation of the user 3 (3VER) and the real world environment 7 superimposed with and / or concurrent with the shared digital content 5, as if the agent 1 were "inside" the screen and looking out at the real world.

[0028] The user 3 is a human user 3 in a real world environment 7 and sees and controls (interacts with) content presented on an end-user display device 10 (a real world display, such as a screen) from a computer. The end-user display device 10 displays a compressed, overlaid and / or blended view of the real world environment presentation (RER) of the shared digital content 5RER (e.g., from a web page via a browser) and the environment (AVE) of the agent 1, including:

[0029] Agent 1’s real-world environment representation (1RER)

[0030] AVE Object 11 Real World Environment Representation (11RER)

[0031] When the user 3 changes the shared digital content 5 via the user interface, the change is reflected in the presentation of the shared digital content 5 in the agent virtual environment 8 of the agent 1. Likewise, when the agent 1 makes a change to an object (item / component of the shared digital content 5) in the agent virtual environment 8, the change is mirrored in the presentation of the shared digital content 5 on the screen 10 of the user 3.

[0032] The agent and its environment can be visualized in different ways on the end-user display device. For example, a 2D plane can be displayed from the agent's perspective by rotating the agent's virtual space. Figure 2 The user-agent interaction is shown from the perspective of agent 1. Agent 1 is simulated within a three-dimensional agent virtual environment 8. Shared digital content 5 is located within the agent virtual environment 8, relatively in front of agent 1 from the user's perspective. Agent 1 perceives a virtual environment presentation of the user 3VER (e.g., via a camera feed), which may optionally also be simulated as a plane or otherwise overlaid / fused in the agent virtual environment 8 or with the shared digital content 5 plane. Agent 1 can thus perceive the relative positioning of the digital content to the user, and ascertain whether the user is viewing the shared digital content 5. Agent 1 is shown extending its hand to touch the shared digital content 5.

[0033] The plane 14 may be located anywhere in the proxy virtual environment 8 . Figure 4 An agent 1 is shown facing the user 3 and a plane 14 containing shared digital content 5 (object) behind the agent 1. The representation of the agent 1 may be shown in front of a real world environment representation (RER) of the shared digital content 5 RER. The user 3 sees the representation of the agent 1 interacting with the shared digital content 5 as if the agent 1 were in front of the screen.

[0034] Figure 3 A schematic diagram of human-computer interaction is shown, wherein an agent 1 facilitates interaction between a user 3 and digital content 4. The user 3 may interact directly with the digital content 4 from the computer 2 by providing input to the computer 2 via any suitable human input device / host input device and / or interface device and receiving output from the computer 2 from any suitable human output device (e.g., a display or speaker). The agent 1 may similarly provide input to the computer 2 and receive output from the computer 2.

[0035] Providing the agent with the perception of objects outside of its simulated environment is similar to providing the agent with "augmented reality" in that it provides the agent with a real-time presentation of the current state of real-world elements and the computer, and blends the presentation of digital content items from the computer, real-world elements, in a unified view of its world. When the virtual world is augmented with real-world objects, this can be more accurately defined as augmented virtualization, which enhances the autonomous agent's vision relative to human vision.

[0036] Agents interact autonomously with digital and real-world input

[0037] The agent may include cognitive, positional, embodied, conversational and dynamic aspects. The agent is embodied, meaning that it has a virtual subject that the agent can express. The agent's body is graphically presented on a screen or other display. The agent may be simulated using a neurobehavioral model (a "brain" or nervous system of biological modeling), which includes multiple modules with coupled computational and graphical elements. Each module represents a biological process and includes computational elements associated with and simulating the biological process and graphical elements that visualize the biological process. Therefore, since no external control is required, the agent is "self-dynamic" and presents naturally occurring automatic behaviors, such as breathing, blinking, looking around, yawning, and moving its lips. Biologically based autonomous animations may be achieved by modeling multiple aspects of the nervous system, including but not limited to sensory and motor systems, reflexes, perception, emotional and regulatory systems, attention, learning and memory, rewards, decisions and goals. The agent's face reflects both the agent's brain and the agent's body, revealing mental states (such as mental attention via the direction of its eyes) and physiological states (such as fatigue, through the position of the eyelids and the color of the skin). The use of neurobehavioral models to animate virtual objects or digital entities is further disclosed in: Sagar, M., Seymour, M., & Henderson, A. (2016), Creating connections using autonomous facial animation, Communications ACM, 59(12), 82-91; and WO 2015016723 A1, which is also assigned to the assignee of the present invention and is incorporated herein by reference. The time-stepping mechanism described in WO 2015016723 A1 can synchronize or coordinate the internal processes of the agent.

[0038] Aspects of the agent can be dynamic, meaning that the agent's future behavior depends on its current internal state. The agent's nervous system, body, and its environment are coupled dynamic systems. Complex behaviors can be generated with or without external input. Complex behaviors can be generated without relying on any internal representation of the agent's external environment. Elements of the agent's surroundings (such as visual stimuli or content items from the user) can form part of a causal network that leads to behavior. Agent behavior and resulting animations can emerge from the bottom up. The dynamics of the agent can be chaotic, making it difficult to predict how the agent will behave in a given situation. The agent's actions rely on a continuous feedback loop from both the real world environment 7 and the agent's virtual environment 8, such as Figure 7. Thus, the agent 1 is in two environments because it receives input 1102 from the real world environment 7, such as audio input (such as via a microphone) from the user and the user's environment (real world environment 7), and visual input (via a camera) from the user. The real world input 1102 simulated for the agent 1 affects the behavior of the agent 1, which results in the agent 1 behavior shown in the agent 1 animation output 1106. For example, when the agent 1 sees a human user via an indirect mechanism, the agent 1 may smile, where the recognition of the user releases a virtual neurotransmitter within the neurobehavioral model, such as dopamine, which can naturally trigger the smile of the agent 1. This output 1106 can in turn affect the real world environment 7, for example by causing a response or emotion in the user in the real world environment 7, which is again input for the agent 1 simulation. The agent virtual environment 8 of the agent 1 is also input 1102 for the agent 1. For example, a shared digital content item within the agent virtual environment can be perceived by the agent 1 as a stream in real time and affects the behavior of the agent 1, which also results in a specific agent 1 animation output 1106. This output 1106 can affect the agent virtual environment 8. For example, agent 1 may change the position of a virtual object within the agent virtual environment 8. The new object position is now input 1102 for the agent 1 simulation to drive the continuous feedback loop.

[0039] The agent may be simulated in any suitable manner and / or presented in any suitable form, such as: human form, fictional human form, anthropomorphic features, robot, animal. The user may be able to select or change the form taken by their agent, or the agent may change its form depending on the user or the real or virtual environment.

[0040] Move to coordinate positioning

[0041] Placing the agent in the agent virtual environment 8 allows for natural-looking "on-the-fly" animations, and allows the agent to freely interact with the surrounding digital content items (shared digital content 5) as if they existed in the agent's 1 environment. This is different from pre-generated or pre-recorded animation clips, which are disadvantageously difficult to use in interactions with unpredictable or dynamic content. An example of a limitation of pre-recorded animations is the use of a simulated arm to reach a specific content item at a location (X, Y). The animation will be limited in functionality given where exactly the content item is located; if the content item moves or changes size, the animation will not be able to change accordingly.

[0042] The agent can be animated using an "on-the-fly" animation algorithm. In an embodied agent, an effector (e.g., a hand) that reaches an end target position (digital object coordinates) is implemented by calculating a vector of joint degrees of freedom that brings the end effector to the target state. The agent 1 can be animated using computational techniques such as inverse kinematics. Inverse kinematics can be approximated using known techniques such as Jacobian or cyclic coordinate descent (CCD). A neural network can be trained to map body positions to render reaching or pointing to an object in a two-dimensional or three-dimensional coordinate space (i.e., learning hand-eye coordination). In one example, a move success predictor uses a deep convolutional neural network (CNN) to determine how a given motion can successfully reach a specified coordinate, and a continuous servomechanism that uses the CNN to continuously update the agent's motion commands.

[0043] Virtual Environment

[0044] The agent is located in the agent virtual environment and can perceive objects and the relative positioning of the objects and the agent in the agent virtual environment. In most embodiments described herein, the agent is located in a three-dimensional agent virtual environment. In one embodiment, the agent virtual environment is a 2D virtual space presented by a 2D pixel array. The 2D agent can be located in the 2D virtual space and interact with shared digital content on the 2D space, which can be displaced horizontally (left or right) and / or vertically (up or down) from the agent.

[0045] exist Figure 1 In FIG. 8 , the agent 1 is located in a 3D space behind a plane 14 (screen) reflecting the shared environment. The agent 1 sees the shared digital content on the plane 14 at the outer boundary / face of the agent virtual environment 8 (the agent virtual environment is a rectangular column). Figure 2 Agent 1 is shown touching a content item directly in front of Agent 1 . Figure 4 The agent 1 is shown in front of a plane 14 in which the shared environment digital content item on the screen is located. The content item / object being the shared digital content can be displaced from the agent 1 horizontally (left, right, front or back) and / or vertically (up or down) in three dimensions relative to the agent 1.

[0046] The coordinates of the plane 14 of the virtual 3D space may be mapped to browser (or other display) coordinates. Thus, when the agent 1 movements associated with the shared digital content are overlaid on the screen, the movements are aligned with the digital content item position.

[0047] The real world coordinates of the display on which the agent is displayed can be changed. For example, the agent displayed on the browser can be moved or resized, such as by the user dragging / resizing the browser. The real world physical size of the display is dynamically changeable, and changing the physical size of the display in the real world environment proportionally updates the rendering of the agent virtual environment.

[0048] The agent virtual environment can be simulated using any appropriate mechanism and can follow certain natural laws. The physical properties of the agent virtual environment can be defined, such as: collision detection between objects and / or agents, physical constraints (e.g., the allowed movement between the joints of an object), atmospheric drag, momentum, gravity, material properties (e.g., elastic items can return to their natural shape after being stretched by an agent).

[0049] The input received by the agent

[0050] The agent receives any suitable visual input from the real world (such as a video stream from a camera), depth sensing information from a camera with range imaging capabilities (such as a Microsoft Kinect camera), auditory input such as from a microphone, input from a biosensor, thermal sensor or any other suitable input device, touch screen input (where the user is pressing the screen). A single input may be provided to the agent, or a combination of inputs may be provided. The agent may spatially relate itself to aspects of the real world, and the user may bring the agent's attention to objects (such as things or people) in the user's real world space.

[0051] The agent may receive input / communication from the user via a computing device. The agent may perceive content items, such as the user's cursor / mouse or touch screen input (which is also input from the real world in a way). The agent may be able to follow the user's mouse movements with their eyes and / or hands. In some embodiments, the agent may be able to perceive keyboard input. For example, a user may communicate with the agent via a keyboard rather than speech.

[0052] The visual system of the agent is provided with visual (e.g., video stream) input. Routines from appropriate interfaces can be used for hardware interfaces. This can be provided via the interface between the programming definition of the agent and the camera and / or pixel stream converter. The interface is provided by a vision module, which provides a module package to collect image data from the camera at each time step of the agent. The vision module may not perform any perceptual processing of the image data (similar to the human "retina").

[0053] The agent can provide output to the real world through the output devices of the computing system, such as visual output on a display, audible output through speakers, or any other suitable means. The agent can also output mouse movements, keyboard strokes, or other interface interactions, such as clicks or presses.

[0054] An agent can have two or more visual systems, allowing the agent to receive visual input from different sources simultaneously without overlaying different visual sources onto each other. For example, an agent can have two visual modules, one for camera input and one for user interface input, and view both simultaneously. Different saliency maps can be applied to each input source. For example, a saliency map that prioritizes faces can operate on a visual system configured to receive real-world input (such as camera input), while a saliency map that prioritizes recognition of text and / or recognition of UI elements can be applied to a visual system configured to receive UI input.

[0055] Digital Content

[0056] Shared digital content associated with a website may include site forms, buttons, text boxes, text fields, video elements, or images. In mixed reality applications such as augmented reality, virtual reality, holograms, etc., digital content items may refer to 3D objects. In applications (such as VR / mobile / computer applications), digital content items may be objects defined by object-oriented languages.

[0057] Digital content can be dynamic (e.g., items of digital content can move on a screen), and agents can follow or direct the movement of such dynamic content items. Digital content items can be static or interactive. Static content items can be text or images. For example, an agent participating in academic tutoring of a user can interact with the text by pointing to a word displayed to the user and asking the user whether the user understands the meaning of the word. Interactive digital content items can respond in some way to the actions of the user and / or the agent. For example, a button is a digital content item that displays additional content when clicked. Digital content items can be presented in two or three dimensions. For example, Fig.15 A car object modeled in 3D and located within the proxy virtual environment is shown.

[0058] Because the agent is embodied in the virtual space along with the content items, the agent can interact with the shared digital content in a number of different ways, using gestures, viewing items, approaching items, manipulating or processing items, or invoking methods associated with items, in a flexible and scalable manner. The agent can:

[0059] Lean their body towards the item and / or tilt their head towards the item.

[0060] View the item with its eyes.

[0061] • Gesture toward an item by pointing at it, shaking their head in the general direction of the item, or waving in the direction of the item.

[0062] Approaching the item by walking toward it or teleporting or floating to a location close to the item.

[0063] Enable agent interaction with shared digital content

[0064] The embodied agents are programmatically defined independently of the digital content with which they interact: there is no central controller controlling both the digital content and the embodied agents. This allows for flexible use of embodied agents with digital content licensed by different third-party vendors. Two mechanisms that can enable agents to perceive new digital content and interact with it locally include computer vision and interaction modules (support interfaces):

[0065] Computer Vision

[0066] In one embodiment, the agent receives input in the form of visual data from a source presented to the user. For example, the agent views the pixels of a screen or 3D virtual space via visual (virtual) images and / or object recognition. The agent may be equipped with visual object recognition such as by standard computer vision / image processing / image recognition / machine learning technology to identify the outline / color of the object / subject in the image or video or object, the subject of the image or video, and interact accordingly. The agent may be equipped with optical character recognition for identifying text and natural language processing for understanding text. In other words, digital content is visually presented to the agent in the same manner as presented to a human user (visual display of pixels). The agent system may include built-in image recognition and / or learning, or it may use third-party services for image recognition.

[0067] Digital content input

[0068] The visual representation of the shared environment may be sent to the proxy in a manner similar to how screen sharing software sends a visual representation of the screen to a remote location. The interface may be configured to send information packets from the computing device to the proxy that describe what is being output by the computing device at any given time. The data may arrive as image files (e.g., JPEGs and GIFs), or the data may arrive as individual pixels assigned to specific X and Y coordinates (and Z coordinates in the case of mixed reality). To minimize the amount of bandwidth, the interface may be configured to send information updates only on portions of the screen that have changed and / or to compress the data that is sent.

[0069] Fig.17 A schematic diagram of a computer vision system using an agent interacting with digital content is shown. The modules involved need not be modular components of code, but can be functional networks of modules driven by highly interconnected neurobehavioral models. At 2901, an end-user computing device user interface (such as a browser) renders an update to the display (e.g., a web page refresh). Via a shared memory buffer 2930, an agent 1 controller 2951 provides pixels from the display as input 2902 to the agent's retina 2903. This is an interface between the programmatic definition of the agent and the pixel stream converter. These pixels can then be provided to a cognitive module 2952, in which a visual inspection module 2904 performs image processing. The concerned / processed portion of the pixel stream can be guided by an attention module 2905, which determines what the agent is paying attention to. Information generated from the processing of pixel data can be passed to a reactive and / or decision-making module 2907. This drives the agent's behavior. For example, after identifying a portion of an image as a button, a reactive and / or decision-making module 2907 can cause the agent to reach out and touch a button 2910. The actions or behaviors performed by the agent are passed to the simulated physiological module 2908, which can have subcomponents for controlling various parts of the body, including arm control 2910. At 2911, the agent controller 2951 can operate to map the interaction between the agent and the digital content into a user 3 interface interaction. In the virtual environment, the "physical" actions of the "agent" in the AVE can be converted into actions on the user 3 interface. For example, when the coordinates of the agent's body intersect with the perceived shared environment plane, the coordinates of the intersection of the plane can be converted into a mouse click or touch pad touch event at the corresponding pixel location on the user interface. Then the event action / back channel 2932 is sent to the computing system as a human input device event (e.g., a mouse click at the corresponding pixel location on the browser). In one embodiment, the chromium embedded framework (CEF) open browser embodiment is suitable for enabling the agent to interact with the site digital content. Off-screen rendering allows the content of the browser window to be output to a bitmap that can be rendered elsewhere.

[0070] The concrete interactions corresponding to the (triggering) events

[0071] Touching a digital content item is one type of interaction that can result in interacting with the digital content item. embodied In other embodiments, an agent may interact with a digital content item, but the invention is not limited in this regard. In other embodiments, a specific gesture of an agent to a digital content item may trigger an event on the item. For example, an agent viewing an item and blinking may trigger an event on the item. Another example is gesturing to a digital content item (e.g., a button).

[0072] Directly control the browser or computing system

[0073] In one embodiment, the agent directly controls the mouse, touchpad, or other primary input device as if the input device were the agent's effector. In other words, the agent can control the input device in the same way that the agent controls its own body / muscle movements. In a computing device, direct control of the computing device by the agent can be achieved through any suitable technology (e.g., such as enabling remote access and remote collaboration on a person's desktop computer through a graphical terminal emulator).

[0074] Saliency Mapping

[0075] The attention module 2905 may include a "saliency" map to direct the agent's attention. A saliency map is a screen presentation of the agent according to importance. A saliency map may define where the agent's attention and focus is. Examples of features that may be distinguishably considered salient or insignificant include:

[0076] User: The agent may include a face detection module to detect faces. Face detection may be relevant to the agent's emotional affect and user interaction loops. The face detection module uses a face tracking and resolution library to find faces in the agent's visual input stream. The presence of a face may be interpreted by the agent as a highly salient visual feature. The resolved facial expressions from any detected faces may be fed into the expression recognition network.

[0077] Movement - A motion detection module may be provided, since the vision module does not attempt any perceptual processing of the video input. The motion detection module may be a component of the agent's visual perception system that compares temporally adjacent video frames to infer simple motion. The resulting "motion map" may be used as a driving element for visual saliency

[0078] Recognition of certain objects or images

[0079] Identification of text; attribution of saliency to certain keywords or text patterns

[0080] ·color

[0081] ·brightness

[0082] ·edge

[0083] The saliency map may be user defined. In one embodiment, the user may interactively communicate to the agent the features that the user wishes the agent to consider salient (to focus on). Where multiple saliency maps are used, each saliency map may be weighted with the ultimate focus of the agent's attention driven by a weighted combination of each active saliency map. In other embodiments, saliency may be defined externally and a manual pointer provided to mark a digital content item for the agent's attention.

[0084] Switching UI controls between user and agent

[0085] Both the user and the agent can control the mouse, keyboard, or other primary input mechanism. In one embodiment, a mechanism for collaboration is provided in the form of a control mechanism that ensures that once either party moves the mouse, the other party is not allowed to move the mouse until the initiated action is completed. The control mechanism can enforce rotation between the user and the agent. In other embodiments, the user and agent can use a dialog to determine who controls the UI (e.g., the user can ask the agent if they can take over control and vice versa).

[0086] Perception Control

[0087] The agent can control its perceptual input. For example, it can choose to look at the user instead of the content, or vice versa. The advantage of enabling the agent to perceive via visual pixel recognition is that it provides freedom / flexibility in what the agent can view / focus on and therefore perceive (using a "fovea" subset / region of the pixel stream displayed to the agent at a higher resolution). The agent can focus on any pixel displayed on the user interface, or any superstructure / aspect of the content displayed on the user interface, such as the patterns created, the colors, or the objects presented.

[0088] Integration with Agent Simulator

[0089] Fig.23 Shown is a class diagram of an embodiment using a CEF browser. The CEF window defines what the user sees on the UI. Variables can be defined in the neurobehavioral model of the agent to store interactions with shared digital content (e.g., browser content). A variable group can be used for user interactions, and another variable group can store agent interactions. The agent's variable group can be set by the runtime through the neurobehavioral modeling language. The runtime host 2304 can set variable monitoring for these two variable groups. When receiving updates to these variables, the runtime host 2304 constructs UI events (e.g., mouse / keyboard events) and sends them to a shared environment VER (which can correspond to a plane in the agent's environment, through which the agent views shared digital content). The shared environment VER is owned by the runtime host, but is rendered off-screen into a buffer, and the runtime host sends the buffer to a neurobehavioral modeling framework (simulator), which is used for the neurobehavioral modeling framework to render content into 3D space).

[0090] When the user interacts with the browser, the UI sends set variable messages to the SDK, for example, user_mousedown, user_mouse_x. The coordinates received from the UI are related to the shared environment 2308 (the agent's window) that is rendered off-screen. The coordinates are converted to xy positions in the browser. The neurobehavioral model method converts the coordinates and determines whether the browser object contains a mouse. The runtime host then constructs the mouse and key events and forwards them to the shared environment (shared browser).

[0091] When the agent interacts with the shared environment 2308, the runtime host 2304 receives callbacks for the monitored changed variables. If there are no user events in the same callback, the agent's interaction is forwarded to the shared browser. Otherwise, the user's interaction overrides the agent's interaction. The NBL variables for shared interactions can be defined as:

[0092] Track mousedown and mouseup events: user_mousedown, user_mouseup, persona_mousedown, persona_rnouseup

[0093] Track keydown and keyup events with variables: user_keydown, user_keyup, persona_keydown, persona_keyup

[0094] To indicate that an event (e.g., a mouse event or a keyboard event) is occurring, the neurobehavioral modeling framework variables (e.g., user_mousseddown / user_mouseup / agent_mousedown / agent_mouseup) are different from the previous time step. Instead of using a 1 / 0 switch to indicate that an event is occurring, a counter counts the event and adds 1 to the previous value each time. Once the counter reaches 1000, the counter is reset to 1. The reason is because down and up (on, off) can be in the same time section to keep track of all events, and the current and previous values ​​of the variables do not need to match. This ensures that events are not lost. Queues can be implemented to facilitate fast and / or simultaneous input / output of events (faster than the agent's time step). Agent internal 2314 can control the rate of agent time stepping and update user interactions in the shared environment VER.

[0095] Interaction Module

[0096] The interactive module can promote the agent to perceive the shared digital content, and define the interactive function visibility of the content items presented in the virtual environment of the agent and transmit it to the agent. The interactive module can be a support library or an application programming interface (API). When the agent 1 decides to take an action, the interactive module 16 converts the action into a command defined locally by the author of the third-party digital content. The agent can interact directly and dynamically with the digital content (e.g., site content, application content or other programmatically defined content) by using the interactive module. The interactive module converts between the digital content defined by the computer program readable information and the agent understandable information.

[0097] The shared digital content project can be presented to the agent as a conceptual object, which is an abstraction of a local (or locally rendered) digital content project. The conceptual object can be defined by certain characteristics (such as virtual world environment coordinates, colors, identifiers, or other things related to the interaction between the agent and the corresponding digital content project). The conceptual object is an object that presents the reality digital content project in an abstract manner across AVE conversions. Agent 1 only needs to understand the "concept" and the metadata (i.e., size, color, positioning / position) associated with the concept.

[0098] The local digital content items exist and are presented to the user in their local format, however the digital content items have additional identification that the agent 1 can use to reference the digital content items. In one embodiment, the agent views the shared digital content items / objects via the interaction module 16, which converts HTML information about these objects and passes it to the agent 1. The interaction module 16 can extract the local digital content item information, such as HTML information, and convert it to the agent 1 so that the agent 1 can understand what the content item is, its characteristics, and what input may be required.

[0099] Figure 5A system diagram for facilitating human-computer interaction between a user 3 and a web page is shown. The system may include a client 512, a digital content provider server 520, and an agent (agent system or simulation) 510, and optionally, communicate with a third-party service 590. The digital content provider may define digital content on a site server 522 and / or a server-side database 524. The site server 522 may provide digital content to a site client 511, such as a site browser, that may be viewed by a user. Human-computer interaction is facilitated by including an interactive module 16 on the client 512. The agent is simulated on an agent system 510, which may be a cloud server. The interactive module 16 processes digital content (e.g., the digital content may be defined by an HIML code) and converts items related to the agent so that they may be perceived by the agent. This may provide the agent with: mapping the content item background on a web page. The agent and the agent's virtual environment reside on the agent system 510. The agent system 510 includes an agent modeling system 513 that simulates an agent within the agent's virtual environment, and an animation rendering system 514 that renders the display of the agent. The knowledge base 515 may provide the agent with a base level of domain knowledge about the environment in which the agent is located and the types of content items with which it may interact. The agent may be supported by a third party service 590, such as a natural language processing system provided by a third party.

[0100] Figure 6 A swim lane process diagram for human-computer interaction is shown. A digital content provider 620 defines a digital content item 621. The digital content provider 620 includes an interaction module 16 linked to the digital content item 621 so that the digital content item can support agent interaction. In order to enhance the digital content with agent 1 interaction, the digital content provider can enable this by linking to or including the interaction module 16 when defining the website. In another embodiment, if there is no link to the interaction module 16, a proxy server can be provided through which the digital content is run, and the digital content is linked to or includes the interaction module 16 that enables interaction via the agent 1.

[0101] The user device 612 locally displays the digital content items to the user, who interacts with the web page, application or other computer program defined by the digital content provider 620. The interaction module 16 converts some digital content items from their local definitions into conceptual objects. The conceptual objects are sent to the agent 1 cloud 610, allowing the agent 1 to conceptualize the digital content 616. The conceptual objects are input 617 for the agent 1 simulation. Therefore, the conceptual objects constitute part of the environment of the agent 1. Therefore, the agent 1 can interact with its behavior or change its behavior based on the conceptual objects. The agent 1 can directly manipulate the digital content 618, for example, press or move the conceptual object. When the agent 1 directly manipulates the conceptual object, the interaction module 16 converts the agent's action into a change to the digital content item. In other words, the interaction module 16 updates 642 the digital content to reflect the agent 1's changes to the content item. The agent can also interact indirectly with the conceptual object, such as by looking toward the location of the conceptual object or gesturing at the location of the conceptual object. The presentation of the agent 1 that directly or indirectly interacts with the conceptual object is rendered 619.

[0102] The agent may not directly perceive the content item in the same way that a user would (such as through pixel recognition). Interaction module 16 may pass a "conceptual object" such as an abstract representation of a content item to the agent. The conceptual object may include basic properties related to agent 1 (the agent 1's interaction with the item), such as a label that defines what the content item is and where it is located. Interaction module 16 may provide agent 1 with a list of functional affordances for the content item in the context of agent 1's virtual environment. The conceptual object corresponding to the content item may include further information defining an object with a "physical quality" that agent 1 can interact with.

[0103] The interaction module 16 can provide "action" support that allows the user 3 to interact with the digital content item via the agent. This "interaction" function provides an abstraction layer for the agent 1 to perform "actions" on the object. Examples of actions are pressing, dragging, pushing, viewing, grabbing, etc. These actions are converted by the interaction module 16 into an operation that can be performed on the digital content item at that moment and in a way that works on the interactive space. Operations that are not converted can be ignored. For example, the agent 1 can "action" press on the site button, which will be converted by the interaction module 16 into a click on the button HTML element. The "push" action on the site element may be ignored, but will cause the ball to move when an action is performed in the 3D interactive space on the ball.

[0104] For example, an agent who wishes to scroll down a web page can send a scroll down command to the interaction module, which in turn can convert the actions of the agent 1 into web-readable commands, such as JavaScript code. The JavaScript code actuates the actions on the web page. Therefore, there is no need for the agent to be able to communicate directly in the site language. This makes the system scalable because the agent 1 can be applied in different contexts.

[0105] In another example, an agent who wishes to enter text into a content item may send an input command to interaction module 16. Interaction module 16 may execute the necessary JavaScript commands to place a cursor within the text field and enter the text that the agent wishes to enter. Thus, in the context of web interaction, the user's view of the digital content item may be a full-fidelity web page with properly stylized site elements (e.g., HTML elements). The agent may have an abstract visualization of the HTML page composed of conceptual objects. Conceptually, this is similar to the agent seeing only a simplified view of aspects of the web page that are relevant to the agent's interaction with the user and the web page.

[0106] In addition to converting information from the site language into information that the agent can perceive, the interaction module 16 can convert the "physical" agent action on the conceptual object into an instruction for moving the corresponding digital content item. The agent manipulates the content item in its virtual environment, rather than directly manipulating the content item via a local method, as if they were physical items. For this effect, the interaction module 16 can also include physical conversion to move HTML elements in a simulated physical manner. For example, by moving an object by a certain amount with a certain number of frames per second to "simulate" physical push, the "physical" type action such as pushing the content item can be converted into an HTML element. Therefore, the various "actions" included in the interaction module 16 can directly implement or simulate (by approximation) the change of an element (such as its position, content or other metadata).

[0107] In some embodiments, agent 1 may send a query to interaction module 16 to obtain additional information about the item. For example, agent 1 wishing to "read" the text within an item (an item in HTML is a text field) may query interaction module 16 to obtain the text within its text field.

[0108] Dynamic Control

[0109] The digital content project may include a site element, which may have a set of parameters established by the document designer to define the initial structure and content of the element. These include the two physical features of the element, such as the absolute or relative spatial positioning of the element within the document, and the attributes of any user text content to be applied to the element, such as font type, font size, font color, and any font attributes such as bold and italic. The document may also be designed to allow the user to reposition one or more of the elements by traditional click-and-drag techniques. In the case where the digital content is in the context of a web page, an interactive module 16 such as a JavaScript interactive module 16 may be provided to enable the agent 1 to modify the physical features and / or attributes of the site element. After rendering the page via dynamic HTML (DHTML), a combination and interaction of several site-related standards (including HTML, CSS (Cascading Style Sheets), DOM (Document Object Model) and scripts), the elements in the HTML page may be controllable. The browser may create a document object model (DOM) that presents the HTML elements on the page when loading the web page. JavaScript may be used to interact with DOM (an interface in the browser that allows programs to access and change the content, structure and style of a document). The JavaScript interaction module 16 may include methods specifically implementing a specific type of interaction between the agent 1 and a web page through the DOM.

[0110] QuerySelector can be used to query the DOM. The interaction module 16 can enable the agent 1 to modify the web page in the following ways:

[0111] Change / delete HTML elements in the DOM or on the page

[0112] Change and / or add CSS styles to elements

[0113] Read and / or change element attributes (href attribute on anchor text, src attribute on image text, alt attribute or any custom attribute)

[0114] Create new HTML elements and insert them into the DOM / page

[0115] Attach event listeners to elements. For example, event listeners can listen for clicks, keypresses, and / or submits, and react to these in JavaScript.

[0116] Although the dynamic control of the web page has been described with reference to the JavaScript interaction module 16, the present invention is not limited in this respect. For example, in another embodiment, JQuery can facilitate the interaction between the agent 1 and the digital content. The interaction / support module can be implemented in any suitable site-related open technology standard.

[0117] Other interactive backgrounds

[0118] Figure 8 User 3 interaction is shown in a virtual reality context such as a virtual reality environment. The methods and systems for user interface interaction described above are similarly applicable to virtual / mixed / augmented reality interactions. A conceptual shared environment can also be provided, including a set of objects that both user 3 and Vent1 can access. The interaction module 16 can be used to convert between agent 1 space and user 3 space. The interaction module 16 can be embedded in a virtual reality application (VR application) with a virtual reality environment (VR environment). The interaction module 16 facilitates visual consistency in the interaction between the agent 1 and the digital content item. Alternatively and / or additionally, the agent can be provided with full-fidelity computer vision of the shared digital content defined by the VR environment (and also with the augmented reality embodiments described below).

[0119] A user 3 in a real world environment 7 views a 3D VR environment 13, which may include shared digital content 5, including 3D objects. The interaction module may convert the digital content from the VR application into conceptual objects 9 perceivable by the agent 1. Thus, the agent 1 may interact with or refer to the conceptual objects 9, directly or indirectly. When the agent 1 interacts directly with the conceptual objects 9, for example, pushing a cylinder along the virtual floor of the agent's 1 environment, the interaction module 16 converts this into changes to the digital objects defined locally in the VR application. Figure 8 The proxy virtual environment 8 is shown to be smaller than the VR environment 13, however the proxy virtual environment 8 of the agent 1 can be coextensive with or larger than the VR environment 13. The agent 1 can pass items in the shared environment to the user 3. For example, the agent 1 can pass a soccer ball (shared digital content) to the user 3. In one embodiment, ray tracing is used to simulate the vision of the agent 1 within the three-dimensional scene. The interface can cast rays from the viewpoint of the agent 1 into the three-dimensional scene, and ray tracing can be performed with the rays to determine whether the object is within the field of view of the agent 1. Therefore, the behavior of the agent 1 can be based on whether the shared digital content 5 is within its field of view.

[0120] Fig. 9A system diagram for human-computer interaction in a virtual reality context is shown. VR digital content items 824 can be defined in a VR application 822 and displayed to a user 3 on a VR display 811. The interaction module 16 converts VR digital content items (e.g., VR objects) into conceptual objects perceptible to the agent 1. Thus, the agent 1 can interact with the perceptible conceptual objects. The interaction module 16 then converts the interaction to reflect corresponding changes in the VR digital content items. The VR application can render a scene facilitated by the interaction module 16 to the user 3, including the agent 1, any aspects of the agent's 1 environment, and the digital content items. The agent 1 system 810 can also include a knowledge base 815 that enables the agent 1 to know the specific domain within which the agent 1 interacts.

[0121] Fig.10 A user interface interaction in an augmented reality context is shown. The interaction is similar to that described with reference to virtual reality, except that the user views virtual digital content through a viewport, shown as a mobile phone screen, for example, which may be superimposed on a view of the real world. Fig.11 A system diagram for human-computer interaction with a mobile application is shown. Content items 824 can be defined in a mobile device 1022 and displayed to a user 3 on a display 1011 such as a mobile device screen. The interaction module 16 converts the mobile application digital content items into conceptual objects that are perceptible to the agent. The agent 1 can therefore interact with the perceptible conceptual objects. The interaction module 16 then converts the interaction to reflect the corresponding changes on the mobile application.

[0122] In one embodiment, the agent 1 is in WEB GL; everything in the scene can be an object that the character can manipulate. WEB GL (Web Graphics API) is a JavaScript API for rendering interactive 3D and 2D graphics in any compatible web browser without the use of plug-ins. A WEB GL compatible browser provides a virtual 3D space into which the agent 1 is projected. This allows the virtual space in which the agent 1 operates to be presented on any compatible web browser or WEB GL compatible device, and allows the agent 1 to interact with web objects and 3D objects in the same virtual space.

[0123] Proxy Rendering vs. Digital Content

[0124] An animation renderer can render animations of the agent and the agent's environment for display to the user. The resulting animation can then be streamed to a UI device (such as a browser) as a video stream. In one embodiment, the agent can be rendered in a restricted area of ​​an end-user display. In a site context, the agent can be bounded in an HTML DIV element. In another embodiment, the display of the agent on the end-user display can be unbounded.

[0125] Pixels can be fused so that the proxy or digital content is transparent and allows viewing what is behind the proxy or digital content, respectively. In the case where the AVE is a 3D environment and the display is a 2D screen, the AVE can be rendered as a 2D animation from the user's point of view. For an interactive user experience, the AVE can be rendered as a moving background or foreground of digital content (e.g., including locally rendered HTL site elements).

[0126] The agent's virtual environment may be presented to the user via one or more perspectives. The agent and / or the user may change the user's viewport to the agent's virtual environment. For example, in a rendered pinhole camera model of the agent's environment, the agent may change the pinhole positioning to change the angle / direction and / or scale of the user's view of the agent's environment. Instead of rendering an animation of the agent's environment (which may be computationally intensive), in some embodiments, a 2D rendering of some view corresponding to the agent's view may be rendered (from the agent's vantage point) and presented to the user.

[0127] Overlay of camera images

[0128] The user may use gestures to draw the agent's attention to or point to / indicate areas on the computer screen. Fig.19An example showing how a user's presentation (3VER and / or 3RER) captured by camera 15 is displayed on screen 10. The presentation can be superimposed on anything else being displayed on screen 10, and can be semi-transparent so that user 3 can see user 3's body and the other contents of screen 10. Alternatively, the background of user 3 can be automatically clipped (using standard image processing techniques) so that only the image or hand of user 3 is displayed on the screen, so that the presentation does not have to be transparent. In another embodiment, the presentation of user 3 is visible only to agent 1. Two buttons A and B are displayed on screen 3120, and user 3's hand 3145 is hovering over button B. The presentation of user 3's hand is visible on the screen. Agent 1 can see the same presentation as user 3 sees, and therefore can also see which button user 3 is drawing attention to. The saliency map can draw attention to the human hand, or move (user 3's hand can move on button B to draw attention to it). Therefore, user 3 can interact with a non-touch screen screen in a similar way to a touch screen. The interface module may be used to convert a user gesture (e.g., a finger click) into an input device event (e.g., a click). In other embodiments, the user 3 may receive some other visual indicator of the location that the user 3 is indicating, such as by the agent 1 looking in that direction, rather than displaying the user 3's presentation on the screen. In a similar manner, the agent 1 may be able to sense where the user 3 is looking at the screen by tracking the user's gaze or based on verbal instructions / instructions from the user 3.

[0129] Change

[0130] Multiple agents can interact with digital content independently. The multiple agents can have conversations with each other and with one or more users. The multiple agents can be simulated in the same virtual environment, or they can be simulated in virtual environments that are different from each other. The multiple agents can have the same sensory abilities, or they can have abilities that are different from each other. One or more agents can interact with multiple users. Any one or more of the users can have a conversation with one or more agents and direct one or more agents to manipulate a user interface that the user interacts with, as described herein.

[0131] Combining Computer Vision and Interaction Modules16

[0132] Computer vision and interaction module embodiments may be combined. In one embodiment, an agent may be perceptually aware of features of a content item such as an image by processing the pixels of the image. This allows the agent to discuss features such as one or more colors of an item.

[0133] Agent knowledge

[0134] The agent can also access object metadata, for example from a provided object database. An example is a catalog of purchase items. The agent 1 can associate digital content items with purchase items in the database catalog and use this information to communicate with the user 3. The agent can control navigation or display aspects of the user 3 interface. For example, in the context of a website, the agent 1 can control which portion(s) of a web page is displayed by scrolling up, down, left or right, or zooming in or out.

[0135] Persistent Agent Browser

[0136] In one embodiment, the agent can understand the nature of certain digital content items so that the agent can be integrated into different digital content sources (e.g., different websites). Therefore, the agent can facilitate user content interaction in a scalable manner on the Internet. Such an agent can be provided via a customized browser. Machine learning techniques can be used to train the agent to understand the nature of the site language, thereby learning the association between content items and actions that can / should be taken relative to such content items. For example, the agent 1 can be trained to identify text fields, regardless of the exact configuration of the text fields, read the user-visible labels of the text fields, and fill in these fields on behalf of the user.

[0137] In one embodiment, the user can teach the agent a digital content item. For example, the user can hover the mouse over the digital content item and name the item. The agent can observe this and associate the name of the item with the presentation of the digital content item (pixel presentation or conceptual presentation provided by the interactive module).

[0138] Embedded Actions and Agent-Aware Locators

[0139] In one embodiment, Fig.18As shown in , digital content can be associated with embedded actions and / or agent-perceivable locators. The agent-perceivable locator locates the digital content item (and can be associated with the spatial coordinates corresponding to the digital content item in the agent virtual environment). The locator can be associated with the metadata describing the digital content item. The locator can replace and / or support a saliency map. In an example of a locator that replaces a saliency map, a locator corresponding to a button is marked with metadata, and the metadata indicates that the button must be clicked by the agent 1. In an example of a locator that supports a saliency map, the locator is placed on the button (and can be automatically generated by reading the HTML content of the web page and assigning the locator to an item with an HTML button tag). The saliency map of the button can be provided together with any other saliency map, such as a color saliency map, and for example, the saliency map can be configured to encourage the agent to click on a red button. Embedded content can be provided in a website that can be visited by an agent but not necessarily a user. For example, embedded content visible to an agent can allow the agent to click on a link that is not visible to the user and navigate to another page or read information that is not visible to the user.

[0140] Conversational Interaction

[0141] Agent 1 can conduct conversations with people using the same verbal and non-verbal means that humans do (such as gestures, facial expressions, etc.). These responses can include computer-generated speech or other audio content played back via one or more speakers of the end-user computing device. The responses generated by agent 1 can be visible to user 3 in the form of text, images, or other visual content to user 3. Agent 1 can conduct conversations with the help of third-party services (such as IBM Watson or Google Dialogue Flow and / or conversation corpora).

[0142] Fig. 20 User 3 is shown. Agent 1 can view User 3. Agent 1 can receive the following information about User 3 and use this information to inform interactions in both the real world and the virtual world:

[0143] The embodied agent can receive camera input and calculate where the user is looking. This can be mapped to a content item or object in the user's space / real world that the user is looking at. The user's gaze can be tracked using the user's eye angle and / or the user's head angle.

[0144] • The embodied agent may additionally track the user's eye movements, head movements and calculate the user's eye and head angles.

[0145] • The embodied agent also receives verbal input including instructions from the user, which in some cases may indicate the embodied agent's actions and / or the embodied agent's gaze. Other input may include text, for example via a keyboard.

[0146] • An embodied agent that has identified the user can follow the user's position (view the user) with their eyes and by leaning towards the user.

[0147] • The embodied agent may be aware of where the user controlled mouse is located on the screen relative to the digital content item. The agent 1 may also be aware of the user's touches, for example through a touch screen monitor.

[0148] • The embodied agent can monitor the user's movements via the camera, particularly the movements of the user's arms, hands and fingers.

[0149] Facial expressions can be used to detect user emotions and adapt accordingly.

[0150] The pitch of the user's voice can be used to detect user information so that the agent can adapt accordingly.

[0151] The agent can ensure that the agent has the user's attention before continuing the conversation.

[0152] The agent may have a memory of past interactions with the user and use this information within the session.

[0153] The agent can use the context of the conversation, digital content items, and user information to resolve ambiguity. The actions taken by the agent can be dynamic, formulated for the user, or context-sensitive, and intention / goal-oriented. The agent can access information sources about the user. For example, the agent can be aware of the user's location (e.g., via a geolocation service) and / or time zone, and use this to guide the interaction accordingly. The agent combines dialogue, emotion, cognition, and memory to create an interactive user experience. An embodiment provides a system for synthesizing the emotions and gesture behaviors of the agent through content interaction. The agent interacts with the user via dialogue and the user's actions to establish the user's goals, beliefs, and expectations, and guides the user accordingly, and the user's actions include eye gaze, eye direction, user's movement, and other inputs received about the user. The agent may include an emotional response module that reacts to the user. In one embodiment, the agent interaction is guided or specified by a learning response (such as reinforcement learning). The specified behavior can be guided by a knowledge base of rules. In one embodiment, the way the agent interacts is guided by the user's psychometric profile.

[0154] In one embodiment, a simulated interaction between an embodied agent and a user may be implemented using a binary turn model applied to the user's and the embodied agent's gaze. The embodied agent may indicate the end of their conversational turn by attempting to look directly at the user during the interaction. In a similar manner, when the embodied agent detects that the user has initiated direct gaze at the embodied agent, the embodied agent may perceive that the user has indicated the end of their turn.

[0155] refer to Fig. 20 , showing an example of taking a turn. User 3 may look 3325 at agent 1 or say agent 1's name or prompt agent 1 by, for example, pointing at agent 1. Once user 3 has agent 1's attention, agent 1 may return eye contact 3330 with user 3, signaling to user 3 that agent 1 has recognized user 3's attention. User 3 may in turn respond with a smile, prompting agent 1 to proceed and take a turn, such as providing information, or otherwise communicating with user 3. Once agent 1 is finished, agent 1 may signal user 3 by pausing and looking directly 3345 at user 3. User 3 may in turn smile and acknowledge agent 1 3350 and take a turn to 3355. When user 3 has finished, user 3 may pause and direct attention 3360 to agent 1, which may take another turn 3365. The above description is merely an example, and user 3's indication may take other forms that allow agent 1 to recognize agent 1's turn. User 3 may, for example, indicate the end of their turn through verbal or non-verbal cues during the interaction. Non-verbal cues may include, for example, smiles, winks, movements of the head, movements of the body including arms, hands and fingers. Likewise, the agent 1's indication may take other forms that allow the user 3 to recognize the user 3's turn. The agent 1 may indicate the end of their turn, for example, by verbal or non-verbal cues during the interaction. ...

[0156] Attention Modeling

[0157] The agent attention model can be implemented as a saliency map of areas in the field of view, where visible locations compete for attention. Those locations that are more active have more saliency. A saliency map is an image showing the unique quality of each location (pixel), and therefore more active locations have more saliency. Several types of saliency that are active in the human brain can be implemented in the embodied agent. These include updating the visual frame for each eye or head movement of the user 3 or the embodied agent 1. Other saliency maps use a reference frame that is stable regardless of head and eye movement. Significant features that can be mapped include the color or brightness or interestingness of stimuli present in the field of view. Other saliency maps that focus on expectations or desires can be created, and based on these expectations or desires, predict where the salient locations on the saliency map may be. As implemented in the embodied agent 1, these saliency maps are combined to derive an aggregate measure of saliency.

[0158] In one embodiment, the attention model implemented includes multiple saliency maps presenting different types of salient targets. The aggregate saliency map therein is a weighted sum of the maps used. It is possible to vary how the various saliency maps are weighted. In one example, the weights may be used such that if both the user and the embodied agent are paying attention to the object, the saliency of the object (thing or person) increases.

[0159] Salience_gaze=weightg1*embededAgent_gaze_map+Weightg2*User_gaze_map+weightg3*(embededAgentgaze_map.*User_gaze_map)

[0160] Salience_point=weightp1*embededAgent_point_map+weightp2*User_point_map+weightp3*(embededAgentpoint_map.*User_poit_map)

[0161] The information used to create the saliency map includes the inputs discussed in the above-mentioned conversational interaction. Non-visual inputs such as auditory and textual inputs can also be applied to the saliency map by mapping these inputs to the visual mapping space. For example, in order to map a user pointing to an object, the system can calculate the location of the user pointing and map the location to the visual mapping space. If the input is auditory, the system calculates the location of the sound from and maps the location to the visual mapping space. These pointing and auditory mappings are combined with the visual mapping. In one embodiment, the attention model includes a sub-map (tracker) that enables the embodied agent to keep tracking the object that has been previously noticed even if there is a subsequent transfer under the current attention.

[0162] Fig.21 A screenshot of the user interface for setting up multiple saliency maps is shown. Fig.21 The weighting of these mappings can be changed by using the slider 2110 shown in. The slider changes the default weighting.

[0163] In some embodiments, multiple visual feeds from different sources can activate visual feature maps, providing an embodied agent with a presentation of multiple visual feeds. Each field can be used to calculate a saliency map that controls visual spatial attention. Each visual feed can be associated with multiple feature maps, making the area where the visual feed is displayed more prominently relative to other areas of the visual feed. For example, a camera visual feed (capturing a user 3 interacting with the embodied agent 1) and a browser window visual feed (capturing a computer browser with which the user 3 and / or agent 1 interacts) both activate visual feature maps. Another visual feed can be provided by the agent's own 3D virtual environment. For example, from the agent's perspective, through ray casting, a 2D plane corresponding to the field of view of the agent 1 with its surroundings can be provided to the agent 1.

[0164] A human-like attention model may be implemented such that the agent 1 can only attend to aspects of one of the visual feeds at any given time. Thus, a single salient region on two or more maps is selected for attention at one time: when attention switching is concerned, the two or more visual fields may be considered a single visual field with two parts. Weightings may be applied across the visual feeds such that some visual feeds are determined to be more salient than others.

[0165] Saliency maps can be applied to verbal cues to help the agent locate the item the user is referencing. For example, keywords such as "left", "right", "up", "down", etc. can be mapped to a "verbal cue" saliency map that highlights the corresponding area of ​​the visual feed as salient. The verbal cue saliency map can be combined with other saliency maps as above to promote joint attention and interaction. For example, if User 3 says: "The button on your left", the verbal cue saliency map can highlight the left half of the screen. This can then be combined with an object saliency map that detects buttons, highlighting the button on the left as the most salient, and thereby highlighting the button that the agent noticed.

[0166] refer to Fig. 22 , shows a system implemented with multiple visual feeds. The system extracts multiple low-level features 3780 from two visual streams 3770, 3775. Then, a center-surround filter 3740 is applied to each map to derive a feature-specific saliency map. Feature maps 3725 are created from these maps, and a combination of maps is created for a specific feature 3720. For example, the user's face or the user's surroundings. A combined saliency map 3710 is then created based on the individual saliency maps or feature maps. These feature-specific maps are combined in a weighted sum to produce a feature-independent saliency map 3710. Then, the system applies a "winner takes all" (WTA) operation, which selects the most active positioning in the saliency map 3710 as the area to be noted. In addition to the camera feed, a sound feed or other feed can be fed into the system to create feature maps 3625, 3725. In one embodiment, a rotation feature map can be incorporated into the system so that the focus of the rotation feature map depends on who is rotating. The rotation map produces a rotation-related deviation in the saliency map.

[0167] Interactive Examples

[0168] Fig.12 Screen shots of an agent facilitating user interaction with a web page are shown. The web page includes shared digital content, which includes several menu items, a search bar, and navigation buttons. Relative to the user, the agent 1 is located in front of the shared digital content. The agent 1 has a perceptual awareness of the shared digital content 5. Therefore, the agent 1 can refer to or point to different content items as part of interacting with the user. The agent 1 can have a dialogue with the user 3 to find out which digital content item the user wants to navigate to next. The agent 1 can turn to see the digital content items that the user has indicated interest in, and trigger navigation to the URL to which the menu content item is linked. The agent can point to a content item and visually press the content item, as if they were clicking on the item.

[0169] Fig.13A screenshot of an agent 1 positioned behind a shared digital content 5 is shown. The example shown is a bank's website showing several credit cards available to the user for selection. The agent 1 can engage in a conversation with the user and ask the user if he wants more information about any of the credit cards displayed as shared digital content 5. The user can click directly on one of the shared digital content items (credit cards) because they are clickable HTML objects that trigger links to further information. Alternatively, the user can ask the agent 1 to provide further information about one of the credit cards. The agent 1 has a sense of the conceptual object representing the shared digital content 5, and can therefore ascertain from the information provided to the agent 1 via the interaction module which credit card the user is interested in. The agent 1 can trigger an action on the conceptual object representing the credit card instead of the user clicking on the credit card, which in turn is converted to clicking on an item on the website via the interaction module. Since the credit card is a clickable image, the agent 1 can use pixel information from the image to ascertain the color of the image, and can therefore understand whether the user 3 refers to the item of interest to them by color. Fig.13 B shows the web page once the user has selected a digital content item. Agent 1 can access metadata related to the digital content item, for example through a digital content provider database, to tell the user more information about the content item. In addition, such information can be displayed to the user in AVE (not shown).

[0170] Fig.14 An agent 1 is shown engaging a user through an interactive menu (sharing digital content 5) to help the user find a suitable credit card. Again, the user can click directly on the menu item, or the user can have a conversation with the agent 1, and the agent 1 can click on the digital content item on the user's behalf. For example, the agent 1 can ask the user "How often do you use your card?" The user can read one of the three displayed menu options: "Always", "From time to time" or "I'm not sure". The agent 1 will match the user's 3 speech to one of the digital content items as text on the digital content item, provided to the agent 1 as an attribute of the corresponding conceptual object of the content item. The user can say something slightly different from the predetermined text, such as "I don't know" instead of "I'm not sure", and the agent 1 can infer the option that the user wishes to select. Fig.14 The option "I'm not sure" is shown to have been selected. The agent 1 reads "The total amount I stated" during the touching (corresponding to the mouse click) of the menu item. The agent 1 is looking directly at the digital content item that the agent is touching. The agent 1 touching the background object triggers a click on the digital content item via the interaction module 16. The user may choose not to go through the menu in sequence, but to skip the first set of options, for example telling the agent 1 that they want a low fee. The agent 1 may select this option and then ask the user about the information from the previous step.

[0171] Fig.15 A sequence of screenshots of Agent 1 is shown facilitating a user to interact with a website to help the user purchase a car. Thus, Agent 1 may walk up to a car and point to features of the car. If the car turns or changes position, Agent 1 may continue to point to the same aspect of the car. Fig.15 A shows agent 1 walking onto the screen from the left. The website includes a top menu that includes digital content 4 and a virtual showroom 1780 showing a virtual car 1760. The virtual showroom is configured to display 3D models of items that can be purchased. Because this is in the virtual environment of agent 1, user 3 may not be able to directly interact with objects in the virtual showroom. Nevertheless, user 3 can indirectly interact with such objects by communicating with agent 1 so that agent 1 manipulates objects in the virtual environment of agent 1. For example, user 3 can ask agent 1 to pick up an item, manipulate an item, change the color of an item, or turn an item so that user 3 can see the item from another angle. Fig.15 B shows that the agent 1 faces the user 3 after entering the virtual showroom and has a conversation with the user 3 to find out what the user 3 is interested in. Fig.15 C shows Agent 1 gesturing towards the virtual car 1750. Since Agent 1 has a perceptual awareness of the location of the car in relation to Agent 1 and the screen, Agent 1 is able to gesture towards the virtual space coordinates of the car. When Agent 1 is in a virtual environment that includes the 3D virtual car, Agent 1 can walk towards the car and point to various features of the car (defined to Agent 1 by object metadata such as coordinates of features and tags). Since both Agent 1 and the car are in the same 3D virtual environment, Agent 1's movement towards the car appropriately reduces the "real world" size of Agent 1 on the screen and increases the realism of the interaction. Fig.15 E shows a menu 2150 of options proposed by Agent 1 to ask User 3 about User 3's interests. Fig.15 F shows the agent 1 selecting a menu item on behalf of the user 3. The agent 1 has touched the item, which triggered a click on the corresponding digital content item. Fig.15 F shows navigation to an image showing the interior of a car. The user's view is now from the perspective of Agent 1, and Agent 1's hands are visible.

[0172] Fig.16 A non-linear interaction sequence that relies on user feedback is shown. Fig.16 A shows that agent 1 displays two purchase options X and Y to the user. On the other hand, Fig.16 B shows that after receiving feedback from the user, Agent 1 softly / temporarily puts Option Y aside, which means that the user has a preference for another Option Y, but is not 100% sure. After receiving feedback from the user that Option Y is definitely not what the user wants, Fig.16C shows that Agent 1 can remove the option by throwing it aside. Fig.16 D shows Agent 1 with an expectant expression while Agent 1 waits for the user to consider whether they wish to return item Y. Fig.16 E shows Agent 1 returning item Y to the screen.

[0173] The agent can facilitate users to learn how to use third-party software to perform specific tasks. For example, in an application such as Photoshop, the agent can have a conversation with the user and show how to navigate the interface of the application by physically interacting with the controllable digital content items on the interface. A general algorithm with steps on how to operate or navigate the user interface can be provided to the agent. The agent can include a knowledge base of tasks, each task is associated with a time sequence of actions, and the knowledge base can be defined by finding a significant item (such as a symbol or text), performing an action on the item (such as clicking), and then proceeding to the next step. When the user requests to perform an action stored in the knowledge base on the agent's UI interaction task, the agent can control a human input device (e.g., a mouse / touch input) to perform these steps.

[0174] The agent can facilitate the user to make a purchase selection for online shopping. The agent can be embedded in an e-commerce platform and help guide the user through the items. In one embodiment, the agent can receive information about the user's profile to guide the interaction. For example, a user can have a profile created from a history of engagement with the e-commerce system (e.g., previous purchases). The user's profile is stored in a broader recommender system, and the agent can use the recommender system to appropriately recommend items to the user. The agent can purchase products on behalf of the user by navigating the user's e-commerce UI. By the agent viewing the user's body language while verbally confirming to proceed with the purchase, accidental or unintentional purchases can be minimized. For example, if the agent sees the user nodding while saying "yes, please buy it", and looking at the product in question, the agent can be confident of its understanding of the user's wishes and proceed with the purchase of the item.

[0175] The agent may include a computer program configured to perform tasks or services for a user of an end-user computing device (personal assistant). Examples of tasks that may be performed by the agent on behalf of a user may include: making a phone call to a user-specified person, launching a user-specified application, sending a user-specified email or text message to a user-specified recipient, playing user-specified music, scheduling a meeting or other event on the user's calendar, obtaining directions to a user-specified location, obtaining scores associated with a user-specified sporting event, posting user-specified content to a social media website or microblogging service, recording a user-specified reminder or note, obtaining a weather report, obtaining the current time, setting an alarm at a user-specified time, obtaining a stock price for a user-specified company, finding a nearby business, performing an Internet search, etc.

[0176] In one embodiment, the agent can play with the user or collaborate with the user in an activity. For example, the agent and the user can collaborate on a drawing together. The drawing itself may be on a shared environment. If the user draws an object such as an apple on the shared space, the agent can identify the object as an apple and talk about the apple. Or it can add drawing lines or visual features to the apple, such as coloring it red. In another embodiment, the agent and the user can interact on the shared environment to collaborate in some other activities, such as playing music. The shared environment may include a virtual musical instrument, such as a xylophone. The user can hit the xylophone keys, and the agent can react accordingly. The agent can be notified of items moving on its shared space, identifying and reacting to objects on the shared space.

[0177] Navigating the Internet, applications, operating systems or any other computer system can be driven by agents and users. Users who visit the site can be searching or browsing (searching for new or interesting things). Agent 1 can promote site browsing and convert site browsing into real-time activities rather than query-based retrieval. In one embodiment, the agent uses interactive dialogue to help users search for sites. The agent can use another search tool "in the back end" (which is not displayed to the user) as the result of a traditional search engine (such as Google). In traditional recommendation systems, users must perform an intellectual "context switch" from the space of browsing web pages to explicit interaction with the search assistant. The embodied agent described herein makes the thought process of the user's browsing activity not interrupted by the need to switch to an independent query interface. Since the agent constantly watches the browsing activity, the agent's recommendation will be real-time because the relevant page appears in the user's browsing activity. The agent can show the user how to navigate or perform an action on an Internet web page. The agent may be able to "preview" links or chains that are not displayed to the user, and thus warn the user about links that are "dead ends" (links that appear that they might be interested in reading the link text or viewing the link image but turn out not to be such), or that a garden path is a sequence of links that provides just enough incentive to keep following the path but ultimately leads to a dead end. In one embodiment, the agent 1 may be provided with a grammatical understanding of the general site language, and may assist the user in searching for online information. The user may ask the agent to scroll up or down, click on a particular search link, enter a URL in a browser, return to a previous page, or navigate in other ways.

[0178] The agent can help the user fill out the form. If the agent has a profile for the user, the agent can automatically fill in the details that the agent knows about the user. The agent may ask the user about fields that the agent is unsure of. In the case of unsuccessful form entry (e.g., the user has missed a constraint on a password), the agent may identify the violated constraint from the error message and ask the user to correct it. The agent may automatically navigate to and / or gesture to the fields that need to be re-entered.

[0179] The shared environment can be used as a canvas for agent 1 to coach user 3, for example, agent 1 can ask user 3 to solve a math equation on the shared environment and show all the work. Agent 1 can use character recognition to perceive the numbers and steps written by the user. Agent 1 can conduct a dialogue and interact with the user about the work on the shared canvas, such as by citing different lines of work, clearing or underlining errors, or adding further work to the shared canvas. In practical topics, agents can help users learn in virtual or augmented reality contexts. For example, to facilitate medical training, agent 1 can use a tool in front of someone, pass the tool to them and let them try it themselves.

[0180] advantage

[0181] Early work in human-computer interaction highlighted the advantages of interacting with a "model world" within a computing device interface where objects and actions resemble / mirror those in the real world, which is more intuitive for human users. Traditional application of artificial agents to user interfaces results in a "magic world" in which the world changes due to the work of hidden hands. The embodiments described herein extend human-computer interaction and artificial intelligence by implementing an interface where the world changes due to visible helpful hands and visually displaying information about the input received by the agent (via the agent's gaze direction / eye direction / body language) and the agent's mental state in an effective manner. Instead of the user seeing interface elements "moving on their own", the user can now visualize the agent's thought processes and actions that lead to interface manipulation. The user can directly observe the agent's autonomous actions, and the agent can observe the actions taken autonomously by the user in the interface.

[0182] An embodied agent that operates directly in the user interface rather than as a "background" or "backend" increases the degree to which the user will perceive the software to act like an assistant. If the user perceives the agent's actions as actions "that I could do myself," the user is more willing to conceptualize the agent as an assistant role.

[0183] In a traditional command line or menu driven interface, the user performs input, types, the system accepts the input, calculates some action, displays the result, and waits for the next input. When the user is ready to input, the system does nothing, and when the system is running, the user does nothing in the interface. The embodiments described herein provide agents that can run independently and concurrently.

[0184] The methods described herein facilitate users to perform tasks operating computing devices through a continuous and / or guided human-computer interaction process. Limiting the capabilities of autonomous agents to actions that can be simulated through traditional input devices or methods can facilitate agents to guide users because agents cannot take shortcuts.

[0185] Embodiments of the present invention may usefully convert between legacy applications (written to communicate with now-often obsolete input / output devices and user interfaces) and user interfaces that include the embodied autonomous agents described herein, such that the logic and data associated with the legacy programs may continue to be used in novel interaction contexts.

[0186] The embodiments described herein avoid the need for synchronization between pre-recorded or pre-defined animations and / or dialogues in the output and the desired behavior generated at runtime. Thus, there is no need to map fixed length speech and / or non-speech segments to runtime conditions and synchronize in space or time to accommodate variable and / or dynamic digital content. There is no need to synchronize speech and non-speech behavior as a proxy simulation including a neurobehavioral model that drives both.

[0187] The embodiments described with reference to agent perception via computer vision can utilize techniques in DOM parsing, computer vision, and / or natural language processing to simulate manual processing that occurs when viewing digital content, so as to automatically extract useful information and interact with digital content. The agent can send messages or requests to an interaction module that embodies actions that are relevant and compatible with the operated object. These actions allow the agent to control and interact with all types of browser-hosted content in an abstract manner. Therefore, user-machine interaction is modeled in a manner that can keep dialog knowledge separate from application knowledge. This opportunity significantly reduces the cost of moving an interactive system from an application domain to a new application domain. Real-time performance and timing control of UI interactions are achieved through the strict time model and architecture of the agent. The real-time response of the agent to user input keeps the delay of each part of the system at a minimum during the on-time execution of the action.

[0188] The advantage of allowing the agent to interact with items in a simulated environment is that this creates interactions that appear more natural and makes the agent 1 alive in the eyes of the user. For example, content items that the agent interacts with can be simulated to have mass and other physical properties. Elements provided by the digital content provider are still rendered locally, as defined by the digital content provider.

[0189] Providing conceptual objects with affordances allows an agent to perceive actions available to the agent without requiring significant cognitive processing / image recognition / reasoning, thereby reducing computational processing power and / or time. An advantage of using interaction modules is that it allows scalable customization of agent-facilitated interactions in different contexts or environments, and enables computer interfaces to adapt to technologies of increasing complexity.

[0190] explain

[0191] The described methods and systems may be used on any suitable electronic computing system.According to the embodiments described below, the electronic computing system utilizes various modules and engines to utilize the methods of the present invention.

[0192] The electronic computing system may include at least one processor, one or more memory devices or interfaces for connecting to one or more memory devices, input and output interfaces for connecting to external devices so that the system can receive instructions from one or more users or external systems and operate on them, a data bus for internal and external communication between the various components, and a suitable power supply. In addition, the electronic computing system may include one or more communication devices (wired or wireless) for communicating with external and internal devices, and one or more input / output devices, such as a display, a pointing device, a keyboard, or a printing device.

[0193] The processor is arranged to perform the steps of the program stored in the memory device as a program instruction. The program instruction enables the execution of the various methods of the present invention as described herein. Any suitable software programming language and toolkit (for example, based on the language and compiler of C) can be used to develop or implement program instructions. In addition, program instructions can be stored in any suitable manner so that they can be transferred to the memory device or read by the processor, for example, stored on a computer-readable medium. Computer-readable medium can be any suitable medium for tangibly storing program instructions, for example, solid-state memory, magnetic tape, optical disk (CD-ROM or CD-R / W), memory card, flash memory, optical disk, disk or any other suitable computer-readable medium.

[0194] The electronic computing system is arranged to communicate with a data storage system or device (eg, an external data storage system or device) in order to retrieve relevant data.

[0195] It should be understood that the system described herein includes one or more elements arranged to perform the various functions and methods described herein. The embodiments described herein are intended to provide the reader with examples of how the various modules and / or engines that make up the elements of the system can be interconnected to enable the implementation of the functions. In addition, the embodiments of the specification explain how the steps of the methods described herein can be performed in system-related details. Conceptual diagrams are provided to indicate to the reader how various data elements are processed by various modules and / or engines at different stages.

[0196] It should be understood that the arrangement and construction of modules or engines may be adapted accordingly to system and user requirements, so that various functions may be performed by modules or engines different from those described herein, and certain modules or engines may be combined into a single module or engine.

[0197] It should be understood that the described modules and / or engines can be implemented and provided with instructions using any suitable form of technology. For example, these modules or engines can be implemented or created using any suitable software code written in any suitable language, where the code is then compiled to produce an executable program that can run on any suitable computing system. Alternatively, or in conjunction with an executable program, a module or engine can be implemented using any suitable mix of hardware, firmware, and software. For example, portions of a module can be implemented using an application specific integrated circuit (ASIC), a system on a chip (SoC), a field programmable gate array (FPGA), or any other suitable adaptive or programmable processing device.

[0198] The methods described herein can be implemented using a general purpose computing system specifically programmed to perform the steps described. Alternatively, the methods described herein can be implemented using a specific electronic computer system, such as a data classification and visualization computer, a database query computer, a graphics analysis computer, a data analysis computer, a manufacturing data analysis computer, a business intelligence computer, an artificial intelligence computer system, etc., wherein the computer has been particularly adapted to perform the steps on specific data collected from an environment associated with a specific field.

[0199] Providing conceptual objects with affordances allows the agent to perceive actions available to the agent without having to engage in significant cognitive processing / image recognition / reasoning.

[0200] An advantage of using the interaction module 16 is that it allows for scalable customization of agent-facilitated interactions in different contexts or environments, and enables computer interfaces to adapt to technologies of increasing complexity. Summary of the invention

[0201] In one aspect, a method for visualizing interactions between an embodied artificial agent and digital content on an end-user display device of an electronic computing device is provided, the method comprising the steps of: creating an agent virtual environment having virtual environment coordinates; simulating digital content in the agent virtual environment; simulating an embodied artificial agent in the agent virtual environment; enabling the embodied artificial agent to interact with the simulated digital content; and displaying the interactions between the embodied simulated agent and the digital content on the end-user display device.

[0202] In one embodiment, the virtual environment is a 3D virtual space, and the virtual environment coordinates are 3D coordinates. Optionally, the interaction is to move or view the digital content project to the digital content project. Optionally, the interaction is to gesture to the digital content project or to touch the digital content project by moving the multi-joint effector. Optionally, inverse kinematics is used to simulate the movement of the multi-joint effector. Optionally, the movement of the multi-joint effector is simulated using the mapping of the joint positions based on the neural network to the target position.

[0203] In another aspect, a method for interacting with digital content on an electronic computing device via an embodied simulated agent is provided, the method comprising the following steps: displaying the digital content to a user on a user interface on the electronic computing device; creating an agent virtual environment having virtual environment coordinates; simulating the digital content in the agent virtual environment; simulating an embodied artificial agent in the agent virtual environment; enabling the embodied artificial agent to interact with the simulated digital content; converting the interaction into an actuation or manipulation of the digital content on the user interface; and displaying the interaction by superimposing the embodied artificial agent on the digital content and displaying the digital content and the superimposed embodied artificial agent on the user interface.

[0204] Optionally, the virtual environment is a 3D virtual space, and the virtual environment coordinates are 3D coordinates. Optionally, the digital content is simulated in the proxy virtual environment as pixels in the proxy virtual environment, and each pixel has a coordinate location in the proxy virtual environment. Optionally, the simulated embodied interaction is an intersection between the coordinates of the pixels corresponding to the digital content and the body of the agent. Optionally, converting the simulated embodied interaction includes: mapping the embodied interaction to an input device event. Optionally, the human input device event is a mouse event, a keyboard event, or a touch screen event.

[0205] In another aspect, a system for facilitating interaction with an electronic computing device is provided, the system comprising: at least one processor device, at least one memory device in communication with the at least one processor; an agent simulator module arranged to be executable by the processor to simulate an embodied agent; an interaction module arranged to be executable by the processor to convert digital content into conceptual objects perceivable by the embodied agent and enable the embodied agent to interact with the digital content; and a rendering module arranged to be executable by the processor to render the digital content, the embodied agent, and the interaction of the embodied agent with the digital content.

[0206] Optionally, the interaction module is also arranged to convert the embodied agent action on the conceptual object into a change to the digital content. Optionally, the interaction module associates the conceptual object with coordinates that present the positioning of the conceptual object relative to the embodied agent. Optionally, the interaction module associates the conceptual object with one or more functional affordances of the corresponding digital content. Optionally, the computing device is site content, and the interaction module is JavaScript code integrated into the site content. On the other hand, there is provided an embodied agent located in a virtual environment, the virtual environment being created on an electronic computing device, the embodied agent being programmed to: receive input from a real-world environment; receive input from a virtual environment; and act according to input from the real-world environment and the virtual environment, wherein the received input from the real-world environment and the virtual environment is received via a continuous feedback loop from both the real-world environment and the virtual environment.

[0207] In another aspect, there is provided an embodied agent simulator implemented on an electronic computing device for simulating interactions with a user, the agent simulator being programmed to: receive user input; receive digital content input including information related to digital content displayed to the user; and simulate a conversation with the user by generating responses to the user based on both the user's natural language statements and the digital content input.

[0208] Optionally, the response and / or input to the user is verbal. Optionally, the verbal response and / or input is auditory or textual. Optionally, the response and / or input to the user is visual. Optionally, the verbal response and / or input is a gesture or facial expression.

[0209] In another aspect, a method for facilitating user interaction with an electronic computing device is provided, the electronic computing device having a display and input device, at least one processor, and a memory for storing instructions, the processor being programmed to: define a virtual environment having virtual environment coordinates; determine coordinate dimensions relative to real-world physical dimensions of a display; simulate an embodied agent in the virtual environment, the agent's position relative to the virtual space being defined by the virtual environment coordinates; simulate one or more digital objects in the agent's virtual environment, the one or more digital objects' positions relative to the virtual environment being defined by the virtual environment coordinates; enable the embodied agent to interact with the one or more digital objects using information about the agent's virtual environment coordinates and the virtual environment coordinates of the virtual objects; and display the interaction between the agent and the one or more digital objects to a user on the display.

[0210] On the other hand, a method is provided for providing an embodied agent with a simulated environment, the simulated environment having substantially real-time perception of continuous visual input from outside the simulated environment, the method comprising the steps of: providing an interface between a programmatic definition of the embodied agent and / or the simulated environment and the continuous visual input; collecting visual input from outside the simulated environment at each time step of the agent's simulation and transmitting the input data to the embodied agent and / or the simulated environment at each time step; and inputting the visual input into the agent's visual system or simulating the visual input within the agent's simulated environment.

[0211] On the other hand, a method for enabling a simulated agent to interact with a user interface is provided, comprising the steps of presenting to the simulated agent digital content displayable to a user via an end-user display; and translating cognitive decisions and / or physical movements of the artificial agent into actuation or manipulation of a human input device that controls the user interface.

[0212] Optionally, converting the simulated embodied interaction comprises mapping the embodied interaction between the artificial agent and the digital content to input device events. Optionally, the human input device events are mouse events, keyboard events or touch screen events.

[0213] On the other hand, a method for interacting with an artificial agent is provided, the method comprising: simulating an artificial agent in an agent virtual space, presenting digital content that will be perceptible to the artificial agent by simulating the digital content in the agent virtual space; displaying the artificial agent virtual space and the digital content to a user on a display; receiving an image of the user from a camera; tracking the user's gaze on the display based on the received image; and simulating an embodied interaction between the artificial agent, the user, and the digital content based on user input including at least the user's gaze.

[0214] Optionally, user gaze is tracked using user eye angle and / or user head angle. Optionally, further comprising tracking eye movements of the user, and wherein the simulated embodied interaction is further based on eye movements of the user.

[0215] Optionally, user eye movement is tracked using the user's eye corners and / or the user's head angle. Optionally, the user input includes verbal input. Optionally, the user input includes auditory or textual input. Optionally, the user input includes touch screen or mouse movement. Optionally, the user input includes visual input. Optionally, the visual input is a gesture or facial expression. Optionally, the gesture includes movement of one or more of an arm, a hand, or a finger. Optionally, the simulated embodied interaction between the artificial agent, the user, and the digital content includes the user directing attention to an object in the digital content of the agent's virtual space.

[0216] Optionally, the method includes tracking the gaze of the artificial agent, and wherein simulating the embodied interaction is further based on a binary turn model applied to the gaze of the user and the gaze of the artificial agent. Optionally, the artificial agent indicates the end of their turn by attempting direct gaze with the user during the interaction. Optionally, when the user initiates direct gaze of the artificial agent, the artificial agent perceives the user indicating the end of its turn. Optionally, the interaction occurs in a virtual reality environment. Optionally, the interaction occurs in an augmented reality environment.

[0217] On the other hand, there is provided an embodied agent simulator for implementing on an electronic computing device to simulate interaction with a user, the embodied agent simulator being programmed to: simulate an artificial agent in an agent virtual space, present digital content that will be perceptible to the artificial agent by simulating the digital content in the agent virtual space; display the artificial agent, the artificial agent virtual space, and the digital content to the user on a display; receive an image of the user from a camera; track the user's gaze on the display based on the received image; and simulate embodied interaction between the artificial agent, the user, and the digital content based on user input including at least the user's gaze.

[0218] Optionally, the user's gaze is tracked using a user eye angle and / or a user head angle. Optionally, further comprising tracking the user's eye movement, and wherein the simulated embodied interaction is further based on the user's eye movement. Optionally, the user eye movement is tracked using a user eye angle and / or a user head angle. Optionally, the user input comprises a verbal input. Optionally, the user input comprises an auditory or textual input. Optionally, the user input comprises a touch screen or a mouse movement. Optionally, the user input comprises a visual input. Optionally, the visual input is a gesture or a facial expression. Optionally, the gesture comprises the movement of one or more of an arm, a hand or a finger. Optionally, the simulated embodied interaction between the artificial agent, the user and the digital content comprises: the user directs attention to an object in the digital content of the agent virtual space. Optionally, the method comprises: tracking the gaze of the artificial agent, and wherein the simulated embodied interaction is further based on a binary turn model applied to the gaze of the user and the artificial agent. Optionally, the artificial agent indicates the end of their turn by attempting to directly gaze with the user during the interaction. Optionally, the artificial agent senses the user indicating the end of its turn when the user initiates direct gaze of the artificial agent. Optionally, the interaction occurs in a virtual reality context. Optionally, the interaction occurs in an augmented reality environment.

[0219] On the other hand, a method for interacting with an artificial agent is provided, the method comprising: simulating an artificial agent in an agent virtual space, presenting digital content that will be perceptible to the artificial agent by simulating the digital content in the agent virtual space; displaying the artificial agent, the artificial agent virtual space, and the digital content to a user on a display; receiving images of the user and the user's environment from a camera; tracking the user's gaze based on the received images; and simulating embodied interaction between the artificial agent, the user, the user's environment, and the digital content based on user input including at least the user's gaze.

[0220] Optionally, the user gaze is tracked using the user eye angle and / or the user head angle. Optionally, further comprising tracking the user's eye movement, and wherein the simulated embodied interaction is further based on the user's eye movement. Optionally, the user eye movement is tracked using the user eye angle and / or the user head angle. Optionally, the user input comprises a verbal input. Optionally, the user input comprises an auditory or textual input. Optionally, the user input comprises a touch screen or a mouse movement. Optionally, the user input comprises a visual input. Optionally, the visual input is a gesture or a facial expression. Optionally, the gesture comprises the movement of one or more of an arm, a hand or a finger. Optionally, the simulated embodied interaction between the artificial agent, the user, the user's environment and the digital content comprises: the user directs attention to an object in the digital content of the agent virtual space or an object in the user's environment. Optionally, the method comprises tracking the gaze of the artificial agent, and wherein the simulated embodied interaction is further based on a binary rotation model applied to the gaze of the user and the artificial agent. Optionally, the artificial agent indicates the end of their rotation by attempting to directly gaze with the user during the interaction. Optionally, the artificial agent senses the user indicating the end of its turn when the user initiates direct gaze of the artificial agent. Optionally, the interaction occurs in a virtual reality environment. Optionally, the interaction occurs in an augmented reality environment.

[0221] A embodied agent simulator for interacting with an artificial agent, the method comprising: simulating an artificial agent in an agent virtual space, presenting digital content that will be perceptible to the artificial agent by simulating the digital content in the agent virtual space; displaying the artificial agent, the artificial agent virtual space, and the digital content to a user on a display; receiving images of the user and the user's environment from a camera; tracking the user's gaze based on the received images; and simulating embodied interaction between the artificial agent, the user, the user's environment, and the digital content based on user input including at least the user's gaze.

[0222] On the other hand: a method of implementing a system for interacting with an artificial agent, the method comprising: simulating an artificial agent in an agent virtual space, displaying the artificial agent and the artificial agent virtual space to a user on a display; receiving an image of the user from a camera; tracking the user's attention; applying an attention model to the user's attention and providing an output to the artificial agent; and simulating an embodied interaction between the artificial agent and the user based on the output of the attention model.

[0223] On the other hand, an embodied agent simulator implemented on an electronic computing device for simulating interactions with a user, the embodied agent simulator being programmed to: simulate an embodied agent in an agent virtual space, display the artificial agent and the embodied agent virtual space to the user on a display; receive an image of the user from a camera; track the user's attention based on the received image; apply an attention model to the user's attention and provide output to the embodied agent; and simulate embodied interactions between the embodied agent and the user based on the output of the attention model.

[0224] Optionally, the attention model is also applied to the attention of the artificial agent. Optionally, the method includes receiving an image of the space of the user from the camera. Optionally, the method includes presenting digital content that will be perceptible to the artificial agent by simulating the digital content in the agent virtual space, the digital content being visible to the user. Optionally, tracking the user attention includes tracking the gaze of the user. Optionally, tracking the user's attention includes tracking the user's eye movement. Optionally, tracking the user's attention includes tracking the user's eye movement on the display. Optionally, the method includes tracking the gaze of the artificial agent, and wherein the attention model is also applied to the gaze of the artificial agent. Optionally, the attention model includes a saliency factor, and objects that are jointly focused on by the artificial agent and the user have increased saliency. Optionally, the object is a person and / or an object. Optionally, the saliency factor includes a weighted saliency gaze factor based on saliency gaze maps of the user and the artificial agent. Optionally, the weighted saliency gaze factor is calculated as: a first weight multiplied by the gaze map of the artificial agent plus a second weight multiplied by the user gaze map plus a third weight multiplied by the gaze map of the artificial agent multiplied by the user gaze map. Optionally, the saliency factor comprises a weighted saliency point factor based on saliency point maps of the user and the artificial agent. Optionally, the weighted saliency point map factor is calculated as: a first weight multiplied by the point map of the artificial agent plus a second weight multiplied by the user point map plus a third weight multiplied by the point map of the artificial agent multiplied by the user point map. Optionally, the attention model is also applied to the user's eye movements, the user's actions, objects in the user's environment, and actions in the user's environment. Optionally, the objects are people and / or things. Optionally, the attention model is also applied to user input. Optionally, the user input comprises auditory or textual input. Optionally, the input comprises verbal input. Optionally, the user input comprises touch screen or user mouse movement. Optionally, the user input comprises visual input. Optionally, the visual input is a gesture or facial expression. Optionally, the gesture comprises movement of one of an arm, a hand, or a finger. Optionally, the attention model is also applied to the actions and actions of the artificial agent in the artificial agent background. Optionally, the user's eye corners and / or the user's head angle are used to track the user's eye movements and gaze. Optionally, the simulated embodied interaction between the artificial agent, the user and the digital content includes the user directing attention to an object in the digital content of the agent virtual space. Optionally, the method includes tracking the gaze of the artificial agent, and wherein simulating the embodied interaction is also based on a binary turn model applied to the gaze of the user and the artificial agent. Optionally, the artificial agent indicates the end of their turn by attempting direct gaze with the user during the interaction. Optionally, the artificial agent indicates the end of their turn by verbal cues during the interaction. Optionally, the artificial agent indicates the end of their turn by non-verbal cues during the interaction.Optionally, the non-verbal cues include a smile, a wink, movement of the head, movement of the body (including arms, hands and fingers). Optionally, when the user initiates direct gaze at the artificial agent, the artificial agent senses the user indicating the end of their turn during the interaction. Optionally, when the user uses a verbal cue, the artificial agent senses the user indicating the end of their turn. Optionally, when the user uses a non-verbal cue, the artificial agent senses the user indicating the end of their turn. Optionally, the non-verbal cues include a smile, a wink, movement of the head, movement of the body (including arms, hands and fingers). Optionally, the interaction occurs in a virtual reality environment and / or an augmented reality environment. Optionally, the artificial agent is not visible on the display during at least a portion of the interaction.

Claims

1. A method for visualizing interaction between an embodied artificial agent and digital content on an end-user display device of an electronic computing device, the method comprising the steps of: creating a proxy virtual environment having virtual environment coordinates; simulating the digital content in the proxy virtual environment; simulating the embodied artificial agent in the agent virtual environment; enabling the embodied artificial agent to interact with the simulated digital content; as well as displaying the interaction between the embodied human agent and the digital content on the end-user display device, Wherein when the end user makes changes to the digital content via the user interface, the changes are reflected in the presentation of the digital content in the proxy virtual environment.

2. The method according to claim 1, wherein: The virtual environment is a 3D virtual space, and the virtual environment coordinates are 3D coordinates.

3. The method according to claim 2, wherein: The interaction is one or more of the group consisting of: moving toward the digital content, viewing the digital content, gesturing toward the digital content, or touching the digital content item by moving a multi-jointed effector.

4. The method according to claim 3, wherein: Digital content is simulated on a plane within the proxy virtual environment.

5. A method for interacting with digital content on an electronic computing device via an embodied human agent, the method comprising the steps of: displaying the digital content to a user on a user interface on the electronic computing device; Creating a proxy virtual environment with virtual environment coordinates; simulating the digital content in the proxy virtual environment; simulating the embodied artificial agent in the agent virtual environment; enabling the embodied artificial agent to interact with the simulated digital content; converting the interaction into an actuation or manipulation of the digital content on the user interface; and displaying the interaction by superimposing, overlapping and / or merging the embodied artificial agent with digital content on the user interface, Wherein when the end user makes changes to the digital content via the user interface, the changes are reflected in the presentation of the digital content in the proxy virtual environment.

6. The method according to claim 5, wherein: The digital content is simulated in the proxy virtual environment as one or more pixels in the proxy virtual environment, each pixel having a coordinate location in the proxy virtual environment.

7. The method according to claim 6, wherein: The interaction is an intersection between coordinates of pixels corresponding to the digital content and coordinates of the body of the embodied artificial agent.

8. The method according to claim 7, wherein: Converting the simulated embodied interactions includes mapping the embodied interactions to input device events.

9. The method according to claim 8, wherein: A human input device event is one or more of the group consisting of: a mouse event, a keyboard event, or a touch screen event.

10. A system for facilitating interaction with an electronic computing device, the system comprising: at least one processor device; at least one memory device in communication with the at least one processor; an agent simulator module arranged to be executable by the processor to simulate an embodied agent; an interaction module arranged to be executable by the processor to convert digital content into conceptual objects perceivable by the embodied agent and enable the embodied agent to interact with the digital content; a rendering module arranged to be executable by the processor to render at least one of: the digital content, the embodied agent, and the interaction of the embodied agent with the digital content, user interface, Wherein when the end user makes changes to the digital content via the user interface, the changes are reflected in the presentation of the digital content in the proxy virtual environment.

11. The system according to claim 10, wherein: The interaction module is further arranged to convert embodied agent actions on the conceptual object into changes to the digital content.

12. The system according to claim 11, wherein: The interaction module associates a conceptual object with coordinates representing a position of the conceptual object relative to the embodied agent.

13. A system according to any one of claims 10 to 12, wherein: The interaction module associates a conceptual object with one or more affordances of corresponding digital content.

14. The system according to claim 13, wherein: The computing device provides site content, and the interaction module is JavaScript code integrated into the site content.

15. An embodied agent tool in a virtual environment created on an electronic computing device, the embodied agent tool being programmed to: receiving input from the real-world environment; receiving input from the virtual environment; and performing actions based on input from the real world environment and the virtual environment, in, receiving received input from the real world environment and the virtual environment in real time via a continuous feedback loop from both the real world environment and the virtual environment, Wherein when the end user makes changes to the digital content via the user interface, the changes are reflected in the presentation of the digital content in the virtual environment.