System and method for real-time assistance

Abstract

The real-time help system provides a visual display of help information on a user's graphical user interface. A help request is received from a client device, and a real-time help provider interface is launched at a real-time help location. Data related to the user's location (including data on an external device in the user's location) is obtained. Indicators are provided to allow the real-time help provider to point to or otherwise indicate items on or off the user interface. Real-time help input is captured at the real-time help provider interface. Instructions are then transmitted to a display of the client device to display the real-time help input as if an agent were present and interacting with or indicating items on or off the screen.

Description

Technical Field

[0001] The present invention relates to logic systems. More specifically, the present invention relates to computer systems and related systems and methods for providing interactive interfaces for real-time help. In other embodiments, one or more methods may be implemented on a data processing device or system such as a computer or other information-enabled device. Background Art

[0002] The discussion of any work, publication, product or activity anywhere in this submission, including any document filed with this application, is not an admission that any such work constitutes prior art. The discussion of any activity, work or publication is not an admission that such activity, work or publication exists or is known in any particular jurisdiction.

[0003] With the increase in complexity and popularity of interactive digital devices (such as computers, tablet computers, smart phones and home appliances), users of computing systems often expect to receive online help. In this case, online help can guide users to perform certain tasks by describing how to perform the tasks verbally or in written text. In some cases, remote control utilizes a personal computer or device. The assistant can move a cursor, enter text, or activate a button on a remote user device or instrument. This remote control method usually shows the assistant a view of the same screen as the user can see.

[0004] Another typical method of providing online assistance to users involves establishing a two-way video call or conference using a computing device with a front-facing camera. A common configuration of such a system involves the user viewing a live image of the caller on his or her computer screen, with, in some cases, an embedded window on the computer screen or displaying the user's own image on the user's screen. Summary of the invention

[0005] Systems and methods according to specific embodiments described herein provide a method of interaction between a user and real-time help, wherein both parties communicate with each other by visually referencing the physical locations of various features of the devices (and in some embodiments, surrounding devices or systems) with which the two parties are communicating.

[0006] The present invention allows a real-time human agent or automated assistant (hereinafter, interchangeably referred to as an "agent") to point to a portion of a user's display as if the user was standing on the other side of the display and the display was a window (or, optionally, in some embodiments, a mirror). The agent can guide the user to various components located on or outside the user's graphical user interface ("GUI") by pointing to features on or outside the user's GUI. In some embodiments, one or more physical features outside the user's display at the user's location or surrounding area are identifiable at the real-time help location. Therefore, the agent or the agent's interface is able to process or identify the geographic location of physical features outside the user's display area without viewing the physical features. In some embodiments, the system may be referred to as geographic real-time support.

[0007] In some embodiments, one or more sensing, detection or data capture devices (such as cameras, touch sensors, thermal sensors or sonar) can be used to detect the user's movement. In some embodiments, such devices can detect the movement of an agent or assistant. Such motion detection can function in addition to any pointing device provided with the computer system.

[0008] In some embodiments, the systems and methods described herein for real-time help provide a more intuitive understanding of instructions related to device features. In some embodiments, a real-time image or animated video image of the agent is shown to the user. In some embodiments, the layout of the agent's pointing device is displayed on the user's screen, as if the agent is pointing to a feature in the user's environment. In some embodiments, an image of the agent or a portion of the agent (such as a hand or arm) can be displayed superimposed on the user's GUI. The image can be translucent so that the user's GUI and elements of the agent are visible to the user.

[0009] The help system or method of the present disclosure can communicate with the user requesting help and the agent providing help through a communication network. The communication network can be any electronic communication network. The network connection can be through any medium, such as cellular service, the Internet, radio, television, etc. The interface of the user or service provider can be any appropriate computing device. For example, the device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, or a television.

[0010] In certain embodiments, methods, systems, and / or devices are described that can be used together or separately to provide the improved real-time help and optional related features described herein.

[0011] In other embodiments, the user system includes operable elements located in predetermined positions known to the system. These elements may include, for example, cameras, microphones, fingerprint readers, or sample box slots. In some implementations, the user interface may graphically display the location of specific elements when instructing the user to take action. These elements may be located in the user's GUI.

[0012] Various embodiments provide methods and / or systems for interacting over a communication network. In some embodiments, a user system or client system is provided with a GUI set, as described herein, which allows a user to view various data or information related to the user system and allows a user to view real-time help. In response to the user's action of requesting help and the completion of appropriate workflow steps, the user system sends the necessary information to the real-time help system to request help. The real-time help system processes the request using the request data and one or more sets of help data. In some embodiments, a client system is provided with an executable code file that allows the client system to operate as described herein.

[0013] An important application and an independent embodiment of the present invention relates to the field of exchanging data through the Internet or private networks. In some embodiments, Internet media protocols and formats such as HTTP, RTTP, XML, HTML, dHTML or VRML can be used. In some embodiments, image, audio or video formats or algorithms can be used. However, through the teachings provided herein, those skilled in the art will understand that the methods and apparatus of the present invention can be advantageously used in other related situations where users access content through communication channels such as modem access systems, institutional network systems and wireless systems.

[0014] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference.

[0015] Other aspects and advantages of the present disclosure will become apparent to those skilled in the art from the detailed description below, in which only illustrative embodiments of the present disclosure are shown and described. It will be appreciated that the present disclosure is capable of other and different embodiments, and that its many details are capable of modification in various obvious respects without departing from the present disclosure. Therefore, the drawings and descriptions are to be regarded as illustrative in nature, and not restrictive. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The included drawings are for illustrative purposes only and are only used to provide examples of possible structures and operations of the inventive systems, apparatuses, methods, and computer program products of the present disclosure for providing an interactive interface for real-time assistance. These drawings do not limit any changes in form and details that may be made by those skilled in the art without departing from the spirit and scope of the implementation of the present disclosure.

[0017] Figure 1 is a flow chart illustrating a method according to a specific embodiment.

[0018] Figure 2 is a block diagram illustrating a virtual transparent window interface according to a specific embodiment.

[0019] Figure 3 is an exemplary client computer system showing a client GUI and integrated devices and external devices according to specific embodiments.

[0020] Figure 4 is an exemplary agent GUI illustrating presentation of a client GUI and presentation of a client integration device and an external device according to a particular embodiment.

[0021] Figure 5 is a block diagram illustrating multiple client systems and a proxy system according to a particular implementation.

[0022] Figure 6 is a block diagram illustrating data interaction between a user and an agent according to a specific embodiment.

[0023] Figure 7 is a block diagram illustrating a representative exemplary logic device in which various aspects of the invention may be embodied.

[0024] Figure 8 An exemplary client computer system showing a client GUI and integrated devices and external devices according to a specific embodiment, and an exemplary agent GUI showing the presentation of a client GUI and the presentation of a client integrated device and external devices according to a specific embodiment. DETAILED DESCRIPTION

[0025] Examples of systems, devices, methods, and computer program products according to the disclosed implementations are described in this section. These examples are provided only to add context and help understand the implementations of the present disclosure. Therefore, it will be apparent to those skilled in the art that implementations can be practiced without some or all of these specific details. In other cases, certain operations are not described in detail to avoid unnecessary confusion to the implementations. Other applications may exist so that the following examples should not be considered as limitations or restrictions on the scope or settings.

[0026] In the detailed description below, reference is made to the accompanying drawings that are part of the specification, and specific implementations are shown in the accompanying drawings by way of illustration. Although these implementations are described in sufficient detail to enable those skilled in the art to practice the disclosed implementations, it should be understood that these examples are not restrictive, and therefore other implementations can be used and changed without departing from the spirit and scope of the disclosed implementations. For example, the operations of the methods shown and described herein are not necessarily performed in the order indicated. It should also be understood that the method may include more or less operations than indicated. In some implementations, the operations described herein as different operations may be combined. On the contrary, the operations that may be described herein as single operations may be implemented as multiple operations.

[0027] Certain implementations of the systems, apparatuses, methods, and computer program products of the present disclosure are configured to provide an interactive interface for real-time help.

[0028] In systems that provide online help, remote desktop control is often used to provide a live agent or automated assistant to view and control the user's screen. However, these systems are often limited. Remote control may be limited to what is visible on the screen, and often only displays the user's screen with icons. The agent cannot directly view or indicate any areas outside the screen. For many users, the help is ineffective.

[0029] For example, a food manufacturer named Acme Foods may operate a research and development laboratory that uses several machines to test, analyze, and identify the ingredients of food and raw material samples. One such machine is operated by a lab technician placing a food sample into a sample tray. The machine then analyzes the food sample and identifies the test results for the food sample. Other machines in the same room as this machine are used for the purpose of preparing food samples for proper analysis. For the lab technician, a machine that produces inconsistent results for food samples is problematic. The lab technician navigates to the help section of the machine, which provides an option to connect to a live agent for help with technical issues. The lab technician and the agent connect, and the live agent is able to remotely control the GUI of the food sample machine to diagnose any problems. Unfortunately, the GUI does not display any indication of the problem. The agent suspects that there is a problem with the food preparation performed in the other machines, but is unable to identify which machines they are or observe the behavior of these machines. In addition, the live agent wants to instruct the lab technician on how to perform touch gestures on the screen of the machine to properly calibrate the machine, but is unable to explain to the lab technician in a simple way what gestures to use and in which area of ​​the screen to operate. Therefore, the lab technician does not get enough help and is unable to solve the problem.

[0030] Some of the disclosed techniques may be implemented to provide a method of interaction between a user and real-time assistance, wherein the two parties communicate with each other by visually referencing the physical locations of various features of the devices that the two parties are communicating with and, in some embodiments, surrounding devices or systems. In some embodiments, the real-time agent may have the ability to physically point to or perform gestures at specific locations on the user's screen. In some embodiments, the real-time agent may also be able to identify, observe, or indicate different devices at the user's location or surrounding area.

[0031] By applying certain embodiments of the disclosed technology, an alternative to the above-described solution is proposed. When a laboratory technician connects to a live agent for help, the live agent can see the user's display as if seen through a window, and the live agent is provided with all GUI elements as if the agent and the laboratory technician are standing at opposite ends of the window. The agent can point to multiple locations of the GUI on the food sample machine and perform gestures at multiple locations of the GUI on the food sample machine. Therefore, in order to demonstrate how to calibrate the machine, the agent performs a touch gesture on the lower part of the screen. The laboratory technician observes the gestures that appear on the screen in real time through the presentation of the finger touch on the screen, and understands how to correctly calibrate the machine. In addition, the agent can observe the presentation of other devices in the user's room. The agent can point to another device that the user in the user's room needs to calibrate correctly so that the food sample machine runs as expected. The laboratory technician also correctly calibrates the machine, thereby solving the problem. Through the technology of real-time help proposed in this article, Acme Foods can solve the technical problems of the machines in its research and development laboratory. In this way, the alternative shows the improvement of real-time help and help related to one or more networked devices.

[0032] operate

[0033] Figure 1 A flow chart of an exemplary method 100 for providing an interactive interface for real-time help is shown.

[0034] At block 110, the system receives a request for help at a live help location and launches an interface of a live help provider. In some embodiments, a live help session is typically initiated when a request for live help is received at an agent's location or workstation. In some embodiments, the live help system provides an interface, such as a GUI, to the agent, which allows the agent to indicate items on the user's GUI as well as items or devices external to the user's GUI. In some embodiments, the agent can see, for example, Figure 4 A representation of a user's GUI and devices external to the user's GUI shown in .

[0035] At block 120, the system obtains data stored in the agent's system representing the layout or location of one or more external devices. In some embodiments, the data about the external devices can be provided by a camera (such as multiple cameras or a fisheye camera) in the user system that captures the devices around the user system.

[0036] At box 130, the interface at the agent's location provides indicators to allow the real-time help provider to point to or otherwise indicate items on the user's GUI and outside the user's GUI. These indicators can be arranged and displayed in a variety of ways by the real-time help system to provide a simple method for the agent to understand the user's location and provide help. In some embodiments, the agent provides help by pointing to or otherwise indicating an object on the user's GUI and pointing to or otherwise indicating a device outside the user's GUI. In some embodiments, a virtual reality environment is presented at the agent's location, allowing the agent to indicate actions related to multiple objects in the user's environment. In some embodiments, an augmented reality environment and a mixed reality environment can be presented at the agent's location. In some embodiments, the agent can use one or more devices that provide a virtual reality environment, an augmented reality environment, or a mixed reality environment, such as headphones, glasses, or a controller.

[0037] At box 140, the real-time help system captures the agent's actions. In some embodiments, the agent's actions may include pointing, gesturing, indicating, or otherwise interacting with one or more elements on or outside the user interface. In some embodiments, the agent's actions include indicating one or more external devices outside the client device. In some embodiments, the real-time help system captures the agent's actions in real time or substantially in real time as the agent performs the action. The action can be captured by one or more detection and capture devices (such as, a pointing device, a keyboard, a motion capture device, or an imaging device). In some embodiments, a motion capture device including a sensing or capture component (such as, a camera, a proximity sensor, a smart glove, a sonar, or an echolocation sensor) can be used.

[0038] At block 150, the real-time help system converts the agent's actions into instructions for display at the user system and transmits these display instructions to the user system. Real-time help is displayed at the user system using one or more of the following: graphical instructions related to help for objects on the user's GUI or devices external to the user's GUI; real-time video showing the agent indicating an object or demonstrating an operation; text entered by the agent; and audio from the agent.

[0039] User Location and User Interface

[0040] As described herein, a user's location generally refers to at least a user's computing system and its integrated components. In other embodiments, a user's location generally may refer to an area near the user's computer system or other area associated with the user's computer system. Such an area may include one or more external devices or apparatuses.

[0041] As described herein, user devices, client devices or instruments may generally be understood as general consumer computing products (such as smart TVs, tablet computers, laptop computers, smart phones, other smart home appliances, etc.). However, in some embodiments, user devices may also be understood as any logic-enabled device that provides a user display and / or interface system, such as but not limited to medical diagnosis or treatment systems, industrial systems, experimental or research systems, and dedicated workstations.

[0042] As further described herein, user locations may include a variety of different devices with known locations or locations that can be determined by reference to a user's GUI. For example, a workbench at a medical diagnostic laboratory station typically has at least one device with a GUI, such as a computer system, and may have dozens of different types of devices spatially arranged around the device in a usual manner. In some embodiments, the device may include electrical components or other components integrated with the user's computer system, such as space for accommodating a sample box or similar device. The device may also include non-integrated devices or objects, such as independent electronic devices, test tubes, or any other device.

[0043] In some embodiments, the location of the user and / or the location of one or more devices at the user's location can be determined by a variety of positioning technologies. For example, the system can determine the location of the user or device by GPS, RFID or NFC technology.

[0044] In some embodiments, multiple devices or users may be connected to each other in a networked or distributed manner. One or more devices may communicate with other devices at the user's location to provide information about their location, status, etc. The agent may receive such networked or distributed data on the device and may use such data to provide assistance. For example, a device may communicate with another connected device that has not been calibrated before performing an operation. The agent may receive such information and then use such information to provide a solution to the user regarding which device at the user's location needs to be calibrated.

[0045] Systems and methods according to particular embodiments as described herein provide real-time help on a graphical user display or GUI that more directly and naturally indicates an object at the user's location, whether the object is on the user's screen or in the user's environment.

[0046] Figure 2 2 is a block diagram showing a virtual transparent window interface according to a specific embodiment. In some embodiments, the "transparent window" interface 210 allows the user 220 to observe instructions and actions on his interface, as if the instructions and actions are being provided by the assistant 230 on the other side of the "transparent window". According to a specific embodiment, the user 220 is like seeing real-time faces and pointing or other actions through the "transparent window". In other embodiments, the user 220 can see representative actions such as finger touches, finger pointing, gestures or hand movements (such as, picking up equipment). In some embodiments, the representative action occurs when the user 220 cannot see the complete projection of the person. Help information can be displayed in a separate "window" on the display screen, or it can be covered to allow the user 220 to see various aspects of the user's GUI and see real-time help actions through the GUI or behind the GUI.

[0047] Figure 3 It is an exemplary client computer system showing the client GUI and integrated devices and external devices according to a specific embodiment. As an example of some specific embodiments only, consider the user system or instrument in a medical laboratory. Such a system may include a screen as shown, optionally include one or more integrated devices (such as, QRC reader) and optionally include one or more external devices (such as, different sample collection tubes). In some embodiments, when real-time help is being provided, the user's GUI displays a real-time image of a real-time agent, which includes a hand 310 of the agent pointing to the user's GUI element or to one or more external devices 320 in the user's environment. In other embodiments, the user's GUI can display the presentation of real-time help gestures, such as, the presentation of fingers or hands. In other embodiments, the user can see representative actions such as finger touch, finger pointing, hand movements (such as, picking up equipment), etc., without seeing the complete projection of the person.

[0048] In some embodiments, real-time help or instructions can use a "mirror interface" that allows the user to see instructions and actions on his interface as if he were looking in a mirror and the actions were being performed by an assistant. That is, the display of real-time help looks like a mirror image of the user's perspective. Such an interface can be used to provide instructions on the use of home medical devices (such as fingerstick devices, bandages, etc.), or for providing touch-up or makeup instructions. In some embodiments, a combination of a mirror interface and a transparent or translucent interface can provide multiple ways to view information.

[0049] In some embodiments, the display of real-time help looks as if the user and real-time help are positioned on the opposite sides of transparent window. In this case, real-time help can have the display of the relative position of not only indicating the content presented on the GUI of the user but also indicating peripheral equipment, and this relative position is shown in mirror image mode. Therefore, the item on the right side of user equipment and display appears on the left side of the real-time help display. The writing from left to right on the user system is shown in the mode of letter reversal from right to left on the real-time help display. Figure 8 An example of this embodiment is shown in . In this case, the user environment (including the GUI) is displayed on the real-time help GUI as if viewed in a mirror or through a transparent plate.

[0050] Live Help Provider Interface

[0051] In some embodiments, the real-time help system or method as described herein can provide some automated instructions or help, and usually will also provide help from a real-time agent. The real-time help interface displays to the help provider a representation of the user's location that is sufficient to enable the real-time help provider to provide meaningful gesture help. In some embodiments, the interface of the real-time help provider can also provide a "transparent window" display to the real-time help assistant, thereby showing the user and the user's location. However, in other embodiments, for the user's privacy or other reasons, the real-time help interface can only display a representation of the user's location.

[0052] Figure 4 is an exemplary agent GUI showing a representation of a client GUI and representations of client integrated devices and external devices according to a specific embodiment. As shown in this embodiment, the agent interface typically includes on its display representations of elements external to the user's display screen. The display allows the help provider to point to or indicate objects in the user's environment by indicating or pointing to graphical representations of objects in the user's environment.

[0053] Depending on the implementation, the systems and methods described herein store or retrieve relative spatial information of various parts of the user's computing device, and optionally also store or retrieve relative spatial information of objects at the user's location. In the diagnostic lab workbench example, this may occur if different labs position key equipment (such as centrifuges) in different arrangements relative to the GUI. In specific implementations, specific spatial information is used to provide more context-sensitive real-time help.

[0054] In addition, if Figure 4 As shown in , the interactions on the agent GUI are converted into appropriate displays on the user's GUI. For example, Figure 4An arrow 410 is shown representing where the agent is pointing on the agent GUI. In this example, the agent is pointing at a representation of an external component 420. What the user sees on his system is a hand icon or other pointing indication 430 pointing at the actual external component 420 at the user's system. In some embodiments, a representation 430 of the user pointer is shown at the agent graphical interface for clarity purposes.

[0055] In some embodiments, an external device may be placed behind the user's GUI. For example, a camera, fingerprint reader, or barcode scanner may be placed behind the GUI. For example, to capture a fingerprint, a finger may be placed at the location of the reader on the GUI. To scan a barcode, an item with a barcode may be placed on the scanner in the GUI. To take a picture, a person or object to be photographed may be positioned at a location in the GUI where the camera is located behind the GUI. Figure 4 The integral device is shown to be located within the scope of the user's GUI. The real-time help GUI has a representation of the user's GUI. The agent can instruct the user to use the integral device located behind the GUI by pointing to the location on the user's GUI where the integral device is located. This Figure 8 , where real-time help points to the location of the device on the user's GUI.

[0056] Example Configuration of Multiple Systems

[0057] Figure 5 is a block diagram showing multiple client systems and proxy systems according to specific embodiments. It should be understood that the various novel systems and methods described herein are only performed in one system, and therefore, multiple systems or multiple parties are not necessary to implement the present invention except as specifically provided in the claims.

[0058] Figure 5Three representative client systems 2 are shown. These systems can be any type of computing system known in the art, and these systems are labeled from 1 to n to represent any number of client systems. As will be understood in the art, a set of GUIs are provided at or to each system through a network such as network 5. These GUIs can be substantially the same in a variety of systems 2, or can be changed according to different systems. According to a specific embodiment, at least one of these GUIs allows a user to request real-time help as described herein, and to view the display of real-time help. The LH module 20 is any code or data at the client for providing real-time help as described herein. In some embodiments, the module 20 is entirely composed of data and executable code transmitted through the network. In other embodiments, the module 20 includes client-specific data, such as user configuration data (whether the user wants real-time help to automatically activate, for example, a camera or any other user-related data). In some embodiments, as discussed above, the module 20 includes location data representing the location of items that are not part of the client GUI but are located outside the area of ​​the display screen. The client system 2 may include one or more user interface components that allow a remote agent to see the user's actions or otherwise perceive the user's actions. Such systems typically include one or more camera components, and may also include other components capable of recording user actions.

[0059] Figure 5 Also shown are three representative agent systems 4. These systems may also be any type of computing system known in the art, and these systems are labeled from 1 to n to represent any number of agent systems. As will be understood in the art, a set of GUIs are provided at or to each system via a network such as a network 5. These GUIs may be substantially the same in a variety of systems 4, or may be changed according to different systems. According to a specific embodiment, at least one of these GUIs allows an agent to respond to a real-time help request and view the display of other items (such as, images of the user's face or body or a variety of devices) at the user's location. HS module 40 is any code or data at an agent system for providing real-time help as described herein. In some embodiments, module 40 is entirely composed of data transmitted over a network and executable code. In other embodiments, module 40 includes client-specific data, such as user configuration data (whether or not the user wishes real-time help to automatically activate, for example, a camera or any other user-related data). In some embodiments, as discussed above, module 40 includes location data representing the location of items that are not part of the client's GUI but are located outside the area of ​​the display screen. Agent system 4 may include one or more interface components that allow a client to observe a remote agent and observe the agent's actions with respect to the client's GUI and environment.

[0060] The network 5 may be any network or combination of networks of devices that can communicate with each other in the real-time help system. For example, the network 5 may be any one or any combination of a LAN (local area network), a WAN (wide area network), a telephone network, a wireless network, a point-to-point network, a star network, a token ring network, a hub network, or other suitable configurations. The network 5 may include a TCP / IP (Transmission Control Protocol and Internet Protocol) network, such as the global interconnected network of networks commonly referred to as the Internet.

[0061] In a particular embodiment, the agent system 4 includes one or more components that allow the agent to indicate objects on or outside the user's GUI. Such components may include a touch screen, a mouse or trackball interface, one or more cameras, or one or more other position sensors.

[0062] In other embodiments, a real-time help server 4 is provided, which can communicate with one or more agent systems and optionally with one or more client systems. The server system may include a primary or backup storage for client data (including location data) and may provide one or more GUIs displayed at the client system or agent system.

[0063] Real-time help client / server data model

[0064] Figure 6 650 is a block diagram illustrating the interaction of user and agent data according to a specific embodiment. The user system or instrument 650 has a GUI interface 651, which displays data according to instructions stored at the system 650 or received through a communication channel 660. The user's GUI 651 can be a customized or proprietary GUI system, or can be rendered in whole or in part using a standardized display such as an HTML-based browser. The user system typically stores identification data 652, which can include one or more system IDs (such as, IP address or MAC address or other ID), system configuration data including the layout of any components outside the GUI. At the beginning of a real-time help session, a request for real-time help is received from the user system or instrument 650, and the request typically includes some indication of the user's identification data 652.

[0065] The agent system 640 receives the request and uses the received data and, depending on the selection, stored user data 644 to generate the agent interface using module 642. The interface module 642 captures input from the agent provided by any input device and converts the input into display commands suitable for the user for real-time assistance and sends these commands using module 643.

[0066] Implementation in programmed information devices

[0067] Figure 7 is a block diagram showing a representative exemplary logic device in which various aspects of the present invention may be embodied. As will be understood by those skilled in the art based on the teachings provided herein, the present invention may be implemented in hardware and / or software. In certain embodiments of the present invention, different aspects of the present invention may be implemented in client-side logic or server-side logic. As will be understood in the art, the present invention or components of the present invention may be implemented in a fixed medium program component containing logic instructions and / or data, which, when loaded into a suitably configured computing device, causes the device to execute according to the present invention. As will be understood in the art, the fixed medium containing the logic instructions may be delivered to a user on a fixed medium to be physically loaded into the user's computer, or the fixed medium containing the logic instructions may be located on a remote server so that the user accesses the remote server via a communication medium to download the program component.

[0068] Figure 7 An information device (or digital device) 700 is shown, which can be understood as a logic device that can read instructions from a medium 717 and / or a network port 719, and the logic device can optionally be connected to a server 720 with a fixed medium 722. As understood in the art, the device 700 can then use these instructions to guide the server or client logic to implement various aspects of the present invention. One type of logic device that can implement the present invention is a computer system as shown in 700, which includes a CPU 707, optional input devices 709 and 711, a disk drive 715, and an optional monitor 705. Such a system can be programmed using a fixed medium 717 or a fixed medium 722 through a network port 719, and the fixed medium 717 or 722 can represent a disk-type optical or magnetic medium, a tape, a solid-state dynamic or static memory, etc. In a specific embodiment, the present invention can be implemented in whole or in part as software recorded on the fixed medium. Network port 719 may also be used to initially receive instructions for programming such a system and may represent any type of communications connection.

[0069] The present invention may also be implemented in whole or in part in a circuit of an application specific integrated circuit (ASIC) or a programmable logic device (PLD). In such a case, the present invention may be implemented in a computer-understandable descriptor language that can be used to create an ASIC or PLD that operates as described herein.

[0070] Other Implementations

[0071] The present invention has been described above with reference to specific embodiments. Other embodiments will be apparent to those skilled in the art. Specifically, the user digital information device is generally shown as a personal computer. However, the digital computing device is intended to be any information device for interacting with remote data applications, and may include devices such as digital technology-enabled televisions, cell phones, personal digital assistants, laboratory equipment, or manufacturing equipment. It should be understood that the examples and embodiments described herein are for illustrative purposes, and the teachings herein will provide those skilled in the art with inspiration for various modifications or changes based on these examples and embodiments, which should be included in the spirit and scope of this application and within the scope of the claims.

[0072] In addition, a variety of different actions can be used to achieve communication between the user and the agent. For example, the user can speak a voice command, indicate a key or screen area, indicate a button or screen area on a related module or component, or make a selection using any pointing device.

[0073] All publications, patents, and patent applications cited herein or filed with this application (including any references filed as part of the Information Disclosure Statement) are hereby incorporated by reference in their entirety.

[0074] This document describes general structures and techniques, as well as more specific implementations that can be used to implement different ways of performing the more general goals.

[0075] Although only certain embodiments are disclosed in detail above, other embodiments are possible, and one or more inventors intend to include these embodiments in this specification. This specification describes specific examples for achieving a more general goal, which can be achieved in another way. This disclosure is intended to be exemplary, and the claims are intended to cover any modifications or substitutions that may be predicted by a person of ordinary skill in the art.

[0076] Furthermore, it is the inventor's intention that only those claims that use the term "means for..." shall be interpreted in accordance with 35 USC 112, sixth paragraph. Furthermore, no limitations in this specification are intended to be incorporated into any claim for interpretation unless such limitations are expressly included in the claim. The logic components described herein may be any type of component, either general purpose or for some specific purpose. The component architecture may be an Intel or AMD based architecture or any other logic processing architecture. Operating systems such as Linux, Unix, Windows, etc. may be included to provide various information processing functions. One or more components may include a handheld computer such as a PDA, a cell phone or a laptop computer, a handheld camera, etc.

[0077] The program may be written in C, Python, Java, Brew, or any other programming language. The program may be loaded on a storage medium, such as a magnetic or optical storage medium, for example, a computer hard drive; a removable disk or medium, such as a memory stick or SD media, a network attached storage (NAS) or other removable medium based on a wired or wireless network or based on Bluetooth; or other removable medium. The program may also be run over a network, for example, where a server or other machine sends a signal to a local machine, which allows the local machine to perform the operations described herein.

[0078] When a specific value is mentioned herein, it should be understood that the value can be increased or decreased by 20% while still falling within the teachings of the present application unless a different range is specifically mentioned. When a specified logical meaning is used, the opposite logical meaning is also intended to be encompassed.

[0079] Software Implementation

[0080] Various embodiments involving methods and / or systems for biometrics or identification information can be implemented in general or dedicated information processing devices or logic-enabled systems using suitable programming languages ​​and any suitable data or format specifications, such as laboratory systems, diagnostic systems, or production systems, programming languages ​​such as Perl, Python, Java, C++, C#, Cobol, C, Pascal, Fortran, PL1, LISP, assembly languages, and data or format specifications such as HTML, XML, dHTML, TIFF, JPEG, tab-delimited text or binary. For the sake of clarity, not all features of the actual implementation are described in this specification. It will be understood that in developing any such actual implementation (as in any software development project), many implementation-specific decisions must be made to achieve the developer's specific goals and sub-goals, such as compliance with system-related and / or business-related constraints, which are different for each implementation. In addition, it will be understood that such development work may be complex and time-consuming, but it is still a routine software engineering work for ordinary technicians who benefit from the present disclosure.

[0081] In addition, as is well known in the art, logic systems and methods as described herein can include various components and different functions in a modular manner. Different embodiments of the present invention can include different mixtures of elements and functions, and various functions can be grouped as a part of various elements. For the sake of clarity, the present invention is described in a system comprising many different novel components and novel combinations of novel components and known components. It should not be inferred that the present invention is limited to the combination of all novel components listed in any illustrative embodiment in this specification sheet.

[0082] Although preferred embodiments of the present invention have been shown and described herein, it is obvious to those skilled in the art that such embodiments are provided only as examples. It is not intended that the present invention be limited by the specific examples provided in the specification. Although the present invention has been described with reference to the foregoing description, the description and illustration of the embodiments herein are not meant to be interpreted in a limiting sense. At this point, many variations, changes and substitutions will occur to those skilled in the art without departing from the present invention. In addition, it should be understood that all aspects of the present invention are not limited to the specific depictions, configurations or relative proportions set forth herein, which depend on various conditions and variables. It should be understood that various alternatives to the embodiments of the present invention described herein may be used when implementing the present invention. Therefore, it should be appreciated that the present invention will also encompass any such substitutions, modifications, variations or equivalents. The appended claims are intended to define the scope of the present invention, and thus encompass methods and structures within the scope of these claims and their equivalents.

Claims

1. A method for providing real-time assistance to a user of a client device, the method comprising: receiving a request for help from said client device at a user's location; Obtain from the client device: external device data identifying additional devices external to and proximate to the client device, the additional devices selected from the group consisting of medical devices, diagnostic devices, laboratory devices, and research devices; and position data representing a spatial relationship of the additional device relative to the client device; in response to receiving the help request, displaying a real-time help provider graphical user interface that depicts a provider graphical representation of the client device and corresponding provider graphical representations of the additional devices in a manner that identifies each of the additional devices at the user's location and depicts a spatial relationship of each of the additional devices relative to the client device; processing inputs from a real-time assistance provider using the real-time assistance provider graphical user interface, wherein each input includes one of the respective provider graphical representations specifying the additional device; as well as Instructions are transmitted to the client device, each instruction is configured to display an indicator at a graphical user interface of the client device, the graphical user interface of the client device depicting a corresponding client graphical representation of the additional device at the user's location in a manner that depicts the spatial relationship of each of the additional devices relative to the client device, the instructions respectively corresponding to input from the real-time help provider, so that each indicator specifies one of the corresponding client graphical representations of the additional device at the graphical user interface of the client device, thereby spatially indicating the additional device among the additional devices that the user wants to operate.

2. The method according to claim 1, further comprising: One or more real-time help elements are displayed or are to be displayed at the graphical user interface of the client device, the one or more real-time help elements presenting the graphical user interface of the client device as if it were a transparent or semi-transparent window, and the real-time help provider appears to be touching, gesturing or indicating from an opposite side of the window.

3. The method according to claim 1, further comprising: Input is received from the real-time help provider, the input coming from one or more of a pointing device, a gesture device, a keyboard, a motion capture device, and an imaging device.

4. The method according to claim 3, wherein: The motion capture device includes one or more of a camera, a proximity sensor, a smart glove, a sonar, and an echolocation sensor.

5. The method according to claim 1, wherein: The real-time help provider is one or both of a remote human assistant and an automated help service.

6. The method according to claim 1, further comprising: A virtual reality environment is presented at the real-time help provider graphical user interface, the virtual reality environment being configured to allow the real-time help provider to indicate one or more actions corresponding to one or more of the additional devices at the user's location.

7. A system for providing real-time assistance to a user of a client device, the system comprising: processor; as well as A memory storing instructions, wherein the instructions are configured to cause the system to: receiving a request for help from said client device at a user's location; Obtain from the client device: external device data identifying additional devices external to and proximate to the client device, the additional devices selected from the group consisting of medical devices, diagnostic devices, laboratory devices, and research devices; and position data representing a spatial relationship of the additional device relative to the client device; in response to receiving the help request, displaying a real-time help provider graphical user interface that depicts a provider graphical representation of the client device and corresponding provider graphical representations of the additional devices in a manner that identifies each of the additional devices at the user's location and depicts a spatial relationship of each of the additional devices relative to the client device; processing inputs from a real-time assistance provider using the real-time assistance provider graphical user interface, wherein each input includes one of the respective provider graphical representations specifying the additional device; as well as Instructions are transmitted to the client device, each instruction is configured to display an indicator at a graphical user interface of the client device, the graphical user interface of the client device depicting a corresponding client graphical representation of the additional device at the user's location in a manner that depicts the spatial relationship of each of the additional devices relative to the client device, the instructions respectively corresponding to input from the real-time help provider, so that each indicator specifies one of the corresponding client graphical representations of the additional device at the graphical user interface of the client device, thereby spatially indicating the additional device among the additional devices that the user wants to operate.

8. The system according to claim 7, wherein: The instructions are also configured to cause the system to: One or more real-time help elements are displayed at the graphical user interface of the client device, the one or more real-time help elements presenting the graphical user interface of the client device as if it were a transparent or semi-transparent window, and the real-time help provider appears to be touching, gesturing, or indicating from an opposite side of the window.

9. The system according to claim 7, wherein: The one or more inputs of the real-time help provider come from one or more of a pointing device, a gesture device, a keyboard, a motion capture device, and an imaging device.

10. The system according to claim 9, wherein: The motion capture device includes one or more of a camera, a proximity sensor, a smart glove, a sonar, and an echolocation sensor.

11. The system according to claim 7, wherein: The real-time help provider is one or both of a remote human assistant and an automated help service.

12. The system according to claim 7, wherein: The instructions are also configured to cause the system to: A virtual reality environment is presented at the real-time help provider graphical user interface, the virtual reality environment being configured to allow the real-time help provider to indicate one or more actions corresponding to one or more of the additional devices at the user's location.

13. A computer program product comprising a non-transitory computer readable medium and computer readable program code executable by one or more processors when retrieved from the non-transitory computer readable medium, the program code comprising instructions configured to: receiving a request for help from a client device located at a user location; Obtain from the client device: external device data identifying additional devices external to and proximate to the client device, the additional devices selected from the group consisting of medical devices, diagnostic devices, laboratory devices, and research devices; and position data representing a spatial relationship of the additional device relative to the client device; in response to receiving the help request, displaying a real-time help provider graphical user interface that depicts a provider graphical representation of the client device and corresponding provider graphical representations of the additional devices in a manner that identifies each of the additional devices at the user's location and depicts a spatial relationship of each of the additional devices relative to the client device; processing inputs from a real-time assistance provider using the real-time assistance provider graphical user interface, wherein each input includes one of the respective provider graphical representations specifying the additional device; as well as Instructions are transmitted to the client device, each instruction is configured to display an indicator at a graphical user interface of the client device, the graphical user interface of the client device depicting a corresponding client graphical representation of the additional device at the user's location in a manner that depicts the spatial relationship of each of the additional devices relative to the client device, the instructions respectively corresponding to input from the real-time help provider, so that each indicator specifies one of the corresponding client graphical representations of the additional device at the graphical user interface of the client device, thereby spatially indicating the additional device among the additional devices that the user wants to operate.

14. The computer program product of claim 13, wherein: The instructions are also configured to perform the following operations: One or more real-time help elements are displayed at the graphical user interface of the client device, the one or more real-time help elements presenting the graphical user interface of the client device as if it were a transparent or semi-transparent window, and the real-time help provider appears to be touching, gesturing, or indicating from an opposite side of the window.

15. The computer program product of claim 13, wherein: The instructions are also configured to perform the following operations: One or more inputs are received from the real-time help provider, the one or more inputs coming from one or more of a pointing device, a gesture device, a keyboard, a motion capture device, and an imaging device.

16. The computer program product of claim 13, wherein: The real-time help provider is one or both of a remote human assistant and an automated help service.

17. The computer program product of claim 13, wherein: The instructions are also configured to perform the following operations: A virtual reality environment is presented at the real-time help provider graphical user interface, the virtual reality environment being configured to allow the real-time help provider to indicate one or more actions corresponding to the additional device at the user's location.

18. A method for providing real-time assistance to a user of a client device, the method comprising: displaying a real-time help provider graphical user interface that depicts a provider graphical representation of the client device at the user location and a corresponding provider graphical representation of the additional device at the user location in a manner that identifies each of the additional devices at the user location and depicts a spatial relationship of each of the additional devices relative to the client device; processing input from a real-time assistance provider using the real-time assistance provider graphical user interface, the input specifying a referenced one of the corresponding provider graphical representations of the additional devices; as well as Instructions are transmitted to the client device, the instructions being configured to display an indicator at a graphical user interface of the client device, the graphical user interface of the client device depicting corresponding client graphical representations of the additional devices at the user location in a manner that depicts the spatial relationship of each of the additional devices relative to the client device, the instructions corresponding to input from the real-time help provider such that the indicator specifies one of the corresponding client graphical representations of the additional devices at the graphical user interface of the client device, thereby spatially indicating a referenced one of the additional devices.

19. The method according to claim 18, further comprising: One or more real-time help elements are displayed or are to be displayed at the graphical user interface of the client device, the one or more real-time help elements presenting the graphical user interface of the client device as if it were a transparent or semi-transparent window, and the real-time help provider appears to be touching, gesturing or indicating from an opposite side of the window.

20. The method of claim 18, further comprising: Input is received from the real-time help provider, the input coming from one or more of a pointing device, a gesture device, a keyboard, a motion capture device, and an imaging device.

21. The method according to claim 20, wherein: The motion capture device includes one or more of a camera, a proximity sensor, a smart glove, a sonar, and an echolocation sensor.

22. The method according to claim 18, wherein: The real-time help provider is one or both of a remote human assistant and an automated help service.

23. The method of claim 18, further comprising: A virtual reality environment is presented at the real-time help provider graphical user interface, the virtual reality environment being configured to allow the real-time help provider to indicate one or more actions corresponding to one or more of the additional devices at the user's location.

24. The method according to claim 18, wherein: The additional equipment is selected from the group consisting of medical equipment, diagnostic equipment, laboratory equipment and research equipment.

25. A system for providing real-time assistance to a user of a client device, the system comprising: processor; as well as A memory storing instructions, wherein the instructions are configured to cause the system to: displaying a real-time help provider graphical user interface that depicts a provider graphical representation of the client device at the user location and a corresponding provider graphical representation of the additional device at the user location in a manner that identifies each of the additional devices at the user location and depicts a spatial relationship of each of the additional devices relative to the client device; processing input from a real-time assistance provider using the real-time assistance provider graphical user interface, the input specifying a referenced one of the corresponding provider graphical representations of the additional devices; as well as Instructions are transmitted to the client device, the instructions being configured to display an indicator at a graphical user interface of the client device, the graphical user interface of the client device depicting corresponding client graphical representations of the additional devices at the user location in a manner that depicts the spatial relationship of each of the additional devices relative to the client device, the instructions corresponding to input from the real-time help provider such that the indicator specifies one of the corresponding client graphical representations of the additional devices at the graphical user interface of the client device, thereby spatially indicating a referenced one of the additional devices.

26. The system of claim 25, wherein: The instructions are also configured to cause the system to: One or more real-time help elements are displayed at the graphical user interface of the client device, the one or more real-time help elements presenting the graphical user interface of the client device as if it were a transparent or semi-transparent window, and the real-time help provider appears to be touching, gesturing, or indicating from an opposite side of the window.

27. The system of claim 25, wherein: One or more of the inputs from the real-time help provider are from one or more of a pointing device, a gesture device, a keyboard, a motion capture device, and an imaging device.

28. The system of claim 27, wherein: The motion capture device includes one or more of a camera, a proximity sensor, a smart glove, a sonar, and an echolocation sensor.

29. The system of claim 25, wherein: The real-time help provider is one or both of a remote human assistant and an automated help service.

30. The system of claim 25, wherein: The instructions are also configured to cause the system to: A virtual reality environment is presented at the real-time help provider graphical user interface, the virtual reality environment being configured to allow the real-time help provider to indicate one or more actions corresponding to one or more of the additional devices at the user's location.

31. The system of claim 25, wherein: The additional equipment is selected from the group consisting of medical equipment, diagnostic equipment, laboratory equipment and research equipment.

32. A computer program product comprising computer readable program code executable by one or more processors when retrieved from a non-transitory computer readable medium, the program code comprising instructions configured to: displaying a real-time help provider graphical user interface that depicts a provider graphical representation of the client device at the user location and a corresponding provider graphical representation of the additional device at the user location in a manner that identifies each of the additional devices at the user location and depicts a spatial relationship of each of the additional devices relative to the client device; processing input from a real-time assistance provider using the real-time assistance provider graphical user interface, the input specifying a referenced one of the corresponding provider graphical representations of the additional devices; as well as Instructions are transmitted to the client device, the instructions being configured to display an indicator at a graphical user interface of the client device, the graphical user interface of the client device depicting corresponding client graphical representations of the additional devices at the user location in a manner that depicts the spatial relationship of each of the additional devices relative to the client device, the instructions corresponding to input from the real-time help provider such that the indicator specifies one of the corresponding client graphical representations of the additional devices at the graphical user interface of the client device, thereby spatially indicating a referenced one of the additional devices.

33. The computer program product of claim 32, wherein: The instructions are also configured to perform: One or more real-time help elements are displayed at the graphical user interface of the client device, the one or more real-time help elements presenting the graphical user interface of the client device as if it were a transparent or semi-transparent window, and the real-time help provider appears to be touching, gesturing, or indicating from an opposite side of the window.

34. The computer program product of claim 32, wherein: The instructions are also configured to perform: One or more of the inputs are received from the real-time help provider, the inputs coming from one or more of a pointing device, a gesture device, a keyboard, a motion capture device, and an imaging device.

35. The computer program product of claim 32, wherein: The real-time help provider is one or both of a remote human assistant and an automated help service.

36. The computer program product of claim 32, wherein: The instructions are also configured to perform: A virtual reality environment is presented at the real-time help provider graphical user interface, the virtual reality environment being configured to allow the real-time help provider to indicate one or more actions corresponding to one or more of the additional devices at the user's location.

Images