Remote assistance workstation, method and system with user interface

Abstract

A remote assistance workstation 12 includes a communication module 54, a user interface (UI) module 52, and a controller 56 coupled to a portable device 14, the portable device including augmented reality (AR) glasses 36 worn by a first responder to perform actions using objects in a scene. The UI module 52 renders a remote assistant graphical user interface (GUI) 100, the GUI including (i) a first pane 96 for displaying a live video stream of a remote assistance request and (ii) a second pane 98 for displaying a 2D representation of an object in the scene, which can be moved within the scene via remote assistant input. The GUI 100 renders a corresponding 3D virtual content item within the first pane relative to a reference point. The controller 56 outputs a remote assistance signal to the portable device 14 to display a 3D virtual content item 38 on the AR glasses in a live view of the scene, appearing at a location determined by the remote assistant input of moving the 2D representation within the second pane 98 to assist the first responder in the scene.

Description

[0001] This application is a divisional application filed on October 12, 2016, with application number 201680063774.6 and entitled “Remote Assistance Workstation, Method and System Having a User Interface for Remotely Assisting Spatial Placement Tasks via Augmented Reality Glasses”. Technical Field

[0002] The present invention relates generally to the provision of remote assistance, and more specifically to remote assistance workstations, methods and systems having a user interface for providing remote assistance for spatial placement tasks via augmented reality glasses. Background Technology

[0003] Various methods and systems are known in the art in which a remote assistant or expert provides guidance to a person in need of assistance using some form of video collaboration or other remote assistance tools. One known method involves interacting with an augmented reality display, such as augmented reality glasses, and is facilitated by receiving a visual recording of the field of view from a first user's augmented reality device and displaying it on the remote assistant's display device. The method also includes receiving an indication of touch input to the remote assistant's display device, and having a processor determine the location of the input within an image, and providing location information about the input to the augmented reality device such that an indication (i.e., an arrow icon) can be imposed on the view provided by the augmented reality at the input location. However, in some situations, such as in medical emergencies, providing arrow icons through this method and system is often insufficient.

[0004] In other words, the use of arrow icons on the monitor is limited, especially in emergencies where first responders need more than just simple two-dimensional (2D) assistance; they need three-dimensional (3D) guidance. For example, to deliver high-quality CPR, a first responder not only needs to position his or her hands properly on the victim's chest, but also needs to perform the movements at the correct depth (i.e., chest compressions) and speed (i.e., the frequency of compressions). Guidelines recommend a depth of 2 inches (2in). Compressions move blood through the body to maintain oxygenation in vital organs. Additionally, sufficient depth essentially causes the heart to tumble between the sternum and spine (traps) and effectively squeeze out blood. Otherwise, inadequate CPR occurs.

[0005] Furthermore, in emergency situations, first responders are able to use currently known CPR metronomes, which guide them to deploy the same depth of compression for both thin and heavy individuals. The problem lies in the fact that deploying too little depth is ineffective, while deploying too much depth can potentially cause damage to the structures being compressed, especially relative to chest compression depths for children. Moreover, rib fractures are known to occur incidentally during CPR, leading to other complications. Another problem in administering CPR is incomplete chest release. While the depth of compression—that is, pressing the chest deep enough—is important, it is also crucial to adequately release pressure on the victim's chest. If the first responder does not adequately lift their hands, the victim's chest remains compressed, and the pumping effect on the heart is diminished.

[0006] In addition to CPR for emergency care, automated external defibrillators (AEDs) are also known. When using an AED, resuscitation must be performed according to a strict and time-critical protocol. Typically, resuscitation of a victim is initiated by informal caregivers (e.g., colleagues, family members, bystanders) because the first minute of an emergency is crucial and time is needed for someone to arrive with the AED. For resuscitation (i.e., pumping motions on the victim's chest, alternating with mouth-to-mouth resuscitation) to be performed correctly, informal caregivers need to perform the actions at the correct depth (i.e., chest compressions) and speed (i.e., compression frequency).

[0007] Furthermore, in conjunction with defibrillation and emergency care using an AED, it is important to note that when the heart is undergoing fibrillation, disordered electrical signals disrupt the synchronized contractions of the heart muscle. An AED device can help bring the heart's electrical signals back into sync by administering a shock. To administer this shock, two adhesive electrode pads need to be positioned and oriented specifically on the victim's body. Specifically, the electrode pads need to be placed diagonally along the body, with the heart positioned between the electrode pads. Optimal placement for adult patients involves attaching the first pad to the upper right chest and then the second pad to the lower left side of the victim. This may be easy for a professional caregiver, but can be tricky for a layperson. Some AEDs provide step-by-step verbal instructions and / or diagrams that help laypeople use the electrode pads and device correctly. However, for a layperson guided only by the diagrams on the electrode pads themselves, placing the pads in the optimal position on a victim requiring emergency care can be difficult.

[0008] In addition to the issues discussed above regarding the application of CPR, another problem with AED use is the incorrect placement of the electrode pads on the victim. For optimal defibrillation, both electrode pads should be correctly placed on the victim's body. Medical professionals generally know how to place the pads relative to body features such as the nipples, navel, and rib line. However, lay first responders are often unaware of this information, making it difficult for the product to quickly convey this complex medical information in a non-dynamic 2D graphic during an emergency. The non-dynamic 2D graphics illustrated on each AED electrode pad are often found to be difficult to reconcile with the victim's body.

[0009] Every second during cardiac arrest is critical. Therefore, there is a need to provide remote assistance workstations, methods, and systems with graphical user interfaces for providing remote assistance to first responders with spatial placement tasks via augmented reality glasses, for example, to reduce the time required to initiate resuscitation or other emergency procedures, and thus improve the victim's chances of survival. Therefore, there is a need for improved methods and apparatus to overcome the problems in the prior art. Summary of the Invention

[0010] According to one aspect of this disclosure, with the advent of augmented reality headsets, it becomes possible for a "person present in the scene" wearing such an augmented reality headset or glasses to connect to a remote expert. The remote expert can then see a live stream captured by a camera in the AR glasses and "seen through the eyes" of the "person present in the scene." Possible use cases include first responders in emergencies being assisted by medical experts from a 911 call center, and field technicians or engineers performing maintenance or repairs consulting with remote experts in the background who have more knowledge of specific machines, processes, and parts availability. The first use case is highly relevant to emergency care, most notably the product-service portfolio that emergency response teams can remotely support first responders. The second use case is highly relevant to, for example, healthcare operations, where maintenance engineers must work on specialized medical equipment such as MRI, CT, iXR and ultrasound scanners, patient monitors, etc. The second use case is also related to lighting operations relative to lighting systems such as office lighting, street lighting, etc. For this type of application, it is crucial to understand how remote experts can use the graphical user interface according to this embodiment to provide virtual 2D and 3D graphical guidance in AR glasses worn by people in the scene.

[0011] According to one embodiment of this disclosure, a graphical user interface used by a remote expert to provide remote assistance via augmented reality glasses includes at least two panes. The first pane includes a live video stream view obtained from a camera in the augmented reality glasses. The second pane includes a representation of a 2D view (e.g., a top-down view or other orientation) illustrating a remote scene in a given orientation and various movable 2D representations of 3D virtual objects associated with the remote assistance being presented, and more specifically, the representations illustrate 2D views of virtual content that can be selected and moved relative to a reference point in the 3D scene. The virtual objects can be selected and moved by an expert assistant within at least the second pane.

[0012] The remote assistant's graphical user interface (GUI) provides remote experts with at least two modes for conveying information to a person present in the scene. In the first mode, the remote expert can highlight any portion of the live video stream in the first pane by selecting (x, y) coordinates, for example, via touching a touchscreen display with the live video stream displayed therein in a first pane, and the exact same portion will be highlighted in stereoscopic AR glasses for the person wearing the AR glasses to see. The virtual highlight is fixed relative to the frame of the live view pane (i.e., for the expert) and relative to the AR glasses. Therefore, this first mode only works when the person in the scene temporarily keeps his or her head still; otherwise, the view on the scene will "slide" relative to the virtual highlight. In the second mode, the remote expert can select and move (i.e., drag) virtual content back and forth in a 2D representation of the scene, as displayed in the second pane. Because the virtual content, as moved by the remote assistant in the second pane of the user interface, is positioned relative to a fixed reference point in the 3D scene, it doesn't matter when the person wearing the AR glasses moves his or her head: the remote expert can move the virtual content without it jumping with the movement of the augmented reality glasses.

[0013] Most importantly, the remote assistant cannot simply move the content within the first pane (i.e., the live video stream) while the first responder is moving their head. This latter situation makes it impossible for the remote expert to keep the 3D virtual content stable within the live video stream of the first pane. Therefore, the remote assistance workstation utilizes the indirect application of 3D virtual content via a second pane (i.e., a 2D top-down view or other desired 2D orientation view). The remote assistant selects and moves a 2D representation of the 3D virtual content within the 2D top-down view of an object or second object in the scene. After the 3D virtual content is placed in the desired position within the 2D top-down view by the remote assistant, the controller of the remote assistance workstation generates a remote assistance signal to make the 3D virtual content appear within the live video stream and remain stable relative to a reference point in the live video stream.

[0014] According to one embodiment, a remote assistance workstation is configured to be operatively coupled to a portable device, the portable device including at least one pair of stereoscopic augmented reality glasses, the portable device being used by a first responder to perform at least one action by using the first object in the scene in conjunction with at least one of an object in the scene and a second object. The remote assistance workstation includes a communication module, a user interface module, and a controller. The communication module is configured to communicate with the portable device in response to a remote assistance request initiated from the portable device; the remote assistance request includes at least a live video stream captured via a camera of the stereoscopic augmented reality glasses in the scene.

[0015] The user interface module is configured to (a) render a remote assistant graphical user interface on a display device and (b) receive remote assistant input from the remote assistant. The remote assistant graphical user interface includes at least (i) a first pane for displaying a live video stream of a remote assistance request, and (ii) a second pane for displaying a 2D representation of a first object at a location in the scene. The rendered 2D representation is movable within the second pane in response to one or more remote assistant inputs. The remote assistant graphical user interface is also configured to render at least one corresponding 3D virtual content within the first pane, relative to at least a reference point within the first pane, corresponding to the 2D representation of the first object at the scene. The reference point is based on the content of the live video stream.

[0016] The controller is configured to generate one or more remote assistance signals output via a communication module to a portable device for displaying at least one 3D virtual content on stereoscopic augmented reality glasses to a first responder within a live view of a scene captured by the AR glasses' camera. The at least one 3D virtual content appears at the correct position relative to a reference point within the live view in response to one or more remote assistant inputs that move a 2D representation of a first object in the scene within a second pane of the remote assistant's graphical user interface, assisting the first responder in performing at least one action using the first object in conjunction with an object or a second object in the scene.

[0017] In another embodiment, the portable device includes a portable medical device, which includes at least one of an automated external defibrillator (AED), a cardiopulmonary resuscitation (CPR) metronome, and an electrocardiogram (ECG) monitor. Furthermore, the first object at the scene includes one or more of the following: (i) the first responder's hand, both hands, or other body parts of the first responder; and (ii) at least one item of the portable device.

[0018] In another embodiment, at least one 3D virtual content includes at least one virtual representation outline of at least one of (i) a first responder’s hand, two hands or other body parts of a first responder, and (ii) a portable device.

[0019] According to another embodiment, the remote assistance workstation further includes one or more of the following: a video rendering module, a 2D graphics view rendering module, a reference point module, and an XY coordinate module. The video rendering module is operable to render at least a live video stream of the remote assistance request within a first pane. The 2D graphics view rendering module is operable to render at least (i) a 2D representation of an object or a second object in the scene, and (ii) a 2D representation of a first object in the scene, within a second pane. The reference point module is operable to establish reference points within the content of the live video stream of the remote assistance request and displayed in the first pane. Finally, the XY coordinate module is operable to establish an XY coordinate system in the first pane based at least on the reference points used for the live video stream of the remote assistance request.

[0020] In additional embodiments, the reference point includes at least one of the following: (i) a face of an object determined by a facial recognition algorithm applied to an image of an object in the live video stream, and (ii) a reference point selected by the remote assistant in the content of the live video stream drawn in the first pane. Furthermore, the reference point selected by the remote assistant can include at least one of the following: (i) a reference point on an object or second object in the scene, (ii) a reference point on a portable device, and (iii) a reference point on the first object. Additionally, the reference point selected by the remote assistant can also include a vertical direction, wherein the vertical direction is selected by rotating the view of the second pane to modify the view such that the vertical direction of the view corresponds to the central axis of the object or second object in the scene.

[0021] In another embodiment, one or more remote assistance signals are configured to display at least one highlight on the stereoscopic augmented reality glasses to a first responder within a live view of the scene captured by the AR glasses' camera. The highlight is displayed in response to at least one remote assistant input, the remote assistant input including the remote assistant selecting XY coordinates in the live video stream displayed in a first pane of the remote expert's graphical user interface. In this embodiment, at least one highlight is displayed at the matching XY coordinates in the augmented reality glasses, as seen by the first responder.

[0022] In another embodiment, the second pane of the remote assistant graphical user interface includes at least one tab for each of a variety of types of actions to be performed in conjunction with an object or second object in the scene. In response to the remote assistant selecting a given tab in the second pane via the remote assistant graphical user interface, a 2D graphics view rendering module renders one or more 2D representations of the first object associated with the given tab within the second pane, the given tab being usable by the remote assistant relative to a corresponding given type of action. For example, one tab may be specific to AED and related actions such as electrode placement, while other tabs may be specific to CPR, triage, etc.

[0023] In one embodiment, the portable device includes a portable medical device comprising at least one of an automated external defibrillator (AED), a cardiopulmonary resuscitation (CPR) metronome, and an electrocardiogram (ECG) monitor. One or more 2D representations of the first object may correspond to one or more of: (i) the hand, both hands, or other body parts of the first responder, and (ii) at least one item of the portable device. Furthermore, one or more 2D representations of at least one item of the portable device may also represent at least one selected from the group consisting of: AED pad placement, CPR compression placement, and ECG sensor placement. Additionally, the remote assistance workstation may include a display device comprising a touchscreen device for receiving remote assistance input.

[0024] According to another approach, one method includes the step of providing remote assistance via a remote assistance workstation and a portable device, wherein the portable device includes at least one pair of stereoscopic augmented reality glasses. The portable device is used by a first responder to perform at least one action using the first object in the scene, in conjunction with at least one of an object in the scene and a second object. Specifically, the method includes at least three steps: operational coupling, drawing and receiving, and generation.

[0025] The first step involves operatively coupling a remote assistance workstation to a portable device via a communication module in response to a remote assistance request initiated from the portable device. The remote assistance request includes at least a live video stream captured by a camera on stereoscopic augmented reality glasses at the scene.

[0026] The second step includes drawing a remote assistant graphical user interface on a display device via a user interface module and receiving remote assistant input from the remote assistant. The remote assistant graphical user interface includes at least (i) a first pane for displaying a live video stream of a remote assistance request, and (ii) a second pane for displaying a 2D representation of a first object in the scene. The drawn 2D representation is movable within the second pane in response to one or more remote assistant inputs, wherein the remote assistant graphical user interface also draws at least one corresponding 3D virtual content within the first pane relative to at least a reference point within the first pane, corresponding to the 2D representation of the first object in the scene. The reference point is based on the content of the live video stream.

[0027] The third step includes generating one or more remote assistance signals via a controller to be output via a communication module to a portable device, the portable device being used to display at least one 3D virtual content on stereoscopic augmented reality glasses to a first responder within a live view of a scene, such as one captured by a camera. The at least one 3D virtual content appears at the correct position relative to a reference point within the live view in response to one or more remote assistant inputs that move a 2D representation of a first object in the scene within a second pane of the remote assistant's graphical user interface, to assist the first responder in performing at least one action in conjunction with an object or a second object in the scene.

[0028] In another aspect, the method includes: wherein the generation via the controller further includes generating one or more remote assistance signals for displaying at least one highlight on stereoscopic augmented reality glasses to a first responder within a live view of a scene captured by a camera in response to at least one remote assistant input, the at least one remote assistant input including the remote assistant selecting XY coordinates in a live video stream displayed in a first pane of a remote expert graphical user interface, and further wherein at least one highlight is displayed at matching XY coordinates in the augmented reality glasses, as seen by the first responder.

[0029] In another aspect, the method includes, wherein the reference point comprises one selected from the group consisting of: (i) a reference point of the face of an object determined by a facial recognition algorithm applied to an image of an object in a live video stream drawn in the first pane, and (ii) a reference point selected by a remote assistant within the content of the live video stream drawn in the first pane. The reference point selected by the remote assistant further includes at least one of: (i) a reference point on an object or a second object in the scene, (ii) a reference point on a portable device, and (iii) a reference point on a first object, and wherein selecting the remote assistant reference point further includes modifying the view by rotating the view of the second pane such that the vertical direction of the view corresponds to the central axis of the object or the second object in the scene.

[0030] In another embodiment of this disclosure, a non-transitory computer-readable medium has instructions that, when executed by a processor, cause the processor to perform the methods discussed herein.

[0031] In another embodiment, the remote assistance system includes a remote assistance workstation as discussed herein, and a portable device. The portable device includes at least one pair of stereoscopic augmented reality glasses, and the first object in the scene includes one or more of the following: (i) a first responder's hand, both hands, or other body parts of the first responder; and (ii) at least one item. Furthermore, the portable device is used by the first responder to perform at least one action in conjunction with at least one of the object and the second object in the scene. The portable device also includes a communication module configured to communicate with the remote assistance workstation. Additionally, at least one item includes at least one artifact used by the first responder to perform at least one action in conjunction with at least one of the object or the second object in the scene. Furthermore, the at least one pair of stereoscopic augmented reality glasses to be worn by the first responder includes a camera for capturing real-time images of the object or the second object in the scene.

[0032] After reading and understanding the following detailed description, other advantages and benefits will become apparent to those skilled in the art. Attached Figure Description

[0033] The embodiments of this disclosure may take the form of various components and component arrangements, as well as various steps and step arrangements. Therefore, the accompanying drawings are for illustrative purposes and should not be construed as limiting the embodiments. In the drawings, similar reference numerals denote the same elements. Furthermore, it should be noted that these drawings may not be drawn to scale.

[0034] Figure 1 This is a block diagram of a remote assistance workstation and a portable device having a user interface for providing remote assistance with a spatial placement task, according to an embodiment of the present disclosure, wherein the portable device includes at least one pair of stereoscopic augmented reality (AR) glasses used by a first responder to perform at least one action in conjunction with an object or a second object in the scene. Figure 2 This is a more detailed block diagram of a remote assistance workstation having a user interface for remote assistance with spatial placement tasks, according to embodiments of the present disclosure. Figure 3 This is a block diagram of a portable device used by a first responder to perform at least one action in conjunction with an object or a second object in a scene, according to an embodiment of the present disclosure. Figure 4This is a combined image and corresponding annotation view of the first and second panes of the graphical user interface of a remote assistance workstation according to an embodiment of the present disclosure, which shows a remote assistance input of a movable 2D representation of 3D content selected and moved within the second pane of the graphical user interface. Figure 5 This is a combined image and corresponding annotated view of objects in a scene, as shown in the augmented reality glasses of a first responder using a portable device, according to embodiments of the present disclosure. Figure 4 The placement of 3D virtual content for each remote assistant input in the second pane of the graphical user interface; Figure 6 This is a combined image and corresponding annotation view of the first and second panes of a remote assistance workstation graphical user interface according to an embodiment of the present disclosure, showing remote assistance input of (x, y) coordinate selection received in the first pane of the graphical user interface; Figure 7 This is a combined image and corresponding annotated view of objects in a scene, as shown in the augmented reality glasses of a first responder using a portable device, according to embodiments of the present disclosure. Figure 6 The placement of the highlight at the corresponding (x, y) coordinates of each remote assistant input in the first pane of the graphical user interface; and Figure 8 This is a flowchart of a method for providing remote assistance with a spatial placement task via a user interface of a remote assistance workstation and a portable device according to embodiments of the present disclosure, wherein the portable device includes at least one pair of stereoscopic augmented reality glasses used by a first responder to perform at least one action in conjunction with an object or a second object in the scene. Detailed Implementation

[0035] Embodiments of this disclosure, along with their various features and advantageous details, will be explained more fully with reference to the non-limiting examples described in detail and / or illustrated in the accompanying drawings and the description below. It should be noted that the features illustrated in the drawings are not necessarily drawn to scale, and as those skilled in the art will know, features of one embodiment may be employed together with other embodiments, even if not explicitly stated herein. Descriptions of well-known components and processing techniques may be omitted to avoid unnecessarily obscuring the embodiments of this disclosure. The examples used herein are intended only to facilitate understanding of how embodiments of the invention can be practiced and to enable those skilled in the art to practice the same manner. Therefore, the examples herein should not be construed as limiting the scope of the embodiments of this disclosure, which is defined only by the claims and applicable law.

[0036] It should be understood that the embodiments of this disclosure are not limited to the specific methodologies, protocols, devices, apparatuses, materials, applications, etc., described herein, as these can vary. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the claimed embodiments. It must be noted that, as used herein and in the claims, the singular forms “a,” “an,” and “described” include multiple references unless the context clearly indicates otherwise.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this disclosure pertain. Preferred methods, apparatus, and materials are described, but any methods and materials similar to or equivalent to those described herein may be used in the practice or testing of the embodiments.

[0038] As is apparent from the content disclosed herein, with the advent of augmented reality (AR) glasses, there is significant interest in applications that allow wearers to consult with remote experts. Because AR glasses are characterized by a camera, a live view of this camera can be transmitted to a remote expert via an internet link or other suitable communication link. Typically, the wearer of AR glasses is the person physically present in the scene who is "hands-on." The remote expert cannot physically be present in the scene to do anything; however, the remote expert usually has more knowledge or access to resources. Therefore, for example, in a medical emergency, the remote expert can request the person present in the scene to do specific things and guide him or her through a process.

[0039] Providing remote assistance to first responders with spatial placement tasks via augmented reality glasses through remote assistance workstations, methods, and systems with the user interface of this disclosure offers several advantages. These advantages include one or more of the following: the person appearing at the scene can be "untrained"; remote experts can provide task support; remote experts can provide emotional support; and with the help of remote experts, the problem can be resolved without a second visit to the scene. In one example, in an emergency, a first responder can connect to an expert from the 911 emergency response team via AR glasses and a remote expert user interface. In another example, a field engineer performing maintenance or repair work can connect to an office to consult with an expert who has more knowledge of a particular machine, process, component availability, etc.

[0040] As understood from this disclosure, embodiments of this disclosure utilize stereoscopic augmented reality (AR) glasses that differ from "glasses with an auxiliary display." In contrast to "glasses with an auxiliary display" (e.g., Google...),... TMThe auxiliary display (often positioned in the upper right corner of the user's field of vision) shows the video stream captured by the camera within the glasses. The display exists alongside the user's actual view in the real world: virtual objects cannot be directly placed into the user's view. Therefore, elements in the camera image can be highlighted for a remote expert, but for a person wearing glasses, these appear on the auxiliary display, not in their direct view in the real world. The drawback is the "indirectness": the user needs to check the auxiliary display to see which element was highlighted by the remote expert, and then subsequently locate the corresponding element in their actual, physical environment.

[0041] Currently, compared to stereoscopic augmented reality (AR) glasses, stereoscopic AR glasses essentially present virtual content in sync with the physical world by being displayed directly in front of the user's eyes. That is, virtual content can be displayed to the user in a way that it appears at a specific location in his or her actual environment, such as a computer-generated hologram. The advantage of such AR glasses is that remote assistants or experts can move virtual content to a specific 3D location (i.e., move the virtual content to a specific location within the 2D representation of the 3D scene, which is then presented or visible in the first responder's AR glasses) for specific remote assistance applications (e.g., in emergencies) to guide the attention of people in the scene, provide directions, show where to go, etc. One problem is that when the live video feed from the camera in the AR glasses is displayed as an image on the remote expert's monitor, it moves with the head position of the person in the scene. This prevents the remote expert from directly manipulating the virtual content in the live view. However, it is very useful for remote assistants or experts to point a virtual cursor at something in the live view, fully trusting that the person in the scene is seeing a virtual pointer on top of the same element in their field of vision.

[0042] In view of the above, the inventors have invented a remote assistant graphical user interface (GUI) for a remote assistant or expert, comprising two panes. The first pane displays a live view from AR glasses. The second pane displays a two-dimensional (2D) view of the 3D scene, wherein the remote assistant or expert can move virtual content. That is, the second pane displays a 2D view of the remote scene (i.e., for a given orientation, such as a top view, side view, etc.) and a graphical representation of various movable virtual objects. The remote assistant GUI preferably provides the remote assistant with at least two modes for conveying information to a person or first responder appearing in the scene.

[0043] In one mode, a remote assistant can highlight any portion of the live video stream in the first pane by selecting (x, y) coordinates, for example, via touching a touchscreen display that has the live video stream displayed thereon in the first pane, and the exact same portion will be highlighted in stereoscopic AR glasses for the person wearing the AR glasses to view. The virtual highlight is fixed relative to the frame of the live view pane (i.e., for experts) and relative to the AR glasses. However, this mode only works if the person in the scene temporarily keeps his or her head still; otherwise, the view on the scene will "slide" relative to the virtual highlight.

[0044] In another mode, the remote assistant can select and move (i.e., drag) one or more desired virtual contents within a 2D representation of the scene. Because the virtual content is positioned relative to a fixed reference point in the 3D scene when moved by the remote assistant in the second pane of the user interface, it doesn't matter whether the person wearing the AR glasses moves their head: the remote assistant can move the virtual content within the second pane, and the virtual content doesn't jump within the live video view of the first pane as the augmented reality glasses move. As the remote assistant moves the virtual content in the second pane, they can see how the virtual content will be presented to the "person in the scene," i.e., via the first pane, which displays a live video stream containing the 3D virtual content included therein.

[0045] According to another embodiment of this disclosure, the remote assistance system includes augmented reality goggles (e.g., AR headsets) at one end and a computer monitor (e.g., a teleprocessing device, workstation, etc.) at the other end. The computer monitor has a 3D pane showing a video at one end and a 2D pane with a graphical representation of objects in the 3D pane (e.g., representing objects in a scene, medical equipment in a scene, or a portable medical device in a scene). The remote assistant manipulates objects in the 2D pane, which are automatically applied in the 3D pane and maintained in the correct position relative to a given reference point within the scene of the video displayed in the 3D pane, corresponding to an actual point or position in the scene.

[0046] In another embodiment, a method for a person (e.g., a first responder) present in a scene using an augmented reality headset (AR headset) connected to a remote assistant includes the remote assistant utilizing a remote assistant graphical user interface (GUI). The GUI includes two panes: a first pane showing a live video stream view from a camera in the AR glasses; and a second pane showing a representation of a 2D view of the remote scene and various movable virtual objects. The GUI provides the remote assistant or expert with two modes for conveying information to the "person present in the scene." In one mode, the remote assistant is able to highlight any portion of the live video stream view in the first pane, and the exact same portion will be highlighted in the glasses of the person wearing the AR glasses. In the other mode, the remote assistant is able to select and move (i.e., drag) one or more desired virtual contents back and forth in the 2D representation of the scene. Because the virtual contents are positioned relative to a fixed reference point in the live video image of the scene, it is irrelevant whether the person wearing the AR glasses moves their head in this case.

[0047] Reference Figure 1-8 In the following discussion, embodiments of this disclosure will be discussed in the context of practical use cases involving remote assistance in cardiac arrest. However, embodiments of this disclosure can also be applied to other types of remote assistance situations, as appropriate for a given situation.

[0048] Now go to Figure 1 This diagram illustrates a block view of a remote assistance system 10, which includes a remote assistance workstation 12 and a portable device 14 coupled to each other via a communication link 16. The communication link 16 can include any suitable network for direct or indirect communication between the remote assistance workstation 12 and the portable device 14, such as a wireless network, the Internet, or other known or developed networks. For example, the communication link 16 can be directly coupled between the remote assistance workstation 12 and the portable device 14, as indicated by reference numerals 18 and 20. Similarly, the communication link 16 can be indirectly coupled between the remote assistance workstation 12 and the portable device 14, for example, via a smartphone 22 located in a given immediate proximity to the portable medical device 14, as indicated by reference numerals 18, 24, and 26. Specific details of communication between the various devices and components discussed herein are preferably implemented using suitable techniques known in the art and will not be discussed further here.

[0049] The remote assistance workstation 12 includes at least one input / output device 28 (e.g., keyboard, mouse, touch input, etc.) and a display 30 used by the remote assistant 32. The remote assistance workstation 12 also includes a user interface, as will be discussed further below, for providing remote assistance to a first responder 34 with a spatial placement task using a portable device 14.

[0050] Still referencing Figure 1 The portable device 14 includes at least one pair of stereoscopic augmented reality (AR) glasses 36 for use by a first responder 34 to perform at least one action in conjunction with an object 42 (e.g., a victim) or a second object (not shown) at scene 40, using a first object 38 at scene 40 (e.g., a pair of AED electrodes, the first responder's hands, both hands, etc.). In one embodiment, the portable device 14 includes a portable medical device comprising at least one of an automated external defibrillator (AED), a cardiopulmonary resuscitation (CPR) metronome, and an electrocardiogram (ECG) monitor. Furthermore, the first object 38 at scene 40 can include one or more of the following: (i) the first responder's hands, both hands, or other body parts of the first responder; and (ii) at least one item of the portable device (i.e., electrodes or a CPR metronome). Figure 1 The portable device 14 may also include a CPR metronome 44. Communication between the augmented reality glasses 36 and the portable device 14 is indicated by reference numeral 46. Similarly, communication between the portable device 14 and the CPR metronome 44 is indicated by reference numeral 48. Furthermore, in the case of the AED electrode 38, the AED electrode is coupled to the portable device 14 via a suitable signal / power line 50. The specific details of the communication and signal / power lines between the various devices and components discussed herein are preferably implemented using suitable techniques known in the art, and therefore will not be discussed further herein.

[0051] During operation, the "person at the scene" or first responder 34 wears stereoscopic augmented reality glasses 36. A data connection is established between the AR glasses 36 and a remote assistance workstation 12 attended by a remote expert or assistant 32. As will be discussed further herein, the remote assistant 32 utilizes a remote assistant graphical user interface comprising at least two panes: a first pane displaying a live video stream from the AR glasses 36, and a second pane displaying a graphical 2D view of the scene (or a portion thereof and for a given orientation, such as a top view, side view, etc.), wherein the 2D representation of 3D virtual objects can be selected and moved back and forth. In other words, the second pane contains an illustrative 2D graphical view of the scene relative to actual objects or objects in the scene and reference points on the actual objects or objects in the scene. Furthermore, as will be discussed further herein, the 2D representation of 3D virtual objects can be selected and moved back and forth by the remote assistant within the second pane.

[0052] Now for reference Figure 2 A more detailed block diagram of a remote assistance workstation 12 is shown, which has a user interface for remote assistance with spatial placement tasks. The remote assistance workstation 12 includes at least a user interface 52, a communication module 54 (e.g., configured to communicate with one or more of a portable medical device 14 and a smartphone 22 located in a given immediate vicinity of the portable device), and a controller 56. In one embodiment, the user interface 52 is configured to communicate at least from a remote assistant 32 ( Figure 1 The remote assistant input is obtained via input / output device 28. User interface 52 includes at least a graphical user interface operatively coupled to controller 56 via signal line 58 for use, for example, in conjunction with a given remote assistance situation during an emergency, as will be discussed further herein. Furthermore, user interface 52 may also include at least one selected from the group consisting of: input / output devices, haptic devices, touchscreens, optical displays, microphones, keypads, keyboards, pointing devices, image capture devices, cameras, audio input / output devices, and any combination thereof, as determined by the requirements of a given remote assistance implementation and / or application. Additionally, in one embodiment, controller 56 also includes one or more microprocessors, microcontrollers, scene programmable gate arrays (FPGAs), integrated circuits, discrete analog or digital circuit components, hardware, software, firmware, or any combination thereof for performing the various functions discussed herein. Controller 56 may also include one or more of the various modules discussed herein. Further details regarding controller 56 will be provided below with reference to the accompanying drawings.

[0053] Still referencing Figure 2The remote assistance workstation 12 may also include one or more of a video rendering module 60, a 2D graphics view rendering module 62, a reference point module 64, and an XY coordinate module 66. The video rendering module 60 is operable to present at least a live video stream of the remote assistance request within a first pane of the remote assistance graphical user interface. The 2D graphics view rendering module 62 is operable to present at least (i) a 2D representation of an object or second object in the scene, and (ii) a 2D representation of a first object in the scene, within a second pane. The reference point module 64 is operable to establish reference points within the content of the live video stream of the remote assistance request and display them in the first pane. Finally, the XY coordinate module 66 is operable to establish an XY coordinate system in the first pane based at least on the reference points used for the live video stream of the remote assistance request. The remote assistance workstation 12 may also include one or more of a database 68 and a memory 70. Each of the database 68 and the memory 70 is operatively coupled to the controller 56 at least, for example, via a signal line 58. In one embodiment, depending on the requirements of a given remote assistant workstation implementation and / or application, modules 60-70 may include one or more of integrated circuits, discrete analog or digital circuit components, hardware, software, firmware, or any combination thereof to perform the various functions discussed herein. Furthermore, one or more modules 60-70 may also include various combinations of one or more of various modules.

[0054] Now for reference Figure 3 The diagram illustrates a block diagram view of a portable device 14 according to an embodiment of the present disclosure, used by a first responder 34 to perform at least one action in conjunction with an object 42 or a second object at scene 40, using a first object at scene 40. In one embodiment, the portable device 14 includes a portable medical device, which includes at least a user interface 72, a communication module 74, and a controller 76. The user interface 72 is configured to at least initiate a request for remote assistance and includes a user interface operatively coupled to at least the controller 76 via a signal line 78 for use in conjunction with receiving remote assistance during an emergency, as discussed further herein. For example, the user interface 72 may include at least one selected from the group consisting of: input / output devices, haptic output devices, touchscreens, optical displays, microphones, keypads, keyboards, pointing devices, image capture devices, cameras, audio output devices, and any combination thereof, as determined by the requirements of a given portable medical device implementation and / or application.

[0055] The communication module 74 is configured to communicate with at least one of (i) the remote assistance workstation 12 and (ii) a smartphone 22 located in a given immediate vicinity of the portable medical device. The communication module 74 is also configured to receive one or more remote assistance signals from the remote assistance workstation 12 in response to a remote assistance request from a first responder, via the remote assistance workstation 12 or a smartphone 22 communicating with the remote assistant workstation. The one or more remote assistance signals from the remote assistance workstation 12 contain information for displaying at least one 3D virtual content to the first responder 34 on the stereoscopic augmented reality glasses 36 within a live view of scene 40 captured by the camera of the AR glasses 36. Furthermore, in response to one or more remote assistant inputs that move a 2D representation of a first object in the scene within a corresponding 2D representation pane of the remote assistant graphical user interface of the remote assistance workstation 12, at least one 3D virtual content appears at the correct position relative to a reference point within the live view, as further discussed herein, to assist the first responder in using the first object in conjunction with an object or a second object in the scene to perform at least one action.

[0056] Communication between the communication module 74 of the portable medical device 14 and the remote assistance workstation 12 is indicated by reference numerals 18 and 20, including network 16 ( Figure 1 Communication between the communication module 74 of the portable medical device 14 and the remote assistance workstation 12 can also occur via the smartphone 22, and is indicated by reference numerals 18, 24, and 26, including network 16 ( Figure 1 Communication between the communication module 74 of the portable medical device 14 and the AR glasses 36 is indicated by reference numeral 46. Communication between the communication module 74 of the portable medical device 14 and the AR glasses 36 is indicated by reference numeral 46. Communication between the communication module 74 of the portable medical device 14 and the CPR metronome 44 is indicated by reference numeral 48. In each case, communication between the various devices and components discussed herein is preferably achieved using suitable techniques known in the art, which will not be discussed further herein.

[0057] Controller 76 is operatively coupled to user interface 72 and communication module 74 via appropriate signal lines indicated by reference numeral 78. Controller 76 is configured to generate control signals output to AR glasses 36 in response to one or more remote assistance signals from remote assistant workstation 12, to display at least one 3D virtual content on AR glasses 36 to a first responder 34 within a live view of scene 40, such as that captured by the camera portion of the AR glasses. Accordingly, the first responder receives remote assistance to perform at least one action by using a first object in conjunction with an object or a second object in the scene. In one embodiment, controller 76 also includes one or more of a microprocessor, microcontroller, scene programmable gate array (FPGA), integrated circuit, discrete analog or digital circuit components, hardware, software, firmware, or any combination thereof, depending on the requirements of a given portable medical device implementation and / or application, for performing the various functions discussed herein. Controller 76 is also capable of including one or more of the various modules discussed herein. Further details regarding controller 76 will be provided below with reference to the accompanying drawings.

[0058] Still referencing Figure 3 The portable medical device 14 may also include one or more of the following: an on / off switch 80, a virtual highlighting drawing module 82, a virtual content drawing module 84, a battery 86, a power source 88, a memory 90, an electric shock button 92 (e.g., for enabling the application of an electric shock via an AED pad electrode), and a GPS module 94. Each of the one or more of the following—the on / off switch 80, the virtual highlighting drawing module 82, the virtual content drawing module 84, the battery 86, the power source 88, the memory 90, the electric shock button 92, and the GPS module 94—is operatively coupled to at least a controller 76, for example, via a signal line 78. The on / off switch 80 includes any suitable switch for powering the portable medical device 70 between on and off states. The virtual highlighting drawing module 82 includes any suitable computer program module for drawing a virtual highlight on AR glasses 36 to a first responder 34 within a live view of scene 40, such as that captured by the camera portion of AR glasses 36. The virtual content rendering module 82 includes any suitable computer program module for rendering at least one 3D virtual content on the AR glasses 36 to a first responder 34 within a live view of a scene 40, such as that captured by the camera portion of the AR glasses 36. It should be understood that the described module can be a computer program module provided in a non-transitory computer-readable medium. The portable medical device 14 is also capable of including an audio speaker (not shown) for a given implementation and / or portable medical device application.

[0059] In one embodiment, battery 86 can include any suitable power source or power supply for a given portable medical device implementation and / or application. Furthermore, energy source 88 can include any suitable power source or power supply for a given portable medical device implementation and / or application. For example, for a portable medical device including an AED device, energy source 88 can include a high-voltage capacitor suitable for storing energy effective for defibrillation shocks, wherein the capacitor is charged by battery 86 via a charging circuit (not shown). Additionally, memory 90 can include any suitable memory device operatively coupled to at least controller 76 for storing information therein and also for retrieving information therefrom at least subsequently.

[0060] The Global Positioning System module 94 includes any suitable GPS module configured to determine the global location of the portable emergency medical device 14. The controller 76 is also configured to determine, based on the global location of the portable emergency medical device 14 and the global location of the smartphone 22, that the smartphone is located in a given immediate vicinity of the portable emergency medical device.

[0061] The portable medical device 14 may also include a pair of AED pad electrodes 38 operatively coupled to a power source 88 for administering an electric shock as an AED device during use of the portable medical device 14. The portable medical device 14 may also include a pair of augmented reality glasses 36, for example, to be worn by a first responder or rescuer during AED and / or CPR procedures in emergency situations. The augmented reality glasses 36 are transmitted via a suitable communication link 46 (e.g., Near Field Communication (NFC), Bluetooth). TM (or other suitable short-range communication link) is operatively coupled to the communication module 74 of the portable medical device 14.

[0062] Furthermore, the portable medical device 14 can include a CPR metronome 44 for use by a first responder or rescuer to perform CPR during emergency treatment. The CPR metronome is connected via a suitable communication link 48 (e.g., Near Field Communication (NFC), Bluetooth). TM(or other suitable short-range communication link) is operatively coupled to the communication module 74 of the portable medical device 14. In one embodiment, the CPR metronome 44 includes an accelerometer-based CPR metronome. The CPR metronome is configured to take chest depth into account to guide the first responder on how deep to press. In another embodiment, the CPR metronome 44 can include an accelerometer-equipped smartphone with a CPR app that takes chest depth into account to guide the first responder on how deep to press via information on a live view of the AR glasses 36 and / or via audio guidance. Note that although the above references an accelerometer-based metronome device, such a device does not always need to be accelerometer-based. Essentially, the metronome device is depth-sensing relative to the body. Therefore, in another embodiment, the augmented reality glasses 36 can be used to assist the first responder in establishing the position of the CPR metronome or the first responder's hands for applying pressure via remote assistance.

[0063] For a better understanding of the embodiments, refer to Figure 1 , 2 And 3, let's consider a use case involving remote assistance in a cardiac arrest situation. First responder 34 at emergency scenario 40 wears stereoscopic AR glasses 36. Telemedicine expert 32 is with a 911 emergency response team. First responder 34's AR glasses 36 are connected via the Internet to telemedicine expert 32's workstation 12. Telemedicine expert 32's operation includes a graphical user interface with two panes (as will also be referenced in this document). Figure 4 Workstation 12 (discussed). A first pane displays a live video stream captured from the stereoscopic AR glasses of a first responder, and includes a depiction of the victim (i.e., an object or structure in the 3D scene of the live video stream) and the surrounding environment (i.e., the environment in the 3D scene of the live video stream). A second pane of the GUI includes a graphical representation of the victim's 2D top-down view (i.e., a 2D representation of an object or structure in the 3D scene of the live video stream), containing virtual content that can be selected and moved around by a remote medical expert using a touch or pointing device within the second pane. According to another embodiment, once selected and moved, the virtual content can also be enabled / deactivated by the remote expert via user input and workstation controller 56 to appear / disappear within the 3D view of the first pane, respectively.

[0064] Telemedicine experts can deliver information to first responders in at least two modes: a highlight mode and a virtual content mode. In highlight mode, the telemedicine expert can touch the live video stream in the first pane, causing a highlight to appear in the first responder's AR glasses (as will be referenced below). Figure 6 and Figure 7(As discussed). In other words, in highlight mode, in response to a telemedicine expert touching (i.e. selecting) a point displayed in the live video in the first pane, workstation 12 outputs a highlight at the corresponding point in the 3D scene appearing in the first responder's AR glasses, at least via controller 56 and X,Y coordinate module 66. The telemedicine expert can also use this highlight mode, at least via controller 56 and reference point module 64, to establish reference points for a virtual content mode, for example, via a snapshot of the scene including reference points.

[0065] In Virtual Content Mode, telemedicine experts can select and move virtual content back and forth and enable / deactivate (e.g., make it appear / disappear) virtual content in the 2D pane, i.e., the virtual content in the second pane of the remote assistant's graphical user interface. In other words, in Virtual Content Mode, in response to the telemedicine expert selecting and moving the virtual content in the second pane of the graphical user interface relative to a 3D scene reference point, there is a corresponding movement of the 3D-rendered virtual content in the live video pane. Furthermore, as used herein, the phrase "movable 2D representation of 3D virtual content" should be understood as follows: In response to the telemedicine expert selecting and moving the 2D representation of 3D virtual content within the 2D graphical view, there is a corresponding movement of the 3D-rendered virtual content in the live video pane (i.e., the first pane) at a position mapped as a function of (i) the 2D representation of an object or object in the scene and (ii) one or more corresponding reference points within the 2D representation (in the second pane) and the live video (in the first pane).

[0066] In one embodiment, the 2D representation of the 3D virtual content includes content specific to (i.e., relevant to) a given remote assistance situation (e.g., a medical emergency - cardiac arrest situation). Furthermore, the 3D virtual content moves relative to a reference point within the live video of the first pane (e.g., the victim's face). One example includes selecting and moving a virtual contour (i.e., an overlapping contour) that indicates where the electrode pads of an automated external defibrillator (AED) should be placed on the victim in a medical emergency scenario. Another example includes selecting and moving a hand icon to indicate where the first responder should apply pressure to the victim's chest to perform CPR compressions / deflation.

[0067] Now for reference Figure 4A combined image and corresponding annotated view of the first and second panes of a remote assistance workstation graphical user interface 100, indicated by reference numerals 96 and 98 respectively, according to an embodiment of the present disclosure, is shown, illustrating remote assistant input received in the second pane 98 of the graphical user interface 100 as a movable 2D representation of 3D content indicated by reference numeral 102. Annotated views are provided to illustrate the demarcation of the first and second panes (96 and 98) more clearly, respectively. In the live video of the first pane 96, the remote assistant 32 views the victim 42 who may require defibrillation as the first responder 34 prepares to attach the AED electrode 38 to the victim's chest. As discussed herein, the second pane 98 includes, for example, a 2D graphical view of the victim in the scene and a movable 2D representation within the second pane 98, for example, of 3D content 102 representing each AED electrode.

[0068] In addition, still refer to Figure 4 In another embodiment, the second pane 98 of the remote assistant graphical user interface 100 includes at least one tab (99, 101, and / or 103) for each of a variety of types of actions (e.g., tabs for various folding or action types) to be performed in conjunction with an object or a second object in the scene. In response to the remote assistant selecting a given tab (99, 101, or 103) in the second pane via the remote assistant graphical user interface, a 2D graphics view drawing module draws one or more 2D representations of a first object associated with the given tab (99, 101, or 103) within the second pane, the one or more 2D representations being usable by the remote assistant relative to a corresponding given type of action. For example, in Figure 4 In the 2D graphical view 98, activity tabs 101 are shown that are specific to AED and related actions such as electrode placement, while other tabs may be specific to CPR (tab 103), triage (tab 99), etc.

[0069] like Figure 4 As shown, the remote assistant 32 selects and moves each of the electrodes 38 within a 2D graphical view of the second pane 98. Within the second pane, for example via a touchscreen within the second pane 98, the 2D representation of the 3D content is controlled to achieve the corresponding placement of 3D outlines or overlapping objects on the live video of the first pane 96, thereby providing remote assistance to a first responder 34 with a corresponding spatial placement task at the scene. In other words, Figure 4 The illustration depicts a remote expert positioned at a workstation with two panes, a live video pane showing a video feed from AR glasses in the scene, and a graphical 2D pane with a 2D representation of 3D virtual content.

[0070] In another embodiment, the second pane can include a hierarchical region (e.g., the upper part of the 2D graphics view 98), from which 3D virtual content is selected and moved to an active region (e.g., the lower part of the 2D graphics view 98), where the selected and moved 3D virtual content becomes active and also appears within the live video pane. Additional features can also be applied, such as the ability to toggle the appearance / disappearance of the 3D virtual content within the live video pane using additional remote assistant input once the 3D virtual content has been selected and moved to the active region.

[0071] Furthermore, the first pane 96 may also include a location information insert 104, wherein the insert 104 indicates the actual address and / or location of the victim 42 in response to a GPS signal obtained via the GPS module 94 of the portable medical device 14. Additionally, the remote assistant graphical user interface 100 may include another pane 106 that presents victim identification information and medical record information from the portable medical device 14 or from other suitable sources to the remote assistant 32 (if available).

[0072] Now go to Figure 5 A combined image and corresponding annotated view of an object 42 at scene 40, as shown in augmented reality glasses 36 of a first responder 34 using portable device 14, are illustrated. According to embodiments of this disclosure, in response to... Figure 4 The remote assistant inputs in the second pane 98 of the graphical user interface 100, and the display 108 in the AR glasses 36 shows the first responder 34 the placement of the 3D virtual content 381 by the remote assistant on a live video view of the first pane 96 of the remote assistant's graphical user interface 100. In other words, in response to the remote expert selecting and moving the virtual content in the 2D pane (i.e., the second pane), the virtual content then appears appropriately in the first responder's AR glasses in a perspective view. Figure 5 In the image view, the first responder's hand can be seen holding one of the AED electrodes in preparation for placing it on the victim's chest, using a remote assistant to guide and / or assist in the spatial placement of the 3D virtual content 38 on the live video view.

[0073] Now for reference Figure 6 The illustration shows a combined image and corresponding annotated view of the first and second panes of a remote assistance workstation graphical user interface 110, indicated respectively by reference numerals 96 and 112. In this embodiment, the second pane 112 includes one or more tabs 105, 107, and 109 that can be selected by a remote assistant during a given remote assistance application, the tabs being similar to those referenced above. Figure 4Those discussed in tabs 99, 101, and 103, or others. For example, in response to... Figure 6 The selection of tab 105 in the middle provides a similar option to the remote assistant. Figure 4 The first pane 96 contains information about the victim's identification and medical records. Additionally, the first pane 96 also includes information such as references... Figure 4 The location information in question is inserted in 104. Additionally, the graphical user interface 110 can include another pane 114 that presents a map showing the victim's geographic location from an appropriate map source (if available).

[0074] Still referencing Figure 6 In the live video of the first pane 96, the remote assistant 32 points to a location within the live video to identify or highlight the desired location 116 within the live scene for the first responder 34. Specifically, Figure 6 This illustration shows remote assistant input, according to an embodiment of the present disclosure, including (x, y) coordinate selection received in a first pane 96 of a graphical user interface 110. In one embodiment, the highlight includes a circular outline, or it may also include a highlight of any geometry suitable for a given remote assistance application.

[0075] Now go to Figure 7 The image shows a combined image and corresponding annotated view of an object 42 at scene 40, as displayed in augmented reality glasses 36 of a first responder 34 using portable device 14. According to an embodiment of this disclosure, the display 118 in the AR glasses 36 shows... Figure 6 The placement of a highlight 116 at the corresponding (x, y) coordinate of each remote assistant input in the first pane 96 of the graphical user interface 110. In other words, the remote assistant can touch (x, y) coordinates in the video pane (i.e., the first pane) (i.e., select (x, y) coordinates) to cause the highlight to appear at the same (x, y) coordinates in the first responder's AR glasses.

[0076] Figure 8 This is a flowchart of a method 150 for providing remote assistance with a spatial arrangement task via a user interface of a remote assistance workstation and a portable device, according to an embodiment of the present disclosure. In a first step, a portable device is provided at a scene, wherein the portable device includes at least one pair of stereoscopic augmented reality glasses, said at least one pair of stereoscopic augmented reality glasses being used by a first responder to perform at least one action in conjunction with an object or a second object at the scene (step 152).

[0077] In the next step (step 154), the method includes operatively coupling a remote assistance workstation to a portable device in response to a remote assistance request initiated from the portable device. For example, a first responder may request remote assistance by selecting one or more designated buttons or soft keys on the portable device, which is configured to send a request for remote assistance to the remote assistance workstation. In one embodiment, the request for remote assistance includes providing the remote assistance workstation with live video, such as that seen from a camera on augmented reality glasses.

[0078] In the next step (step 156), the method includes presenting a remote assistant graphical user interface (GUI) on a display device of the remote assistance workstation and receiving remote assistant input, the remote assistant GUI including a first pane for displaying a live video stream of a remote assistance request and a second pane for displaying a 2D representation of a first object in the scene.

[0079] In the next step (step 158), the method includes generating one or more remote assistance signals to be output to a portable device, the portable device being used to display at least one 3D virtual content on stereoscopic augmented reality glasses to a first responder in a live view in response to one or more remote assistant inputs that select and move a 2D representation of a first object within a second pane of the remote assistance GUI.

[0080] Embodiments of this disclosure advantageously enable the quick and intuitive selection and movement of 3D virtual content. For example, a remote assistant can select and move virtual content hovering over a 2D plane above the victim (i.e., hovering from the victim's top-down view). Figure 2 D represents the virtual content in the second pane above. Based on facial recognition, the orientation of the victim's body can be derived, and the 2D XY coordinate system of the screen (i.e., in the second pane) can be aligned with the XY coordinate system of the body (i.e., in the first pane), where the Y-axis is the midline of the body, such that when the remote assistant moves the 2D representation of the 3D virtual object in the Y direction on the screen (i.e., in the second pane), the virtual object moves parallel to the victim's midline.

[0081] In an emergency, the first responder's perspective can change as they provide emergency assistance to the victim. When the first responder moves their head (and will do so due to the nature of the task), the guide (remote expert / assistant) can select and move 3D virtual content on a 2D screen (i.e., in a second pane), and the selected virtual content will be stabilized in the first responder's AR glasses according to perspective. In other words, in response to the 3D virtual content selected and positioned within the 2D screen, the perspective positioning of the 3D virtual content is unaffected by the movement of the first responder or remote expert / assistant (unless, of course, the remote expert / assistant actively, purposefully, or definitively selects and moves the 2D representation of the 3D virtual content).

[0082] Although only a few exemplary embodiments have been described in detail above, those skilled in the art will readily recognize that many modifications can be made to the exemplary embodiments without substantially departing from the novel teachings and advantages of the embodiments of this disclosure. For example, although embodiments of this disclosure have been described herein in the context of emergency care, the embodiments are equally applicable to other use cases involving remote assistance using augmented reality (AR). Therefore, all such modifications are intended to be included within the scope of the embodiments of this disclosure as defined in the following claims. In the claims, the module plus function clauses are intended to cover the functions described herein that perform the recited functions, and include not only structural equivalents but also equivalent structures.

[0083] Furthermore, any reference numerals placed in parentheses within one or more claims should not be construed as limiting the claims. The words “comprising” and “including” do not exclude the presence of elements or steps other than those listed as a whole in any claim or specification. A singular reference to an element does not exclude a plural reference to such an element, and vice versa. One or more of the embodiments may be implemented by means of hardware comprising several different elements and / or by means of a suitably programmed computer. In device claims enumerating several modules, several of these modules may be implemented by the same hardware. Although specific measures are recited in mutually different dependent claims, this does not indicate that combinations of these measures cannot be advantageously used.

Claims

1. A remote assistance workstation (12) configured to be operatively coupled to a portable device including at least one pair of stereoscopic augmented reality glasses (36), the portable device being used by a first responder (34) to perform at least one action by using a first object (38) in conjunction with at least one of (i) an object (42) and (ii) a second object in a scene (40), the remote assistance workstation (12) comprising: A communication module (54) is configured to communicate with the portable device in response to a remote assistance request initiated from the portable device; The remote assistance request includes at least a live video stream captured by the camera of the stereoscopic augmented reality glasses (36) at the scene; A user interface module (52) configured to (a) draw a remote assistant graphical user interface (100) on a display device and (b) receive remote assistant input from a remote assistant (32), wherein the remote assistant graphical user interface includes at least: (i) A first pane (96) for displaying the live video stream of the remote assistance request, and (ii) A second pane (98) for displaying a 2D representation of the first object (38) at the scene, wherein the drawn 2D representation is movable within the second pane in response to one or more remote assistant inputs, the remote assistant graphical user interface (100) further for drawing at least one 3D virtual content item in the first pane (96) corresponding to the drawn 2D representation of the first object (38) at the scene in the second pane, relative to at least a reference point in the first pane (96), wherein the reference point is based on the content of the live video stream, and the at least one 3D virtual content item moves accordingly in the first pane in response to the drawn 2D representation being selected and moved relative to the reference point within the second pane; A controller (56) generates one or more remote assistance signals to be output to the portable device (14) via the communication module (54) in response to one or more remote assistant inputs, displaying at least one 3D virtual content item to the first responder (34) on the stereoscopic augmented reality glasses (36) in a live view of the scene as captured by the camera of the stereoscopic augmented reality glasses (36), such that the at least one 3D virtual content item appears at the correct position relative to the reference point in the live view, to assist the first responder (34) in using the first object (38) in conjunction with the object (42) or the second object at the scene (40) to perform the at least one action, wherein the one or more remote assistant inputs move the drawn 2D representation of the first object at the scene within the second pane.

2. The remote auxiliary workstation (12) according to claim 1 further includes: A video drawing module (60) is configured to draw at least the live video stream of the remote assistance request within the first pane (96); A 2D graphics view drawing module (62) is configured to draw at least (i) a 2D representation of the object (42) or the second object at the scene (40) and (ii) a 2D representation of the first object (38) at the scene within the second pane (98); Reference point module (64), wherein the reference point module can be used to establish reference points within the content of the live video stream of the remote assistance request and displayed in the first pane (96); and XY coordinate module (66), wherein the XY coordinate module can be used to establish the XY coordinate system in the first pane (96) based at least on the reference point of the live video stream in response to the remote assistance request.

3. The remote assistance workstation (12) of claim 1, wherein, The portable device includes a portable medical device (14), which includes at least one of the following: an automated external defibrillator (AED), a cardiopulmonary resuscitation (CPR) metronome, and an electrocardiogram (ECG) monitor; and wherein the first object (38) at the scene (40) includes one or more of the following: (i) the hand of the first responder, the hands of the first responder, or other body parts of the first responder, and (ii) at least one of the portable devices.

4. The remote assistance workstation (12) according to claim 2, further wherein, The at least one 3D virtual content item includes at least one virtual representation outline, the virtual representation outline representing at least one of the following: (i) the first responder's hand, the first responder's two hands or other body parts of the first responder, and (ii) the at least one item of the portable device.

5. The remote assistance workstation (12) according to claim 1, wherein, The reference point includes at least one of the following: (i) the face of the object determined by a facial recognition algorithm applied to an image of the object in the live video stream, and (ii) a reference point selected by a remote assistant within the content of the live video stream drawn in the first pane (96).

6. The remote assistance workstation (12) according to claim 5, further wherein, The reference point selected by the remote assistant includes at least one of the following: (i) a reference point on the object (42) or the second object at the scene (40), (ii) a reference point on the portable device, and (iii) a reference point on the first object (38).

7. The remote assistance workstation (12) according to claim 6, wherein, The reference point selected by the remote assistant also includes a vertical direction, wherein the vertical direction is selected by modifying the view by rotating the view of the second pane (98) such that the vertical direction of the view corresponds to the central axis of the object (42) or the second object at the scene (40).

8. The remote assistance workstation (12) according to claim 1, wherein, The one or more remote assistance signals are also configured to display at least one highlight (116) on the stereoscopic augmented reality glasses (36) to the first responder (34) within the live view of the scene captured by the camera in response to at least one remote assistant input, the remote assistant input including the remote assistant selecting the XY coordinates in the live video stream displayed in the first pane (96) of the remote assistant graphical user interface (100, 110), and wherein the at least one highlight (116) is displayed at the matching XY coordinates in the augmented reality glasses (36) as seen by the first responder (34).

9. The remote assistance workstation (12) according to claim 1, further wherein, The second pane (98) includes at least one tab (99, 101, 103) for each of a plurality of types of actions to be performed in conjunction with the object (42) or the second object at the scene (40), wherein, in response to the remote assistant (32) selecting a given tab (99, 101, 103) in the second pane (98) via the remote assistant graphical user interface (100), the 2D graphics view drawing module (62) draws one or more 2D representations of the first object (38) associated with the given tab (99, 101, 103) within the second pane (98), the one or more 2D representations being available for use by the remote assistant (32) with respect to actions of a corresponding given class.

10. The remote assistance workstation (12) according to claim 9, wherein, The portable device includes a portable medical device (14) comprising at least one of the following: an automated external defibrillator (AED), a cardiopulmonary resuscitation (CPR) metronome, and an electrocardiogram (ECG) monitor; furthermore, wherein the one or more 2D representations of the first object (38) correspond to one or more of the following: (i) the hand of the first responder, the hands of the first responder, or other body parts of the first responder, and (ii) at least one of the portable devices.

11. The remote assistance workstation (12) according to claim 10, wherein, One or more 2D representations of the at least one item of the portable device also represent at least one selected from the group consisting of: AED pad placement, CPR compression placement, and ECG sensor placement.

12. The remote assistance workstation (12) according to claim 1, further comprising the display device (30), wherein, The display device includes a touchscreen device for receiving the remote assistance input.

13. A method (150, 152) for providing remote assistance via a remote assistance workstation and a portable device, wherein, The portable device includes at least one pair of stereoscopic augmented reality glasses, the portable device being used by a first responder to perform at least one action by using the first object in the scene in conjunction with at least one of (i) an object and (ii) a second object in the scene, the method comprising: In response to a remote assistance request initiated from the portable device, the remote assistance workstation is operatively coupled (154) to the portable device via a communication module; the remote assistance request includes at least a live video stream captured via a camera of the stereoscopic augmented reality glasses in the scene; The remote assistant graphical user interface (156) is drawn on a display device via a user interface module and receives remote assistant input from the remote assistant, wherein the remote assistant graphical user interface includes at least: (i) A first pane, comprising a 3D pane for displaying the live video stream of the remote assistance request, and (ii) A second pane comprising a 2D pane for displaying a 2D representation of the first object at the scene, wherein the drawn 2D representation is movable within the second pane in response to one or more remote assistant inputs, wherein the remote assistant graphical user interface further draws at least one 3D virtual content item in the first pane corresponding to the drawn 2D representation of the first object at the scene in the second pane, relative to at least a reference point within the first pane, wherein the reference point is based on the content of the live video stream, and the corresponding movement of the at least one 3D virtual content item in the first pane in response to the drawn 2D representation being selected and moved relative to the reference point within the second pane; and One or more remote assistance signals are generated via the controller (158) and output to the portable device via the communication module in response to the input of the one or more remote assistants on the augmented reality glasses. The at least one 3D virtual content item is displayed to the first responder in a live view of the scene captured by the camera of the augmented reality glasses, such that the at least one 3D virtual content item appears in the correct position relative to the reference point in the live view, to assist the first responder in performing the at least one action in conjunction with the object or the second object in the scene. The one or more remote assistants move the drawn 2D representation of the first object in the scene in the second pane.

14. The method (150) according to claim 13, wherein, The portable device includes a portable medical device, which includes at least one of the following: an automated external defibrillator (AED), a cardiopulmonary resuscitation (CPR) metronome, and an electrocardiogram (ECG) monitor; and wherein the first object in the scene includes one or more of the following: (i) the hand of a first responder, the hands of a first responder, or other body parts of a first responder, and (ii) at least one of the portable devices.

15. The method (150) according to claim 14, wherein, The 3D virtual content item includes a virtual representation outline, which represents at least one of the following: (i) the first responder's hand, the first responder's two hands or other body parts of the first responder, and (ii) at least one of the portable devices.

16. The method (150) according to claim 13, wherein, The generation via the controller also includes generating one or more remote assistance signals in response to at least one remote assistant input to display at least one highlight on the stereoscopic augmented reality glasses to the first responder within the live view of the scene captured by the camera, the at least one remote assistant input including the remote assistant selecting XY coordinates in the live video stream displayed in the first pane of the remote assistant graphical user interface, further wherein the at least one highlight is displayed at a matching XY coordinate in the augmented reality glasses as seen by the first responder.

17. The method (150) according to claim 13, wherein, The reference point includes one selected from the group consisting of: (i) Reference points of the face of the object determined by a face recognition algorithm applied to an image of the object drawn in the live video stream in the first pane; and (ii) The reference point selected by the remote assistant within the content of the live video stream drawn in the first pane.

18. The method (150) according to claim 17, further wherein, The reference point selected by the remote assistant includes at least one of the following: (i) a reference point on the object or the second object in the scene, (ii) a reference point on the portable device, and (iii) a reference point on the first object; and wherein selecting the remote assistant reference point further includes modifying the view by rotating the view of the second pane such that the vertical direction of the view corresponds to the central axis of the object or the second object in the scene.

19. A non-transitory computer-readable medium having instructions that, when executed by a processor, cause the processor to perform the method according to claim 13.

20. A remote assistance system (10), comprising: The remote assistance workstation (12) according to claim 1; as well as A portable device, wherein the portable device includes at least one pair of stereoscopic augmented reality glasses (36), and the first object (38) in the scene includes one or more of the following: (i) a first responder's hand, the first responder's hands or other body parts of the first responder, and (ii) at least one item; furthermore, wherein the portable device is used by the first responder (34) to perform at least one action in conjunction with at least one of the object (42) and the second object in the scene (40), wherein the portable device also includes a communication module (76) configured to communicate with the remote assistance workstation (12), wherein the at least one item includes at least one artifact, the at least one artifact being used by the first responder (34) in conjunction with performing the at least one action on the object (42) or the second object in the scene (40), and wherein the at least one pair of stereoscopic augmented reality glasses (36) to be worn by the first responder (34) includes a camera for capturing real-time images of the object (42) or the second object in the scene (40).

Images