Method, system and computer program for automatic generation of self-guided augmented reality session plans from remotely guided augmented reality sessions

Abstract

To provide a method, a system, and a program for executing a self-guided augmented reality (AR) session only by a local user guided by a prerecorded instruction.SOLUTION: A method is used to receive data recorded during a remotely guided ART session, and to generate a self-guided AR session plan. The data are segmented to time segments corresponding to a step executed by a local user during the remotely guided ART session. Steps are detected by using at least one of detection of utterance for showing a shift from a step to another step by applying natural language understanding (NLU) algorithm to a conversation included in data, detection of a difference in a position for showing a shift by analyzing a position of an annotation included in data, and detection of a posture shift by analyzing posture data of a camera included in data, and a self-guided AR session plan is generated based on segmented data and three-dimensional expression of a scene.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to the field of Augmented Reality (AR). [Background technology]

[0002] AR-based remote guidance (or "assistance") is a relatively new application of AR technology. An expert user located at a remote location assists and guides a local user in handling physical equipment in the local user's vicinity by overlaying digital content on real-time video captured by the local user's camera. As the local user watches this video of his or her physical environment on a video display, the video is simultaneously transmitted to a remote user who may add textual and graphical annotations at various locations on the video using computer software, and these annotations are immediately transmitted back to the local user and overlaid on the video he or she is watching. Additional guidance may be simultaneously provided by the remote user using text, voice, or video communication between the two users, or a combination thereof. Summary of the Invention [Problem to be solved by the invention]

[0003] To eliminate the need for a remote expert user to provide guidance every time a local user needs to handle a particular device, self-guided AR sessions have been proposed and implemented: these sessions can be run solely by the local user guided by pre-recorded instructions.

[0004] The foregoing examples of the related art and limitations associated therewith are intended to be illustrative and not exhaustive. Other limitations of the related art will become apparent to those of skill in the art upon reading this specification and studying the drawings. [Means for solving the problem]

[0005] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools, and methods that are intended to be exemplary and explanatory only and not limiting in scope.

[0006] One embodiment is directed to a method comprising: operating at least one hardware processor to receive data recorded during a remote-guided augmented reality (AR) session between a remote user and a local user, the recorded data comprising images of a scene captured by a camera of the local user, camera pose data, annotations made by at least one of the remote and local users, and a conversation between the remote user and the local user; automatically generating a three-dimensional (3D) representation of the scene based on the images and camera pose data, and automatically associating annotations with positions within the 3D representation; automatically segmenting at least a portion of the data into a plurality of temporal segments corresponding to a plurality of steps performed by the local user in the scene during the remote-guided AR session, where the plurality of steps are automatically detected using at least one of: (a) applying a natural language recognition (NLU) algorithm to the conversation to detect utterances indicative of step-to-step transitions; (b) analyzing positions associated with the annotations to detect position differences indicative of step-to-step transitions; and (c) analyzing the camera pose data to detect pose transitions indicative of step-to-step transitions. A self-guided AR session plan is automatically generated based on the segmented data and the 3D representation, where the AR session plan includes step-to-step AR guidance on how to perform multiple steps in the scene or a similar scene.

[0007] Another embodiment is directed to a system comprising at least one hardware processor and a non-transitory computer-readable storage medium having program code embodied thereon, the program code executable by the at least one hardware processor to receive data recorded during a remote-guided augmented reality (AR) session between a remote user and a local user, the recorded data comprising images of a scene captured by a camera of the local user, camera pose data, annotations made by at least one of the remote and local users, and a conversation between the remote user and the local user; automatically generating a three-dimensional (3D) representation of the scene based on the images and the camera pose data, and automatically associating annotations with positions within the 3D representation; and automatically segmenting at least a portion of the data into a plurality of temporal segments corresponding to a plurality of steps performed by the local user in the scene during the remote-guided AR session. wherein the steps are automatically detected using at least one of: (a) applying a natural language recognition (NLU) algorithm to the conversation to detect utterances indicative of step-to-step transitions; (b) analyzing positions associated with the annotations to detect position differences indicative of step-to-step transitions; and (c) analyzing camera pose data to detect pose transitions indicative of step-to-step transitions. Automatically generating a self-guided AR session plan based on the segmented data and the 3D representation, wherein the AR session plan includes step-to-step AR guidance for how to perform the steps in the scene or a similar scene.

[0008] A further embodiment is directed to a computer program product comprising a non-transitory computer-readable storage medium having program code embodied thereon, the program code being executable by at least one hardware processor to perform the receiving step of: receiving data recorded during a remote-guided augmented reality (AR) session between a remote user and a local user, the recorded data comprising images of a scene captured by a camera of the local user, camera pose data, annotations made by at least one of the remote and local users, and a conversation between the remote user and the local user; automatically generating a three-dimensional (3D) representation of the scene based on the images and the camera pose data, and automatically associating annotations with positions within the 3D representation; and automatically segmenting at least a portion of the data into a plurality of temporal segments corresponding to a plurality of steps performed by the local user in the scene during the remote-guided AR session. wherein the steps are automatically detected using at least one of: (a) applying a natural language recognition (NLU) algorithm to the conversation to detect utterances indicative of step-to-step transitions; (b) analyzing positions associated with the annotations to detect position differences indicative of step-to-step transitions; and (c) analyzing camera pose data to detect pose transitions indicative of step-to-step transitions. Automatically generating a self-guided AR session plan based on the segmented data and the 3D representation, wherein the AR session plan includes step-to-step AR guidance for how to perform the steps in the scene or a similar scene.

[0009] In some embodiments, a self-guided AR session plan is configured to be executed by a computerized AR system, the self-guided AR session plan comprising program instructions configured to cause the computerized AR system to transition between steps of step-to-step AR guidance in response to at least one of user commands and machine vision detection of step completion.

[0010] In some embodiments, the step-to-step AR guidance includes at least some of the annotations localized according to their associated positions within the 3D representation.

[0011] In some embodiments, the step-to-step AR guidance includes at least a portion of a conversation provided as at least one of audio and text.

[0012] In some embodiments, the steps performed by the local user in the scene include manipulation of at least one physical object, and generating the 3D representation of the scene each includes generating at least one 3D model of the at least one physical object, and the AR guidance from step to step includes guidance on how to manipulate the at least one physical object in the scene or at least one similar physical object in a similar scene, respectively.

[0013] In some embodiments, the method or the program code further comprises or is executable to train a machine learning NLU model to detect utterances indicative of step-to-step transitions based on manually labeled conversation samples from the remote-guided AR session, and application of the NLU algorithm to the conversation is based on the trained machine learning NLU model.

[0014] In some embodiments, the automated detection of steps uses a fusion of at least two of (a), (b) and (c).

[0015] In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the figures and by study of the following detailed descriptions. [Brief description of the drawings]

[0016] Exemplary embodiments are illustrated in the reference figures. Dimensions of components and features shown in the figures have generally been chosen for convenience and clarity of presentation and are not necessarily shown to scale. The figures are listed below.

[0017] [Figure 1] FIG. 1 is a flow diagram of a process involving a remotely guided AR session and generating a self-guided AR session plan from the remotely guided AR session.

[0018] [Diagram 2] FIG. 1 is a block diagram of a system for generating a self-guided AR session plan.

[0019] [Diagram 3] 1 is a flow chart of a method for generating a self-guided AR session plan.

[0020] [Figure 4A] FIG. 1 is a semi-pictorial view of an exemplary personal computing device during a remotely guided AR session. [Figure 4B] FIG. 1 is a semi-pictorial view of an exemplary personal computing device during a remotely guided AR session.

[0021] [Figure 5A] FIG. 13 is an additional semi-pictorial view of an exemplary personal computing device during a remotely guided AR session. [Figure 5B] FIG. 13 is an additional semi-pictorial view of an exemplary personal computing device during a remotely guided AR session. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Disclosed herein are techniques for automatically generating a self-guided AR session plan from a remotely guided AR session, which may be embodied in methods, systems, and computer program products.

[0023] Advantageously, since the technology leverages existing recordings of past remotely guided AR sessions, there is no need to specifically prepare, perform, and record an organized performance to generate a self-guided AR session plan. Rather, the technology intelligently processes data from existing remotely guided AR session recordings to automatically generate a self-guided AR session plan. This is, of course, superior to performing a self-guided AR session by mere playback of such recordings, which may be significantly longer than actually necessary and which typically includes redundant data.

[0024] The technique may be particularly useful when a particular technical procedure performed by a local user under the guidance of a remote user involves multiple steps, such as the manipulation of one or more physical objects by the local user. Advantageously, a recording of such a remotely guided AR session may be automatically segmented into multiple segments corresponding to the performed steps. In one example, the steps may be detected using a natural language recognition (NLU) algorithm that detects utterances indicative of a transition from one step to another in a conversation performed during the remotely guided AR session. In another example, the steps may be detected by analyzing the positions of annotations made during the remotely guided AR session to detect position differences indicative of a transition from one step to another. In a further example, the steps may be detected by analyzing camera pose data from the remotely guided AR session to detect pose transitions indicative of a transition from one step to another.

[0025] A self-guided AR session plan can then be automatically generated based on the segmented data. The plan can include step-to-step AR guidance to be performed by a computerized AR system worn or otherwise used by the user for how to perform the multiple steps. The plan can further include program instructions configured to transition the computerized AR system between steps of the step-to-step AR guidance in response to one or more user commands or machine vision detection of one or more step completions, or both.

[0026] Reference is now made to FIG. 1, which is a flow diagram of a process 100 involving a remotely guided AR session and generating a self-guided AR session plan from the remotely guided AR session.

[0027] In the process 100, the AR session data is received, transmitted, or processed, or a combination thereof, by a remotely assisted session manager 102 during a remotely assisted AR session. The session manager 102 represents one or more computing devices that manage a remotely assisted AR session between a local user 104 and a remote user 106. For example, the session manager 102 may be a computing device located in close proximity to the local user 104 and communicates with a personal computing device (not shown) of the remote user 106 over a network. As a counter example, the session manager 102 may be a computing device located in close proximity to the remote user 106 and communicates with a personal computing device (not shown) of the local user 104 over a network. As a further example, the session manager 102 may be a server-type computing device located remotely from the users 104 and 106 and accessible to them over a network via their personal computing devices. The network may be, for example, a LAN (local area network) or a WAN (wide area network such as the Internet).

[0028] The session manager 102 or the personal computing device of the local user 104 may include or be functionally associated with a camera 108 that captures still images or video, or both, of a physical scene 110 adjacent to the local user. For reasons of simplicity, the term "image" is used hereinafter to denote either a still image, a video frame, or both.

[0029] Scene 110 may include any indoor or outdoor area where a procedure is performed by local user 104 with respect to one or more objects, such as object 112. By way of example, scene 110 may be an area within a factory where a particular piece of equipment (e.g., a machine, or any type of device) undergoes maintenance by local user 104, or the engine compartment of an automobile where a particular automobile part is being serviced by a local user.

[0030] Object 112 may actually include multiple smaller objects, such as separate parts or designated regions of a larger object. In the example shown, these smaller objects are a rectangular button 112a and a screw 112b. For ease of explanation, the term "object" is used hereinafter to describe any type of object that may be located in a scene, be it an entire device or a portion or region of a device.

[0031] Images captured by camera 108 may be transmitted to remote user 106 for viewing, or may be displayed to local user 104, for example, on the screen of a portable computing device, wearable display, or the like.

[0032] The session manager 102 or the personal computing device of the local user 104 may further obtain or calculate camera pose data (sometimes referred to in the art as “camera position and orientation data”) defined relative to a set of three-dimensional (3D) coordinates of the scene 110 in accordance with conventional techniques.

[0033] The session manager 102, or a personal computing device of any one of the users, may further obtain or reconstruct a 3D point cloud of the scene 110 defined relative to a set of 3D coordinates of said scene from camera pose data or images, or both, in accordance with conventional techniques.

[0034] The session manager 102 or the personal computing device of the remote user 106 may further manage the creation and transmission of annotations, such as text or graphic annotations, or both, that the remote user creates and associates with various locations within the image he or she views. The locations of these annotations may then be translated by the session manager 102 or the personal computing device of the remote user 106 relative to a set of 3D coordinates of the scene 110. The annotations and their 3D coordinates may then be transmitted to the personal computing device of the local user 104 and displayed to him or her as an augmentation layer on top of the scene 110 at the corresponding location.

[0035] For example, during a remotely assisted AR session, the remote user 106 may use one or more input devices (e.g., a keyboard, a pointing device, a touch screen, etc.) to overlay annotations on displayed objects of the scene 110. For example, if the remote user 106 wishes to instruct the local user 104 to remove a screw 112b from the object 112, then the remote user may draw a graphical annotation at the location of the screw in one or more of the images, such as an arrow pointing to the location, a shape surrounding the location, or any other type of visual marking of the location. The location is typically a point, but could also be a circumscribed two-dimensional (2D) area, or even a circumscribed 3D space.

[0036] The remote user 106 may attach to the graphical annotation text that describes the annotated object or text that instructs the local user 104 how to manipulate it, or both, thereby creating a combined text-graphic annotation. In some cases, however, the remote user 106 merely draws a graphical annotation at the location of the object and provides the explanation or instructions only in a text or voice conversation that he or she simultaneously conducts with the local user 104.

[0037] In a remotely assisted AR session, annotations are typically made by the remote user 106, but they can be made by the local user 104 as well. For example, the local user 104 can draw an arrow pointing to an object and ask the remote user 106 a question via text, voice, or video chat about the nature of the object or how it should be manipulated.

[0038] The session manager 102 may further facilitate the transmission of conversation between the local user 104 and the remote user 106 during the remote-guided AR session, such as in the form of text chat, voice chat, or face-to-face video chat, or a combination thereof, between the local user 104 and the remote user 106.

[0039] The session manager 102 may record in the AR session database 114 all or a portion of the data acquired, used, transmitted, generated, or calculated, or a combination thereof, during or in relation to the multiple remotely guided AR sessions, such as images, camera pose data, scene 3D coordinates, scene 3D point clouds, annotations, and conversations. If the remotely guided AR session includes an audio or video chat between its participants, then a transcript of such conversation may be generated by the voice / text engine and stored in the AR session database 114 in place of or in addition to the actual audio or video files. For ease of explanation, the term "conversation" is used to describe either the text of the conversation originally carried out in text, or a transcript of the audio or video conversation, or both.

[0040] The stored data, or parts of it, may be time-stamped to enable later temporal correlation between different modalities of data. For example, transcripts may be stored together with metadata in the form of time-stamps for each message, utterance, sentence, etc., indicating when that message / utterance / sentence was sent during the conversation. Similarly, for example, time-stamps may be stored as metadata with each of the annotations and each of the images, respectively, indicating when they were created or taken.

[0041] A system may be provided that generates self-guided AR session plans 116 from recorded remotely guided AR sessions. The system 116 may receive data from the AR session database 114 and automatically generate such self-guided AR session plans, each suitable for performance in multiple scenes similar to or even the same scene as the scene in which the respective remotely guided AR session was recorded.

[0042] In some embodiments, system 116 generates each self-guided AR session plan based solely on data recorded during one or more remotely guided AR sessions, and not based on any other user input.

[0043] Reference is now made to Figure 2, which illustrates a detailed block diagram of a system 116 according to an embodiment. The system 116 may include one or more hardware processors (e.g., CPUs) 200, a random access memory (RAM) 202, and one or more non-transitory computer-readable storage devices 204.

[0044] The storage device 204 may store program instructions and / or components configured to operate the hardware processor 200. The program instructions may include one or more software modules, such as a self-guided AR session plan generation module 206. The software components may include an operating system having various software components and / or drivers that manage and control general system tasks (e.g., memory management, storage device control, power management, networking, etc.) as well as facilitate communication between various hardware and software components.

[0045] System 116 may operate by loading instructions of self-guided AR session plan generation module 206 into RAM 202, where they may be executed by processor 200. The instructions of self-guided AR session plan generation module 206 may cause system 116 to receive recorded data of a remotely guided AR session, process it, and output a self-guided AR session plan 210 that is configured to be executed by a computerized AR system (not shown) worn or otherwise used by a user.

[0046] System 116 as described herein is merely an exemplary embodiment of the invention and may in fact be implemented in hardware only, software only, or a combination of both hardware and software. System 116 may have more or fewer components and modules than those shown, may combine two or more of the components, or may have a different configuration or arrangement of components. System 116 may include any additional components that enable it to function as an operational computer system, such as a motherboard, data bus, power supply, network interface card, display, input devices (e.g., keyboard, pointing device, touch-sensitive display), etc. (not shown). Additionally, the components of system 116 may be co-located or distributed, or the system may operate as one or more cloud computing "instances," "containers," or "virtual machines," or combinations thereof, as known in the art.

[0047] The instructions of the self-guided AR session plan generation module 206 will now be described with reference to the flowchart of FIG. 3, which illustrates a method 300 of generating a self-guided AR session plan according to an embodiment.

[0048] The steps of method 300 may be performed either in the order in which they are presented or in a different order (or even in parallel), so long as the order allows for the input required for a particular step to be obtained from the output of a previous step. Additionally, the steps of method 300 are performed automatically (e.g., by system 116 of FIGS. 1 and 2) unless specifically noted otherwise.

[0049] At step 302, data recorded during a remotely guided AR session may be received. The session took place between at least one remote user and at least one local user. Simply for simplicity of the following description, a single remote user and a single local user will be referred to below.

[0050] The received data may include, for example, images of the scene captured by the local user's camera, camera pose data, annotations made by at least one of the remote and local users, conversations between the remote and local users, or any other data stored in the AR session database 114 (of FIG. 1), such as scene 3D coordinates, scene 3D point clouds, etc., or combinations thereof.

[0051] In step 304, a 3D representation of the scene can be generated based on the image and camera pose data using conventional techniques. This 3D representation can be useful in certain embodiments when the 3D point cloud generated during the remotely guided AR session is a rough and inaccurate representation of the scene, which is sufficient only for simple tasks such as overlaying annotations in approximately correct positions. A 3D representation generated offline, without the real-time constraints of an ongoing AR session, can be much more accurate and can therefore be used to generate a robust self-guided AR session plan. However, if the 3D point cloud generated during the remotely guided AR session is sufficiently accurate, the 3D representation of the scene can be generated from the point cloud instead of being generated from the image and camera pose data.

[0052] Optionally, each generated 3D representation of the scene is or includes at least one 3D model of at least one physical object. For example, if the remotely guided AR session is directed to the maintenance of a desktop printer, then step 304 may include generating a 3D model of the printer. In this way, a subsequently generated self-guided AR session plan (described below) may not depend on the same model of the desktop printer's environment to be maintained, such as the cabinet or shelf where the printer is located, the furniture around the printer, etc.

[0053] In step 306, the annotation may be associated with a preferred location within the generated 3D representation, for example by correlating the original location of the annotation, defined during the remote-guided AR session in a set of 3D coordinates of the scene, with the generated 3D representation and its own set of coordinates.

[0054] Optionally, this step may also include comparing the annotations and their positions with a "digital twin" of the relevant object in the scene, if such a digital twin is available. A digital twin, as known in the art, is a computer-simulated 3D model of an object, which typically includes text labels naming or describing the various parts of the object, or both. Digital twins are often generated to be used as a reference for engineers or other people interested in learning about the complexity of the object. Since digital twins typically include expertly defined labels, these labels may be used in step 306 to enhance, modify, or add to, or combine with, the annotations made in the remote-guided AR session. For example, if a particular annotation reads "button," but the corresponding label in the digital twin (i.e., a label located at substantially the same object location as the annotation) is "main power button," the latter may be preferred. Following a comparison between the two, a final text for each annotation may be selected based on one or more criteria, such as the longest text, the shortest text, a concatenation of both texts, or simply selecting only the label text to override the annotation's text.

[0055] Optionally, the final selected text may be used to replace any text present in the conversation that is identical or similar to the text of the associated annotation. For example, if the annotation "button" in a particular segment is overridden with the digital twin label "main power button", then all occurrences of the text "button" in that conversation may be replaced with "main power button" in the same segment.

[0056] In step 308, at least a portion of the received data (e.g., the modality or modalities, or a portion of the data available for each of the modalities) may be segmented into multiple temporal segments corresponding to multiple steps performed by the local user in the scene during the remotely guided AR session. As a simple example, if the local user presses button 112a, removes screw 112b, cleans the screw hole, and replaces the screw in the hole (i.e., performing four object manipulations), then the data may be segmented into four temporal segments - one for each of these four steps.

[0057] Step 308 may be better understood with reference to Figure 4, which is a semi-pictorial view of an exemplary personal computing device 400 (in this example, a smartphone) of a remote or local user during a remotely assisted AR session. Specialized software running on device 400 may facilitate the session by interfacing with the device's camera or cameras and its network (e.g., Internet) connectivity capabilities (e.g., performing text, voice, or video chat, or a combination thereof), and by providing a graphical user interface (GUI) with the functionality required for the session.

[0058] One area on the display of device 400 may display an image or video 402 captured by the local user's camera. Another area may display a conversation 404 between the remote user and the local user, which in this illustrative example is conducted as a text chat.

[0059] The displayed conversation 404 includes messages spanning the time period between 9:59:30 and 10:02:08 (hours:minutes:seconds). John, a local user, asks Jane, a remote user, how to replace a rusty screw. Jane guides John through the steps required to replace the screw, the first two of which are shown: turning off the power to the unit and removing the rusty screw. The remote guidance provided by Jane includes both text instructions and graphical annotations that she places next to where, in this example, the "OFF" button and the screw are displayed on her screen. The steps performed by John include the manipulation of three physical objects displayed in the scene: the "OFF" button pressed, the unit turned off (by pressing a button), and the screw removed. However, the relationship between steps and object manipulations is not necessarily one manipulation per step. In certain embodiments, it is quite possible that a step may include multiple manipulations of either the same object or different objects.

[0060] In the exemplary scenario of FIG. 4, it may be beneficial to segment the recorded data into two segments: a first segment from 9:59:30 to 10:01:21, corresponding to the first step (turning off the unit), and a second segment from 10:01:44 to 10:02:08, corresponding to the second step (removing the screw). Of course, the segments can overlap, if desired, so that each new segment also includes brief parts of the previous segment, serving as a reminder to the self-guided user. Also, when segmenting the recorded data, it is possible to omit data portions (of one or more modalities) that are considered unnecessary for the self-guided session, such as niceties exchanged between users of a remote-guided session, or, in a more general sense, anything that is not a direct command to perform a particular action in the scene.

[0061] Now referring back to the flow chart of figure 3, in order to detect such existing steps in the data, one or more of the following sub-steps may be performed.

[0062] In sub-step 308a, machine learning natural language recognition (NLU) algorithms may be applied to the conversation to detect utterances made by the local user or the remote user, or both, that indicate transitions from step to step. For example, with tentative reference to FIG. 4, the local user may utter (by voice or by typing text) an expression such as "I pressed," where the past tense of the verb "press" indicates the completion of a step, or an expression such as "What's next?", which is one of many typical expressions that people tend to use to indicate that they have completed a step. As a result, the remote user may utter an expression or words such as "First..." or "Now...," which are words that people typically use when starting a description of a new step.

[0063] The NLU algorithm may detect such utterances utilizing a previously trained machine learning NLU model based on manually labeled speech samples from remotely guided AR sessions. The samples may be labeled by human reviewers who are instructed to label (i.e., mark) those words or expressions in the samples that are believed to indicate a transition from one step to another, such as the completion of a step or the start of a new step, or both. Training the model based on these labeled samples may be performed using conventional techniques.

[0064] In some embodiments, the NLU algorithm is an intent classification algorithm (sometimes referred to as an "intent analysis" or "intent detection" algorithm) that is configured to analyze a conversation to detect words, expressions, or sentences, or combinations thereof, that can serve as a basis for expressing an intent to proceed to a next step and / or reporting completion of a current step.

[0065] Alternatively, or in addition to using an NLU algorithm, transitions from step to step may be detected by searching for words or expressions, or both, from the conversation in a custom dictionary that includes words or expressions, or both, predefined to indicate such transitions. The search may be extended to cover synonyms and / or grammatical inflections of such words or expressions, or both.

[0066] In sub-step 308b, the positions (in the 3D representation of the scene) associated with the annotations may be analyzed to detect position differences indicative of transitions from step to step. For example, with tentative reference to Fig. 4, the fact that the "push" and "remove" annotations are associated with different positions in the 3D representation of the scene may indicate that they belong to two different steps. That is, subsequent annotations made at positions significantly apart in the scene may indicate that the focus of the participating user has shifted from one object to another, and possibly from one step to another.

[0067] Thus, in sub-step 308b, one segment may be defined as the period from (and including) the creation of the "push" annotation to (but not including) the creation of the "remove" annotation, and a subsequent segment may be defined as the period from (and including) the creation of the "remove" annotation to (but not including) the creation of the next annotation (not shown in FIG. 4). However, as generally described above with respect to step 308, the segments may overlap, if desired.

[0068] The detection of the difference in annotation positions is based on a predefined threshold, so that very close (below the threshold) positions do not trigger step transition detection and vice versa. The threshold may be defined, for example, in units of physical distance in the real scene (e.g., X inches). However, since the overall size of the scene or the object being manipulated may vary significantly between remotely guided AR sessions, the threshold may be defined for the mean, median, mode, or different statistical measure of the distance between subsequent annotations made in the session. For example, in a scene with manipulations made on a car, the average distance between multiple pairs of subsequent annotations may be 2 feet, while in a scene with manipulations made on a desktop printer, the average distance may be 2 inches. Thus, a suitable threshold may be, for example, 50%-70% of the average distance. More broadly defined, the threshold may be set at 20%-90% of the mean, median, mode, or different statistical measure of the distance between subsequent annotations. In further embodiments, the threshold may be set at a value beyond the aforementioned range of 20%-90%.

[0069] Additionally or alternatively to the distance threshold, a time threshold may be set, so that annotations made very close together (below the time threshold) will not trigger step transition detection, and vice versa. As with the distance threshold, the time threshold may be set hard, in seconds, or flexibly, as a percentage of a statistical measure (e.g., mean, median, mode) of the time difference between all subsequent pairs of annotations.

[0070] In sub-step 308c, the camera pose data may be analyzed to detect pose transitions indicative of transitions from step to step. Similar to the analysis of annotation positions in the previous sub-step 308b, the camera pose data is yet another source that allows understanding of the shifting focus of the user during a remotely guided AR session. If the local user's camera maintains approximately the same pose for a period of time, then undergoes a substantial pose change and is maintained for another period of time, this may indicate that two different steps have been performed by the local user.

[0071] 5A and 5B, which show the same object as in FIGS. 4A and 4B on the display of a smartphone 500, if the camera pose changes from looking at the front of the object (FIG. 5A) to looking at the side of the object (FIG. 5B), this is likely to indicate a first pose associated with a first step and a second pose associated with a second step. For example, the second pose may enable the local user to see (and potentially manipulate) a hole 502 in the left facet of the object that is not visible in the first pose.

[0072] Detection of such pose transitions is based on thresholds of differences in camera position or orientation, or both, and, optionally, also on temporal thresholds that filter out poses that are not long-maintained (e.g., camera pose movements that are more likely to be due to accidental or otherwise inconsequential movements than movements indicative of actual focus movements by the local user). These thresholds can be either conventionally defined strictly in units of distance / angle (corresponding to the position / orientation of the pose data) and units of time, respectively, or flexibly defined as a percentage of a certain statistical measure of pose change.

[0073] Optionally, detecting steps according to step 308 may include fusion of two or all three of sub-steps 308a, 308b and 308c. Fusion may include, for example, adding one or more steps that are detected by one or more of the three (NLU, annotation position analysis, camera pose analysis) techniques but not by one or more of the three different techniques. For example, if detection by the NLU algorithm produces two segments with a long time gap between them, and detection by the camera pose analysis produces a segment exactly at that long gap (or even if it slightly overlaps with the adjacent segment), then all three segments may be utilized. Another option is to go through an automatic voting process, where the segmentation is decided by majority vote among the three different techniques, or by a random technique in case of equal votes for any particular segment.

[0074] At step 310, a self-guided AR session plan (hereinafter "session plan") may be generated based on the segmented data and the 3D representation. The session plan may include step-to-step AR guidance on how to perform multiple steps in the scene where the remotely guided AR session took place, or in similar scenes (e.g., scenes containing the same or similar objects requiring the same or similar manipulations).

[0075] The session plan may include a set of program instructions configured to be executed (312) by a computerized self-guided AR system, which in some embodiments is fully or partially wearable or portable by its user. The system may include at least a camera, a display, and a computerized device that cooperate to analyze a scene imaged by the camera, track relevant objects therein, and overlay information in real time as an augmentation layer on the displayed scene and / or otherwise provide such information to the user.

[0076] The information provided by the self-guided AR system, in this embodiment, is guidance included in a session plan. The guidance may include instructions for how to manipulate one or more physical objects displayed in the associated scene in each of a number of steps. These instructions may include, for example, at least a portion of the annotations or at least a portion of the text of the conversation that took place during the remote-guided AR session, or both. Additionally or alternatively, the instructions may not include the annotations or verbatim text from the conversation, or both, but rather an automatically processed version thereof, such as a summary generated by a text summarization algorithm or the like.

[0077] For example, with tentative reference to FIG. 4A and FIG. 4B, a first step of a session plan generated based on the illustrated dialogue and annotations may be the following text: "Press the main power button indicated by the arrow. Wait until the unit powers down." Such text may be the result of applying a text summarization algorithm to the dialogue in the segment corresponding to the first step, as well as the use of the digital twin to override the word "press" with the more descriptive word "main power button." At the same time as providing the user with that text, an arrow may be displayed as an augmentation layer on the scene displayed to the user, pointing to the location of the main power button. Similarly, any other suitable graphical annotation may be displayed at or near the location of the main power button. Optionally, the guidance text itself may be displayed as an augmentation layer, for example, at or near the location of the main power button, or further away from the button (so as not to obstruct the view), but with an arrow, line, or different graphic connector that visualizes the attributes of the textual guidance relative to the button's location. Alternatively, the guidance text may be displayed not as an augmentation layer on the scene, but in a different area of ​​the computerized display used by the user. In a further option, the guidance, or certain portions thereof, may be converted to audio using a text-to-speech algorithm and output to the user using one or more speakers, headphones, etc.

[0078] In order to enable the self-guided AR system to track relevant objects in the scene that are not favorable for correct overlay of annotations, instructions, etc., the session plan may also include a generated 3D representation of the scene or a generated 3D model of the object, or both, encoded in a format readable by the self-guided AR system.

[0079] Advantageously, the step-by-step aspect of the session plan may cause the self-guided AR system to initiate the display of guidance associated with each step in response to user commands, or machine vision detection of the completion of the previous step, or both.

[0080] In one option, the program instructions of the session plan may be configured to transition the self-guided AR system between steps of the guidance in response to the user speaking, typing, or otherwise issuing a command to proceed to the next step. For example, the program instructions may consist of a list of words or expressions, or both, that, when spoken by the user and captured by the microphone of the computerized AR system, cause it to proceed to the next step of the guidance, such as "go ahead," "next," "continue," or "keep going." Additionally or alternatively, the NLU model trained in sub-step 308a to detect utterances indicative of step-to-step transitions may again be used to detect utterances indicative of a user's request to proceed the self-guided AR system to the next step of the guidance.

[0081] In a further option, the program instructions of the session plan may display buttons, e.g., labeled "Next Step," "Continue," etc., that the user can activate to proceed to the next step in the AR guidance. The buttons may be displayed, for example, on a touch-sensitive display of the user's computerized device, or may be overlaid as an augmentation layer on the displayed scene, or both.

[0082] In yet another alternative, the program instructions of the session plan may be configured with a machine vision algorithm that analyzes the imaged scene in real time to detect a physical change in the scene that signifies that a step has been completed. With tentative reference to Figures 4A and 4B, the machine vision algorithm may detect that a button protruding from the top surface in Figure 4A is subsequently depressed and protruding slightly in Figure 4B. The program instructions may be configured to associate this particular visual detection with the completion of a step that involves pressing the button, and may then cause the computerized AR system to automatically proceed to the next step of the AR guidance.

[0083] The machine vision algorithm may be trained based on visual changes to objects shown in various segments of the recorded data. Training may be fully automatic in some embodiments: a training set may be automatically generated from images taken during a remotely guided AR session that have been automatically labeled to be associated with a completed or not completed step. For example, with tentative reference to FIG. 4A and FIG. 4B, images taken between 9:59:30 and 10:00:41 may be automatically labeled as not associated with the completion of the first step, since they were taken before the local user had an opportunity to perform the first step operation (pressing a button). In contrast, images taken after 10:00:55, for example up to 10:01:44 (a time when the user may have begun performing the second step), may be automatically labeled as associated with the completion of the first step, since they were taken after the local user reported the completion of the first step (pressing a button). Training may then be performed automatically using conventional techniques based on the labeled images.

[0084] Optionally, the training session is performed step by step to generate a step-specific machine vision model configured to detect the particular operation expected to occur in that step. Thus, when the session plan is executed, the machine vision algorithm may utilize the pre-generated machine vision model associated with the currently performed step to analyze the captured images and detect the modeled operation therein in real time.

[0085] Optionally, even if a machine vision algorithm is used to detect step completion, the program instructions may still cause the self-guided AR system to obtain explicit user confirmation before proceeding to the next step. For example, when the machine vision algorithm detects that the main power button has been pressed, the self-guided AR system may display a confirmation button and / or audibly ask the user whether to proceed, and wait for a button activation or a verbal command by the user, or both, respectively, before proceeding.

[0086] The present invention may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include computer readable storage medium(s) having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0087] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices having instructions recorded thereon, and any suitable combination of the foregoing. A computer-readable storage medium as used herein is not intended to be interpreted as being itself a transitory signal, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a wave guide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electric signal transmitted through a wire. Rather, the computer-readable storage medium is a non-transitory (non-volatile) medium.

[0088] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0089] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0090] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0091] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus produce means for implementing the functions / acts specified in a block or blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium that may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored may include an article of manufacture including instructions that implement aspects of the functions / acts specified in a block or blocks of the flowcharts and / or block diagrams.

[0092] Furthermore, the computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device and cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device may implement the functions / operations specified in a block or blocks of the flowcharts or block diagrams, or both.

[0093] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special purpose hardware-based system that performs the specified functions or acts or executes a combination of special purpose hardware and computer instructions.

[0094] The description of a numerical range should be considered as specifically disclosing not only the individual numbers within that range but also all the possible subranges. For example, a description of a range from 1 to 6 should be considered as specifically disclosing not only the individual numbers within that range, e.g., 1, 2, 3, 4, 5, and 6, but also the subranges such as 1 to 3, 1 to 4, 1 to 5, 2 to 4, 2 to 6, 3 to 6, etc. This applies regardless of the breadth of the range.

[0095] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to most appropriately explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the industry, or to enable other skilled in the art to understand the embodiments disclosed herein.

Claims

1. At least one hardware processor, Receiving data recorded during a remote-guided augmented reality (AR) session between a remote user and a local user, the recorded data comprising: an image of a scene captured by a camera of the local user; pose data of the camera; and An annotation created by at least one of the remote user and the local user; A conversation between the remote user and the local user; receiving the automatically generating a three-dimensional (3D) representation of the scene based on the images and the camera pose data, and automatically associating the annotations with positions in the 3D representation; and automatically segmenting at least a portion of the data into a plurality of temporal segments corresponding to a plurality of steps performed by the local user in the scene during the remotely guided AR session, the plurality of steps comprising: (a) applying a natural language recognition (NLU) algorithm to the conversation to detect utterances indicative of transitions from step to step; (b) analyzing the positions associated with the annotations to detect position differences indicative of transitions from step to step; (c) analyzing the camera pose data to detect pose transitions indicative of step-to-step transitions; and automatically detecting the segmentation using at least one of the following: automatically generating a self-guided AR session plan based on the segmented data and the 3D representation, the self-guided AR session plan including step-to-step AR guidance on how to perform the plurality of steps in the scene or a similar scene; The method of claim 1, comprising the steps of:

2. the self-guided AR session plan is configured to be executed by a computerized AR system; The self-guided AR session plan comprises: User commands, Machine vision detection of step completion and 2. The method of claim 1, comprising program instructions configured to transition between steps of the AR guidance from step to step in response to at least one of:

3. The method of claim 1 or 2, wherein the step-to-step AR guidance includes at least some of the annotations localized according to their associated positions within the 3D representation.

4. The method of claim 1 , wherein the step-to-step AR guidance includes at least a portion of the conversation provided as at least one of audio and text.

5. the steps performed by the local user in the scene include manipulating at least one physical object; said generating said 3D representation of said scene each includes generating at least one 3D model of said at least one physical object; the step-to-step AR guidance includes guidance on how to manipulate the at least one physical object in the scene or at least one similar physical object in the similar scene, respectively.

5. The method according to any one of claims 1 to 4.

6. The method further comprises training a machine learning NLU model to detect utterances indicative of step-to-step transitions based on manually labeled speech samples from a remotely guided AR session; the application of the NLU algorithm to the conversation is based on the trained machine learning NLU model; 6. The method according to any one of claims 1 to 5.

7. The method of claim 1 , wherein the automatic detection of steps uses a fusion of at least two of (a), (b) and (c).

8. (i) at least one hardware processor; (ii) a non-transitory computer readable storage medium having program code embodied therein, the program code comprising: Receiving data recorded during a remote-guided augmented reality (AR) session between a remote user and a local user, the recorded data comprising: an image of a scene captured by a camera of the local user; pose data of the camera; and An annotation created by at least one of the remote user and the local user; A conversation between the remote user and the local user; receiving the automatically generating a three-dimensional (3D) representation of the scene based on the images and the camera pose data, and automatically associating the annotations with positions in the 3D representation; and automatically segmenting at least a portion of the data into a plurality of temporal segments corresponding to a plurality of steps performed by the local user in the scene during the remotely guided AR session, the plurality of steps comprising: (a) applying a natural language recognition (NLU) algorithm to the conversation to detect utterances indicative of transitions from step to step; (b) analyzing the positions associated with the annotations to detect position differences indicative of transitions from step to step; (c) analyzing the camera pose data to detect pose transitions indicative of step-to-step transitions; and automatically detecting the segmentation using at least one of the following: automatically generating a self-guided AR session plan based on the segmented data and the 3D representation, the self-guided AR session plan including step-to-step AR guidance on how to perform the plurality of steps in the scene or a similar scene; a non-transitory computer-readable storage medium executable by the at least one hardware processor to perform A system comprising:

9. the self-guided AR session plan is configured to be executed by a computerized AR system; The self-guided AR session plan comprises: User commands, Machine vision detection of step completion and 10. The system of claim 8, further comprising program instructions configured to transition between steps of the AR guidance from step to step in response to at least one of:

10. The system of claim 8 or 9, wherein the step-to-step AR guidance includes at least some of the annotations localized according to their associated positions within the 3D representation.

11. The system of claim 8 , wherein the step-to-step AR guidance includes at least a portion of the conversation provided as at least one of audio and text.

12. the steps performed by the local user in the scene include manipulating at least one physical object; said generating said 3D representation of said scene each includes generating at least one 3D model of said at least one physical object; the step-to-step AR guidance includes guidance on how to manipulate the at least one physical object in the scene or at least one similar physical object in the similar scene, respectively. A system according to any one of claims 8 to 11.

13. The program code is further executable by the at least one hardware processor to train a machine learning NLU model to detect utterances indicative of step-to-step transitions based on manually labeled speech samples from a remotely guided AR session; the application of the NLU algorithm to the conversation is based on the trained machine learning NLU model; A system according to any one of claims 8 to 12.

14. 14. The system of claim 8, wherein the automatic detection of steps uses a fusion of at least two of (a), (b) and (c).

15. The processor:

1. A method for receiving data recorded during a remote-guided augmented reality (AR) session between a remote user and a local user, the recorded data comprising: an image of a scene captured by a camera of the local user; pose data of the camera; and An annotation created by at least one of the remote user and the local user; A conversation between the remote user and the local user; receiving the signal; automatically generating a three-dimensional (3D) representation of the scene based on the images and the camera pose data, and automatically associating the annotations with positions in the 3D representation; 2. Automatically segmenting at least a portion of the data into a plurality of temporal segments corresponding to a plurality of steps performed by the local user in the scene during the remotely guided AR session, the plurality of steps comprising: (a) applying a natural language recognition (NLU) algorithm to the conversation to detect utterances indicative of transitions from step to step; (b) analyzing the positions associated with the annotations to detect position differences indicative of transitions from step to step; (c) analyzing the camera pose data to detect pose transitions indicative of step-to-step transitions; an automatic segmentation step, the automatic segmentation being detected automatically using at least one of automatically generating a self-guided AR session plan based on the segmented data and the 3D representation, the self-guided AR session plan including step-to-step AR guidance on how to perform the steps in the scene or a similar scene; A computer program that executes the following:

16. the self-guided AR session plan is configured to be executed by a computerized AR system; The self-guided AR session plan comprises: User commands, Machine vision detection of step completion and 16. The computer program product of claim 15, comprising program instructions configured to transition between steps of the AR guidance from step to step in response to at least one of:

17. 17. The computer program product of claim 15 or 16, wherein the step-to-step AR guidance comprises at least some of the annotations localized according to their associated positions within the 3D representation.

18. 18. The computer program product of claim 15, wherein the step-to-step AR guidance includes at least a portion of the conversation provided as at least one of audio and text.

19. the steps performed by the local user in the scene include manipulating at least one physical object; said generating said 3D representation of said scene each includes generating at least one 3D model of said at least one physical object; the step-to-step AR guidance includes guidance on how to manipulate the at least one physical object in the scene or at least one similar physical object in the similar scene, respectively.

19. A computer program according to any one of claims 15 to 18.

20. further causing the processor to train a machine learning NLU model to detect utterances indicating step-to-step transitions based on manually labeled speech samples from a remotely guided AR session; the application of the NLU algorithm to the conversation is based on the trained machine learning NLU model; 19. A computer program according to any one of claims 15 to 18.

Images