Methods for augmented reality application for adding annotations and interfaces to control panels and screens

The AR-based remote assistance system addresses the limitations of existing AR technologies by using image tracking and anchor images to create interactive overlays, enhancing user guidance and reducing errors in complex interface interactions.

JP2024169556AActive Publication Date: 2024-12-05FUJIFILM BUSINESS INNOVATION CORP
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2024160657
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-02-10
Filing Date
2024-09-18
Publication Date
2024-12-05
Estimated Expiration
2040-09-15

AI Technical Summary

Technical Problem

Existing AR technologies for remote assistance require users to follow complex verbal instructions, are prone to errors, and lack efficient methods for asynchronous guidance, especially when interacting with complex interfaces like MFDs.

Method used

An AR-based remote assistance system that uses image tracking and anchor images to generate overlays on a user's screen, allowing remote experts to provide interactive and asynchronous guidance by recording procedures during live streams for later playback.

Benefits of technology

Enhances user understanding by providing clear, interactive AR overlays that guide users through complex tasks, reducing errors and the need for real-time communication, and enabling asynchronous learning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024169556000001_ABST
    Figure 2024169556000001_ABST
Patent Text Reader

Abstract

To provide methods for generating an overlay display to which annotations can be added on a screen using an augmented reality technique.SOLUTION: A method of the present invention includes: pausing a video received for display on a second device from a first device on the second device; while pausing the video received from the first device on the second device, accepting input including annotations for the paused video on the second device; and generating an augmented reality (AR) overlay corresponding to a portion of the paused video on a display of the first device in response to input made to the portion of the paused video on the second device if a pause duration of the input exceeds a threshold.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates generally to Augmented Reality (AR) systems, and more specifically to systems and methods for generating AR-enabled control panels and screen interfaces. [Background technology]

[0002] In the related art, AR applications have been implemented to provide an interface for a user to operate a vehicle's dashboard or stereo system. Other applications may utilize AR during an Internet browsing session to add overlays to web pages to aid in navigating the Internet. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 9,679,144 [Patent Document 1] U.S. Pat. No. 10,089,769 [Non-patent literature]

[0004] [Non-Patent Document 1] HENDERSON, S. et al. "Augmented Reality for Maintenance and Repair" Air Force Research Laboratory Human Effectiveness Directorate Warfighter Readiness Research Division Logistics Readiness Branch Wright Patterson AFB OH 45433-7604, August 2007 [Non-Patent Document 2] PALMARINI, R. et al. "A Systematic Review of Augmented Reality applications in Maintenance" Robotics and Computer-Integrated Manufacturing, February 2018, pp. 215-228, Volume 49 [Non-Patent Document 3] KLINKER, G. et al. "Augmented maintenance of powerplants: A prototyping case study of a mobile AR systern" [Non-Patent Document 4] YEH, T. et al. "Creating Contextual Help for GUIs Using Screenshots" VIST'11, October 16-19, 2011, Santa Barbara, California, USA [Non-Patent Document 5] BAIRD, K. et al. "EVALUATING THE EFFECTIVENESS OF AUGMENTED REALITY AND WEARABLE COMPUTING FOR A MANUFACTURING ASSEMBLY TASK" Thesis for Master of Science in Industrial and Systems Engineering, June 21, 1999, Blacksburg, Virginia, USA [Non-Patent Document 6] GATTULLO, M. et al. "From paper manual to AR manual: do we still need text?" 27th International Conference on Flexible Automation and Intelligent Manufacturing, FAIM2017, June 27-30, 2017, pp. 1303-1310; Modena, Italy [Non-Patent Document 7] STOCK, I. et al. "Metadata based Authoring for Technical Documentation" SIGDOC'05, 21-23 September 2005, Coventry, UK [Non-Patent Document 8] TANG, Q. et al. "The implementation of an AR (augmented reality) approach to support mammographic interpretation Training - an initial feasibility study" Proceedings Volume 10136, Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment, March 10, 2017 [Non-Patent Document 9] DAI, Y. et al. "Tipper. Contextual Tooltips that provide Seniors with Clear, Reliable Help for Web Tasks" CHI'15 Extended Abstracts, April 18-23, 2015, Seoul, Korea, pp. 1773-1778. [Non-Patent Document 10] +Rescue by LogMeln "https: / / www.logmeinrescue.com / " Accessed January 14, 2020 [Non-Patent Document 11] ScopeAR "https: / / www.scopear.com / " Accessed January 14, 2020 [Non-Patent Document 12] Vuforia Chalk AR "https: / / chalk.vuforia.com / " Accessed January 14, 2020 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention aims to provide a method for generating an overlay display on a screen to which annotations can be added, using augmented reality technology. [Means for solving the problem]

[0006] The embodiments described herein are directed to AR implementations that can stream a calibrated view of a display (e.g., a computer screen, a touch liquid crystal display (LCD), a digital control panel, an appliance control panel) and add live (real-time) or automated agents overlays that guide the user through the process (e.g., which button to click or tap on the screen, where to enter text, etc.). An embodiment includes registration, i.e., detecting the boundaries of the object of interest to properly display the AR overlay on the screen even as the user moves the camera. In another embodiment, the creation of marks can be based on the underlying content and automatically removed once an action is performed. In another embodiment, automatic detection of occlusions is performed to display the instruction overlay in a realistic manner. Finally, an automated process can capture existing video material (e.g., a how-to video for an appliance's LCD display such as a multi-function device (MFD)) and extract anchor images to use to start the registration process.

[0007] In an embodiment, the AR interface can be extended to live remote assistance tasks, where a remote expert connects with people who share a live stream from a mobile device or a head-mounted device to diagnose and solve problems on-site. With the introduction of live streaming services, live remote assistance is becoming one way for users to troubleshoot increasingly difficult problems. The tools in the background art focus on allowing remote users to annotate and demonstrate solutions, but do not consider the time and effort that users need to spend to follow instructions. Instructions often need to be repeated until the user fully understands, and sometimes two hands are required to operate physical equipment, making streaming live video from a mobile device difficult. To address these issues, an embodiment makes available an AR-based tool for remote assistance interfaces that can automatically record procedures during a live stream that users can watch asynchronously.

[0008] Aspects of the disclosure include a method that may include stabilizing a video received from a first device for display on a second device, and generating an augmented reality (AR) overlay on a display of the first device that corresponds to the portion of the stabilized video in response to input made to the portion of the stabilized video on the second device.

[0009] Fixed display of the video received from the first device for display on the second device may include identifying one or more anchor images within the video, determining a target object that is a two-dimensional surface based on the identified one or more anchor images, and performing perspective correction of the video based on the target object that is the two-dimensional surface for display on the second device.

[0010] Identifying the one or more anchor images in the video may include searching a database for one or more anchor images that match one or more images in the video.

[0011] Identifying the one or more anchor images in the video may include detecting a rectangle on the video received from the first device, matching a three-dimensional plane to two-dimensional points of the detected rectangle, tracking the three-dimensional plane that matches the two-dimensional points of the detected rectangle, and receiving a selection of the one or more anchor images in the video via the second device.

[0012] The method may further include cropping the object of interest in the video for display on the second device.

[0013] The target object may be a display screen.

[0014] Generating the AR overlay on the display of the first device corresponding to the portion of the frozen video may be performed live in response to input made to the portion of the frozen video on the second device.

[0015] Generating the AR overlay on the display of the first device corresponding to a portion of the frozen displayed video may include generating the AR overlay on the display of the second device in response to the input, and, if an instruction to provide the AR overlay to the first device is received, transmitting the AR overlay to the first device for display.

[0016] The method may further include tracking one or more hands and fingers in the video and occluding portions of the AR overlay that overlap the one or more hands and fingers on the display of the first device.

[0017] Freezing the video received from the first device for display on the second device may include pausing the video on the display at the second device, and the input may include annotations.

[0018] Generating an augmented reality (AR) overlay on the display of the first device corresponding to a portion of the frozen video may include generating a video clip including the annotation when a pause in the input exceeds a timeout threshold and providing the video clip on the display of the first device.

[0019] The method may further include playing the video clip on the display of the first device when it is determined that the first device is placed down, and providing the video for display on the second device when it is determined that the first device is in a user's hand.

[0020] The AR overlay may include instructions for moving a cursor of the first device from a first location to a second location.

[0021] The method may further include storing the AR overlay for playback by the first device.

[0022] Aspects of the disclosure include a non-transitory computer-readable medium storing instructions for performing a process, the instructions including: pinning a video received from a first device for display on a second device; and, in response to input made to the portion of the pinned-displayed video on the second device, generating an augmented reality (AR) overlay on a display of the first device that corresponds to the portion of the pinned-displayed video.

[0023] Aspects of the present disclosure include a system comprising: means for pinning a video received from a first device for display on a second device; and means for generating, in response to input made to the portion of the video pinned on the second device, an augmented reality (AR) overlay on a display of the first device that corresponds to the portion of the video pinned. [Brief description of the drawings]

[0024] [Figure 1] 1 illustrates an exemplary flow for overlaying an AR interface and / or annotations on a screen or control panel, according to an embodiment. [Diagram 2] 1 illustrates an exemplary overlay on a device panel captured from a user device, according to an embodiment. [Diagram 3] 1 illustrates an example of a three-dimensional overlay node with transitions, according to an embodiment. [Figure 4] 1 shows a perspective corrected panel according to an embodiment. [Diagram 5] 13 shows an example of an overlay where a hand and finger mask is implemented such that the overlay is placed under the hand or finger. [Figure 6] 13 illustrates an example of recording and playing an AR interface, according to an embodiment. [Figure 7] 1 shows a flow diagram of an annotation and recording process, according to an embodiment. [Figure 8] 1 illustrates an example of a computing device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The following detailed description provides details of the figures and embodiments of the present application. Descriptions of elements that overlap between reference numbers and figures are omitted for clarity. Terms used throughout the description are provided as examples and are not intended to be limiting. For example, the use of the term "automatic" may include fully automatic or semi-automatic implementations that include user or administrator control over certain aspects of the implementation, depending on the desired implementation of the person skilled in the art implementing the present application. Selections may be made by a user via a user interface or other input means, or may be implemented via a desired algorithm. The embodiments described herein may be utilized alone or in combination, and the functionality of the embodiments may be implemented through any means according to the desired implementation.

[0026] Remotely helping customers troubleshoot sophisticated appliances such as MFDs can pose several challenges: voice-only communication is error-prone, and dispatching a service engineer to a customer's site can be time-consuming and costly.

[0027] To address this situation, many electronics manufacturers create how-to videos. When videos are not sufficient, customers still need live assistance from a service engineer. In an embodiment, an AR system is provided that is configured to provide an AR overlay on a screen or control panel, such as a computer / smartphone screen when a customer is installing drivers for a new MFD, or an LCD screen of an MFD when a worker is touching buttons to configure the MFD. In particular, the embodiment takes advantage of the surfaces of the screen and control panel, which are essentially two-dimensional surfaces, to provide an overlay that is superior to annotations and related technology implementations.

[0028] In background technology implementations, a customer installs screen sharing software that allows the remote engineer to view the customer's screen, control the customer's screen, and move a remote cursor to guide the customer. In many cases, users can only rely on video, such as taking a picture of the LCD or control panel with a smartphone to show the remote engineer what the customer sees.

[0029] In such background art implementations, installing screen sharing software on a personal computer can be problematic: for example, the customer may already have requested assistance in installing other software, the company may not easily allow the installation of new software, the computer may not be connected to the Internet, or there may not be a screen sharing application for mobile devices.

[0030] Additionally, with a video stream, the remote engineer can become confused if the user moves the phone, and communication can be severely impaired by being limited to verbal instructions (e.g., "Right, click this red button on the bottom left, and no, not this button, it's that one, and then press all of these buttons at the same time and hold them down for three seconds").

[0031] To address these issues, embodiments support an AR interface and overlay system for control panels and screens (e.g., computer screens, touch screens, MFDs, or typical digital control panels found on appliances such as microwave ovens, car stereo systems, etc.) Using only a mobile device utilizing the AR interface of the embodiments described herein, a user can point the mobile device's camera at the screen / LCD / panel and a remote engineer can interactively add overlay instructions to guide the user.

[0032] 1 illustrates an exemplary flow for overlaying an AR interface and / or annotations on a screen or control panel, according to an embodiment. The flow begins at 100 when a local user connects to a remote assistance system via a user device.

[0033] In an embodiment, the system performs image tracking as a basis for detecting and tracking a screen or control panel. At 101, the system searches a database for anchor images that match the streamed content. Depending on the desired implementation, data can be automatically or manually populated into the database of anchor images that represent the objects to be detected. An anchor image is an image that has been processed to extract key points.

[0034] If the screen or LCD display is static and of a known device (such as the LCD panel of a known MFD), the reference image may be pre-added to the application or may be retrieved from an online database and downloaded to the application. For example, in the case of an MFD, there may be a series of images showing the LCD control panels of specific MFD devices, so that the application can automatically detect and track these types of control panels as soon as they appear in the camera's field of view. Similarly, a series of images may be created for generic standard laptop models. Thus, if an anchor image is found within the application or can be retrieved from an online database, such an anchor image is used at 103.

[0035] If no anchor image is found (102), the application also supports dynamic registration of unseen objects or LCD displays, in which case the rectangle detector can be used in conjunction with the AR plane detector. Specifically, when a service engineer or local user taps on the screen, the application can be configured to run a rectangle or quadrangle detector in the current frame and test the projection of the four corners in three-dimensional space that intersect with the known AR plane. A three-dimensional plane that matches the two-dimensional points of the rectangle is then created and tracked in three-dimensional space by the AR framework, which then selects an anchor image at 104.

[0036] Once these reference images are established, the video frames captured by the application are perspective corrected so that a stable version of the area can be seen by the remote engineer, and an Augmented Reality Overlay (ARO) can be created at 105. The remote assistant can then annotate the stream at 106, after which the application system determines if there is an object occluding the screen at 107. If so, the annotation is hidden at 109, otherwise the annotation is revealed at 108.

[0037] Once the application detects and tracks the anchor, the remote engineer can click on the screen to create an overlay. The marks are sent to the application and displayed at the corresponding location in AR. In a working example, the tracked three-dimensional rectangle uses the WebView as a texture, and the marks made by the remote engineer are recreated in HyperText Markup Language (HTML) so that what both users see matches.

[0038] Depending on the desired implementation, the overlaid marks can be masked and displayed on top of the display surface to enhance the AR experience. Such an embodiment is useful when the device is touch-sensitive (either a digital touch screen or physical buttons) and obscures parts of the display surface during customer interaction.

[0039] In an embodiment, the application facilitates dynamic overlays, allowing a service engineer to create an overlay with multiple steps (e.g., "Now type into this text box, then click the OK button"). In that case, the service engineer clicks / tap on the text box, then navigates to the OK button and clicks / tap on it. The overlay is then sent to the customer as an animation of what to do, showing movement from the current customer position to the text box (e.g., an arc following the outline of the text box is highlighted), then another arc moving from the text box to the OK button. The steps can be numbered to make the order of actions to be performed more clear and to allow the customer to replay the steps performed (which would not be possible if multiple overlays and mouse positions were transferred in real time).

[0040] Unlike traditional screen sharing, dynamic overlays can be convenient for end users because users may not always be able to follow the entire sequence while looking at the display. A user may want to first see the sequence in AR and then perform the steps while facing the actual display. Additionally, some steps require multiple button presses that may not be easily conveyed using real-time overlays. With the dynamic overlays described herein, a service engineer can comfortably create a sequence of steps and send them to a remote customer after they are properly created. This asynchronous video collaboration in an otherwise synchronous video collaboration is similar to what users can do in a text-based chat system: create and edit a text message before committing and hitting "send."

[0041] In embodiments, different types of overlays can be used. For example, some actions require dragging a finger or mouse pointer over a path, while other actions simply involve moving the finger / mouse to another location. Some types of overlays can represent these differences, for example light vs. thick arrows. Depending on the desired implementation, the overlays can be augmented by text tooltips.

[0042] In an embodiment, detection of the current mouse / cursor location can also be made easier. Like holding a child's hand, the AR overlay can take into account the current finder / cursor location to show the user where to go next. For example, during a software installation process, it may not be clear where the user's cursor should be placed. Some UI elements require a click inside a text box first. If a service engineer specifies a click within an area, but the user cursor is outside the area, the application can automatically display an arc from the current user mouse location to the location of the text box to make it clear that the cursor needs to move there first.

[0043] In an embodiment, automatic overlays may also be facilitated. In an embodiment, instructions received during a live session may be recorded and later played back. For example, if the application detects that the object being videoed contains the same anchor image, it may automatically suggest playing a previously recorded overlay instead of repeatedly calling a service engineer. This functionality allows customers to operate the device themselves without the need for live communication with a service engineer.

[0044] In an embodiment, a check can also be made to determine whether an action was performed. In some scenarios, a button may need to be pressed for several seconds. When an engineer creates an overlay, they can specify a time period, rather than having to press an area for a required amount of time (e.g., 10 seconds). However, the user must hold that button down for the specified amount of time. In addition to displaying the duration in a tooltip, an embodiment can also facilitate counting how long the cursor / fingertip was held at a specified location.

[0045] FIG2 illustrates an example overlay on a device panel captured from a user device, according to an embodiment. As shown in FIG2, real-time rectangle detection is utilized to track the control panel captured by the user device. A framework according to an embodiment can be used to generate and apply three-dimensional overlay nodes, and texturing plane nodes can be used in any view. FIG3 illustrates an example three-dimensional overlay node with transitions, according to an embodiment.

[0046] For network communication, the user device can act as a web server and web socket server using appropriate libraries. Frames captured by the application are sent as images to the remote engineer, and the marks made are sent back to the application and recreated on the web view to be used as textures. For two-way audio communication, a WebRTC (Web Real-Time Communication) based solution can be used between the web browser and the application. Once the three-dimensional plane is fitted and then tracked by the AR framework, the frames are perspective corrected and sent to the remote engineer. Figure 4 shows a perspective corrected panel according to an embodiment. With perspective correction, the remote engineer sees a live camera view, adjusted with a crop of the display, captured by the end user of the application. The remote engineer can create any overlays.

[0047] Through the embodiments, an AR system can be provided that overlays an AR interface on a two-dimensional surface of a live scenario, in particular detecting occluding hands and mouse / fingers position to create an overlay that helps guide the user. FIG. 5 shows an example of an overlay where a hand and finger mask is implemented so that the overlay is placed under the hand or finger. Depending on the desired implementation, the hand and finger mask can also be implemented to track the hand and place the overlay under the hand or finger. Such a mask can be obtained via a segmentation network or using a hand tracking model that tracks the hand or finger in real time. Thus, if there is an object occluding the screen at 107 in FIG. 1, the annotation added at 109 can be hidden.

[0048] In another embodiment, the AR remote assistance system can also generate viewing instructions for the system. Accumulating a shared visual representation of the work environment can be useful in addressing many problems in the field. With step-by-step instructions from the expert, the user needs to complete the task, which can be difficult. During this period, the user must put down the device or ignore the visual input. Furthermore, the user may forget the exact details of how to perform a particular step, requiring the remote expert to repeat the instructions.

[0049] To address such issues, in an embodiment, the AR interface can be extended to support the function of asynchronous tutorial procedure creation in a live remote assistance system. FIG. 6 shows an example of recording and playback of an AR interface according to an embodiment. In an embodiment, instructions from a remote expert are automatically or manually saved as their own video clips during a live video call. Then, when the local user needs to complete the procedure, the saved video clips can be displayed in a separate video player to complete the task. While completing the task, the remote expert can see a live view of the recording that the local user is watching in a sub-window. The local user can switch to a live camera view at any time.

[0050] In an embodiment, video clip sequences are generated automatically whenever a remote expert is actively using a keyboard, mouse, or other peripheral device. A remote expert can also create sequences manually.

[0051] FIG. 7 shows a flow diagram of an annotation and recording process, according to an embodiment. At 700, a local user connects to a remote assistant. During the connection, multiple functions can be supported. The local user can share a stream with the remote assistant at 701. In such an embodiment, the local user streams content from a mobile device or a heads-up device to the remote user. In this case, when the remote expert starts providing annotations to the user stream, recording of a new clip automatically starts in the background while the live video session continues. The system records until the remote expert stops annotating the stream and times out. The remote expert can optionally pause the user video to add more expressive annotations depending on the desired implementation. As shown in FIG. 7, the remote assistant can watch the stream at 704 and add annotations as needed. During the stream, the remote assistant can pause annotations at 707. If annotations are paused for a threshold period (e.g., a few seconds), they time out at 709. At that point, the system saves the video clip as a procedure at 711.

[0052] In another example procedure, the remote assistant shares the stream with the local user at 702. Sometimes, for example, if the local user is trying to solve a problem with a software system, the remote expert may share his screen to show how to solve a particular problem using his software tools. In this case, the remote expert actively uses his mouse and keyboard to show a "procedure" that the system can record. Again, a timeout is used to determine the end of the procedure. The remote assistant begins interacting with the stream at 705 by providing annotations or controlling an on-screen interface or panel. The flow can continue with the saving of the video clip, as shown at 707 onwards. In another example, the remote expert can manually create a video clip by clicking a button in the interface. This can be useful if the remote user wants to create a clip using his video camera or load an external clip.

[0053] In another embodiment, the local user may put down the user device to perform a function indicated by the remote assistant at 703. The putting down of the user device can be detected based on the accelerometer, gyroscope, or through other hardware of the device according to a desired implementation. Even if the user is trying to hold the device, the background process system can detect slight anomalies in the accelerometer and gyroscope data and determine that the device is being held. However, when the user puts down the device, the accelerometer and gyroscope data becomes static and the background process can determine that the device is not in the user's hand. In this way, the system can automatically switch between displaying the recorded procedure (when the device is put down) and the live stream (when the device is in the user's hand). When the putting down of the device is detected at 706, the application switches to the procedure view. The procedure view is maintained until the local user picks up the device at 708. The application then switches back to the live view at 710.

[0054] These approaches can be combined to help a local user complete a difficult task. For example, when interacting with a complex interface, a remote expert can annotate the user's live stream and automatically create one clip. Then, while the user pauses to complete the task, the remote expert can annotate the same or similar interface in his or her own stream and automatically create another clip. Or, he or she can manually load another clip from a recorded stream of another user who has worked through the same problem.

[0055] Similarly, a local user can switch between live video streaming and reviewing clips using automatic or manual methods depending on the desired implementation.

[0056] By default, the system turns off the local user microphone when the local user is reviewing clips, and by default, the most recently recorded clips are shown first. Additionally, the user can use standard vertical swipe gestures to move between different media clips and horizontal swipe gestures to navigate within clips. In this way, the local user can seamlessly switch the device from a live streaming tool to a lightweight tutorial review tool.

[0057] If the user is streaming from a heads-up display, they can issue a verbal command to switch between the live streaming and review interface. On a mobile device, the user can issue a verbal command or press a button to switch interfaces.

[0058] Through the embodiments described herein, a remote assistance system can be facilitated that automatically records procedures during a live stream that can be viewed asynchronously by a user.

[0059] FIG. 8 illustrates an example of a computing device, according to an embodiment. The computing device may be in the form of a laptop, a personal computer, a mobile device, a tablet, or other device, depending on the desired implementation. The computing device 800 may include a camera 801, a microphone 802, a processor 803, a memory 804, a display 805, an interface (I / F) 806, and an orientation sensor 807. The camera 801 may include any type of camera configured to record any type of video, depending on the desired implementation. The microphone 802 may include any type of microphone configured to record any type of audio, depending on the desired implementation. The display 805 may include a touch screen display configured to receive touch input to facilitate instructions to perform the functions described herein, or a regular display, such as a liquid crystal display (LCD) or any other display, depending on the desired implementation. The I / F 806 may include a network interface to facilitate connection of the computing device 800 to external elements, such as servers and any other devices, depending on the desired implementation. The processor 803 can be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units, depending on the desired implementation. The orientation sensor 807 can include any form of gyroscope and / or accelerometer configured to measure any type of orientation measurement, such as tilt angle, orientation relative to x, y, z, axis, acceleration (e.g. gravity), depending on the desired implementation. The orientation sensor measurements can also include gravity vector measurements to indicate the gravity vector of the device, depending on the desired implementation. The computing device 800 can be utilized as a device for a local user or as a device for a remote assistant, depending on the desired implementation.

[0060] In an embodiment, the processor 803 is configured to continuously display a video received from a first device (e.g., a local user device) in a fixed manner for display on a second device (e.g., a remote assistant device), and in response to an input made to a portion of the video that is fixedly displayed on the second device, generate an augmented reality (AR) overlay on the display of the first device that corresponds to the portion of the video that is fixedly displayed, for example as shown in Figures 3-5.

[0061] Depending on the desired implementation, the processor 803 may be configured to stationary and fixed-point the video received from the first device for display on the second device by identifying one or more anchor images in the video, determining a target object for the two-dimensional display based on the identified one or more anchor images, and performing perspective correction of the video based on the target object being a two-dimensional surface for display on the second device as described in FIG. 1. As described herein, the target object may include a two-dimensional panel surface such as a panel display (e.g., displayed on an MFD), a keypad, a touch screen, a display screen (e.g., on a computer, mobile device, or other device), and other physical or displayed interfaces depending on the desired implementation. The anchor image may include a button, dial, icon, or other object that is presumed to be on a panel surface depending on the desired implementation.

[0062] Depending on the desired implementation, the processor 803 may be configured to crop the video to the object of interest for display on the second device, as shown in Figure 4. In this manner, the video may be cropped such that only the display screen, panel display, or other object of interest is provided to the second device.

[0063] Depending on the desired implementation, the processor 803 is configured to identify one or more anchor images in the video by searching a database for one or more anchor images that match one or more images in the video, as described in 101 of FIG. 1. The database can be stored and accessed remotely in a storage system, a server, or in other ways depending on the desired implementation. In an embodiment, the AR overlay can also be stored in the database for retrieval and future playback on the first device.

[0064] The processor 803 is configured to identify one or more anchor images in the video by detecting a rectangle on the video received from the first device, matching a three-dimensional plane to the two-dimensional points of the detected rectangle, tracking the three-dimensional plane that matches the two-dimensional points of the detected rectangle, and receiving a selection of one or more anchor images in the video via the second device as described in FIG. 1. Since most panels and interfaces displayed tend to be in the form of a rectangle or square, in the embodiment, a rectangle or square detector known in the art is utilized, but the detector can be modified according to a desired implementation. For example, in an embodiment including a circular interface, a circular surface detector can be utilized instead. Furthermore, after the rectangle or square is detected, the three-dimensional plane can be mapped (e.g., mapping to the corners of the rectangle) to the two-dimensional points of the detected rectangle / rectangle, which can then be tracked according to a desired implementation known in the art. Once the panel is detected, the user can select an anchor image (e.g., a button, dial, etc. on the panel) that can be incorporated into the AR system in real time.

[0065] As shown in FIGS. 1 to 6, in order to facilitate real-time interaction between the remote assistant and the local user, an AR overlay corresponding to a portion of the fixedly displayed video may be live on the display of the first device in response to input made to the portion of the fixedly displayed video of the second device. In another embodiment, the generation of the AR overlay may be delayed and deployed asynchronously depending on the desired implementation of the remote assistant. The remote assistant may confirm the AR overlay on his / her device and then provide an instruction (such as touching a confirmation button) to the device to transmit the AR overlay to the local user device for display. In this manner, the remote assistant may create AR annotations or provide other AR overlays to preview before deploying them to the local user. In an embodiment, the input made to the portion of the fixedly displayed video may include free-form annotations. Furthermore, if the AR overlay includes selecting a particular panel button or moving a cursor to click a particular portion, the AR overlay may include an indication for moving the cursor of the first device from a first location to a second location. Such an indication may be implemented in any manner (e.g., by an arrow following a path, a line, etc.) depending on the desired implementation.

[0066] As shown in Figure 5, the processor 803 can be configured to track one or more hands and fingers in the video and occlude portions of the AR overlay that overlap the one or more hands and fingers on the display of the first device. The tracking of the hands and / or fingers can be performed through any desired implementation. Such an implementation allows the AR overlay to be displayed in a realistic manner on the device of the remote user.

[0067] As shown in FIG. 7, the processor 803 can be configured to pause the video on the display of the second device to cause the video received from the first device to be pinned for display on the second device. In such an embodiment, depending on the desired implementation, the remote assistant can pause the video stream to make annotations or provide other AR overlays. Additionally, the processor 803 can be configured to generate an AR overlay corresponding to the pinned portion of the video on the display of the first device by generating a video clip including annotations and providing the video clip on the display of the first device, as shown at 707, 709, 711 in FIG. 7, if the pausing of the input exceeds a timeout threshold. The timeout can be configured depending on the desired implementation.

[0068] The processor 803 may be configured to play a video clip on a display of the first device when the first device is determined to be placed, and to provide the video for display on the second device when the first device is determined to be in the user's hand, as shown at 703, 706, 708, and 710 in FIG. 7 .

[0069] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a sequence of defined steps leading to a desired end state or result. In the illustrative examples, the steps performed require specific quantities of physical manipulations to achieve a specific result.

[0070] Unless otherwise indicated, and as will be apparent from the preceding description, descriptions using terms such as "processing," "computing," "calculating," "determining," "displaying," and the like throughout the specification may include operations and processing of a computer system or other information processing device that manipulates and transforms data represented as physical (electronic) quantities in a register or memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system or other information storage, transmission, or display device.

[0071] The embodiments may also relate to an apparatus for carrying out the operations of the present specification. The apparatus may be specially constructed for the required purposes, or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. A computer-readable storage medium may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memory, random access memory, solid-state devices and drives, or other suitable tangible or non-transitory media for storing electronic information. A computer-readable signal medium may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation that includes instructions for performing the operations of a desired implementation.

[0072] Various general-purpose systems may be used with the programs and modules according to the embodiments herein, or it may be convenient to construct more specialized apparatus to perform the desired method steps. Moreover, the embodiments are not described with reference to a particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the embodiments described herein. Instructions in the programming language may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0073] As known in the art, the operations described above can be performed by hardware, software, or any combination of software and hardware. Various aspects of the embodiments can be implemented using circuits and logic devices (hardware), while other aspects can be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, cause the processor to perform a method for implementing the present application. Furthermore, some embodiments of the present application may be performed exclusively in hardware, while other embodiments may be performed exclusively in software. Furthermore, the various functions described may be performed in a single unit or distributed among multiple components in any of a variety of ways. When performed by software, the method may be executed by a processor, such as a general purpose computer, based on instructions stored on a computer-readable medium. If necessary, the instructions may be stored on the medium in compressed and / or encrypted form.

[0074] Other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and components of the described embodiments may be used alone or in any combination. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the present application being indicated by the following claims.

Claims

1. pausing, on the second device, the video received from the first device for display on the second device; accepting input, including annotations, for the video paused on the second device; generating, on the display of the first device, an augmented reality (AR) overlay corresponding to the portion of the paused video in response to an input made to the second device during the portion of the paused video if a duration of the pause of the input exceeds a threshold; The method includes:

2. 2. The method of claim 1 , wherein generating the AR overlay on the display of the first device corresponding to the portion of the paused video is performed live in response to input made to the portion of the paused video on the second device.

3. generating, on the display of the first device, the AR overlay corresponding to a portion of the paused video; generating the AR overlay on the display of the second device in response to the input; if an instruction to provide the AR overlay to the first device is received, transmitting the AR overlay to the first device for display; The method of claim 1 , comprising:

4. Tracking one or more hands and fingers in the video; occluding a portion of the AR overlay that overlaps the one or more hands and fingers on the display of the first device; The method of claim 1 further comprising:

5. generating an augmented reality (AR) overlay on the display of the first device corresponding to the portion of the paused video; generating a video clip including the annotation if a pause time of the input exceeds the threshold; presenting said video clip on said display of said first device; The method of claim 1 , comprising:

6. if it is determined that the first device is located, playing the video clip on the display of the first device; providing the video for display on the second device if it is determined that the first device is in the user's hand; The method of claim 5 further comprising:

7. The method of claim 1 , wherein the AR overlay includes instructions for moving a cursor of the first device from a first location to a second location.

8. The method of claim 1 , further comprising: storing the AR overlay for playback by the first device.

Citation Information

Patent Citations

  • Electronic still camera

    JP2001136418A

  • Collaborative dynamic video commenting device and method

    JP2002507027A

  • Remote support method and system, and program

    JP2011034315A

  • Video instruction display method, system, terminal, and program for synchronously superposing instruction image on imaged moving image

    JP2015179947A

  • Display system, display device, terminal device, and program

    JP2019079314A