Systems and methods for augmented reality applications

Abstract

Systems and methods for augmented reality applications. Example implementations herein describe systems and methods for providing a platform that facilitates augmented reality (AR) overlays, which can involve: stabilizing a video received from a first device for display on a second device; and, in response to input to a portion of the stabilized video at the second device, generating an AR overlay on a display of the first device that corresponds to the portion of the stabilized video.

Description

Technical Field

[0001] The present disclosure relates generally to augmented reality (AR) systems, and more particularly to systems for generating usable control panels and screen interfaces utilizing AR. Background Art

[0002] In the implementation of related technology, there are AR applications that provide interfaces so that users can operate the dashboard or stereo system in the vehicle. In other applications, Internet browsing sessions can use AR to add overlays to web pages that assist people in navigating the Internet. Summary of the invention

[0003] Example implementations described herein relate to AR implementations that allow for a rectified view of a streaming display (e.g., a computer screen, a touch liquid crystal display (LCD), a digital control panel, a control panel of an appliance) and an immediate or automatic agent to add overlays to guide people in processing (e.g., which button to click or tap on the screen, where to enter text, etc.). Example implementations relate to registration: detecting the boundaries of an object of interest so that the AR overlay is appropriately displayed on the screen even when the user moves the camera. In another example implementation, the creation of markers is based on the content below and can be automatically removed when an action is taken. In another example implementation, occlusions are automatically detected to display the instruction overlay below in a realistic manner. Finally, the automatic process allows the ingestion of existing video footage to extract an anchor image for initializing the registration step (e.g., a guide video of an LCD display on an appliance such as a multi-function device (MFD)).

[0004] In an example implementation, the AR interface can be extended to instant remote assistance tasks, where a remote expert connects with a person sharing a live stream from a mobile device or head-mounted device to diagnose and fix problems on the spot. As live streaming services become more popular, instant remote assistance has become a way for users to solve increasingly difficult problems. Tools in the related art focus on allowing remote users to annotate or demonstrate solutions, but do not take into account the time and effort users need to spend in following their instructions. Users will often need to repeat the instructions before they can fully understand them, and in some cases they will need to use both hands to manipulate the physical device, making it difficult to simultaneously stream live video from a mobile device. To address these issues, the example implementation facilitates an AR-based tool for a remote assistance interface that can automatically record steps during a live stream that the user can watch asynchronously.

[0005] Aspects of the present disclosure relate to a method that may include: stabilizing a video received from a first device for display on a second device; and generating an augmented reality (AR) overlay corresponding to a portion of the stabilized video on a display of the first device in response to input made to the portion of the stabilized video at the second device.

[0006] Aspects of the present disclosure relate to a non-transitory computer-readable medium storing instructions for performing processing, the instructions including: stabilizing a video received from a first device for display on a second device; and generating an augmented reality (AR) overlay corresponding to a portion of the stabilized video on a display of the first device in response to input made to a portion of the stabilized video at the second device.

[0007] Aspects of the present disclosure relate to a system comprising: a device for stabilizing a video received from a first device for display on a second device; and a device for generating an augmented reality (AR) overlay corresponding to a portion of the stabilized video on a display of the first device in response to input of the portion of the stabilized video at the second device. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An example process for overlaying an AR interface and / or annotations on a screen or control panel according to an example implementation is illustrated.

[0009] Figure 2 An example overlay on a device panel captured from a user device according to an example implementation is illustrated.

[0010] Figure 3 An example of a 3D overlay node with transitions according to an example implementation is illustrated.

[0011] Figure 4 A perspective-corrected panel according to an example implementation is illustrated.

[0012] Figure 5 An example of an overlay is illustrated in which hand and finger masks are implemented so that the overlay is placed under the hand or fingers.

[0013] Figure 6 Examples of recording and playback of AR interfaces according to example implementations are illustrated.

[0014] Figure 7 A flow chart for annotating and recording a process according to an example implementation is illustrated.

[0015] Figure 8 An example of a computing device according to an example implementation is illustrated. DETAILED DESCRIPTION

[0016] The following detailed description provides details of the drawings and example implementations of the present application. For clarity, the figure marks and descriptions of redundant elements between the drawings are omitted. The terms used throughout the specification are provided as examples only and are not intended to be limiting. For example, the use of the term "automatic" may involve a fully automatic implementation or a semi-automatic implementation involving control of some aspects of the implementation by a user or administrator, depending on the implementation desired by a person of ordinary skill in the art who practices the implementation of the present application. The selection can be made by the user through a user interface or other input means, or can be implemented by a desired algorithm. The example implementations described herein can be used individually or in combination, and the functions of the example implementations can be implemented by any means according to the desired implementation.

[0017] Remotely helping customers troubleshoot advanced appliances such as MFDs can involve multiple challenges. For example, voice-only interaction can be error-prone, while dispatching a service engineer to a customer site can be lengthy and costly.

[0018] To address this situation, many appliance manufacturers have written instructional videos. If the video is not enough, the customer will still need on-site assistance from a service engineer. In an example implementation, there is an AR system that is configured to provide AR overlays on screens and control panels such as a customer's computer / smartphone screen when the customer is installing a drive for a new MFD, or on the LCD screen of the MFD when the operation involves touching buttons to configure the MFD. Specifically, the example implementation utilizes the surfaces of the screens and control panels, which are essentially 2D surfaces, to provide annotations and better overlays than related art implementations.

[0019] In the implementation of related art, the customer installs screen sharing software that allows the remote engineer to view the customer screen and control the customer screen or move a remote cursor to guide the customer. In addition, users often have to resort to video, that is, by taking an image of the LCD or control panel with a smartphone and letting the remote engineer see what the customer sees.

[0020] In such related art implementations, the problem with installing screen sharing software on a personal computer is that the customer is already seeking help to install other software, their company may not easily allow installation of new software, the computer may not be connected to the Internet, or there may not be a screen sharing application for mobile devices.

[0021] Additionally, with video streaming, remote engineers can become disoriented as users move their phones around, and communication can be greatly impaired by being limited to verbal instructions (e.g., “yes, click this red button in the bottom left corner, no, not this one, that one, then press all of them at the same time and hold for 3 seconds”).

[0022] To address such issues, example implementations facilitate an AR interface and overlay system involving control panels and screens (e.g., computer screens, touch screens, conventional digital control panels seen on MFDs, or appliances such as microwave ovens, car stereos, etc.) Simply by virtue of a mobile device utilizing the AR interface of the example implementations described herein, a user is able to point the mobile device camera at their screen / LCD / panel, allowing a remote engineer to interactively add overlay instructions to guide them.

[0023] Figure 1 An example process for overlaying an AR interface and / or annotations on a screen or control panel according to an example implementation is illustrated. The process begins when, at 100 , a local user connects to a remote assistance system through his user device.

[0024] In an example implementation, the system performs image tracking as a basis for detecting or tracking a screen or control panel. At 101, the system searches a database for an anchor image that matches the streamed content. Depending on the desired implementation, the database of anchor images representing the objects to be detected can be populated automatically or manually. An anchor image is an image that has been processed to extract key points.

[0025] For static cases where the screen or LCD display is from a known device (e.g., an LCD panel of a known MFD), reference images are pre-populated in the application, or reference images are obtained from an online database and downloaded into the application. For example, for MFDs, there are sets of images showing LCD control panels of specific MFD devices, so that the application automatically detects and tracks these types of control panels whenever they appear in the camera's field of view. Similarly, sets of images can be made for popular standard laptop models. Therefore, if an anchor image is found within the application or can be obtained from an online database, such an anchor image is used at 103.

[0026] If no anchor image is found (102), the application also supports dynamic registration of never-seen objects or LCD displays, in which case a quadrilateral detector can be used with the AR plane detector. Specifically, when a service engineer or local user taps the screen, the application can be configured to execute a rectangle or quadrilateral detector on the current frame and test the projection of the four corners in 3D space for intersection with a known AR plane. A 3D plane matching the 2D points of the rectangle is then created and tracked in 3D space by the AR framework, and an anchor image is subsequently selected at 104.

[0027] Once these reference images are established, the video frames captured by the application are perspective corrected so that the remote engineer sees a stable version of the area, creating an augmented reality overlay (ARO) at 105. The remote assistant is then able to provide annotations on the stream at 106, from which the application system determines if there are objects obscuring the screen at 107. If so, the annotations are hidden at 109, otherwise the annotations are shown at 108.

[0028] Once the app detects and tracks the anchor, the remote engineer can create an overlay by tapping the screen. The marker is sent to the app and displayed in AR at the corresponding location. In the example implementation, the tracked 3D rectangle uses a WebView as a texture, and the marker created by the remote engineer is recreated in Hypertext Markup Language (HTML), allowing the content seen by the two users to match.

[0029] Depending on the desired implementation, to enhance the AR experience, the overlaid markers can be masked so as to appear above the display surface. Such an example implementation can be helpful when the device is a touch panel (digital touch screen or physical buttons) where the customer may confuse some parts of the display surface during interaction.

[0030] In an example implementation, the application can facilitate dynamic overlays, which allows the service engineer to create an overlay that contains more than one step (e.g., "Fill in this text box here, then click the OK button"). In this case, the service engineer clicks / taps on the text box, then moves to the "OK" button and clicks / taps on it. Only then is the overlay sent to the customer as an animation of what is to be performed, showing the movement from the current customer position to the text box (e.g., an arc, then highlighting the outline of the text box), and then another arc jumping from the text box to the OK button. The steps can be numbered to make it more obvious to follow the sequence of actions and let the customer replay the steps to be taken (which would not be possible if the overlay and mouse position were transmitted in real time).

[0031] Unlike traditional screen sharing, dynamic overlays can be useful to end users because the user may not be able to follow the entire sequence while always watching the display. The user may want to see the sequence in AR first, then focus on the actual display and perform the steps. In addition, some steps may require pressing and holding several buttons, which may not be easy to convey again using a real-time overlay. With the dynamic overlay described in this article, a service engineer can easily create a series of steps and send them to a remote customer after they are correctly created. This asynchrony in an otherwise synchronous video collaboration is similar to what users can do in a text-based chat system: compose and edit a text message without submitting it before pressing "send".

[0032] In example implementations, various types of overlays can be utilized. For example, some actions require dragging a finger or mouse pointer on a path, while other actions simply represent moving the finger / mouse to another location. For example, several types of overlays can depict these differences, such as a light arrow versus a bold arrow. Depending on the desired implementation, the overlay can be enhanced with a text tooltip.

[0033] The example implementation can also facilitate current mouse / cursor position detection. Just like one would hold a child's hand to teach her, the AR overlay can take into account the current finder / cursor position and show the user where they need to go next. For example, during a software installation, it may not be clear where one's cursor needs to be: some UI elements require a click inside a text box first. If the service engineer defines a click inside an area, but the user cursor is outside of that area, the app can automatically display an arc from the current user mouse position to the text box position, making it clear that the cursor needs to move here first.

[0034] The example implementation can also facilitate automatic overlays. In the example implementation, the steps received during the live session can be recorded and played back later. For example, if the application detects that the object being videoed contains the same anchor image, it can automatically suggest playing back the previously recorded overlay instead of repeatedly calling the service engineer. This feature enables the customer to operate the device on their own without the need for live communication with the service engineer.

[0035] The example implementation can also check to determine if the action was performed. In some cases, it may be necessary to press the button for a few seconds. When the engineer creates the overlay, he does not need to press the required time (e.g., 10 seconds) on the area, but can specify the duration. However, the user must press the button and hold it for the specified time. In addition to displaying the duration in the tooltip, the example implementation can also facilitate counting how long the cursor / fingertip is assisted at the specified location.

[0036] Figure 2 An example overlay on a device panel captured from a user device according to an example implementation is illustrated. Figure 2 As shown, real-time quadrilateral detection is used to track the control panel captured by the user device. The framework can be used to generate and apply 3D overlay nodes according to the example implementation, and the textured plane node can be used with any visual Figure 1 Use together. Figure 3 An example of a 3D overlay node with transitions according to an example implementation is illustrated.

[0037] For network communication, the user device can act as a network server and Websocket server by using appropriate libraries. The frames captured by the application are sent to the remote engineer as images, and the created markers are sent back to the application and recreated in the web view used as textures. For two-way audio, a WebRTC-based solution can be utilized between the web browser and the application. Once the 3D plane is prepared and subsequently tracked by the AR framework, the frame is perspective corrected and sent to the remote engineer. Figure 4 A perspective-corrected panel according to an example implementation is illustrated. With perspective correction, the remote engineer sees a cropped and corrected live camera view of the display as captured by the end user in the application. The remote engineer is able to create arbitrary overlays.

[0038] By way of example implementation, there can be an AR system that specifically overlays an AR interface on a 2D surface in an immediate scene, and specifically occludes hands and detects the position of a mouse / fingers to create an overlay that helps guide the user. Figure 5 An example of an overlay is illustrated in which a hand and finger mask is implemented so that the overlay is placed under the hand or fingers. Depending on the desired implementation, a hand and finger mask can also be implemented to track the hand so that the overlay can be placed under the hand or fingers. Such a mask can be obtained by a segmented network or by using a hand tracking model that tracks the hand or fingers in real time. Therefore, if in Figure 1 If there is an object blocking the screen at 107 , the added annotation can be hidden at 109 .

[0039] In another example implementation, the AR remote assistance system can also generate viewing steps for the system. Visual representations shared in the work environment can help solve many problems on the spot. Incremental instructions from experts require users to complete tasks, sometimes arduous tasks. During this time, users must put down their devices or ignore their visual input. In addition, users may forget the exact details of how to perform a specific step, requiring remote experts to repeat their instructions.

[0040] To address this issue, the sample implementation is able to enhance the AR interface to facilitate the creation of asynchronous tutorial steps with real-time remote assistance. Figure 6 An example of recording and replaying an AR interface according to an example implementation is illustrated. In an example implementation, during an instant video call, instructions from a remote expert are automatically or manually saved as their own video clip. Then, when the local user needs to complete the step, they can view the saved video clip in a separate video player to complete the task. When they complete their task, the remote expert can see an instant view of the recording they were watching in a subwindow. The local user can switch back to the instant camera view at any time.

[0041] In an example implementation, video clip steps are automatically generated as long as the remote expert is actively using their keyboard, mouse, or other peripherals. The remote expert can also create steps manually.

[0042] Figure 7 A flow chart for annotating and recording processes according to an example implementation is illustrated. At 700, a local user connects to a remote assistant. During the connection, a variety of functions can be facilitated. At 701, the local user can share a stream with the remote assistant. In such an example implementation, the local user will stream content to the remote user from their mobile or heads-up device. In this case, once the remote expert begins annotating the user stream, a new clip recording automatically begins in the background while the live video session continues. The system records until the remote expert stops annotating the stream and a timeout is reached. The remote expert can optionally pause the user video to add more expressive annotations, depending on the desired implementation. Figure 7 As shown, at 704, the remote assistant can watch the stream and add annotations as needed. During the stream, at 707, the remote assistant can pause the annotation. If the annotation is paused for a threshold period of time (e.g., a few seconds), then at 709, a timeout is reached. At this point, at 711, the system saves the video clip as a step.

[0043] In another example implementation, at 702, the remote assistant shares the stream to the local user. Sometimes, for example, when a local user is trying to solve a problem with a software system, the remote expert may share their own screen to show how a specific problem can be solved using their own software tools. In this case, the remote expert actively uses their own mouse and keyboard to demonstrate the "steps" that the system can record, again using a timeout to determine the end of the step. At 705, the remote assistant begins to interact with the stream by providing annotations or controlling the interface or panel on their screen. The process can continue with saving video clips, as shown in 707 forward. In another example implementation, the remote expert can also click a button on their interface to manually generate a video clip. This is useful when the remote user wishes to create a clip using their own camera or load an external clip.

[0044] In another example implementation, at 703, the local user can put down the user device to perform the function indicated by the remote assistant. It is possible to detect the putting down of the user device based on the desired implementation based on the accelerometer, gyroscope, or other hardware of the device. Even if the user is still trying to hold the device, the background processing system can detect slight irregularities in the accelerometer and gyroscope data to determine that the device is being held. However, once the user puts down the device, the accelerometer and gyroscope data become static, causing the background processing to determine that the device is no longer in the user's hand. In this way, the system can automatically switch between a view showing recorded steps (when the device is put down) and an instant stream (when the device is in the user's hand). At 706, once it is detected that the device is put down, the application switches to the step view. Keep the step view until the local user picks up the device at 708. Then, the application switches back to the instant view at 710.

[0045] These approaches can be combined to help local users complete challenging tasks. For example, a remote expert can annotate a user's live stream while they interact with a complex interface, automatically creating a clip. Then, when the user pauses to complete the task, the remote expert can annotate the same or similar interface on their own stream, automatically creating another clip. They might also manually load another clip from a stream recorded by another user working on the same problem.

[0046] Similarly, the local user can switch between the live video stream and the playback clip using either automatic or manual methods, depending on the desired implementation.

[0047] The system defaults to lowering the local user's microphone when they are reviewing a clip. It also defaults to showing the most recently recorded clip first. In addition, the user can use the standard vertical swipe interaction to move between different media clips, and the horizontal swipe interaction to navigate within a clip. In this way, the local user can seamlessly switch their device from a live stream to a lightweight tutorial review tool.

[0048] If the user is streaming from a heads-up display, they can issue verbal commands to switch between the live stream and the playback interface. On a mobile device, the user can switch interfaces using verbal commands or button presses.

[0049] Through example implementations described herein, a remote assistance system can be facilitated that automatically records steps during a live stream that a user can asynchronously watch.

[0050] Figure 8An example of a computing device according to an example implementation is illustrated. According to the desired implementation, the computing device can be in the form of a laptop, a personal computer, a mobile device, a tablet or other device. The computing device 800 can include a camera 801, a microphone 802, a processor 803, a memory 804, a display 805, an interface (I / F) 806 and an orientation sensor 807. According to the desired implementation, the camera 801 can include any type of camera configured to record any form of video. According to the desired implementation, the microphone 802 can relate to any form of microphone configured to record any form of audio. According to the desired implementation, the display 805 can relate to a touch screen display configured to receive touch input for the execution of instructions for the functions described herein, or a conventional display such as a liquid crystal display (LCD) or any other display. According to the desired implementation, the I / F 806 can include a network interface that facilitates connecting the computing device 800 to external elements such as servers and any other devices. According to the desired implementation, the processor 803 can be in the form of a hardware processor such as a central processing unit (CPU), or a combination of hardware and software units. Depending on the desired implementation, the orientation sensor 807 can relate to any form of gyroscope and / or accelerometer configured to measure any type of orientation measurement such as tilt angle, orientation relative to x, y, z, orientation, acceleration (e.g., gravity), etc. Depending on the desired implementation, the orientation sensor measurements can also relate to gravity vector measurements to indicate the gravity vector of the device. Depending on the desired implementation, the computing device 800 can function as a local user's device or a remote assistant's device.

[0051] In an example implementation, the processor 803 is configured to stabilize a video received from a first device (e.g., a local user device) for display on a second device (e.g., a remote assistant device); and for input on a portion of the stabilized video on the second device, such as Figure 3-Figure 5 As shown, an augmented reality (AR) overlay corresponding to the portion of the stabilized video is generated on the display of the first device.

[0052] Depending on the desired implementation, the processor 803 can be configured to stabilize a video received from a first device for display on a second device by identifying one or more anchor images in the video; determining an object of interest of a 2D surface based on the identified one or more anchor images; and Figure 1As described in , perspective correction is performed on a video based on an object of interest on a 2D surface for display on a second device. As described herein, the object of interest can include a 2D panel surface, such as a panel display (e.g., as displayed on an MFD), a keypad, a touch screen, a display screen (e.g., on a computer, mobile device, or other device), and other physical or display interfaces, depending on the desired implementation. The anchor image can include a button, a dial, an icon, or other object expected to be located on the panel surface, depending on the desired implementation.

[0053] Depending on the desired implementation, the processor 803 can be configured to crop the video to the object of interest for display on the second device, such as Figure 4 In this way, the video can be cropped so that only the display screen, panel display or other object of interest is provided to the second device.

[0054] Depending on the desired implementation, the processor 803 is configured to identify one or more anchor images in the video by searching a database for one or more anchor images that match one or more images in the video, such as in Figure 1 As described at 101 of . Depending on the desired implementation, the database can be stored and accessed remotely in a storage system, a server, etc. In an example implementation, the AR overlay can also be stored in the database for retrieval and future playback by the first device.

[0055] The processor 803 is configured to identify one or more anchor images in the video by detecting a quadrilateral on the video received from the first device; matching the 3D plane to the 2D points of the detected quadrilateral; tracking the 3D plane matched to the 2D points of the detected quadrilateral; and Figure 1 As described in , a selection of one or more anchor images in a video is received by a second device. In an example implementation, because most panels and displayed interfaces tend to be in the form of quadrilaterals or rectangles, quadrilateral or rectangle detectors known in the art are utilized, but the detector can be modified according to the desired implementation. For example, in an example implementation involving a circular interface, a circular surface detector can be utilized instead. In addition, after the quadrilateral or rectangle is detected, the 3D plane is mapped to the 2D points of the detected quadrilateral / rectangle (e.g., mapped to the corners of the rectangle) so that it can be tracked according to any implementation known in the art. Once the panel is detected, the user can select an anchor image (e.g., a panel button, a dial, etc.) that can be incorporated into the AR system in real time.

[0056] like Figures 1 to 6As shown in , it is possible to respond to an input made to a portion of a stabilized video at a second device, and to instantly perform an AR overlay corresponding to the portion of the stabilized video on the display of the first device, so as to facilitate real-time interaction between the remote assistant and the local user. In another example implementation, the generation of the AR overlay can be deployed delayed or asynchronously according to the desired implementation of the remote assistant. The remote assistant can confirm the AR overlay on their own device, and then provide instructions to their device (e.g., touching a button for confirmation) to send the AR overlay to the local user device for display. In this way, the remote assistant can make AR annotations or provide other AR overlays, and preview them before deploying them to the local user. In an example implementation, the input made to a portion of the stabilized video can include free-form annotations. In addition, if the AR overlay involves selecting a specific panel button or moving a cursor to click on a specific portion, the AR overlay can involve an indication for moving the cursor of the first device from a first position to a second position. Such an indication can be implemented in any manner according to the desired implementation (e.g., via an arrow that tracks a path, via a line, etc.).

[0057] like Figure 5 As shown, the processor 803 can be configured to track one or more of the hands and fingers in the video; and block the portion of the AR overlay that overlaps with one or more of the hands and fingers on the display of the first device. Hand and / or finger tracking can be achieved through any desired implementation. Through such an example implementation, the AR overlay can be presented on the remote user's device in a realistic manner.

[0058] like Figure 7 As shown, the processor 803 can be configured to stabilize the video received from the first device for display on the second device by pausing the video on the display on the second device. In such an example implementation, the remote assistant can pause the video stream to annotate or provide other AR overlays according to the desired implementation. In addition, the processor 803 can be configured to: Figure 7 As shown in 707, 709 and 711 of the present invention, for a pause in the input that exceeds a timeout threshold, a video clip with annotations is generated; and the video clip is provided to a display of the first device to generate an AR overlay corresponding to a portion of the stabilized video on the display of the first device. The timeout can be set according to the desired implementation.

[0059] The processor 803 can be configured to, for a first device determined to be put down, replay the video clip on the display of the first device; and for a first device determined to be in the user's hand, provide the video to the second device for display, such as Figure 7 As shown in 703, 706, 708 and 710.

[0060] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are a means used by those skilled in the art of data processing to convey the essence of their innovations to others skilled in the art. An algorithm is a defined series of steps to achieve a desired end state or result. In an example implementation, performing the steps requires physical manipulation of tangible quantities to achieve tangible results.

[0061] Unless otherwise specifically stated, it will be apparent from the discussion that it should be understood that throughout the specification, discussions utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," etc., may include actions and processes of a computer system or other information processing device that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly presented as physical quantities within the computer system's memories or registers or other information storage, transmission, or display devices.

[0062] Exemplary implementations may also relate to devices for performing the operations herein. The device may be specially constructed for the desired purpose, or it may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such a computer program may be stored in a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. A computer-readable storage medium may include tangible media, such as, but not limited to, optical disks, disks, read-only memories, random access memories, solid-state devices and drives, or any other type of tangible or non-temporary media suitable for storing electronic information. A computer-readable signal medium may include a medium such as a carrier wave. The algorithms and displays presented herein are not inherently related to any particular computer or other device. A computer program may include a pure software implementation, which includes instructions for performing the operations desired to be implemented.

[0063] According to the examples herein, various general purpose systems can be used with programs and modules, or it may prove convenient to construct more specialized equipment to perform the desired method steps. In addition, the exemplary implementations are not described with reference to any particular programming language. It will be understood that the teachings of the exemplary implementations as described herein can be implemented using various programming languages. The instructions of the programming language can be executed by one or more processing devices (e.g., a central processing unit (CPU), a processor, or a controller).

[0064] As known in the art, the above operations can be performed by some combination of hardware, software or software and hardware. Circuits and logic devices (hardware) can be used to implement various aspects of exemplary implementations, while other aspects can be implemented using instructions (software) stored on machine-readable media, which will cause the processor to perform the method for implementing the implementation of the present application if the instructions are executed by a processor. In addition, some exemplary implementations of the present application can be performed only in hardware, while other exemplary implementations can be performed only in software. In addition, the various functions described can be performed in a single unit, or can be distributed on multiple components in any number of ways. When executed by software, the method can be performed by a processor such as a general-purpose computer based on instructions stored on a computer-readable medium. If necessary, instructions can be stored on the medium in a compressed and / or encrypted format.

[0065] In addition, other implementations of the present application will be apparent to those skilled in the art in view of the specification and the teachings of practicing the present application. The various aspects and / or components of the described example implementations may be used alone or in any combination. It is intended that the specification and example implementations be considered as examples only, and the true scope and spirit of the present application are indicated by the appended claims.

Claims

1. A method for augmented reality application, the method comprising the following steps: stabilizing video received from a first device for display on a second device; as well as generating, in response to input of a portion of the stabilized video at the second device, an augmented reality (AR) overlay corresponding to the portion of the stabilized video on a display of the first device, The step of generating an augmented reality (AR) overlay corresponding to the portion of the stabilized video on a display of the first device comprises the following steps: In response to the input, generating the augmented reality (AR) overlay on a display of the second device; In response to receiving an instruction to provide the augmented reality (AR) overlay to the first device, sending the augmented reality (AR) overlay to the first device for display; For pauses in the input that exceed a timeout threshold, generating a video clip including annotations; and The video clip is provided on a display of the first device.

2. The method according to claim 1, wherein: The step of stabilizing the video received from the first device for display on the second device comprises the following steps: identifying one or more anchor images in the video; determining an object of interest of the 2D surface based on the identified one or more anchor images; and Based on the object of interest on the 2D surface, perspective correction is performed on the video for display on the second device.

3. The method according to claim 2, wherein: The step of identifying one or more anchor images in the video comprises the step of searching a database for one or more anchor images that match the one or more images in the video.

4. The method according to claim 2, wherein: The step of identifying one or more anchor images in the video comprises the following steps: detecting a quadrilateral on the video received from the first device; Match the 3D plane to the 2D points of the detected quadrilateral; tracking the 3D plane matched to the 2D points of the detected quadrilateral; and A selection of the one or more anchor images is received in the video by the second device. 5 . The method according to claim 2 , further comprising the step of: cropping the video to the object of interest for display on the second device.

6. The method according to claim 2, wherein: The object of interest is a display screen.

7. The method according to claim 1, wherein: In response to input of the portion of the stabilized video at the second device, generating an augmented reality (AR) overlay corresponding to the portion of the stabilized video on a display of the first device is performed on the fly.

8. The method according to claim 1, further comprising the steps of: tracking one or more of a hand and a finger in the video; as well as A portion of the augmented reality (AR) overlay that overlaps one or more of the hand and fingers is obscured on a display of the first device.

9. The method according to claim 1, wherein: The step of stabilizing the video received from the first device for display on the second device comprises the steps of pausing the video on a display on the second device, and Wherein, the input includes comments.

10. The method according to claim 1, further comprising the steps of: For the first device determined to be put down, replaying the video clip on a display of the first device; as well as For the first device determined to be in the user's hand, the video is provided to the second device for display.

11. The method according to claim 1, wherein: The augmented reality (AR) overlay includes instructions for moving a cursor of the first device from a first position to a second position.

12. The method according to claim 1, further comprising the steps of: The augmented reality (AR) overlay is stored for playback by the first device.

Images