A Program for a Web-Based Remote Assistance System with Context- and Content-Aware 3D Hand Gesture Visualization

Abstract

To attain more intuitive and natural display on a mobile device in superimposing the expression of a user hand on an image on the mobile device.SOLUTION: Example implementations described in a description are directed to the transmission of hand information from a user hand or other object to a remote device via browser-to-browser connections, such that the hand or other object is oriented correctly on the remote device based on orientation measurement values received from the remote device. Such example implementations can facilitate remote assistance in which the user of the remote device needs to view the hand or object movement as provided by an expert for guidance.SELECTED DRAWING: Figure 5a

Description

[Technical field]

[0001] The present disclosure relates to a program for a real-time video-based gesture assistance system. [Background technology]

[0002] Current related technology video conferencing systems allow people to continue to collaborate even when they are in different locations, but in remote assistance scenarios where a remote expert assists a local customer in performing a complex physical task with guidance conveyed using related technology video and audio communication media, there is a high risk that the customer will misinterpret the intent and instructions from the expert, resulting in inefficient collaboration experience and performance.

[0003] Non-verbal communication such as body language and hand gestures in the related art can significantly improve the performance of remote collaborative activities in remote assistance or collaborative scenarios. The effect of using visual cues of hand gestures in remote assistance or collaborative scenarios has been tested in implementations of the related art. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Arcore. https: / / developers.google.com / ar / . Accessed: 2019-06-03. [Non-Patent Document 2] Arkit. https: / / developer.apple.com / augmented-reality / . Accessed: 2019-06-03. [Non-Patent Document 3] Colmap. https: / / colmap.github.io / . Accessed: 2019-06-03. [Non-Patent Document 4] OpenCV. https: / / opencv.org / . Accessed: 2019-06-03. [Non-Patent Document 5] Scope AR. https: / / www.scopear.com. Accessed: 2019-06-03. [Non-Patent Document 6] Teamviewer pilot. https: / / www.teamviewer.com / en-us / solutions / augmented-reality-remote-support / #gref. Accessed:2019-06-03 [Non-Patent Document 7] Upskill skylight. https: / / upskill.io / skylight / . Accessed: 2019-06-03. [Non-Patent Document 8] ViewAR. https: / / www.viewar.com / . Accessed: 2019-06-03. [Non-Patent Document 9] Vuzix remote assist. https: / / www.Vuzix.com / Vuzix-Remote. Accessed: 2019-06-03. [Non-Patent Document 10] ALEM, L. et al. "Remote Tele-assistance System for Maintenance Operators in Mines" 2011 Underground Coal Operators' Conference, February 10-11, 2011 (8 pages) [Non-Patent Document 11] AMORES, J. et al. "Show Me: A Remote Collaboration System that Supports Immersive Gestural Communication" CHI, 2015, April 18-23, 2015, Seoul, Republic of Korea, pp. 1343- 1348.

Non-Patent Document 12

Non-Patent Document 13

Non-Patent Document 14

Non-Patent Document 15

[0005] An object of the present invention is to display a user's hand expression on a mobile device in a more intuitive and natural way when the expression is superimposed on an image on the mobile device. [Means for solving the problem]

[0006] The exemplary implementations described herein take into account several context and content aspects that can be obtained by analyzing the view of the customer workspace. Such analysis can be used to modify the visualization parameters of the hand model (e.g., size, orientation, color, etc.), which can improve the collaboration experience and enhance task performance.

[0007] An exemplary implementation described herein includes a remote assistance system that facilitates remote experts to provide guidance using hand gestures to customers performing physical tasks at different locations. The system is built on a web-based real-time media communication framework that allows customers to use common smartphones to send live video feeds to experts. The experts can then see a view of the customer workspace and show their hand gestures on the footage displayed in real time. The expert's hand gestures are captured with a hand tracking device and visualized with a three-dimensional (3D) hand model that is rigged on the live video feed. The system is accessible through a web browser and does not require the installation of prerequisite software on the customer device (such as a common smartphone). The system enhances the collaboration experience by utilizing context and content information on the customer's environment for navigation guidance based on visualization aspects of the hand model (e.g., orientation, scale, and color) and audio cues. The system also supports capture and playback of the expert's hand gestures.

[0008] In an exemplary implementation described herein, there is a lightweight web-based system that supports remote guidance using hand gestures from a remote expert to a local customer. The system uses web-based real-time media communication technology (e.g., WebRTC) to enable a local customer to use a common smartphone device to share a view of the customer workspace with a remote expert helper via a video conferencing system. The hand gesture data from the expert can be transferred and visualized in real-time in a shared live video stream in a web browser. The system is accessible by end users through most modern web browsers by simply browsing a specific web page, so there is no need to install any prerequisite software on the user's smartphone device. The expert-side system utilizes a tracking device (e.g., a hand tracking device) to track the expert's hand movements. Furthermore, the exemplary implementation described herein is not limited to hand tracking and can be extended to other types of tracking (e.g., feet, tools) according to the desired implementation.

[0009] Aspects of the disclosure include a program for executing a process on a mobile device including a direction sensor and a camera, the instructions including transferring image from the camera and measurements from the direction sensor to another device, receiving hand information related to a user's hand from the other device, and overlaying a representation of the user's hand onto the image displayed on the mobile device based on the received hand information, where the representation of the user's hand is further overlaid onto the image displayed on the mobile device in an orientation determined by the measurements from the direction sensor.

[0010] Aspects of the disclosure include a method of performing a process on a mobile device including a direction sensor and a camera, the instructions including transferring image from the camera and measurements from the direction sensor to another device, receiving hand information related to a user's hand from the other device, and overlaying a representation of the user's hand onto an image displayed on the mobile device based on the received hand information, where the representation of the user's hand is further overlaid onto the image displayed on the mobile device in an orientation determined by the measurements from the direction sensor.

[0011] Aspects of the disclosure include a mobile device including a direction sensor, a camera, and a processor, the processor including instructions for transferring image from the camera and measurements from the direction sensor to another device, receiving hand information related to a user's hand from the other device, and overlaying a representation of the user's hand onto an image displayed on the mobile device based on the received hand information, where the representation of the user's hand is further overlaid onto the image displayed on the mobile device in an orientation determined by the measurements from the direction sensor.

[0012] Aspects of the present disclosure include a mobile device including a direction sensor means, a camera means, means for transferring an image from the camera and measurements from the direction sensor to another device, means for receiving hand information relating to a user's hand from the other device, and means for overlaying a representation of the user's hand onto an image displayed on the mobile device based on the received hand information, where the representation of the user's hand is further overlaid onto the image displayed on the mobile device in an orientation determined by the measurements from the direction sensor.

[0013] Aspects of the disclosure include a program for executing a process on a device communicatively coupled to a tracking device, the instructions including receiving video and orientation sensor measurements from a mobile device, transferring hand information associated with a user's hand generated based on measurements obtained from the tracking device, and overlaying a representation of the user's hand onto a video on the device based on the transferred hand information, where the representation of the user's hand is further overlaid onto the video on the device in an orientation determined by measurements from the orientation sensor.

[0014] Aspects of the disclosure include a method of a device communicatively coupled to a tracking device, the method including receiving video and orientation sensor measurements from a mobile device, transferring hand information associated with a user's hand generated based on measurements obtained from the tracking device, and overlaying a representation of the user's hand onto a video on the device based on the transferred hand information, where the representation of the user's hand is further overlaid onto the video on the device in an orientation determined by measurements from the orientation sensor.

[0015] Aspects of the disclosure include a device communicatively coupled to a tracking device, the device including a processor configured to execute instructions: receiving video and orientation sensor measurements from a mobile device; transferring hand information associated with a user's hand representation generated based on measurements obtained from the tracking device; and overlaying the user's hand representation onto a video on the device based on the transferred hand information, where the user's hand representation is further overlaid onto the video on the device in an orientation determined by measurements from the orientation sensor.

[0016] Aspects of the disclosure include a device communicatively coupled to a tracking device, the device including means for receiving video and orientation sensor measurements from a mobile device, means for transferring hand information associated with a user's hand generated based on measurements obtained from the tracking device, and means for overlaying a representation of the user's hand onto a video on the device based on the transferred hand information, the representation of the user's hand further overlaying the representation of the user's hand onto the video on the device in an orientation determined by measurements from the orientation sensor.

[0017] Aspects of the disclosure include a program storing server instructions, the instructions including receiving a first connection from the mobile device, receiving a second connection from another device communicatively coupled to the tracking device, and establishing a third connection between the mobile device and the other device to facilitate transfer of video and orientation sensor measurements from the mobile device to the other device and to facilitate transfer of hand information from the other device to the mobile device.

[0018] Aspects of the disclosure include a method including receiving a first connection from a mobile device, receiving a second connection from another device communicatively coupled to the tracking device, and establishing a third connection between the mobile device and the other device to facilitate transfer of video and orientation sensor measurements from the mobile device to the other device and to facilitate transfer of hand information from the other device to the mobile device.

[0019] Aspects of the disclosure include a server including means for receiving a first connection from the mobile device, means for receiving a second connection from another device communicatively coupled to the tracking device, and means for establishing a third connection between the mobile device and the other device to facilitate transfer of video and orientation sensor measurements from the mobile device to the other device and to facilitate transfer of hand information from the other device to the mobile device.

[0020] The video may be live video from the camera of the mobile device, the measurements may be live measurements from the orientation sensor, and the hand information may include hand skeleton joint measurements and hand visualization parameters determined from live movements of the user's hand.

[0021] Overlaying the representation of the user's hand onto the image displayed on the mobile device based on the hand information may include generating a 3D hand model of the user's hand as the representation based on the hand skeletal joint measurements, and adjusting one or more of a color and a size of the 3D model for overlaying onto the image based on hue information of the image and one or more of objects detected on the image.

[0022] The instructions may further include establishing a browser-to-browser connection via a web browser from the mobile device to another web browser of the other device, and transferring the video from the camera and the measurements from the orientation sensor to the other device, and receiving the hand information from the other device may occur via the browser-to-browser connection.

[0023] The instructions may further include transferring recorded audio from a microphone to the other device, and outputting the audio received from the other device.

[0024] The video may be previously recorded video and the measurements from the direction sensor may be previously recorded measurements.

[0025] Aspects of the present disclosure may include a program storing instructions for executing a process of a device communicatively coupled to a tracking device, the instructions causing a computer to execute a method including receiving video and orientation sensor measurements from a mobile device; transferring hand information associated with a user's hand from the other device, the hand information being generated based on measurements obtained from the tracking device; and overlaying a representation of the user's hand onto the video for display by the device based on the hand information, the representation of the user's hand being overlaid on the video in an orientation determined from the measurements from the orientation sensor.

[0026] The video may be live video from a mobile device, the directional sensor measurements may be live directional sensor measurements from the mobile device, and the hand information may include hand skeleton joint measurements and hand visualization parameters determined from live measurements of the user's hand from the tracking device.

[0027] Overlaying the representation of the user's hand onto the image for display by the device based on the hand information may include generating a 3D hand model of the user's hand as the representation based on the hand skeletal joint measurements, and adjusting one or more of a color and a size of the 3D model for overlaying onto the image based on hue information of the image and one or more of detected objects on the image.

[0028] The instructions may further include establishing a browser-to-browser connection via a web browser from the device to another web browser on the mobile device, and receiving the video and the orientation sensor measurements from the mobile device and transferring hand information from the device may occur via the browser-to-browser connection.

[0029] The instructions may further include transferring recorded audio from a microphone to the mobile device, and outputting the received audio from the mobile device.

[0030] The video may be previously recorded video and the directional sensor measurements may be previously recorded directional sensor measurements.

[0031] Aspects of the disclosure may include a program storing server instructions that cause a computer to execute a method including receiving a first connection from a mobile device, receiving a second connection from another device communicatively coupled to a tracking device, and establishing a third connection between the mobile device and the other device to facilitate transfer of video and orientation sensor measurements from the mobile device to the other device and to facilitate transfer of hand information from the other device to the mobile device.

[0032] The first connection and the second connection may be received via a web browser, and establishing the third connection may include establishing a direct browser-to-browser connection between the mobile device and the other device.

[0033] The video may be live video from the mobile device, the directional sensor measurements may be live directional sensor measurements from the mobile device, and the hand information may include hand skeleton joint measurements and hand visualization parameters determined from live measurements of the user's hand from the tracking device.

[0034] The video may be previously recorded video and the directional sensor measurements may be previously recorded directional sensor measurements. [Brief description of the drawings]

[0035] [Figure 1a] 1 illustrates an example system according to an example implementation. [Figure 1b] 1 illustrates an example system according to an example implementation. [Diagram 2] 1 illustrates a web-based real-time communication framework for a system according to an exemplary implementation. [Diagram 3] 1 illustrates an exemplary web page where a user can initiate a session, according to an exemplary implementation. [Figure 4a] 1 illustrates a dialogue box and corresponding alignment according to an exemplary implementation. [Figure 4b] 1 illustrates a dialogue box and corresponding alignment according to an exemplary implementation. [Figure 4c] 1 illustrates a dialogue box and corresponding alignment according to an exemplary implementation. [Figure 5a] 1 illustrates exemplary visualizations of pointing gestures and interaction spaces in various configurations according to an exemplary implementation. [Figure 5b] 1 illustrates exemplary visualizations of pointing gestures and interaction spaces in various configurations according to an exemplary implementation. [Figure 5c] 1 illustrates exemplary visualizations of pointing gestures and interaction spaces in various configurations according to an exemplary implementation. [Figure 6a] 1 illustrates adaptive coloring or shading, according to an example implementation. [Figure 6b] 1 illustrates adaptive coloring or shading, according to an example implementation. [Figure 7a] 13 illustrates an example of adjustment to finger size according to an exemplary implementation. [Figure 7b] 13 illustrates an example of adjustment to finger size according to an exemplary implementation. [Figure 8a] 4 shows examples of detected blobs with their corresponding bounding boxes highlighted, according to an exemplary implementation. [Figure 8b] 4 illustrates an example of detected blobs with their corresponding bounding boxes highlighted, according to an exemplary implementation. [Figure 9]13 illustrates an example of a replay function of an object to access previously captured operations, according to an exemplary implementation. [Figure 10] 1 illustrates an example of a mobile device, according to an exemplary implementation. [Figure 11] 1 illustrates an example of a system including a tracking device, according to an exemplary implementation. [Figure 12] 1 illustrates an exemplary computing environment including an exemplary computing device suitable for use with some exemplary implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The following detailed description of the embodiments of the present disclosure provides diagrams of the present disclosure and further details of exemplary implementations. Reference numbers and descriptions of redundant elements in the figures have been omitted for simplicity. Terms used throughout the specification are provided by way of example and are not intended to be limiting. For example, the use of the term "automatic" may include fully automatic or semi-automatic implementations including user or administrator control over certain aspects of the implementation, depending on the desired implementation of the skilled artisan implementing the present application. Selection may be made by a user via a user interface or other input means, or may be implemented via a desired algorithm. The exemplary implementations as described herein may be utilized alone or in combination, and the functionality of the exemplary implementations may be implemented through any means according to the desired implementation.

[0037] Described herein is a web-based remote assistance system that uses different aspects of context and content to enhance hand gesture visualization and other visualizations.

[0038] 1a and 1b show an example system according to an example implementation. Specifically, 1a shows the system setup at the expert side, and 1b shows the system setup at the customer side. In 1a, the remote expert watches a live video feed sent by the customer and moves his / her hand over the tracking device 100 to provide hand gesture-based guidance. In 1b, the customer uses a smartphone to share a view of the customer workspace and an object in the view (printer 102) with the remote expert and is guided by viewing the expert's hand gesture movements 103 on the smartphone screen.

[0039] The remote expert uses a tracking device 100 (e.g., configured to track hands, tools, etc.) placed on a desk and has a computer monitor 101 in front of the tracking device 100. A live video feed transmitted from the customer's camera (e.g., from the smartphone device shown in FIG. 1b) is displayed on the monitor 101, from which the expert can view the customer workspace and objects in the customer workspace (e.g., the printer 102 in FIG. 1b) in real time. By viewing the live video and performing an interaction via the tracking device 100, the expert provides hand or tool gesture-based guidance (e.g., guidance to check the document feed roller with a pointing gesture). In an exemplary implementation including hand gestures, the hand gestures are visualized on a rigged 3D hand model 103 with a semi-transparent effect on the video displayed simultaneously on both the expert device and the customer device. Viewing the expert's 3D hand model 103 on the smartphone screen as in FIG. 1b, the customer can be instructed on a physical task related to the object (e.g., check a message on the printer monitor, check the document feed roller, etc.).

[0040] FIG. 2 illustrates a web-based real-time communication framework of the system according to an exemplary implementation. As illustrated in FIG. 2, the system is built on the Web Real-Time Communication Framework (WebRTC). A WebRTC intermediate signaling server 200 establishes a peer-to-peer (P2P) connection between the customer and the expert for real-time media communication, and each end user can participate in this direct media communication session by browsing a web page hosted by the server using a compatible modern web browser known in the related art. Further description of the customer device is provided in FIG. 10, described below. Further description of the expert system is provided in FIG. 11, described below. As described herein and illustrated in FIG. 2, the P2P connection facilitates the exchange of information such as video (e.g., live video), device orientation, audio, hand skeletal joints, and hand model visualization parameters.

[0041] 3 shows an exemplary web page where a user can initiate a session according to an exemplary implementation. Data communication between the two peers and data access to the user devices (e.g., camera, microphone, tracking devices) are controlled by WebRTC and JavaScript® application programming interfaces (APIs) in the user web browser, so the exemplary implementation described herein does not require the installation of any prerequisite software or plugins on these devices or browsers.

[0042] To generate and display the 3D hand model in real time in the user's web browser, the exemplary implementation described herein utilizes the Three.js API based on WebGL. Three.js uses the JavaScript language as part of the website, so there is no need to install a web browser plug-in. Hand skeleton joint data obtained from the expert's tracking device is used to rig the 3D hand model in the Three.js scene. To synchronize the perspective view of the rendered 3D hand model and its pose between the client and the expert, the expert sets the hand skeleton joint data and hand model visualization parameters provided by the Three.js perspective camera parameters (e.g., field of view, view direction, position, etc.). The hand skeleton joint data and hand model visualization parameters set by the expert are then transferred to the client, as shown by the arrows showing the data flow between the client and the expert in FIG. 2. The transferred hand skeleton joint data and Three.js camera parameters are used to rig the client's hand model and display it in the same perspective view in the client's web browser. In an exemplary implementation, because the 3D hand model data is already embedded in the web page, the system does not need to send large amounts of 3D mesh data to the customer, and instead transfers only a small amount of hand skeletal joints and camera parameters to the customer.

[0043] The system of the present disclosure can also utilize context and content information of the customer environment and the target object to change visualization aspects of the hand model (such as color, size, or orientation). Such visualization and utilization of context and content is managed by a context and content aware visualization module on the expert side, which processes both the media data transferred from the customer device and the data captured from the expert device, and updates the visualization parameters of the hand model that are transferred to and applied to the hand model on the customer device.

[0044] Figures 4a-c show interaction boxes and corresponding alignments according to an example implementation: in particular, Figure 4a shows an example interaction box on a tracking device of the expert system, Figure 4b shows an example of aligning the sensor interaction space with the customer's camera view space, and Figure 4c shows an example of gravity-aware positioning of the interaction space.

[0045] In the exemplary implementation described herein, accurate positioning of the expert's virtual hand displayed relative to the customer's real environment and the target object therein is important for the expert to convey accurate guidance and information by performing the required physical action on the object. To facilitate such an exemplary implementation, we utilize appropriate positioning between the interaction space of the expert's tracking device (and the virtual hand therein) and the customer's camera view space. In the exemplary implementation described herein, we 1) snap the interaction space to the customer's real-world ground with a context-aware alignment of the interaction space based on the gravity direction of the customer's environment. This is an arrangement for manipulating the expert's virtual hand in a more intuitive and natural way, just as the expert would present physically in the customer's environment. 2) change the scale or color of the virtual hand with a content-aware alignment based on the visual appearance and objects of the customer scene. This is an arrangement for making it easier for the customer to follow the expert's instructions by viewing the expert's virtual hand movements with the correct scale and prominent color related to the object.

[0046] Aspects of the context considered in this disclosure are the orientation of the customer's camera view relative to the customer environment, and for aligning the expert's virtual hand displayed on the video according to the physical world of the customer environment. Such an exemplary implementation helps the expert to perform a more natural and intuitive interaction by the tracking device's sensors, and allows the expert to move his / her hand as if he / she were actually in the customer workspace. More precisely, the exemplary implementation uses the inertial sensor of the customer's smartphone to estimate the direction of gravity in the customer workspace, which is used to continuously change the interaction space orientation of the customer's camera view space (and the virtual 3D hand model therein) and stabilize its relative pose with respect to the ground of the customer workspace, allowing the customer to freely move the handheld camera.

[0047] Figure 4a shows a 3D box representation 402 on a tracking device 401 as an interaction space. The front of the box can be color encoded, color, or grayscale encoded (e.g., red or a red hue). Figure 4b shows an example of aligning the interaction space to a customer's camera without regard for gravity, where the interaction space is placed in front of the camera and its canonical axes are aligned to the axes of the camera.

[0048] Figure 4c shows gravity-aware alignment by aligning the vertical axis of the interaction space with the direction of gravity in the customer's camera space. With this gravity-aware alignment, the movements from the expert's real hand in the physical world match the movements of the virtual hand model in the customer's physical world. This helps the expert to easily and intuitively perform interactions on the tracking device and manipulate the virtual hand in the customer's camera view space.

[0049] Figures 5a-c show exemplary visualizations of pointing gestures and interaction spaces in various configurations according to an exemplary implementation. Specifically, Figures 5a and 5c show hand movements on a tracking device in a professional environment (top box), manipulating a virtual hand to point to the top edge of a cube in a customer workspace (bottom box). The top boxes in Figures 5a and 5c represent pointing gestures made on the tracking device. The bottom boxes in Figures 5a and 5c show a visualization of a 3D hand model and an interaction box representation superimposed on the customer's view.

[0050] Without gravity-aware alignment, the expert sees the motion of his or her virtual hand rendered on a live video feed, often moving his or her hand diagonally upwards, bringing the tip of the virtual hand's index finger towards the top edge of the cube (Figure 5a). With gravity-aware alignment, the expert can move their hand as if they were physically present in the client workspace and moving their hand parallel to the top edge of the cube (Figure 5c).

[0051] Another limitation of aligning the interaction space without awareness of gravity is that it often leads to inadvertent visualization of the hand in a pointing gesture, especially when the palm is nearly parallel to the sensor. For example, Fig. 5b shows that the tip of the index finger of the hand model is obscured by the palm and other fingers due to the viewpoint of the camera being directly behind the hand model. To address this, the expert must intentionally rotate the hand to show a full view of the index finger of the virtual hand, as in the top box of Fig. 5a. However, this also often leads the expert to bring the hand too close to the tracking device's sensor, resulting in fewer visual cues for the sensor, making tracking poses more difficult and reducing tracking performance. For example, the index finger may be obscured from the tracking device's view by the other fingers, as in the top box of Fig. 5a.

[0052] Exemplary implementations are further configured to adapt the appearance and size of the hand based on the content of the displayed object.

[0053] 6a and 6b illustrate adaptive coloring or contrast changes according to an exemplary implementation. Depending on the desired implementation, the contrast of the virtual hand against the background can be adjusted by adjusting the highlight color of some or all parts of the hand based on the color of the background / object. In the example of FIG. 6a and 6b, all parts of the hand are adjusted, but only parts (e.g., thumb and index finger) can also be adjusted according to the desired implementation. FIG. 6a and 6b illustrate the appearance of an adapted virtual hand superimposed on a plastic teeth model by changing the contrast, which makes the hand stand out more from the background. Alternatively, depending on the desired implementation, the highlight color can also be changed (e.g., from red to blue) according to the desired implementation. In such an exemplary implementation of selecting the highlight color of the hand, the average hue value in the camera image near the virtual hand can be taken, and the complementary hue value is utilized for contrast. Thus, in an exemplary implementation, the hue value of the hand model is adapted from red to blue to increase its visibility.

[0054] 7a and 7b show examples of adjustments to finger size according to an exemplary implementation. By performing content analysis on a 2D canvas image of the camera feed in a web browser window, an exemplary implementation can adapt the finger size based on the estimated size of an object near the pointing index finger. For example, in FIG. 7a, the finger is too large and it is difficult to tell which of two objects on a circuit board it is pointing at. In FIG. 7b, the finger size is adjusted and it is clear that the object on the left is the target object.

[0055] 8a and 8b show examples of detected blobs with corresponding bounding boxes highlighted according to an exemplary implementation. To estimate the size of objects in the vicinity of the finger, an exemplary implementation applies blob detection. One exemplary process of blob detection is to use contour detection on a thresholded binarized image. The result of applying the thresholded binarized image on the image of FIG. 8a is shown in FIG. 8b, which shows the objects detected in the vicinity of the index finger in FIG. 7a and FIG. 7b and their sizes, displayed as bounding boxes. The size of the finger can be adjusted by scaling it to be the average size of the objects detected in the vicinity.

[0056] In another exemplary implementation of adjusting the size of the hand in the image, the 3D position of the object in the real-world space can be estimated, and as a result, a virtual hand is placed near the object, and the scale of the hand in the camera image is automatically adjusted based on the distance between the camera and the object (or the hand placed nearby). The 3D position of the object in space can be estimated by reconstructing a 3D model of the scene (including the object) from a sequence of camera images using Structure from Motion (SfM) techniques known to those skilled in the art or deep learning-based depth map estimation methods according to a desired implementation. Another approach is to directly obtain the 3D pose of the object with respect to a pre-constructed 3D reference model. The model is constructed from a 3D point cloud using SfM, and each 3D point is associated with a local feature descriptor. The 6-DOF pose of the object can be obtained by establishing a 2D-3D local feature correspondence between the camera image of the object and the reference model, and solving a Perspective-n-Point (PnP) problem based on the correspondence.

[0057] In an exemplary implementation, context-aware audio cues can be provided further according to the desired implementation. In an exemplary implementation, there can be a type of setup that makes it easier for the expert to navigate in a reconstructed 3D environment based on compositing the scene as the customer moves the smartphone, which effectively expands the field of view of the customer's smartphone.

[0058] Additionally, an exemplary implementation can be configured to address the issue of the expert's hand movements being outside the field of view of the customer's smartphone. Since the expert is pointing or talking to help guide the smartphone towards the customer's viewpoint of viewing the expert's hand movements, it may be better to use another modality. One technique is to use audio cues to indicate the direction in which the expert's hands are currently located. The system determines the relative direction and synthesizes a stereo signal that is added to the video stream transmitted to the customer. Using stereo headphones, the customer can hear the direction in which the smartphone is moved and see the expert's hand movements.

[0059] FIG. 9 illustrates an example of an object replay feature to access previously captured actions, according to an exemplary implementation. Specifically, FIG. 9 facilitates such functionality by providing a replay icon. An exemplary implementation can further facilitate hand motion capture and replay according to a desired implementation. Often, customers ask the same question for a particular problem. An example given in FIG. 9 is the task of cleaning the rollers of a printer. To save the cost of repeated use of an expert's time, frames of rendered hand actions can be recorded to capture the task and later replayed by the customer. An exemplary scenario is a customer pointing a smartphone at a target object. The system recognizes the object and, if there is an associated captured task, displays a rendered icon over the current smartphone camera view, as shown in FIG. 9. The customer can tap the icon to replay the rendered hand action over the current smartphone camera view. In such an exemplary implementation, both the video and the directional sensor measurements or directional measurements can be stored for later use and processing by the expert's system, such that the video replay is associated with the recorded directional measurements.

[0060] The exemplary implementation differs from prior related art in that it uses a web-based system that does not require special applications or plug-ins to be installed on the customer device or web browser, but rather is accessible for use on any internet-connected device, most web browsers known to those skilled in the art.

[0061] Exemplary implementations further facilitate context- and content-aware visualization of hands in that visual aspects of the hand models, such as pose, color, scale, etc., are adapted according to context and content information on the customer environment and target object.

[0062] FIG. 10 illustrates an example of a mobile device included in an exemplary implementation. The mobile device 1000 may include a camera 1001, a microphone 1002, a processor 1003, a memory 1004, a display 1005, an interface (hereinafter, referred to as I / F) 1006, and a direction sensor 1007. The camera 1001 may include any type of camera configured to record any form of video according to a desired implementation. The camera and the direction sensor may be separate. The microphone 1002 may include any type of microphone configured to record any form of audio according to a desired implementation. The display 1005 may include a touch screen display configured to receive touch input in terms of facilitating instructions to perform the functions described herein, or a conventional display such as a liquid crystal display (LCD) or any other display according to a desired implementation. The I / F 1006 may include a network interface facilitating connection of the mobile device 1000 to external elements such as servers and any other devices according to a desired implementation. The processor 1003 may be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units, according to a desired implementation. The orientation sensor 1007 may include any form of gyroscope and / or accelerometer configured to measure any type of orientation measurement, such as tilt angle, orientation relative to x, y, z, approach, or acceleration (e.g., gravity), according to a desired implementation. The orientation sensor measurements may also include a gravity vector measurement to indicate the direction of gravity on the device, according to a desired implementation.

[0063] The processor 1003 may be configured to forward the video from the camera and measurements from the direction sensor 1007 to another device (such as the device shown in FIG. 11) via the I / F 1006, as shown in FIG. 1b and FIG. 2. As shown in FIG. 2, the processor 1003 may receive hand information related to a representation of a user's hand from another device and overlay the representation of the user's hand on a video displayed on the mobile device based on the received hand information, overlaying the representation of the user's hand on the video in an orientation determined by measurements from the direction sensor 1007. In an example implementation including live video as shown in FIG. 1b and described herein, the video may be live video from the camera of the mobile device and the measurements may be live measurements from the direction sensor. The hand information may also include hand skeletal joint measurements and hand visualization parameters determined from live movements of the user's hand, as shown in FIG. 4a-4c and FIG. 5a-5c.

[0064] In an exemplary implementation, the processor 1003 may be configured to overlay a representation of the user's hand based on the hand information onto an image displayed on a mobile device (e.g., display 1005) by generating a 3D hand model of the user's hand as a representation based on hand skeletal joint measurements, as shown in Figures 5a-5c, 6a, 6b, 7a, 7b, and 8a-8b, and adjust one or more of the color and size of the 3D model to be overlaid on the image based on hue information of the image and one or more of objects detected on the image.

[0065] 2 and 3, the processor 1003 may be configured to establish a browser-to-browser connection via a web browser from the mobile device to another web browser on another device, and the processor 1003 is configured to transfer video from the camera 1001 and measurements from the orientation sensor 1007 to the other device and receive hand information from the other device via the browser-to-browser connection.

[0066] 2, audio may be transferred from a customer device or an expert device. In such an example implementation, the processor 1003 may be configured to transfer recorded audio from the microphone 1002 to another device (e.g., via the I / F 1006) and output audio received from the other device through any conventional sound speaker, according to a desired implementation.

[0067] As shown in FIG. 9, for example, for an example implementation including previously recorded footage, the footage may be previously recorded footage and the measurements from the directional sensor may be previously recorded measurements.

[0068] 11 illustrates an example of a system including a tracking device according to an exemplary implementation. The system may include a device 1100, such as a computer, laptop, tablet, or other device, according to a desired implementation. Such a device may include a camera 1101, a microphone 1102, a processor 1103, a memory 1104, a display 1105, and an interface (hereinafter, referred to as I / F) 1106. Also, the hardware may be similar to the hardware of the mobile device 1000 of FIG. 10 or may be modified according to a desired implementation.

[0069] The tracking device 1110 may include sensors 1111 that may be configured to track motion and may be in the form of any motion tracking sensor, such as infrared, radio frequency, etc. The tracking device 1110 may be configured to generate hand information, such as hand skeletal joints and camera visualization parameters, according to any desired implementation compatible with Three.js.

[0070] As shown in FIG. 2, the processor 1103 receives video and orientation sensors from the mobile device via the I / F 1106, and transfers hand information associated with a representation of a user's hand from another device, generated based on measurements obtained from a tracking device (e.g., sensor 1111), as shown in FIG. 4a-4c and FIG. 5a-5c. As shown in FIG. 1a, FIG. 4a-4c, FIG. 5a-5c, FIG. 6a, FIG. 6b, FIG. 7a, FIG. 7b, FIG. 8a, and FIG. 8b, the processor 1103 may be configured to overlay a representation of the user's hand onto the video displayed on the device based on the transferred hand information. The representation of the user's hand is overlaid onto the video displayed on the device in an orientation determined by measurements from the orientation sensors. The orientation is determined by determining the direction of gravity in the customer's environment from the measurements of the orientation sensors.

[0071] Depending on the desired implementation, the video may be live video from a mobile device, as shown in Figures 1b and 2. The orientation sensor measurements may be live orientation sensor measurements from a mobile device. The hand information may include hand skeleton joint measurements and hand visualization parameters determined from live measurements of the user's hand from a tracking device.

[0072] Depending on the desired implementation, the processor 1103 may be configured to overlay a representation of the user's hand onto the image displayed on the device based on the hand information by generating a 3D hand model of the user's hand as a representation based on the hand skeletal joint measurements, as shown in Figures 6a-6b, 7a-7b, and 8a-8b, and adjusting one or more of a color and a size of the 3D hand model so as to be overlaid on the image based on one or more of hue information of the image and detected objects on the image.

[0073] 2 and 3, the processor 1103 may be configured to establish an inter-browser connection via a web browser from the device to another web browser on a mobile device via the inter-browser connection receiving video and orientation sensor readings from the mobile device and transferring hand information from the device.

[0074] The processor 1103 may be configured to transfer recorded audio from the microphone 1102 to the mobile device via the I / F 1106 and output audio received from the mobile device through any conventional sound speaker, according to a desired implementation.

[0075] As shown in FIG. 9, the video may be previously recorded video and the directional sensor measurements are previously recorded directional sensor measurements.

[0076] 12 illustrates an exemplary computing environment with an exemplary computing device suitable for use in some exemplary implementations, such as a server (e.g., WebRTC signaling server 200) that establishes a connection between a customer device and an established device. The computing device 1205 in the computing environment 1200 can include one or more processing units, cores, or processors 1210, memory 1215 (e.g., RAM, ROM, etc.), internal storage 1220 (e.g., magnetic, optical, solid-state storage, and organic), or an I / O interface 1225. Any of these can be coupled onto a communication mechanism or bus 1230 for communicating information or can be embedded in the computing device 1205. The I / O interface 1225 is also configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0077] The computing device 1205 may be communicatively coupled to an input / user interface 1235 and an output device / interface 1240. Either or both of the input / user interface 1235 and the output device / interface 1240 may be wired or wireless interfaces and may be removable. The input / user interface 1235 may include any device, component, sensor, or interface, physical or virtual, that can be used to provide input. For example, buttons, touch screen interfaces, keyboards, pointing / cursor controls, microphones, cameras, braille displays, motion sensors, optical readers, etc. The output devices / interfaces 1240 may include displays, televisions, monitors, printers, speakers, braille displays, etc. In some example implementations, the input / user interface 1235 and the output device / interface 1240 may be embedded in the computing device 1205 or physically coupled to the computing device 1205. In other example implementations, other computing devices may function as or provide the functionality of the input / user interface 1235 and the output device / interface 1240 of the computing device 1205 .

[0078] Examples of computing devices 1205 may include, but are not limited to, highly mobile devices, mobile devices, and devices not designed for mobility, including, for example, smart phones, devices in vehicles and other machines, devices carried by humans and animals, tablets, notebooks, laptops, personal computers, portable televisions, radios, other computers, information kiosks, televisions with one or more embedded processors coupled thereto, and the like.

[0079] Computing device 1205 may be communicatively coupled (e.g., via I / O interface 1225) to external storage 1245 and network 1250 for communicating with any number of network components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 1205 or any connected computing device may be a server, a client, a thin server, a general purpose machine, a dedicated machine, etc.

[0080] I / O interface 1225 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem, cellular network protocols, etc.) for communicating information to and from at least all connected components, devices, and networks in computing environment 1200. Network 1250 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0081] The computing device 1205 can use and / or communicate using computer usable or computer readable media, including transitory and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD-ROMs, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0082] The computing device 1205 may be used to implement techniques, methods, applications, processes, or computer-executable instructions in some exemplary computing environments. The computer-executable instructions may be obtained from, stored in, or retrieved from a transitory medium. The executable instructions may originate from one or more of any programming, scripting, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0083] The one or more processors 1210 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logic unit 1260, an application programming interface (API) unit 1265, an input unit 1270, an output unit 1275, and an inter-unit communication mechanism 1295 for different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements may vary in design, function, configuration, or implementation and are not limited to the provided description. The one or more processors 1210 can be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units.

[0084] In some example implementations, when information or instructions for execution are received by API unit 1265, it may be communicated to one or more other units (e.g., logic unit 1260, input unit 1270, output unit 1275). Illustratively, logic unit 1260 may be configured to control information flow between units and, in some example implementations described above, direct services provided by API unit 1265, input unit 1270, output unit 1275. For example, the flow of one or more processes or implementations may be controlled by logic unit 1260 alone or in conjunction with API unit 1265. Input unit 1270 may be configured to obtain inputs for computations described in example implementations, and output unit 1275 may be configured to provide outputs based on computations described in example implementations.

[0085] As shown in Figures 2 and 3, the one or more processors 1210 receive a first connection from the mobile device (e.g., through a browser and selection of an "I am a customer" button in Figure 3). Then, receive a second connection from another device communicatively coupled to the tracking device (e.g., through a browser and selection of an "I am an expert" button in Figure 3). Thus, the one or more processors 1210 may be configured to establish a third connection between the mobile device and another device to facilitate transfer of video and orientation sensor measurements from the mobile device to the other device and to facilitate transfer of hand information from the other device to the mobile device. As shown in Figure 3, the first and second connections may be received via a web browser, and establishing the third connection may include establishing a direct browser-to-browser connection between the mobile device of Figure 10 and another device, such as the system of Figure 11.

[0086] As shown in Figures 1a, 1b, and 2, the video may be live video from a mobile device, the directional sensor measurements may be live directional sensor measurements from a mobile device, and the hand information may include hand skeleton joint measurements and hand visualization parameters determined from live measurements of the user's hand from a tracking device.

[0087] As shown in FIG. 9, the video may be previously recorded video and the directional sensor measurements may be previously recorded directional sensor measurements.

[0088] The exemplary implementation of the present disclosure relating to measuring a user's hand can be extended to objects other than a user's hand, such as a user's foot, a stylus, any kind of tool, etc., according to the desired implementation, as long as the shape of the object is known and can be converted into skeletal joints and visualization parameters to facilitate the desired implementation.

[0089] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a sequence of defined steps leading to a desired end state or result. In exemplary implementations, the steps performed require actual amounts of physical manipulation to achieve some substantial result.

[0090] Unless otherwise indicated, it will be apparent from the description that throughout this specification, descriptions utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," and the like may include the actions and processing of a computer system or other information processing device that manipulates and transforms data represented as physical (electronic) quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system, or other information storage, transfer, or display device.

[0091] Exemplary implementations may also relate to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the required purposes, or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. Computer-readable storage media may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memory, random access memory, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation that includes instructions for performing the operations of a desired implementation.

[0092] Various general-purpose systems can be used with the programs and modules according to the examples described herein, or it may prove convenient to construct more specialized apparatus to perform the desired method steps. Moreover, the example implementations are not described with reference to any particular programming language. A variety of programming languages ​​may be used to implement the teachings of the example implementations described herein. Instructions in one or more programming languages ​​may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0093] As known in the art, the operations described above may be performed by hardware, software, or a combination of software and hardware. Various aspects of the exemplary implementations may be implemented using circuits and logic devices (hardware). Other aspects may be implemented by a processor executing instructions (software) stored on a machine-readable medium to perform the methods described herein. Also, some exemplary implementations described herein may be performed exclusively in hardware, while other exemplary implementations may be performed exclusively in software. Furthermore, various functions described herein may be performed in a single unit or may be distributed across multiple components in multiple ways. When performed by software, the methods may be executed by a processor, such as a general purpose computer, based on instructions stored on a computer-readable medium. If necessary, the instructions may be stored on the medium in compressed or encrypted form.

[0094] Moreover, other implementations described herein will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present disclosure. Various aspects and components of the example implementations described herein may be used alone or in any combination. It is intended that the specification and example implementations be considered as examples only, with a true scope and spirit of the present disclosure being indicated by the following claims.

Claims

1. 1. A program for executing a process on a mobile device including a direction sensor and a camera, the instructions comprising: transmitting the image from the camera and the measurements from the direction sensor to another device; receiving hand information associated with a hand of a user from the other device; and overlaying a representation of the user's hand onto an image displayed on the mobile device based on the hand information, wherein the representation of the user's hand is further overlaid onto the image displayed on the mobile device in an orientation determined by the measurements from the direction sensor.

2. the video is live video from the camera of the mobile device and the measurements are live measurements from the direction sensor; The program of claim 1 , wherein the hand information comprises hand skeletal joint measurements and hand visualization parameters determined from live movements of the user's hand.

3. 3. The program of claim 2, wherein overlaying the representation of the user's hand onto the image displayed on the mobile device based on the hand information comprises: generating a 3D hand model of the user's hand as the representation based on the hand skeletal joint measurements; and adjusting one or more of a color and a size of the 3D hand model for overlaying onto the image based on one or more of hue information of the image and objects detected on the image.

4. The instruction: establishing, via a web browser, a browser-to-browser connection from the mobile device to another web browser of the other device; The program of claim 1 , wherein the transferring of the image from the camera and the measurements from the direction sensor to the other device and the receiving of the hand information from the other device are performed via the browser-to-browser connection.

5. The instruction: Transferring the recorded audio from the microphone to the other device; The method of claim 1 , further comprising: outputting the audio received from the other device.

6. The program of claim 1 , wherein the video is a previously recorded video and the measurements from the direction sensor are previously recorded measurements.

7. 1. A program for executing a process on a device communicatively coupled to a tracking device, the instructions comprising: receiving video and orientation sensor measurements from a mobile device; transferring, from another device, hand information associated with the user's hand, the hand information being generated based on measurements obtained from the tracking device; and overlaying a representation of the user's hand onto the image for display by the device based on the hand information, wherein the representation of the user's hand is overlaid onto the image in an orientation determined by the measurements from the direction sensor.

8. 8. The program of claim 7, wherein the video is live video from a mobile device, the direction sensor measurements are live direction sensor measurements from the mobile device, and the hand information includes hand skeleton joint measurements and hand visualization parameters determined from live measurements of the user's hand from the tracking device.

9. 9. The program of claim 8, wherein overlaying the representation of the user's hand onto the image for display by the device based on the hand information includes: generating a 3D hand model of the user's hand as the representation based on the hand skeletal joint measurements; and adjusting one or more of a color and a size of the 3D hand model for overlaying onto the image based on one or more of hue information of the image and detected objects on the image.

10. The instruction: establishing a browser-to-browser connection from the device to another web browser on the mobile device via the web browser; The program of claim 7 , wherein receiving the video and the orientation sensor measurements from the mobile device and transferring hand information from the device occurs via the browser-to-browser connection.

11. The instruction: transferring recorded audio from a microphone to the mobile device; The method of claim 7 , further comprising: outputting the audio received from the mobile device.

12. 8. The program of claim 7, wherein the video is a previously recorded video and the direction sensor measurements are previously recorded direction sensor measurements.

Images