DEVICE AND METHOD FOR HAND-BASED USER INTERACTION IN VR AND AR ENVIRONMENTS

The method and device enhance VR/AR user interfaces by using an overlay with a center indicator and neutral zone to enable intuitive hand gestures, reducing accidental interactions and improving usability.

JP7591042B2Active Publication Date: 2024-11-27INTERDIGITALCE PATENT HLDG SAS
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2022518181
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-09-20
Filing Date
2020-09-07
Publication Date
2024-11-27
Estimated Expiration
2040-09-07

Smart Images

  • Figure 0007591042000001
    Figure 0007591042000001
  • Figure 0007591042000002
    Figure 0007591042000002
  • Figure 0007591042000003
    Figure 0007591042000003
Patent Text Reader

Abstract

A virtual reality (VR) or augmented reality (AR) system detects, through a camera, when the position of a user's hand matches a predefined position, and in response to that detection, renders an overlay including a crown that the user can interact with to lock onto an object or scene. The system then detects and renders a center indicator of the crown, tracks the user's hand, enables or disables manipulation depending on the position of the center indicator relative to the neutral zone, realizes manipulation in response to movement of the user's hand, and also detects unlocking of the object or scene.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates generally to user interfaces, and more particularly to user interfaces for Virtual Reality (VR) and Augmented Reality (AR) environments. [Background technology]

[0002] This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present disclosure that are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. As such, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] VR and AR technologies provide immersive experiences to users who run such applications on suitable displays, such as smart TVs, PCs, projectors, head-mounted displays (HMDs), etc. Through a user interface, a user can typically interact with the VR or AR environment, for example, to manipulate objects therein. To do this, user input needs to be received, which can be done by a user tracking system, for example using sensors (wearable or non-wearable) that detect color and depth. For example, a connected smart TV with a color / depth camera can track the shape and position of each element of the user's body, which can enable the use of hand gestures in a spontaneous interaction system. This can provide a user with a way to interact with AR or VR scenes or objects, for example in broadcast TV programs, content items from the Internet, or content shared during video conferences.

[0004] Traditional user interfaces may overlay an image of a user's hands, which makes interacting with buttons and other menu items relatively simple. Interacting with objects is more difficult because objects do not always react as they would in the real world.

[0005] Another conventional solution provides a representation of the user's hand position on the screen, allowing the user to interact with virtual objects to zoom in / out and rotate the objects.

[0006] However, such user interfaces tend to be difficult to use, especially for new users, as they are not always intuitive, especially when it comes to interacting with objects.

[0007] It can therefore be appreciated that a solution that addresses at least some of the shortcomings of user interfaces for AR and VR is desirable. The present principles provide such a solution. Summary of the Invention

[0008] In a first aspect, the present principles are directed to a method in a device for rendering a virtual or augmented reality environment, the method including: rendering, in at least one hardware processor of the device, an overlay on a display, calculating a position of a center indicator of the overlay based on a detected position of a user's hand, enabling at least one operation based on the position of the center indicator relative to a neutral zone, and performing the enabled operation in response to a movement of the user's hand.

[0009] In a second aspect, the present principles are directed to a device for rendering a virtual or augmented reality environment, the device comprising at least one hardware processor configured to render an overlay on a display, calculate a position of a central indicator of the overlay based on a position of a user's hand, enable at least one operation based on the position of the central indicator relative to a neutral zone, and perform the enabled operation in response to movement of the user's hand.

[0010] In a third aspect, the present principles are directed to a computer program product stored on a non-transitory computer readable medium and comprising program code instructions executable by a processor to implement the steps of the method according to any of the embodiments of the second aspect. [Brief description of the drawings]

[0011] Features of the present principles will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which: [Figure 1] 1 illustrates a virtual reality (VR) / augmented reality (AR) system in accordance with an embodiment of the present principles. [Diagram 2] 1 illustrates a method in accordance with an embodiment of the present principles. [Diagram 3] 1 illustrates a first example of such an overlay in accordance with present principles. [Figure 4] 1 illustrates a second example of such an overlay in accordance with present principles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] 1 is an illustration of a virtual reality (VR) or augmented reality (AR) interactive system 100 in accordance with an example embodiment. The interactive system 100 includes a rendering device 110, a display 120, and a camera 130.

[0013] Although rendering device 110, display 120, and camera 130 are shown as separate entities, in different variations they are combined in any possible manner. For example, these three entities may be incorporated into a single device such as a television, with display 120 and rendering device 110 being integral but camera 130 being separate, or rendering device 110 being separate from the display including the camera.

[0014] Display 120 is operable to display the video received from the rendering device to a user (not shown), and may be, for example, a television, a computer screen, a head mounted display (HMD), a tablet, or a smartphone.

[0015] The camera 130, which may be a depth camera and a color camera, is operable to capture images and / or video of what is in front of its opening. In the interactive system 100, the camera 130 will typically be positioned to capture the environment including the user. Other gesture detectors that detect the position of the user's hands are also possible, such as one or more sensors worn by the user.

[0016] The rendering device 110 may be, for example, a computer, a set-top box, a game console, a tablet, a smartphone, etc. The rendering device 110 typically includes a user input interface 111, at least one hardware processor ("processor") 112, a memory 113, a network interface 114, and an output interface, such as a display interface 115 or a speaker. The interfaces are at least partially implemented in hardware. It will be understood that parts of the device 110 that are not useful for understanding the present principles are not shown for simplicity of explanation.

[0017] The user input interface 111 may be wireless and configured to receive input from an input device (not shown), but may be an interface to a touch screen, keyboard and / or mouse, etc. The user input interface is configured to provide the received input to the processor 112. The processor 112 is configured to execute program code instructions for implementing a method according to the present principles described with reference to FIG. 2. The memory 113 is configured to store program code instructions for execution by the processor 112 and to store data. The network interface 114 may be wired or wireless and is configured to enable communication between the processor 112 via a connection 140 and at least one device, such as a server in an external network (not shown), to receive computer programs or content for display to a user. Such content may be, as a non-limiting example, a video program to be used. The output interface may be wired or wireless and may be configured to output content for display on the display 120. As mentioned above, if the device 110 is a television, the display 120 is typically included in the device. Note that the television may also have a display interface for displaying information on a separate screen (not shown).

[0018] A prominent aspect of the present principles is the use of overlays over VR / AR scenes.

[0019] 2 illustrates a method according to an embodiment of the present principles. It is assumed that the user is "inside" the AR / VR scene, i.e., the user can interact with the scene. For illustrative purposes, it is also assumed that the user is not interacting with any objects in the scene.

[0020] In step S210, the user interface (120 in Figs. 1 and 2) of the interactive system (100 in Fig. 1) detects the position of the user's hand and detects when the position matches a predefined position. The predefined position may be one of multiple positions having the same meaning. An example of such a predefined position may be when the user extends his / her hand in front of the body.

[0021] If a match is found, the user interface renders an overlay on the AR / VR scene in step S220. Figure 3 illustrates a first example of such an overlay 300 in accordance with the present principles. The overlay 300 may be at least partially translucent, and in this example is intended to be interacted with using two hands.

[0022] The overlay 300 may include a center indicator 310 , a neutral zone 320 (shaded diagonally in grey) and a coronal portion 330 including a first locking position 341 and a second locking position 342 .

[0023] The central indicator 310 indicates the center of the coronal portion 330. When the central indicator 310 is located inside the neutral zone 320, certain user operations are prohibited, as described below. Conversely, operations may be enabled only when the central indicator 310 is located inside the neutral zone 320. The neutral zone 320 may be invisible (or more transparent) when the central indicator 310 is located inside, and may be visible or more opaque when the central indicator 310 is outside.

[0024] Lock positions 341, 342 are preferably hand shapes, indicating where a user must place their hands to lock or unlock a scene or object, as described below.

[0025] The user's hand positions are represented by right hand indicator 351 and left hand indicator 352 illustrated in exemplary positions.

[0026] In step S230, the user interface waits for the user to lock onto the object with a particular hand gesture, which in this example is placing the hand on (or at least essentially on) the lock positions 341, 342 for a given period of time, in other words, waiting for the user to place the hand indicators 351, 352 on top of the lock positions 341, 342 for a given period of time, which could be 0.5 seconds, 1 second, 1.5 seconds, etc.

[0027] In an embodiment, the lock positions 341, 342 flash when there is no lock. Additionally, the user interface may render text to inform the user how to lock onto an object.

[0028] Upon locking onto the object, the user interface detects and renders, in step S240, the position of the center indicator 310, preferably essentially continuously. In particular, the user interface can detect whether the center indicator 310 is inside or outside the neutral zone 320. When locked, the center indicator 310 typically corresponds, possibly with some delay, to the center of the user's hands, i.e., the center of the hand indicators 351, 352. Note that the crown 330 typically follows the movement of the center indicator 310.

[0029] Figure 4 illustrates a second example of an overlay 400 in accordance with the present principles. The components of overlay 400 are the same as those shown in overlay 300 of Figure 3, with the difference being that lock positions 341, 342 are filled in to indicate a lock to an object or scene, and center indicator 310 and crown 330 have followed the user's hand, such that center indicator 310 is now outside of neutral zone 320 (note that in the figure it appears as if the neutral zone has moved).

[0030] In step S250, the user interface enables and / or disables user operation depending on the position of the center indicator 310 relative to the neutral zone 320.

[0031] For example, when inside, the user actions allowed are: Zooming in or out (i.e., moving objects closer or further away) by spreading or bringing the hands closer together, respectively (possibly limited to when the hand indicators 351, 352 are essentially at the same height); and optionally constraining the hand indicators 351, 352 when they are inside the coronal portion 330 to rotate the object (as if they were on a steering wheel) through a circular movement of the corresponding hand; When external, the user actions allowed are: Rotating the object horizontally by moving the hand in the corresponding horizontal direction or by moving the hand in the corresponding vertical direction, or This may involve moving the camera (i.e. the user's viewpoint) around within the scene.

[0032] As can be appreciated, the neutral zone can make the AR / VR scene less susceptible to, for example, accidental rotations, since small horizontal or vertical hand movements have no effect while the center indicator 310 is in the neutral zone 320.

[0033] The user interface may indicate operation opportunities, for example, by rendering icons indicating enabled operations. In addition, the icons may be rendered differently (e.g., grayed out or faded) for disabled operations, as is conventional in user interfaces. The icons may be rendered for a limited period of time before disappearing or fading. Preferably, the icons are rendered in a peripheral area so as not to obscure large portions of the scene or objects.

[0034] It should be noted that other user operations, such as moving an object, may be accomplished by the user interface.

[0035] In step S260, the user interface responds that user actions are tracked through detection of hand positions. User actions can include those described with reference to step S250. Further actions can include unlocking (as described with reference to step S270) and hiding and showing the overlay 300 when the user crosses his hands, i.e., brings the right hand to the left of the left hand (note that this action can be disabled to allow for larger circular rotations, e.g., 180°, when the user performs a rotation through a circular movement of both hands. Alternatively, the hand indicators 351, 352 can be prevented from switching sides, such that the left side is always to the left of the right side or in the same vertical position as the right side (e.g., except when performing a circular movement).

[0036] Note that when the center indicator 310 passes through the neutral zone 320 on its way to the other side, horizontal or vertical rotation of the object may be temporarily disabled.

[0037] In step S270, the user interface detects the unlocking of an object or scene. This may be done by the user placing the hand indicators 351, 352 over the lock positions 341, 342 for a given time. As can be seen, the user may do the same as when locking onto an object or scene. Unlocking may also require the center indicator 310 to be in the neutral zone 320.

[0038] In a variant suitable for one-handed use, unlocking occurs in the same manner, but the overlay may include a single lock position over which the user must place a hand indicator for a given period of time to lock a scene or object. In a variant, the enabled user operations may be limited to moving objects.

[0039] It will thus be appreciated that the present principles can be used to provide an AR / VR system that provides a user with information regarding enabled user operations, and through the use of a neutral zone, to provide an AR / VR experience that is less susceptible to "jitter" caused by small hand movements.

[0040] It should be understood that the elements shown in the figures may be implemented in various forms of hardware, software, or a combination thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory, and input / output interfaces.

[0041] The present description illustrates the principles of the present disclosure, and it will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the present disclosure and are included within its scope.

[0042] All examples and conditional language recited herein are intended for educational purposes to aid the reader in understanding the principles of the present disclosure and concepts contributed by the inventors to further the art, and should not be construed as being limited to such specifically recited examples and conditions.

[0043] Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. In addition, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0044] Thus, for example, those skilled in the art will appreciate that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the principles of the present disclosure. Similarly, any flow charts, flow diagrams, or the like will be understood to represent various processes that may be executed by a computer or processor, substantially represented on a computer-readable medium, whether or not such a computer or processor is explicitly shown.

[0045] The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. If provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. Furthermore, explicit use of the terms "processor" or "controller" should not be construed as referring exclusively to hardware capable of executing software, but may implicitly include, but is not limited to, digital signal processor (DSP) hardware, read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage.

[0046] Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their functions may be performed through the operation of program logic, through the operation of dedicated logic, through program control and interaction with dedicated logic, or even manually, with the particular technique being selectable by the implementer as more particularly understood from the context.

[0047] In the claims herein, any element expressed as a means for performing a certain function is intended to encompass any way of performing that function, including, for example, a) a combination of circuitry that performs that function, or b) software in any form, including firmware, microcode, etc., in combination with appropriate circuitry for executing the software to perform the function. The disclosure as defined by such claims resides in the fact that the functionality provided by the various recited means is combined and brought together in the manner which the claims call for. Thus, any means which can provide those functionality are regarded as equivalent as shown herein.

Claims

1. 1. A method in a device for rendering a virtual reality or augmented reality environment, comprising: rendering the overlay on the display by a rendering means; enabling, by an enabling means, at least one user operation of a plurality of user operations based on whether a position of a center indicator of the overlay is inside or outside a neutral zone of the overlay, the position of the center indicator being calculated as a center of a detected position of a user's hand; A method in which a first set of operations is permitted only when the position of the center indicator is inside the neutral zone and a second set of operations is permitted only when the position of the center indicator is outside the neutral zone.

2. The method of claim 1 , further comprising: rendering the overlay in response to detecting a match between a detected hand position and at least one predefined position.

3. The method of claim 1 , further comprising: performing, by the executing means, the enabled user operation in response to the detected user's hand movement.

4. 2. The method of claim 1, wherein the overlay includes the center indicator, the neutral zone, a coronal portion having first and second locking positions, and right and left side indicators, the first and second locking positions indicating where the user should overlay the right and left side indicators to lock onto or unlock a lock on to an object or scene.

5. 2. The method of claim 1, wherein the first set of operations is one of zooming in, zooming out, and rotating in the plane of the display, and the second set of operations is one of horizontal rotation, vertical rotation, and translation.

6. 2. The method of claim 1, further comprising rendering, by the rendering means, a first icon having a first visual aspect indicating an enabled operation and a second icon having a second visual aspect for a disabled operation.

7. 1. A device for rendering a virtual reality or an augmented reality environment, comprising: Rendering the overlay on the display; and and enabling at least one user operation of a plurality of user operations based on whether a position of a center indicator of the overlay is inside or outside a neutral zone of the overlay, the position of the center indicator being calculated as a center of a detected position of a user's hand; the at least one hardware processor is configured to enable a first operation only when the position of the center indicator is inside the neutral zone and to enable a second operation only when the position of the center indicator is outside the neutral zone.

8. The device of claim 7 , wherein the at least one hardware processor is further configured to calculate the position of the center indicator based on a detected position of a user's hand.

9. 8. The device of claim 7, wherein the first operation is one of zooming in, zooming out, and rotating in the plane of the display, and the second operation is one of horizontal rotation, vertical rotation, and translation.

10. 8. The device of claim 7, wherein the at least one hardware processor is further configured to render a first icon having a first visual aspect indicating an enabled operation and a second icon having a second visual aspect for a disabled operation.

11. The device of claim 7 , wherein the device comprises at least one of the display and a camera configured to capture the position of the user's hand.

12. 8. The device of claim 7, wherein the at least one hardware processor is further configured to perform enabled user operations in response to detected user hand movements.

13. A non-transitory computer readable medium storing program code instructions which, when executed by a processor, implement the steps of the method according to at least one of claims 1 to 6.

Citation Information

Patent Citations

  • Mediated reality

    EP3260950A1

  • Detection of gestures made with at least two control objects

    JP2016511492A

  • Display control method and program for causing computer to execute the method

    JP2017187952A

  • Method for providing virtual space, program and recording medium

    JP2017200494A

  • Head-Tracked User Interaction with Graphical Interface

    US20130246967A1