Apparatus and method for hand-based user interaction in VR and AR environments

Abstract

A virtual reality (VR) or augmented reality (AR) system detects via a camera when the position of a user's hand matches a predefined position and, in response, renders an overlay including a crown with which the user can interact to lock onto an object or scene. The system then detects and renders a center indicator of the crown that tracks the user's hand, enables or disables actions based on the position of the center indicator relative to a neutral zone, performs actions in response to user hand movement, and also detects unlocking of an object or scene.

Description

Technical Field

[0001] The present disclosure relates generally to user interfaces, and more particularly to user interfaces for virtual reality (VR) and augmented reality (AR) environments. Background Art

[0002] This section is intended to introduce the reader to various aspects of the art that may be relevant to various aspects of the present disclosure described and / or claimed below. It is believed that this discussion will help provide the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Therefore, it should be understood that these statements should be read in this light, and not as admissions of prior art.

[0003] VR and AR technologies provide users with an immersive experience of running such applications on a suitable display, such as a smart TV, PC, projector, head-mounted display (HMD), etc. Through a user interface, a user is typically able to interact with the VR or AR environment, for example to manipulate objects therein. To do this, user input must be received, which can be accomplished, for example, by a user tracking system using sensors (wearable or non-wearable) or cameras that detect color and depth. For example, a connected smart TV equipped with a color / depth camera may be able to track the shape and position of each part of the user's body, which may enable the use of gestures in a naturally interactive system. This may provide a way for users to interact with AR or VR scenes or objects in, for example, a broadcast TV program, content items from the internet or content shared during a video conference.

[0004] Conventional user interfaces may overlay an image of the user's hand, which makes interacting with buttons and other menu items relatively simple. Interacting with objects is more difficult because objects don't always react as they would in the real world.

[0005] Another conventional solution provides an indication of the position of the user's hand on the screen and enables the user to interact with the virtual object to zoom in / out and rotate the object.

[0006] However, such user interfaces tend to be difficult to use, especially for new users, particularly when interacting with objects, as they are not always intuitive.

[0007]

[0006] It will be appreciated, therefore, that a solution is desired that addresses at least some of the shortcomings of user interfaces for AR and VR. The present principles provide such a solution. Summary of the Invention

[0008] In a first aspect, the present principles relate to a method for rendering a virtual reality or augmented reality environment in a device. The method includes, in at least one hardware processor of the device: rendering an overlay on a display; calculating a position of a center indicator of the overlay based on a detected position of a user's hand; enabling at least one action based on the position of the center indicator relative to a neutral zone; and performing the enabled action in response to a movement of the user's hand.

[0009] In a second aspect, the present principles relate to an apparatus for rendering a virtual reality or augmented reality environment, the apparatus comprising at least one hardware processor configured to: render an overlay on a display; calculate a position of a center indicator of the overlay based on a position of a user's hand; enable at least one action based on the position of the center indicator relative to a neutral zone; and perform the enabling action in response to user hand movement.

[0010] In a third aspect, the present principles relate to a computer program product stored on a non-transitory computer-readable medium and comprising program code instructions executable by a processor to implement the steps of the method according to any embodiment of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The features of the present principles will now be described by way of non-limiting example with reference to the accompanying drawings, in which:

[0012] Figure 1 A virtual reality (VR) / augmented reality (AR) system according to an embodiment of the present invention is shown;

[0013] Figure 2 A method according to an embodiment of the present invention is shown;

[0014] Figure 3 A first example of such a covering according to the principles of the present invention is shown; and

[0015] Figure 4 A second example of such a covering in accordance with the principles of the present invention is shown. DETAILED DESCRIPTION

[0016] Figure 1 is a depiction of a virtual reality (VR) or augmented reality (AR) interactive system 100 according to an exemplary embodiment. The interactive system 100 includes a rendering device 110, a display 120, and a camera 130.

[0017] The rendering device 110, display 120, and camera 130 have been depicted as separate entities, but in various variations, they can be combined in every possible way. For example, the three entities can be incorporated into a single device such as a television, with the display 120 and rendering device 110 together and the camera 130 separate, or the rendering device 110 can be separated from the display including the camera.

[0018] The display 120 operates to display the video received from the rendering device to a user (not shown).The display 120 may be, for example, a television, a computer screen, a head mounted display (HMD), a tablet computer, or a smartphone.

[0019] The camera 130, which may be a depth and color camera, operates to capture images and / or video of anything in front of its aperture. In the interactive system 100, the camera 130 will typically be arranged to capture the environment including the user. Other gesture detectors that detect the position of the user's hands are also possible, such as one or more sensors worn by the user.

[0020] Rendering device 110 can be, for example, a computer, a set-top box, a game console, a tablet computer, a smartphone, or the like. Rendering device 110 typically includes a user input interface 111, at least one hardware processor ("processor") 112, a memory 113, a network interface 114, and an output interface, such as a display interface 115 or a speaker. The interfaces are at least partially implemented in hardware. It should be understood that for the sake of brevity, components of device 110 that are not useful for understanding the principles of the present invention are not shown.

[0021] The user input interface 111 may be wireless and configured to receive input from an input device (not shown), but the user input interface may also be a touch screen, an interface to a keyboard and / or a mouse, etc. The user input interface is configured to provide the received input to the processor 112. The processor 112 is configured to execute program code instructions to perform the operations according to the reference. Figure 2 The method of the present invention described in detail. The memory 113 is configured to store program code instructions for execution by the processor 112 and to store data. The network interface 114 can be wired or wireless and is configured to allow communication between the processor 112 and at least one device such as a server (not shown) in an external network via a connection 140 to receive a computer program or content for display to a user. Such content can be a video program, which will be used as a non-limiting example. The output interface can be wired or wireless and is configured to output content for display on the display 120. As already mentioned, in the case where the device 110 is a television, the display 120 is typically included in the device. It should be noted that the television may also have a display interface for displaying information on a separate screen (not shown).

[0022] A salient aspect of the present principles is the use of an overlay on top of the VR / AR scene.

[0023] Figure 2 A method according to an embodiment of the present invention is shown. It is assumed that the user is "inside" the AR / VR scene, that is, the user is able to interact with the scene. For exemplary purposes, it is also assumed that the user has not yet interacted with any objects in the scene.

[0024] In step S210, the interactive system ( Figure 1 100) of the user interface ( Figure 1 and Figure 2 120) detects the position of the user's hand to detect when the position matches a predefined position. The predefined position can be one of multiple positions with the same meaning. An example of such a predefined position can be when the user extends his hand in front of his body.

[0025] When a match is detected, the user interface renders an overlay on the AR / VR scene in step S220 . Figure 3 A first example of such a cover 300 according to the principles of the invention is shown. The cover 300, which may be at least partially translucent, is in the example intended for interaction using two hands.

[0026] The cover 300 may include a center indicator 310 , a neutral zone 320 (diagonally shaded gray), and a crown 330 including a first locking position 341 and a second locking position 342 .

[0027] The center indicator 310 indicates the center of the crown 330. When the center indicator 310 is located within the neutral zone 320, certain user actions are inhibited, as will be described. Conversely, actions are enabled only when the center indicator 310 is located within the neutral zone 320. The neutral zone 320 may be invisible (or more transparent) when the center indicator 310 is located within, and become visible or more opaque when the center indicator 310 is located outside.

[0028] The preferably hand-shaped lock positions 341 , 342 indicate where a user must place their hand to lock or unlock a scene or object, as will be described.

[0029] The positions of the user's hands are represented by the right hand pointer 351 and the left hand pointer 352 shown in exemplary positions.

[0030] In step S230, the user interface waits for the user to lock onto the object through a specific gesture, which in this example places the hand above the lock position 341, 342 (or at least substantially above it) for a given time period; in other words, to place the hand indicator 351, 352 above the lock position 341, 342 for a given time period, which can be 0.5 seconds, 1 second, 1.5 seconds, etc.

[0031] In one embodiment, when there is no lock, the lock position 341, 342b flashes. In addition, the user interface can render text to inform the user how to lock onto the object.

[0032] While locked onto an object, the user interface preferably substantially continuously detects and renders the position of the center indicator 310 in step S240. Specifically, the user interface can detect whether the center indicator 310 is located inside or outside the neutral zone 320. When locked, the center indicator 310 generally corresponds to the center of the user's hands, i.e., the center of the hand indicators 351 and 352, possibly with some hysteresis. It should be noted that the crown 330 generally follows the movement of the center indicator 310.

[0033] Figure 4 A second example of a cover 400 according to the principles of the present invention is shown. Figure 3 300, except that the lock positions 341, 342 have been filled in to indicate lock onto an object or scene, and the center indicator 310 and crown 330 have followed the user's hand so that the center indicator 310 is now outside the neutral zone 320 (note that this appears in the figures as if the neutral zone has moved).

[0034] In step S250 , the user interface enables and / or disables user actions based on the position of the center indicator 310 relative to the neutral zone 320 .

[0035] For example, when located internally, enabling user actions may include:

[0036] zooming in or out (i.e., making an object closer or further away) by extending or contracting the hands, respectively (possibly limited to when the hand pointers 351, 352 are at substantially the same height),

[0037] as well as

[0038] Rotating the object by corresponding circular movements of the hands (as they would on a steering wheel), possibly limited to when the hand indicators 351 , 352 are inside the crown 330 ; and

[0039] When external, enabling user actions can include:

[0040] Rotate the object horizontally by moving your hand in the corresponding horizontal direction, or rotate the object vertically by moving your hand in the corresponding vertical direction, or

[0041] • The camera (ie, the user's viewpoint) moves around the scene.

[0042] As will be appreciated, the neutral zone may make the AR / VR scene less prone to accidental rotation, for example, because small horizontal or vertical hand movements will have no effect when the center indicator 310 is in the neutral zone 320 .

[0043] The user interface can indicate the possibility by, for example, rendering an icon indicating that the action is enabled. Alternatively, the icon can be rendered in a different aspect (e.g., grayed out or faded) than that used to disable the action, as is conventional in user interfaces. The icon can be rendered for a limited period of time before disappearing or fading. Preferably, the icon is rendered in a peripheral area so as not to obscure a large portion of the scene or object.

[0044] It should be noted that other user actions such as moving an object may be implemented by the user interface.

[0045] In step S260, the user interface responds to the user actions tracked by hand position detection. The user actions may include the user actions described with reference to step S250. Additional actions may include unlocking (described with reference to step S270) and hiding and showing the overlay 300 when the user crosses their hands, i.e., bringing the right hand to the left of the left hand (and it should be noted that this can be disabled when the user performs a rotation by a circular movement of the hand to enable a large circular rotation, such as 180°). Alternatively, the hand indicators 351, 352 can be prevented from switching sides so that the left side is always on the left, or in the same vertical position as the right side (except, for example, when performing a circular movement).

[0046] It should be noted that as the center indicator 310 progresses through the neutral zone 320 to the other side, horizontal or vertical rotation of the object may be temporarily disabled.

[0047] In step S270, the user interface detects unlocking of the object or scene. This can be accomplished by the user placing hand indicators 351, 352 over lock positions 341, 342 for a given period of time. As can be seen, the user can do the same for locking onto an object or scene. Unlocking may also require that the center indicator 310 be within the neutral zone 320.

[0048] In a variant adapted for one-handed use, the overlay may include a single locking position over which the user should place the hand pointer for a given time in order to lock the scene or object, while unlocking is the same way. In a variant, the enabling user action may be limited to moving objects.

[0049] It will therefore be appreciated that the principles of the present invention may be used to provide an AR / VR system that provides information to a user about enabled user actions and also provides an AR / VR experience that is less susceptible to "jitter" caused by small hand movements through the use of a neutral zone.

[0050] Should be understood that the elements shown in the drawings can be implemented by various forms of hardware, software or a combination thereof. Preferably, these elements are implemented with a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory and input / output interface.

[0051] This description illustrates the principles of the present disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the present disclosure and are included within its scope.

[0052] All examples and conditional language recited herein are intended for educational purposes to help readers understand the principles of the present disclosure and the concepts of the inventors' further technical contributions, and should be construed as not being limited to such specifically recited examples and conditions.

[0053] In addition, all statements of the principles, aspects, and embodiments of the present disclosure, as well as specific examples thereof, set forth herein, are intended to encompass structural and functional equivalents thereof. Additionally, it is intended that such equivalents include currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0054] Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual diagrams of illustrative circuitry embodying the principles of the present disclosure. Similarly, it will be appreciated that any flow charts, flowcharts, and the like represent various processes that may be substantially represented in a computer-readable medium and executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0055] The functions of the various elements shown in the drawings may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. Furthermore, explicit use of the term "processor" or "controller" should not be construed to refer only to hardware capable of executing software, and may implicitly include, but not be limited to, digital signal processor (DSP) hardware, read-only memory (ROM), random access memory (RAM), and non-volatile storage devices for storing software.

[0056] Other conventional and / or custom hardware may also be included. Similarly, any switches shown in the figures are conceptual only. Their functions may be performed through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, as will be more specifically understood from the context, the particular technique being selectable by the implementer.

[0057] In the claims herein, any element expressed as a means for performing a specified function is intended to encompass any manner of performing that function, including, for example: a) a combination of circuit elements that perform that function; or b) any form of software, thus including, firmware, microcode, etc., combined with appropriate circuitry for executing that software to perform that function. The disclosure as defined by such claims resides in the fact that the functionality provided by the various recited means is combined and taken together in the manner invoked by the claims. Therefore, any means that can provide those functions are considered equivalent to those shown herein.

Claims

1. A method for rendering a virtual reality or augmented reality environment in a device, the method comprising, in at least one hardware processor of the device: Rendering an overlay on the display; The method is characterized in that it further comprises: For a locked object in the virtual reality or augmented reality environment, a first set of user actions is enabled only when the position of the center indicator of the overlay is within the neutral zone of the overlay, and a second set of user actions is enabled only when the position of the center indicator is outside the neutral zone, and the position of the center indicator is calculated as the center of the detected position of the user's hand. 2 . The method of claim 1 , further comprising rendering the overlay in response to detecting a match between the detected hand position and at least one predefined position. The method of claim 1 , further comprising performing an enabling user action in response to detecting user hand movement.

4. The method of claim 1 , wherein the cover includes the center indicator, the neutral zone, a crown, a first locking position and a second locking position, and a right-hand indicator and a left-hand indicator, wherein the first locking position and the second locking position indicate positions where the user should superimpose the right-hand indicator and the left-hand indicator to lock onto an object or scene or unlock the locking portion onto an object or scene. 5 . The method of claim 1 , wherein the first set of user actions includes at least one of zooming in, zooming out, and rotating in the plane of the display, and the second set of user actions includes at least one of rotating horizontally, rotating vertically, and moving. 6 . The method of claim 1 , further comprising rendering a first icon having a first visual aspect indicating an enabled action and a second icon having a second visual aspect for disabling the action.

7. An apparatus for rendering a virtual reality or augmented reality environment, the apparatus comprising at least one hardware processor configured to: Rendering an overlay on the display; The device is characterized in that the at least one hardware processor is also configured to: for a locked object in the virtual reality or augmented reality environment, enable a first set of user actions only when the position of the center indicator of the overlay is within the neutral zone of the overlay, and enable a second set of user actions only when the position of the center indicator is outside the neutral zone, and the position of the center indicator is calculated as the center of the detected position of the user's hand. 8 . The apparatus of claim 7 , wherein the at least one hardware processor is further configured to calculate the position of the center indicator based on a detected position of a user's hand.

9. The apparatus of claim 7, wherein the first set of user actions comprises at least one of zooming in, zooming out, and rotating in the plane of the display, and the second set of user actions comprises at least one of horizontally rotating, vertically rotating, and moving.

10. The apparatus of claim 7, wherein the at least one hardware processor is further configured to render a first icon having a first visual aspect indicating an enabled action and a second icon having a second visual aspect for disabling an action.

11. The device of claim 7, wherein the device comprises at least one of the display and a camera configured to capture a position of the user's hand. 12 . The apparatus of claim 7 , wherein the at least one hardware processor is further configured to perform an enabling user action in response to detecting user hand movement.

13. A non-transitory computer-readable medium storing program code instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images