System, device and method for providing user interface of virtual reality environment

Abstract

The present application relates to systems, devices and methods for providing a user interface for a virtual reality environment. A device that can be connected to a near-eye display or to a virtual reality helmet and can be connected to a portable computing platform with a processor is provided herein. The device may include: an illuminator configured to illuminate the vicinity of a user wearing a helmet or a near-eye display with patterned light; and an IR camera configured to capture reflections of the patterned light from at least one object located near the user, wherein the processor is configured to: establish data and power connections between the device and the portable computing platform and the near-eye display; and generate a depth map of the object based on the reflections.

Description

[0001] This application is a divisional application of an application filed on December 18, 2015, with application number 201580074404.8 and invention name “System, device and method for providing a user interface for a virtual reality environment”. Technical Field

[0002] The present invention relates generally to virtual reality (VR) environments and, more particularly, to apparatus and methods for enabling a natural interface with a VR environment using body gestures and / or postures. Background Art

[0003] Before setting forth the background of the present invention, it may be helpful to set forth definitions of certain terms that will be used hereinafter.

[0004] The term "virtual reality" (VR) as used herein is defined as a computer-simulated environment that can simulate the physical presence of places in the real or imagined world. Virtual reality can recreate sensory experiences, including virtual taste, sight, smell, hearing, touch, etc. Many traditional VR systems use near-eye displays for presenting 3D virtual environments.

[0005] As used herein, the term "augmented reality" (AR) is defined as a vivid direct or indirect view of a physical, real-world environment, elements of which are enhanced (or supplemented) by computer-generated sensory input (e.g., sound, video, graphics, or GPS data). This is related to the broader concept of mediated reality, in which the view of reality is modified (perhaps even diminished rather than enhanced) by computers.

[0006] The term "near-eye display" as used herein is defined as a device that includes a wearable projection display that is typically stereoscopic in that each eye is presented with a slightly different field of view in order to create a 3D perception.

[0007] The term "virtual reality helmet", sometimes referred to as "goggles", is a surround visual interface that displays computer output. Typically, the computer display information is presented as a three-dimensional representation of the real world environment. In addition to a simple structure for holding a computer display (perhaps in the form of a smartphone), the goggles may or may not include optical devices.

[0008] Figure 1 A conventional VR system according to the prior art is shown, wherein a user 10 wears a head mounted stereoscopic display 12 (e.g., Oculus Rift TM), which projects a composite image 16 of the scene onto each eye of the user 10. Typically in VR, each eye receives the composite image 16 at a slightly different angle to create a 3D perception in the user's brain (stereo pairs are not shown for simplicity). In addition, the head-mounted display 12 can be provided with sensors such as accelerometers or gyroscopes (not shown) that can detect the user's viewing angle or gaze direction in real time. The VR system can therefore adjust the image 16 to fit the user's head orientation. Because this adjustment is made in real time, the illusion of virtual reality can be achieved that mimics head movements in the real world.

[0009] In addition to tracking the user's viewpoint as described above, the VR system provides further interaction with the virtual world via an input device such as a joystick 14 (or a mouse, touchpad, or even a keyboard). Such an input device may enable the user 10 to perform various actions. For example, a cursor 18 indicated on an image 16 on a VR scene may be controlled by the user 10 via the joystick 14. Naturally, the use of such a conventional input device would disrupt the overall experience of the VR user. In addition, the use of the cursor 18 limits the user experience to a standard computer user interface known from a personal computer. Summary of the invention

[0010] According to some embodiments of the present invention, a user may wear a near-eye display, and a device according to an embodiment of the present invention may be mounted on the near-eye display as an accessory. The device may include an illuminator (e.g., a laser emitter) and a capture unit (e.g., a camera) having a field of view wide enough to include any surroundings of the user. In operation, the near-eye display is configured to project a synthesized scene onto both eyes of the user. The illuminator may illuminate the user and / or the vicinity of the user with patterned light. Reflections of the patterned light may be captured by a capture device and then analyzed by, for example, a computer processor that may be located in a smartphone as part of the near-eye display to generate a three-dimensional visual representation of the position and orientation of an object such as a user's hand that changes in real time.

[0011] In another embodiment, the device can benefit from the camera of a smartphone, which can be configured by dedicated software to capture images of a hand or other gesture object controlled by the user. At the same time, the processor of the smartphone is configured to calculate a depth map of the same hand or gesture object based on the data captured by the device. The processor is also configured to merge the data of the depth map and the visible image into a 3D image of the hand that can be imported into the corresponding position of the virtual image. In this way, the user's actual hand is imported into the VR scene while maintaining its 3D properties.

[0012] This application also includes the following:

[0013] 1) A device that can be connected to a virtual reality helmet and can be connected to a portable computing platform having a processor, the device comprising:

[0014] an illuminator configured to illuminate the vicinity of a user wearing the virtual reality helmet with patterned light; and

[0015] a camera configured to capture reflections of the patterned light from at least one object located near the user,

[0016] Wherein, the processor is configured to:

[0017] establishing a data and power connection between the device and the portable computing platform; and

[0018] A depth map of the object is generated based on the reflections.

[0019] 2) The apparatus of 1), wherein the illuminator and the camera operate in the infrared (IR) range.

[0020] 3) The device according to 1), wherein the processor is configured to:

[0021] generating a 3D image of the at least one object based on the depth map and the 2D image of the at least one object; and

[0022] The 3D image of the at least one object is superimposed on a virtual reality (VR) scene generated by the processor and viewable by the user.

[0023] 4) The device according to 1), wherein the total power consumption of the device is approximately less than 1 watt.

[0024] 5) A device capable of being connected to a virtual reality helmet and capable of being connected to a portable computing platform having a processor and a camera, the device comprising:

[0025] an illuminator configured to illuminate a vicinity of a user wearing the near-eye display with patterned light,

[0026] Wherein, the processor is configured to:

[0027] establishing a data and power connection between the device and the portable computing platform;

[0028] instructing the camera to capture reflections of the patterned light from at least one object located near the user, and

[0029] A depth map of the object is generated based on the reflections.

[0030] 6) The device according to 5), wherein the total power consumption of the device is approximately less than 500 milliwatts.

[0031] 7) The device according to 5), wherein the portable computing platform and the virtual reality helmet are integrated into a single component.

[0032] 8) The apparatus of 1), wherein the portable computing platform further comprises a visible light camera, and wherein the 2D image of the at least one object is obtained from the visible light camera of the portable computing platform.

[0033] 9) The apparatus according to 3), further comprising: a visible light camera, and wherein the 2D image of the at least one object is obtained from the visible light camera.

[0034] 10) The device according to 1), wherein establishment of data and power connections between the device and the portable computing platform is performed as "plug and play".

[0035] 11) The apparatus according to 1), wherein the at least one object is a body part of the user.

[0036] 12) The device according to 3), wherein superimposing the 3D image of the at least one object on a virtual reality (VR) scene further comprises: aligning the 3D image of the at least one object with the user's perspective of the VR scene.

[0037] 13) The apparatus of 1), wherein the portable computing platform is selected from the group consisting of: a smartphone, a laptop personal computer (PC); and a tablet PC.

[0038] 14) The apparatus of 1) wherein the illuminator and the camera present a horizontal field of view of approximately 55 degrees and a vertical field of view of approximately 43 degrees in a "landscape" configuration, and wherein the illuminator and the camera present a horizontal field of view of approximately 43 degrees and a vertical field of view of approximately 55 degrees in a "portrait" configuration.

[0039] 15) A method comprising:

[0040] connecting a device to a virtual reality helmet and a portable computing platform having a processor, the device including an illuminator and a camera;

[0041] establishing a data and power connection between the device and the portable computing platform;

[0042] illuminating a vicinity of a user wearing a near-eye display with patterned light;

[0043] capturing a reflection of the patterned light from at least one object located proximate to the user; and

[0044] A depth map of the object is generated based on the reflections.

[0045] 16) The method according to 15), wherein the illuminator and the camera operate in the infrared (IR) range.

[0046] 17) The method according to 15) further includes: generating a 3D image of the at least one object based on the depth map and the 2D image of the at least one object; and superimposing the 3D image of the at least one object on a virtual reality (VR) scene generated by the processor and viewable by the user.

[0047] 18) The method of claim 16), wherein the total power consumption of the device is approximately less than 1 Watt.

[0048] 19) A method comprising:

[0049] connecting a device to a virtual reality helmet and connecting the device to a portable computing platform having a processor and a camera, the device including an illuminator configured to illuminate a vicinity of a user wearing the virtual reality helmet with patterned light;

[0050] establishing a data and power connection between the device and the portable computing platform; and

[0051] instructing the camera to capture reflections of the patterned light from at least one object located near the user, and

[0052] A depth map of the object is generated based on the reflections.

[0053] 20) The method according to 19), wherein the total power consumption of the device is approximately less than 500 milliwatts.

[0054] 21) The method according to 19), wherein the portable computing platform and the virtual reality helmet are integrated into a single component.

[0055] 22) The method of 19), wherein the portable computing platform further comprises a visible light camera, and wherein the 2D image of the at least one object is obtained from the visible light camera of the portable computing platform.

[0056] 23) The method according to 19) further includes: generating a 3D image of the at least one object based on the depth map and the 2D image of the at least one object; and superimposing the 3D image of the at least one object on a virtual reality (VR) scene generated by the processor and capable of being viewed by the user.

[0057] 24) A method according to 19), wherein establishment of data and power connections between the device and the portable computing platform is performed as "plug and play".

[0058] 25) The method according to 19), wherein the at least one object is a body part of the user.

[0059] 26) The method according to 23), wherein superimposing the 3D image of the at least one object on a virtual reality (VR) scene further comprises: aligning the 3D image of the at least one object with the user's perspective of the VR scene.

[0060] 27) A device capable of being connected to a near-eye display and capable of being connected to a portable computing platform having a processor, the device comprising:

[0061] an illuminator configured to illuminate a vicinity of a user wearing the near-eye display with patterned light; and

[0062] a camera configured to capture reflections of the patterned light from at least one object located near the user,

[0063] Wherein, the processor is configured to:

[0064] establishing data and power connections between the device, the near-eye display, and the portable computing platform; and

[0065] A depth map of the object is generated based on the reflections.

[0066] 28) The device according to 1), wherein the processor is configured to:

[0067] generating a 3D image of the at least one object based on the depth map and the 2D image of the at least one object; and

[0068] The 3D image of the at least one object is superimposed on a virtual reality (VR) scene generated by the processor and viewable by the user through a near-eye display.

[0069] These additional and / or other aspects and / or advantages of embodiments of the invention are set forth in the detailed description which follows; may be inferred from the detailed description; and / or may be learned by practice of embodiments of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] For a better understanding of embodiments of the invention and to show how the invention may be practiced, reference will now be made, by way of example only, to the accompanying drawings in which like reference numerals refer to corresponding elements or parts throughout.

[0071] In the attached picture:

[0072] Figure 1 A diagram showing a virtual reality system and its environment according to the prior art;

[0073] Figure 2 is a perspective view showing a device attached to its surroundings according to some embodiments of the present invention;

[0074] Figure 3 is a diagram illustrating a non-limiting implementation of an apparatus according to some embodiments of the present invention;

[0075] Figure 4 is a block diagram illustrating an apparatus according to some embodiments of the present invention;

[0076] Figure 5A and 5B is a diagram showing an aspect of an embodiment according to the present invention;

[0077] Figure 6 is a diagram showing a device operating in its surrounding environment according to some embodiments of the present invention;

[0078] Fig. 7A , 7B and 7C are block diagrams of possible patterned light generators according to embodiments of the present invention; and

[0079] Figure 8 is a diagram illustrating an aspect of patterned light according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] With specific reference now to the detailed drawings, it should be emphasized that the particular examples shown are exemplary in nature and are presented only for the purpose of illustrative discussion of the preferred embodiments of the present technology, and in order to provide what is believed to be the most practical and easy to understand description of the principles and conceptual aspects of the present technology. In view of this, no attempt is made to show the structural details of the present invention in more detail than is necessary for a basic understanding of the present technology, and the description with reference to the drawings makes it clear to those skilled in the art how the various forms of the present invention can be implemented in practice.

[0081] Before describing in detail at least one embodiment of the present technology, it should be understood that the present invention is not limited to the details of the construction and arrangement of the components set forth in the following description or shown in the accompanying drawings. The present technology may be applicable to other embodiments or practiced or implemented in various ways. Likewise, it should be understood that the words and terms used herein are for descriptive purposes and should not be considered as limiting.

[0082] Figure 2 2 is a perspective view of a device 210 packaged and attached to its direct environment 200, which is a virtual reality (VR) helmet 240 (goggles) configured for use with a mobile device such as a smart phone 220, according to some embodiments of the present invention. The device 210 may include an illuminator (e.g., a laser emitter) such as an infrared (IR) pattern illuminator 212, a camera such as an IR camera 214. The device 210 may further include a pre-processor (not shown) configured to perform initial image processing. The device 210 is also configured to physically and electronically interface with the smart phone 220 and with a near-eye display 240 that together forms a VR helmet. Such a VR helmet (goggles) can be arranged for use with a smart phone, as known in the art, and typically includes an optical device (not shown - facing the opposite side) that can transmit the display of the smart phone, a sleeve 250 for accommodating the smart phone 220 (which may include a camera 222), and a strap 230 for fastening the VR helmet (goggles) to the user's head. However, it should be understood that the device 210 may be used with a mobile device such as the Samsung Gear VR TM and Oculus Rift TM Near-eye display interface connection.

[0083] In operation, device 210 can be used as a means to enhance a VR headset by adding the ability to view 3D images of a user's hands or other body parts within a VR scene. This is performed by device 210, which provides the necessary hardware to attach to a near-eye display and the necessary software executed on a VR device processor to control the device and analyze the data captured by it. Economically, some embodiments of the present invention eliminate the need for VR-specific equipment and will further save costs.

[0084] Advantageously, the device 210 provides only minimal hardware, such as a patterned light illuminator and a camera for capturing the reflection of the pattern. An interface to the smartphone 220 can provide the required power and transmit the acquired data to the smartphone 220, which is performing all the processing, utilizing its computing power. Thus, according to some embodiments of the present invention, installing a VR system becomes very easy and uses off-the-shelf components such as smartphones and goggles.

[0085] Figure 3 301 is a mechanical diagram of a non-limiting exemplary embodiment of a device according to some embodiments of the present invention. The perspective view shows a printed circuit board (PCB) 301 on which components are mounted and can be physically attached to a base unit 305. The main components of the exemplary implementation are a laser emitter 360 coupled to a laser driver 320, which is configured to feed sufficient power and possible modulation to the laser emitter 360. A camera 330 (which may be a VGA camera) is configured to receive reflections of patterned light. According to one embodiment, the PCB may include an IR filter 350 covering the camera 330. In some embodiments, data (such as data related to patterned light reflections received by the camera 330) is initially processed by a signal processor 310, for example, and then transmitted to a third party processor (not shown) via a port (such as a USB port 340), for example, the data can be transmitted to a USB port of a smart phone via a USB protocol and interact with an application executed on the processor of the smart phone. Power for the device can also be provided via the USB port 340. Sockets for additional cameras and various integrated circuits (ICs) such as a initial image processor (TIP) are also attached to the PCB and are shown here.

[0086] In some embodiments, the camera 330 can be omitted from the device and the camera of an external portable computing platform (smartphone) can be used instead. The inventors have tested several prototypes of the device and have measured a low power consumption of 1 Watt for the device with the camera 330, and a power consumption of 550 milliwatts or even lower for the device without the camera 330.

[0087] Figure 4 4 is a block diagram illustrating the architecture of the device described above and its immediate environment according to some embodiments of the present invention. The device 400 may include an IR illuminator configured to illuminate a scene with patterned light; an IR camera 406 configured to receive reflections of the patterned light and a preprocessor configured to perform initial processing of data from the IR camera 406. The device 400 is also configured to interface with a portable computing platform such as a smart phone 410, which may include a processor 414 and software modules associated with the device 400 and executed by the processor 414. The smart phone 410 is further connected to a near-eye display 420, which is configured to present a synthetic scene to a user and also present a 3D image of a body part (e.g., a hand) of the user. It should be noted that the platform can also be a laptop personal computer (PC), a tablet computer, etc.

[0088] In operation, the reflected IR pattern illuminated by the illuminator 402 is captured by the IR camera 406, and after some initial processing by the pre-processor 404, the data is transmitted to the processor 414 and the dedicated software 416 of the smart phone 410, and the processor 414 generates a depth map of the user's body part based on the reflected pattern. The visible light camera 412 can be configured to capture a 2D color image of the same body part (e.g., hand) of the user illuminated by the patterned light. Thus, the processor 414 can use the depth map and the 2D color image of the body part to generate a 3D color image of the captured body part. Finally, the processor 414 is configured to overlay the generated 3D color image of the captured body part onto the synthesized VR scene, so that the user will be able to view the VR scene and his or her captured body part (e.g., hand) as a 3D color image through the near-eye display 420, just as it is positioned and oriented in real life.

[0089] Figure 5A and 5B is a diagram illustrating an aspect of some embodiments according to the present invention. A user 500 is seen wearing a device 510A according to some embodiments of the present invention. It should be noted that for simplicity, only the device is shown here without its accompanying helmet. Figure 5A The fields of view of the illuminator and IR camera of device 510A are shown.The area covered by patterned light 530A, shown in the portrait configuration, substantially overlaps with the area covered by IR camera 520A. Figure 5B A similar overlapping field of view is shown for a landscape configuration, but with the IR camera rotated 90°, and the illuminator may also be rotated or alternatively have its pattern generating surface stretched to effectively become landscape. The use of a landscape orientation is advantageous for VR applications where the full range of the hand is important for achieving and monitoring natural postures and gestures. During experiments, the inventors have found that in a portrait configuration, a horizontal field of view of approximately 43° and a vertical field of view of approximately 55° for embodiments of the present invention produced good results. Similarly, a horizontal field of view of approximately 55° and a vertical field of view of approximately 43° produced good results for a landscape configuration. It should be understood that other fields of view may be used with other devices in accordance with other embodiments of the present invention.

[0090] Figure 6 6 is a schematic diagram showing a device according to an embodiment of the present invention in its common environment. The user 10 wears a near-eye display 630, and the device 600 can be mounted on the near-eye display 630 as an additional device as described above. The device 600 may include an illuminator (e.g., a laser emitter) 620 and a capture unit (e.g., a camera) 610 as described above, and its field of view is wide enough to include any surroundings of the user.

[0091] In operation, the near-eye display 630 is configured to project a synthesized scene onto both eyes of the user 10. The illuminator 610 may illuminate the vicinity of the user 10 with patterned light 624. Reflections of the patterned light may be captured by the capture device 610 and then analyzed by a computer processor (not shown here), which may be located, for example, on a smartphone attached to the near-eye display 630. The smartphone's visible light camera (or alternatively a component of the device 600) is configured to capture a 2D image of the user's hand or other gesture object controlled by the user 10. At the same time, the smartphone's processor is configured to calculate a depth map of the hand or gesture object, and then merge the data of the depth map and the visible image into a 3D image of the hand 662 at the corresponding position that can be imported into the virtual image 660. In this way, the 3D image of the user's 10 hand is overlaid on the VR scene while maintaining its 3D properties.

[0092] Figures 7A to 7C is a diagram showing various embodiments of implementing an illuminator (patterned light source) in the form of a multi-line pattern generator. It therefore comprises a laser source and an optical device that transforms the laser beam into multiple lines. To provide more details on how structured light can be achieved, the entire contents of the following applications are incorporated herein by reference by way of illustration only: U.S. Patent Application No. 13 / 497,586, WIPO Patent Application Publication No. WO 2013088442 and U.S. Provisional Case Application Nos. 61 / 863,510, 61 / 894,471 and 61 / 926,476.

[0093] Fig. 7A The configuration of a pattern generator system 700A according to a first embodiment of the present invention is shown. The system 700A may include a light source 710A such as a laser diode light source and a single optical lens element 720A. The optical lens element 720A may provide the following three functions (not specifically in this order) to generate a pattern such as a multi-line pattern in space or on a surface or object: a) line generation, b) multiplication, and c) collimation.

[0094] According to a first embodiment of the present invention, a single optical lens element 720A is provided, which includes at least two surfaces (a first optical surface 725A and a second optical surface 735A). In a first cross section such as a YZ cross section of the first optical surface, the optical lens element 720A has a positive optical power. The positive optical power is used to collimate the laser light in the slow divergence portion.

[0095] In a second cross-section, such as an XZ cross-section of the first surface 725A or the second surface 735A of the lens, the optical lens element 720A has a line generator. The line generator can be in the form of a positive optical power (e.g., an aspheric lens, a cylindrical lens, or a diffractive element) or a negative optical surface or a combined negative / positive surface, etc. The single optical element 700A also includes a beam splitting element formed in a first cross-section, such as a YZ cross-section of the second surface 735A.

[0096] Figure 7B 700B according to a second embodiment of the present invention. The system 700B may include a light source 710B such as a laser diode light source and two optical lens elements (a first lens element 720B and a second lens element 740B). In at least one cross section (e.g., fast axis or slow axis), for example, in a YZ cross section of a first surface 725B of the first lens, the element 720B has positive optical power, and in another cross section of the first optical element (i.e., an XZ cross section), a multiplication function is provided in the second surface 727B for splitting the light beam provided by the laser source 710B.

[0097] Adjacent or near the first lens element 720B, a second lens element 740B is disposed that is configured to generate a pattern such as a line pattern. The line pattern may be disposed at a first cross section of a first surface 745B of the second element 740B, for example in a YZ cross section of the first surface 745B.

[0098] Figure 7C The configuration of a pattern generator system 700C according to a third embodiment of the present invention is shown. System 700C may include a light source 710C such as a laser diode light source and two optical lens elements (a first lens element 720C and a second lens element 730C). In at least one cross section, for example, in a YZ cross section of a first surface of a first lens, element 720C has positive optical power.

[0099] Adjacent to or near the first lens element 720C, a second lens element 730C is provided for performing two functions: 1) line generation, and 2) multiplication. For example, a pattern such as a line pattern is formed at the multiplication function of a first cross section (i.e., a YZ cross section) of the first surface 745B and another cross section (i.e., an XZ cross section) of the second surface 755B.

[0100] According to some embodiments of the present invention, the first element and the second element may be coupled to each other by welding using gluing techniques known in the art.

[0101] According to another embodiment of the invention, the diffractive surface of the second lens element faces the surface of the first lens element to protect the sensitive diffractive surface and prevent any contact of unwanted external elements with the diffractive surface.

[0102] According to some embodiments of the present invention, positive optical surfaces, combined negative / positive surfaces or diffractive surfaces may be used to form the line generating function.

[0103] According to an exemplary embodiment, tracking of hand movements is performed using a light pattern designed to detect hand movements (e.g., fine movements of fingers and thumbs). The use of structured light may be, for example, as disclosed in WIPO PCT Patent Application Publication No. WO 2013 / 088442, the entire contents of which are incorporated herein by reference.

[0104] Specifically designed light patterns allow tracking of movements, even in 2D video data, which differs from 3D depth maps, without easily separating the hands from the rest of the body.

[0105] Optionally, the light pattern can be specifically designed to track the movement of the fingers of the hand in two-dimensional video data (e.g., video images transmitted from a conventional camera stream). More specifically, the light pattern can be designed so that the fingers (i.e., fingers and thumb) and the palm can be detected and tracked in the two-dimensional video data based on the pattern due to the distortion of the fingers.

[0106] Optionally, the light pattern has continuous features in a first direction (e.g., X-axis) and has discontinuous (e.g., periodic) features in a direction substantially perpendicular to the first direction (e.g., Y-axis). In one example of such a pattern, the light pattern includes a plurality of stripes arranged parallel (or approximately parallel) to each other.

[0107] Figure 8 is a diagram illustrating an aspect of patterned light according to an embodiment of the present invention.

[0108] The following is a more detailed description of how the generated patterned light can be used to track gestures made by the user. According to an exemplary embodiment, a sensor (not shown here) can be positioned at a certain Y-axis distance, such as near an emitter that projects a stripe pattern onto the hand 810 and the background 820 (e.g., a table surface, a wall, etc. on which the hand is located). The position of the sensor is selected so as to create a triangulation effect between the camera, the light projector, and the light reflected back from the user's hand 810 and the background 820.

[0109] The triangulation effect results in discontinuities in the pattern at points along the stripe that have a significant depth offset from the object projected with the light pattern. The discontinuous segments divide (i.e., divide) the stripe into two or more stripe segments, such as segment 831 located at the hand, segment 832 located to the left of the hand, and segment 833 located to the right of the hand.

[0110] The bar segment produced by this depth shift can be located at the outline of the palm or finger of the user's hand, which is located between the camera and the user's body. That is, the user's finger or palm divides the stripe into two or more bars. Once such a bar segment is detected, it is convenient to follow the bar segment to the end of the bar segment.

[0111] Thus, the device can analyze the two-dimensional video data to generate clusters of segments. For example, the device can identify in the light pattern one or more clusters of segments generated by segmentation of stripes of fingers of the hand, such as cluster 841 of four segments reflected from the middle finger of the hand. Thus, the device tracks the movement of the finger by tracking the clusters of segments generated by segmentation of stripes by the finger or by tracking the segments of at least one cluster.

[0112] The clusters of segments generated by dividing (i.e., splitting) the stripes by the finger head include segments with overlap on the X-axis. Optionally, the segments in the cluster also have similar lengths (derived from the finger thickness) or are relatively adjacent in the Y-axis coordinate.

[0113] In the X-axis, the segments may completely overlap for a finger positioned straight in the XY plane, or partially overlap for a diagonally positioned finger. Optionally, the device further identifies a depth movement of the finger, such as by detecting a change in the number of segments in the tracked cluster. For example, if the user extends the user's center finger, the angle between the finger and the plane of the light projector and camera (the XY plane) changes. Thus, the number of segments of cluster 841 is reduced from four to three.

[0114] Optionally, the device further identifies in the light pattern one or more clusters of one or more segments resulting from segmentation of the stripes by the palm of a hand.

[0115] The cluster of segments generated by dividing the stripes by the palm includes an upper segment 831 that overlaps the cluster of segments of the fingers of the user's hand on the X axis. The upper segment 831 overlaps the four finger clusters on the X axis, but does not exceed the minimum and maximum X values ​​of the bottom segments of the four finger clusters.

[0116] The cluster of segments produced by the palm segmentation of the stripes further includes several segments directly below segment 831 that significantly overlap segment 831. The cluster of segments produced by the palm segmentation of the stripes further includes a longer segment extending to the bottom of the user's thumb segment cluster 851. It should be understood that the orientation of the finger and palm clusters may be different depending on the particular hand position and rotation.

[0117] Advantageously, even when source power is limited, as is the case with smartphones, the power consumption of the device is sufficiently reduced to enable uninterrupted and continuous operation. As described above, in contrast to prior art solutions involving energy blocking, some embodiments of the present invention utilize pattern generation based on interference in which energy is spatially transferred. Using interference bottom pattern generation is more energy efficient. As further explained above, the generation of the depth map is the basis segment of the reflection that enables partial pattern processing with reduced computational intensity. This also helps to reduce overall power consumption. In some embodiments, the autofocus feature of the smartphone can be used to provide initial data about the range of the object being projected with the pattern. This also helps to reduce power consumption. All of the low power features described above significantly contribute to making it possible to interface the smartphone and near-eye display as a VR helmet from a power consumption perspective.

[0118] In the above description, an embodiment is an example or implementation of the invention. The various appearances of "one embodiment", "an embodiment" or "some embodiments" do not necessarily refer to the same embodiments.

[0119] Although various features of the present invention may be described in the context of a single embodiment, features may also be provided separately or in any suitable combination. Conversely, although the present invention may be described in the context of separate embodiments herein for clarity, the present invention may also be implemented as a single embodiment.

[0120] Reference in the specification to "some embodiments," "embodiments," "one embodiment," or "other embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments of the invention, but not necessarily all embodiments of the invention.

[0121] It is to be understood that the phraseology and terminology employed herein should not be interpreted as limiting and are for the purpose of description only.

[0122] The principles and uses of the teachings of the present invention may be better understood with reference to the accompanying description, drawings and examples.

[0123] It should be understood that the details set forth herein do not constitute limitations on the application of the present invention.

[0124] Furthermore, it is to be understood that the invention can be carried out or practiced in various ways and that the invention can be implemented in embodiments other than those outlined in the above description.

[0125] It should be understood that the terms "comprises", "comprising", "consisting of" and grammatical variations thereof do not exclude the addition of one or more components, features, steps or integers or combinations thereof, and the terms will be interpreted as specifying the components, features, steps or integers.

[0126] If the specification or claims refer to "an additional" element, that does not preclude there being more than one of the additional element.

[0127] It should be understood that if a claim or specification refers to "a" or "an" element, such reference should not be construed as meaning there is only one of the element.

[0128] It should be understood that where the specification states that a component, feature, structure, or characteristic “may,” “might,” “can,” or “could” be included, that specific component, feature, structure, or characteristic is not required to be included.

[0129] Where applicable, although state diagrams, flow charts, or both may be used to describe embodiments, the invention is not limited to these diagrams or corresponding descriptions. For example, the flow need not pass through each of the boxes or states shown, or in exactly the same order as shown and described.

[0130] The method of the present invention can be implemented by executing or completing manual, automatic or selected steps or tasks.

[0131] The descriptions, examples, methods, and materials presented in the claims and specification should not be construed as limiting but merely illustrative.

[0132] Unless defined otherwise, technical and scientific terms used herein have the same meaning as generally understood by one of ordinary skill in the art to which this invention belongs.

[0133] The present invention can be implemented in testing or practice using methods and materials equivalent or similar to those described herein.

[0134] Although the present invention has been described with respect to a limited number of embodiments, these embodiments should not be construed as limiting the scope of the present invention, but rather as examples of some preferred embodiments. Other feasible variations, modifications, and applications are also within the scope of the present invention. Therefore, the scope of the present invention should not be limited by what has been described so far, but rather by the appended claims and their legal equivalents.

Claims

1. A device capable of being connected to a portable computing platform, the device comprising: an illuminator configured to illuminate a scene using structured light; a camera configured to capture a reflection of the structured light from at least one object in the scene; as well as The preprocessor is configured as: performing initial image processing of data associated with the captured reflections from the camera; as well as transmitting the initial image processed data to a processor of the portable computing platform to generate a depth map of the at least one object based on the reflections, the device communicating with the portable computing platform and a near-eye display forming a virtual reality helmet via the pre-processor; wherein the structured light comprises a light pattern having continuous features in a first direction and discontinuous features in a second direction perpendicular to the first direction; and wherein the position of the camera relative to the illuminator is selected such that the captured reflection of the structured light has at least one discontinuity along the continuous feature that is associated with at least one depth offset of the at least one object.

2. The device according to claim 1, wherein: The illuminator and the camera operate in the infrared IR range, the camera being covered by an IR filter.

3. The device according to claim 1, wherein: The device also includes: An interface is used to connect the device to the portable computing platform, and the interface provides power to the portable computing platform.

4. The device according to claim 1, wherein: The portable computing platform is selected from the group consisting of: a smart phone, a laptop personal computer (PC), and a tablet PC.

5. The device according to claim 1, wherein: The near-eye display connected to the device is configured to present a virtual reality (VR) scene and at least one 3D image of the at least one object obtained based on the depth map to a user.

6. The device according to claim 5, wherein: The at least one object includes at least one body part of the user.

7. The device according to claim 5, wherein: The field of view of the camera includes the scene surrounding the user of the near-eye display.

8. The apparatus of claim 1, wherein: The luminaire includes a light source coupled to at least one optical element, and The at least one optical element generates the structured light using light emitted from the light source.

9. A method for providing a user interface for a virtual reality environment, comprising: connecting a device to a portable computing platform, the device comprising an illuminator and a camera; illuminating a scene with structured light emitted from the illuminator; capturing, by the camera, a reflection of the structured light from at least one object in the scene; performing initial image processing of data associated with the captured reflections from the camera; transmitting the initial image processed data to a processor of the portable computing platform to generate a depth map of the at least one object based on the reflections, the device communicating with the portable computing platform and a near-eye display forming a virtual reality helmet via a pre-processor; wherein the structured light comprises a light pattern having continuous features in a first direction and discontinuous features in a second direction perpendicular to the first direction; and wherein the position of the camera relative to the illuminator is selected such that the captured reflection of the structured light has at least one discontinuity along the continuous feature that is associated with at least one depth offset of the at least one object.

10. The method according to claim 9, further comprising: Power is provided from the device to the portable computing platform via an interface connecting the device to the portable computing platform.

11. The method according to claim 9, further comprising: Structured light is generated using at least one optical element of the illuminator, the structured light having continuous features in a first direction and discontinuous features in a second direction perpendicular to the first direction, the at least one optical element comprising an aspherical lens of the illuminator.

12. The method according to claim 9, further comprising: Structured light is generated using at least one optical element of the illuminator, the structured light having continuous features in a first direction and discontinuous features in a second direction perpendicular to the first direction, the at least one optical element comprising a cylindrical lens of the illuminator.

13. The method according to claim 9, further comprising: Structured light is generated using at least one optical element of the illuminator, the structured light having continuous features in a first direction and discontinuous features in a second direction perpendicular to the first direction, the at least one optical element comprising a diffractive element of the illuminator.

14. The method according to claim 9, further comprising: Structured light is generated using at least one optical element of the illuminator, the structured light having continuous features in a first direction and discontinuous features in a second direction perpendicular to the first direction, the at least one optical element comprising a beam splitting element of the illuminator.

15. The method according to claim 9, further comprising: The structured light is generated using at least one optical element of the luminaire, the at least one optical element comprising at least one of a positive optical surface and a negative optical surface of the luminaire.

16. The method according to claim 9, wherein: The depth map includes information about at least one body part of a user wearing the near-eye display coupled to the device.

Images