Deployment of dynamic vision sensor hybrid elements, SLAM or safety flaps in methods for tracking controllers and simultaneous body tracking

Abstract

The position and orientation of the controller is determined from a known configuration of two or more light sources relative to each other and relative to the controller body and from a signal from an output of a dynamic vision sensor (DVS) generated in response to a change in light output from the light sources. The output signal is indicative of a time event at a corresponding photosensitive element in an array in the DVS and an array position of the photosensitive element. The position and orientation of the one or more objects are determined from signals generated by the two or more light sensitive elements as other light reaches the two or more light sensitive elements.

Description

Technical Field

[0001] Aspects of the present disclosure relate to game controller tracking, and in particular, aspects of the present disclosure relate to game controller tracking using dynamic vision sensors. Background Art

[0002] Modern virtual reality (VR) and augmented reality (AR) implementations rely on accurate and fast motion tracking for user interaction with the device. AR and VR often rely on information about the position and orientation of a controller relative to other objects. Many VR and AR implementations rely on inertial measurements made by accelerometers or gyroscopes within the controller combined with visual detection of the controller by an external camera to determine the controller's position and orientation.

[0003] Some of the earliest implementations use infrared light detected by an infrared camera with a defined detection radius on a game controller pointed at the screen. The camera captures images at a moderately fast rate of 200 frames per second, and the position of the infrared light is determined. The distance between the infrared lights is predetermined, and based on the relative position of the infrared light in the camera image, the position of the controller relative to the screen can be calculated. Accelerometers are sometimes also used to provide information about relative three-dimensional changes in the position or orientation of the controller. These previous implementations rely on a fixed position of the screen and the controller pointing to the screen. In modern VR and AR implementations, the screen can be placed close to the user's face in a head-mounted display that moves with the user. Therefore, having an absolute light position (also called a lighthouse) becomes undesirable because the user must set an independent lighthouse point, which requires additional setup time and limits the user's range of movement. In addition, even the moderately fast frame rate of the infrared camera at 200 frames per second is not fast enough to provide smooth feedback for movement. In addition, this simple setup itself is not suitable for use with more modern inside-out detection methods (such as room mapping and hand detection).

[0004] Recent implementations use cameras and accelerometers in conjunction with trained machine learning algorithms that are trained to detect hands, controllers, and / or other body parts. For smooth motion detection, a high frame rate camera must be used to generate image frames for body part / controller detection. This generates a large amount of data that must be processed quickly to achieve a smooth update rate. Therefore, expensive hardware must be used to process the frame data. Additionally, a large portion of the frame data in each of the frames is considered unnecessary because it is not relevant to motion tracking.

[0005] It is within this context that aspects of the present disclosure are presented. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The teachings of the present invention may be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

[0007] Figure 1 is a diagram depicting an implementation of game controller tracking using DVS with a single sensor array according to an aspect of the present disclosure.

[0008] Figure 2 is a diagram depicting an implementation of game controller tracking using DVS with a dual sensor array according to an aspect of the present disclosure.

[0009] Figure 3 is a diagram depicting an implementation of game controller tracking using a combined DVS with a single sensor array and camera according to an aspect of the present disclosure.

[0010] Figure 4 is a diagram showing a DVS tracking the movement of a game controller having two or more light sources according to an aspect of the present disclosure.

[0011] Figure 5 is a diagram depicting an implementation of head tracking or other device tracking using a controller including a DVS with a single sensor array according to an aspect of the present disclosure.

[0012] Figure 6 is a diagram depicting an implementation of head tracking or other device tracking using a game controller including a DVS with a dual sensor array according to an aspect of the present disclosure.

[0013] Figure 7 is a diagram illustrating an implementation of head tracking or other device tracking using a controller including a combined DVS with a single sensor array and camera according to an aspect of the present disclosure.

[0014] Figure 8 is a flow chart depicting a method for motion tracking with DVS using one or more light sources and a light source configuration fitting model according to an aspect of the present disclosure.

[0015] Fig. 9 is a flow chart illustrating a method for motion tracking with DVS using time-stamped light source position information according to an aspect of the present disclosure.

[0016] Fig. 10A is a diagram depicting the basic form of an RNN with a layer of nodes, each of which is characterized by an activation function, an input weight, recurrent hidden node transition weights, and output transition weights according to an aspect of the present disclosure.

[0017] Fig. 10B is a simplified diagram illustrating that an RNN according to aspects of the present disclosure can be considered as a series of nodes with the same activation function over time.

[0018] Fig. 10C Depicted are example layouts of convolutional neural networks, such as CRNNs, in accordance with aspects of the present disclosure.

[0019] Fig. 10D A flow chart depicting a method for supervised training of a machine learning neural network in accordance with aspects of the present disclosure is shown.

[0020] Fig.11A is a diagram depicting a hybrid DVS with multiple co-located sensor types in accordance with aspects of the present disclosure.

[0021] Fig. 11B is a diagram of a hybrid DVS with multiple sensor types arranged in an array in a checkerboard pattern in accordance with aspects of the present disclosure.

[0022] Fig. 11C is a schematic cross-sectional diagram of a hybrid DVS having multiple sensor types arranged in an array in a pattern in accordance with aspects of the present disclosure.

[0023] Fig.11D is a schematic cross-sectional view of a hybrid DVS having multiple filter types arranged in an array in a certain pattern in accordance with aspects of the present disclosure.

[0024] Fig.12 is a diagram illustrating a hybrid DVS including multiple sensor types with inputs split with an optical splitter in accordance with aspects of the present disclosure.

[0025] Fig.13 is a diagram depicting a hybrid DVS with multiple sensor types including inputs separated by micro-electromechanical (MEMS) mirrors in accordance with aspects of the present disclosure.

[0026] Fig.14 is a diagram illustrating a hybrid DVS with multiple sensor types including temporal filtered inputs in accordance with aspects of the present disclosure.

[0027] Fig.15 is a diagram depicting body tracking with a DVS in accordance with aspects of the present disclosure.

[0028] Fig.16A is a diagram depicting a headset with a safety shutter door in accordance with aspects of the present disclosure.

[0029] Fig. 16B is a diagram illustrating a headset with a sliding safety shutter in accordance with aspects of the present disclosure.

[0030] Fig. 16C is a diagram depicting a headset with a louvered safety shield in accordance with aspects of the present disclosure.

[0031] Fig.16D is a diagram illustrating a headset with a fabric safety shield in accordance with aspects of the present disclosure.

[0032] Fig.16E is a diagram illustrating a headset with a liquid crystal safety shield according to aspects of the present disclosure.

[0033] Fig.17 is a diagram depicting finger tracking with a DVS and a controller in accordance with aspects of the present disclosure.

[0034] FIG. 18A to FIG. 18B is a schematic diagram illustrating gaze tracking within the context of aspects of the present disclosure.

[0035] Fig.19 is a system block diagram of a system for tracking with a DVS according to aspects of the present disclosure. DETAILED DESCRIPTION

[0036] Although the following detailed description contains many specific details for illustrative purposes, it will be appreciated by any person skilled in the art that many variations and modifications to the following details are within the scope of the present invention. Therefore, the exemplary embodiments of the present invention described below are set forth without losing the generality of the claimed invention and without imposing limitations on the claimed invention.

[0037] introduction

[0038] Recently a new type of vision system called a dynamic vision system (DVS) has been developed that uses only the changes in light intensity of an array of light-sensitive pixels to resolve changes in the scene. DVS has an extremely fast update rate, and rather than providing a stream of image frames, DVS provides a near-continuous stream of the locations of changes in pixel intensity. Each change in pixel intensity can be called an event. This has the added benefit of greatly reducing irrelevant data output.

[0039] The two or more light sources may provide continuous updates about the position of the DVS camera relative to the position indicator light at an update rate determined by the speed of the flashes of light. In some embodiments, the two or more light sources may be infrared light sources and the DVS may use infrared light sensitive pixels. Alternatively, the DVS may be sensitive to the visible spectrum and the two or more light sources may be visible light sources or visible light at known wavelengths. In embodiments with a DVS sensitive to visible light, the DVS may also be sensitive to motion occurring within its field of view (FOV). The DVS may detect changes in light intensity caused by light reflections from moving surfaces. In embodiments using an infrared sensitive DVS, an infrared illuminator light may be used to detect movement in the FOV by reflection.

[0040] Implementation

[0041] Figure 1 An example of an implementation of game controller tracking using a DVS 101 with a single sensor array according to an aspect of the present disclosure is depicted. In the illustrated implementation, the DVS is mounted to a headset 102, which may be part of a head mounted display. A controller 103 including two or more light sources is within the field of view of the DVS 101. In the illustrated example, the controller 103 includes four light sources 104, 105, 106, and 107. These light sources have a known configuration relative to each other and relative to the controller 103. Here, there is a DVS with a single photosensitive array. Therefore, the four light sources can be used to accurately determine the position and orientation of the controller 103 relative to the DVS 101. The known information about the light sources may include the distance between each of the light sources relative to each of the other light sources and the position of each of the light sources on the controller 103. As shown, the three light sources 104, 105, 106 may describe a plane, and the light source 107 may be out of plane relative to the plane described by the three light sources 104, 105, 106. The light sources here have a known configuration, for example but not limited to, the first light source 104 is located on the front upper left side of the controller and the second light source 105 is located on the front upper right side of the controller, the third light source 106 is located on the rear upper left side of the controller and the fourth light source 107 is located on the bottom front center of the controller. In the case of four light sources, a DVS with a single photosensitive array may be able to determine the movement of the controller in the X-axis, Y-axis and Z-axis. In addition, an inertial measurement unit (IMU) 108 may be coupled to the controller 103. As an example, the IMU 108 may include an accelerometer configured to measure acceleration relative to one, two or three axes. Alternatively, the IMU may include a gyroscope configured to sense changes in rotation relative to one, two or three axes. In some embodiments, the IMU may include both an accelerometer and a gyroscope. The IMU 108 can be used to improve movement, position and orientation determination using information from the DVS 101 using a processor. The processor may be located in the headset 102, a game console or other computing device (not shown). The DVS 101, headset 102, and IMU 108 can be operably coupled to a processor 110, which can be located on the headset 102, the controller 103, or a separate device such as a personal computer, laptop, tablet, smartphone, or game console. The processor can implement tracking as described herein, for example, as described below with respect to Figure 4 , Figure 8 and Fig. 9 In addition, the processor 110 can control the flashing of the light sources 104, 105, 106.

[0042] During operation, a DVS 101 with a photosensitive array can detect movement of light sources 104, 105, 106, 107 with the photosensitive array, and changes in light detected by the photosensitive array can be sent to a processor. In some embodiments, the light sources can be configured to turn on and off in a predetermined pattern using, for example but not limited to, a circuit system and / or a signal from a processor. The predetermined pattern can be used by the processor to determine the identity of each light source. The identity of the light source can include a known position relative to the controller and relative to other light sources. In other embodiments, each light source can be configured to turn off and on in a predetermined pattern, and the pattern can be used to determine the identity of that particular light source. In some embodiments, the processor can fit the light sources to the known configuration of the controller to the events detected by the photosensitive array.

[0043] The DVS may have a near-continuous update rate that may be discretely approximated to about 1 million updates per second. The DVS with its high update rate may be able to resolve extremely fast flash patterns of light sources. The flash rate is limited primarily by the Nyquist frequency, which is half the sampling rate of the DVS. The light source may flash at a duty cycle that is suitable for the DVS's detection of the flashes. Generally speaking, the "on" time of the flashes should be long enough so that the DVS can always detect them. In addition, small differences in the flash rate may be detectable due to the high update rate.

[0044] The light sources 104, 105, 106, 107 may be broad visible spectrum light, such as an incandescent lamp or a white light emitting diode. Alternatively, the light sources 104, 105, 106, 107 may be infrared light, or the light sources may have a specific spectral distribution detectable by the DVS 101. The DVS 101 may include a photosensitive array configured to detect the emission spectrum of the light sources 104, 105, 106, 107. For example, but not limited to, if the light source is infrared light, the photosensitive array of the DVS may be sensitive to infrared light, or if the light source has a specific emission spectrum, the photosensitive array may be configured to have increased sensitivity to the specific emission spectrum of the light source. In addition, for example, but not limited to, the photosensitive array of the DVS may be insensitive or exclude light of other wavelengths not emitted by the light source, for example, if the light source is infrared light, the photosensitive array may be configured to detect only infrared light.

[0045] Figure 2 An example of an implementation of game controller tracking using a DVS with a dual sensor array according to an aspect of the present disclosure is shown. In this implementation, the headset 203 includes a first DVS 201 and a second DVS 202. Alternatively, the headset 203 may include a DVS with a first photosensitive array 201 and a second photosensitive array 202. The general functionality of the light source and DVS is similar to that described above with respect to Figure 1As described above. Information from a second DVS or second array may be combined with information from the first array to provide a better fit to the controller orientation and some depth information. The two DVS or photosensitive arrays may have partially overlapping fields of view, allowing the use of binocular parallax.

[0046] Two DVS or two photosensitive arrays provide binocular vision for depth sensing. This further allows for a reduction in the number of light sources. The first DVS and the second DVS or the first photosensitive array and the second photosensitive array may be separated by a known distance, such as but not limited to about 50 mm to 100 mm or greater than 100 mm. More generally, the separation is large enough to provide sufficient parallax for the desired depth sensitivity but not so large that there is no overlap between the fields of view. As shown, the controller 207 may include a first light source 204, a second light source 205, and a third light source 206. A fourth light source coupled to the controller may not be necessary because the information from the two DVS or two arrays provides sufficient information to determine the position and orientation of the controller. The controller may include an IMU 208, which may provide additional inertial information for improving position and orientation determination.

[0047] The first DVS 201, the second DVS 202, the headset 203, and the IMU 208 are operably coupled to a processor 210, which may be located on the headset 203, the controller 207, or a separate device such as a personal computer, laptop, tablet, smartphone, or game console. The processor may implement tracking as described herein, for example, as described below with respect to Figure 4 , Figure 8 and Fig. 9 In addition, the processor 210 can control the flashing of the light sources 204, 205, 206.

[0048] Although Figure 2 Two DVS or two photosensitive arrays are shown, but aspects of the present disclosure are not limited in this regard. The device may include any number of DVS or photosensitive arrays. For example, but not limited to, three DVS or a DVS with three separate photosensitive arrays may allow the use of only two light sources coupled to the controller. The third DVS or photosensitive array may not be co-linear with the other two DVS or arrays. Similar to binocular parallax, each additional DVS or photosensitive array may be separated by a known distance and have a partially overlapping field of view, allowing for greater use of the parallax effect. In addition, some embodiments may include multiple DVSs, each with multiple photosensitive arrays. For example, but not limited to, there may be two DVSs, each with two separate photosensitive arrays.

[0049] Figure 3An example of an implementation of game controller tracking using a combined DVS with a single sensor array and camera according to an aspect of the present disclosure is depicted. In this implementation, a DVS 301 is supplemented with a camera 302. The DVS 301 and the camera 302 may be coupled to a headset 303. The DVS 301 and the camera 302 may have partially overlapping fields of view or share the same field of view. The controller may include three or more light sources 304, 305, 306 on the controller 307. The DVS 301 and the camera 302 may be used together to determine the position and orientation of the controller. Frames from the camera 302 may be interpolated using events from the DVS 301. The image frames may also be used to perform simultaneous localization and mapping to improve controller orientation and position determination. Additionally, the image frames from the camera may be used to perform inside-out tracking of a user, such as hand tracking or foot tracking, using a machine learning algorithm. The IMU 308 may provide additional inertial information for further improving the determination of the position and orientation of the controller.

[0050] The first DVS 301, the second DVS 302, the headset 303, and the IMU 308 are operably coupled to a processor 310, which may be located on the headset 303, the controller 307, or a separate device such as a personal computer, laptop, tablet, smartphone, or game console. The processor may implement tracking as described herein, for example, as described below with respect to Figure 4 , Figure 8 and Fig. 9 In addition, the processor 310 can control the flashing of the light sources 304, 305, 306.

[0051] Figure 5An example of an implementation of head tracking or other device tracking using a controller including a DVS with a single sensor array according to an aspect of the present disclosure is depicted. Here, a DVS 507 is mounted to a controller 508. A headset 501 within the field of view of the DVS 507 is tracked using a light source attached to the headset. The headset 501 may include two or more light sources, here four light sources 502, 503, 504, and 505. The four light sources provide accurate information for determining position in three dimensions with a single DVS. A smaller number of light sources may be used with additional DVSs or cameras. The position of each light source relative to the headset 501 may be known to the system and may be of some importance in providing relevant information to the DVS. Here, light sources 502, 503, and 504 describe a plane; light source 505 is located out of the plane of the other light sources 502, 503, and 504. This facilitates detection of the position, orientation, or movement of the headset in three dimensions. The headset 501 may include an IMU 506 that may improve position and orientation estimates with information from the DVS 507. Additionally, the controller 508 may include an IMU 509. Information from the controller IMU 509 may be used to further improve the position and orientation determination of the headset, for example, but not limited to, the IMU information may be used to determine whether the controller is moving relative to the headset and the speed or acceleration of that movement.

[0052] The headset 501, IMU 506, DVS 507, controller IMU 509 are operably coupled to a processor 510, which may be located on the headset 501, controller 508, or a separate device such as a personal computer, laptop, tablet, smartphone, or game console. The processor may implement tracking as described herein, for example, as described below with respect to Figure 4 , Figure 8 and Fig. 9 In addition, the processor 510 can control the flashing of the light sources 502, 503, 504, 505.

[0053] Figure 6An example of an implementation of head tracking or other device tracking using a game controller including a DVS with a dual sensor array according to an aspect of the present disclosure is shown. Here, a controller 605 is coupled to two DVSs 606, 607 or a single DVS with two photosensitive arrays 606 and 607. As discussed above, the two DVSs or two photosensitive arrays may be separated by a suitable distance, such as between 500 mm and 1000 mm, and have a partially overlapping field of view. This allows depth determination using a parallax effect. In addition, a headset 601 may include three light sources 602, 603, 604 and an IMU 608. The use of two DVSs or two separate arrays 606, 607 may allow the use of less than four light sources, such as but not limited to three light sources. Two light sources 602, 603 may describe a line, and a third light source 604 may not be on a line with the other two light sources 602, 603. Information from an IMU 605 coupled to the headset 601 may be used to improve position and orientation determination.

[0054] The headset 601, DVS 606, DVS 607, and IMU 608 are operably coupled to a processor 610, which may be located on the headset 601, the controller 605, or a separate device such as a personal computer, laptop, tablet, smartphone, or game console. The processor may implement tracking as described herein, for example, as described below with respect to Figure 4 , Figure 8 and Fig. 9 In addition, the processor 610 can control the flashing of the light sources 602, 603, 604.

[0055] Figure 7 An example of an implementation of head tracking or other device tracking using a controller including a combined DVS with a single sensor array and an image camera according to an aspect of the present disclosure is depicted. Here, a DVS 707 and a camera 708 are coupled to a controller 706. The camera 708 and the DVS 707 may have partially overlapping fields of view or may have completely overlapping fields of view. In some implementations, the camera pixels and the DVS photosensitive elements may share the same photosensitive array, thereby acting as an integrated DVS and camera.

[0056] Three or more light sources 702, 703, 704 may be coupled to the headset 701. For example, but not limitation, the light sources may be integrated into the headset housing, each light source may be an LED, incandescent, halogen, or fluorescent light emitter mounted to a circuit board within or on the headset housing. In some embodiments, a single light emitter may form multiple light sources using plastic or glass light pipes or optical fibers that split the light from the single emitter into two or more light sources on the headset housing.

[0057] The three or more light sources 702, 703, 704 may be configured to turn on and off in response to an electronic signal. In some embodiments, the three or more light sources may turn on and off in a predetermined sequence. The DVS 707 may generate an event each time a light source 702, 703, 704 moves or flashes. The camera 708 generates an image frame of its field of view at a set frame rate. A high update rate DVS may allow the image frames generated by the camera to be interpolated with DVS events.

[0058] In some embodiments, the IMU 705 may also be coupled to the headset 701. The headset 701, IMU 705, DVS 707, camera 708, and IMU 608 may be operably coupled to a processor 710, which may be located on the headset 701, controller 706, or a separate device such as a personal computer, laptop, tablet, smartphone, or game console. The processor may implement tracking as described herein, for example, as described below with respect to Figure 4 , Figure 8 and Fig. 9 In addition, the processor 710 can control the flashing of the light sources 702, 703, 704.

[0059] In addition, in some embodiments, camera 708 can be a depth camera, such as a depth time-of-flight (DTOF) sensor. A DToF camera acquires a depth image by measuring the time required for light to travel from a light source to an object in a scene and back to a pixel array. As an example, and not a limitation, a DToF camera can operate using continuous wave (CW) modulation, which is an example of an indirect time-of-flight (ToF) sensing method. In a CW ToF camera, light from an amplitude modulated light source is backscattered by an object in the camera's field of view (FOV), and the phase shift between the transmitted waveform and the reflected waveform is measured. By measuring the phase shift at multiple modulation frequencies, a depth value for each pixel can be calculated. The phase shift is obtained by measuring the correlation between the transmitted waveform and the received waveform at different relative delays using intra-pixel photon mixing demodulation.

[0060] The DTOF system generally includes an illumination module and an imaging module. The illumination module consists of a light source, a driver for driving the light source at a high modulation frequency, and a diffuser for projecting a light beam from the light source to a designed field of illumination (FOI). The DToF illumination module may include one or more light sources, which may be, for example but not limited to, amplitude modulated light emitters, such as vertical cavity surface emitting lasers (VCSELs) or edge emitting lasers (EELs). The imaging module may include an imaging lens assembly, a bandpass filter (BPF), a microlens array, and an array of photosensitive elements that convert incident photon energy into electronic signals. The microlens array increases the amount of light reaching the photosensitive element, while the BPF reduces the amount of ambient light reaching the photosensitive element and the microlens array.

[0061] operate

[0062] Figure 4 A DVS according to one aspect of the present disclosure tracks the movement of a game controller having two or more light sources. A DVS 401 has a controller 402 within its field of view. As shown, the controller 402 includes a plurality of light sources coupled to the body of the controller. The plurality of light sources may be configured to be turned off and on again at a predetermined rate. Each flash of a light source within the field of view of the DVS 401 may generate one or more events 403 at the DVS. In the event 403 shown, all light sources have been changed from an off state to an on state, whereby the event shows that all light sources are in an on state. Alternatively, depending on the sensitivity of the DVS, a change in the brightness of the light source may be sufficient to trigger the event 403. It should be noted that in other embodiments, the light may be turned off and on at different rates or at different times, and thus, each event may correspond to non-all illuminated sources. The time of the event generated by the flash and its corresponding position within the array may be recorded in a memory (not shown). In some embodiments, the position and orientation of the controller 402 may be reconstructed based on one or more events from the DVS.

[0063] In some embodiments, each light source may flash in a predetermined time sequence. The DVS may output the time at which each event occurred along with each event as, for example, but not limited to, a timestamp. The time at which the event occurred may be used together with the predetermined time sequence to determine the identity of each light in the event, for example, which event corresponds to which light source position. In the example shown, one or more events 403 output by the DVS 401 show a first light 406, a second light 407, a third light 408, and a fourth light 409 detected by a photosensitive array. As discussed above, the identity of each light source may be determined based on the information output by the DVS and the predetermined flash sequence of the light source. For example, but not limited to, the photosensitive array of the DVS 401 may detect a light event 403 at time T+1, and the predetermined sequence may provide that the light source 406 is turned on at T+1, thereby determining that the light event corresponds to the light source 406. The predetermined sequence may be stored in a memory, for example, as a table listing the sequence timing and position of each light source. In some embodiments, the predetermined sequence may be encoded in the flashing of the light itself, for example, but not limited to, the sequence flashing of each light may indicate its identity. For example, a light source labeled 1 may flash in a Morse code sequence indicating the number one. The identity of the light event can then be recovered by analyzing the light event. Alternatively, the sequence information can come from the light source itself or a driver of the light source that indicates when the light source is turned on or off. Alternatively, the light source can be turned on and off simultaneously, and then a machine learning algorithm can be applied to the detected light event 403 to fit the controller gesture 402 to the event and the known configuration of the light source.

[0064] Additionally, information from the events, such as the size, intensity, and separation of the light, can be used to determine the orientation and position of the light sources. The system may have information defining the size, position on the controller body, and intensity of each light source. Based on the differences between the detected size, intensity, and separation; the position and orientation can be determined with improved accuracy. Additionally, if one or more additional DVS or photosensitive arrays are present, parallax information can be used to further enhance the position and orientation determination.

[0065] During operation, the user may change the position and orientation of the controller 410. The relatively high update rate of the DVS may allow it to capture a series of events 411 as the light event moves 412 during changes in position and orientation. The movement of the light event here is Figure 4 410. The determined position and orientation of the controller 410 may be updated as the detected light events move, or may be updated at regular intervals. The positions and times of the events detected by the light sensitive elements of the DVS due to the flashes of the first light 415, the second light 416, the third light 417, and the fourth light 418 may be fitted to the new position and orientation of the controller. Additionally, inertial information from the IMU may be used to improve the movement and position and orientation of the controller 410.

[0066] Figure 8 The flowchart in depicts a method for motion tracking with a DVS 801 using one or more light sources and a light source configuration fitting model according to one aspect of the present disclosure. In this embodiment, the light sources (depicted as LEDs) can be turned on and off simultaneously, individually, or in a predetermined sequence. The movement of the light sources or the flashing of one or more of the light sources generates an event 802 from the DVS 801. Generally, an event includes an electronic signal that relays the following information: the time interval at which the event occurred, the location within the array of light sensing elements of the DVS 801 where the event occurred, and binary data (e.g., 1 or 0) corresponding to a change in light intensity greater than a certain detection threshold. Each event can be processed at 803 to format the event into a usable form, such as but not limited to placing the event location within a data array, associating a timestamp with the event, and aggregating multiple events. As an example of an aggregated event, all events that occurred at various elements in the array within a predefined time interval can be combined into a single data structure for analysis.

[0067] The processed events may then be analyzed to associate detected DVS events with corresponding LED pulses 804. In some embodiments, each light source may be turned off and on at a unique predefined time interval, and thus, the timestamps of the aggregated events may be used to determine the predefined time interval from the events to associate a particular light source with a particular event. For example, but not limited to, the spatial pattern of aggregated events occurring within a predefined time interval or sequence of time intervals may be analyzed to determine if the pattern is consistent with a pulse of an LED. Event patterns that are too large or too small or too irregular in shape may be excluded as LED events. Additionally, the timing of the events may also be analyzed; events that are too short or too long may also be excluded.

[0068] The trained machine learning model 805 can be applied to the processed events. The model 805 can include information about the configuration of the light source, such as the size of the light source and its relative position relative to the body of the controller. The machine learning model can be trained with training event data, which has the corresponding masking position and orientation of the controller, which will be discussed in a later section. The trained machine learning model is applied to the processed event data to determine the correspondence 806 between the detected pulse 804 and the posture 808. The trained machine learning model can fit the posture 808 (e.g., position and orientation) of the controller to one or more processed events, such as the detected LED pulse 804. Alternatively, a fitting algorithm can be applied to the processed events instead of the trained model 805. The fitting algorithm can use a manually developed model of the light source to fit the position and orientation of the controller to the processed event. Alternatively, the fitting algorithm can be a hypothesis and test type algorithm that tries all possible permutations of light correspondences and finds the best fit using redundant light sources. Afterwards, a tracking / prediction algorithm can be applied to keep tracking the light sources. In addition, the predicted current posture can be used to predict the next posture 809. Inertial data from the IMU 807 may be fused 810 with the predicted pose 808 to generate a final predicted position and orientation of the controller. Fusion may be performed by a trained machine learning algorithm that is trained to use inertial data to improve controller position and orientation. Alternatively, fusion may be performed by, for example, but not limited to, a Kalman filter or nonlinear optimization.

[0069] Fig. 9A method for motion tracking with a DVS using time-stamped light source position information is shown according to an aspect of the present disclosure. In this embodiment, the timing of events output by a DVS 901 is used to determine the position and orientation of a controller. Here, the light sources are depicted as LEDs, and each LED may be turned on at different times, as depicted in the graph. A DVS having an LED within a field of view may generate an event 902 each time the LED is turned on or off. Each event may include an electronic signal corresponding to information such as a time interval at which the event occurred, a location within an array of light sensing elements of the DVS 901 where the event occurred, and binary data (e.g., 1 or 0) corresponding to a change in light intensity greater than a certain detection threshold. Each event may be processed 903 to format the event into a usable form, such as, but not limited to, placing the event location in a data array and aggregating multiple events (e.g., by combining multiple events occurring at various elements in the array within a predefined time interval into a single data structure), as discussed above. The processed events may then be analyzed to detect corresponding LED pulses 904. For example, but not limited to, the shape and size of the events can be analyzed for regularity and fit of the light source, events with shapes that are too large or too small or too irregular can be excluded as LED events, and noise suppression such as event averaging can be performed to remove random events. In addition, the timing of the events can also be analyzed; events that are too short or too long can also be excluded, or multiple events can be condensed into a single event by eliminating events that are co-localized with the initial event and are close in time but after the initial event.

[0070] Once the events are processed, the individual LED positions 905 can be determined. Determining the individual LED positions can be performed using a time sequence of LED on and off. For example, but not limited to, the time of one or more events can be compared to a known time sequence of LED flashes. The known time sequence can be, for example, a table with the LED on and off times and positions on the controller body for each LED or a timestamp from an LED driver about when each LED is turned on or off. If timestamps are used, the timestamps may be related to the LED positions on the controller body. Based on the time sequence, LED position information, and processed event information, the matching position and orientation of the controller can be determined. The IMU data 907 can be combined with previously determined LED positions and inertial data from the IMU by a Kalman filter 908. The Kalman filter can predict the position of the light source based on the inertial information from the IMU, which can be combined with the position information determined from the LED time sequence to improve the movement data and produce a final posture 909 and improve future estimates.

[0071] General Neural Network Training

[0072] According to various aspects of the present disclosure, the tracking system may use machine learning using a neural network (NN). For example, the training model 805 discussed above may use machine learning, as discussed below. The machine learning algorithm may use a training data set that may include input from the DVS, such as events or processed events with known controller positions and orientations as labels. In addition, the machine learning algorithm using the NN may perform fusion between the controller position and orientation determined based on the DVS information and the inertial information from the IMU. The training set used for fusion may be, for example, but not limited to, potential controller positions and orientations and inertial data with final positions and orientations. In some embodiments, the machine learning algorithm may be trained to perform simultaneous localization and mapping (SLAM) with a training set having objects (such as ground, landmarks, and body parts with hidden labels). The hidden labels may include the identity of the object and its relative position. As generally understood by those skilled in the art, SLAM technology generally solves the problem of building or updating a map of an unknown environment while simultaneously tracking the position of an agent within its position environment.

[0073] The NN may include one or more of several different types of neural networks and may have many different layers. By way of example and not limitation, the neural network may be composed of one or more convolutional neural networks (CNNs), recurrent neural networks (RNNs), and / or dynamic neural networks (DNNs). The motion decision neural network may be trained using the general training methods disclosed herein.

[0074] By way of example, and not limitation, Fig. 10A The basic form of an RNN that can be used, for example, in a trained model 805 is depicted. In the example shown, the RNN has a layer of nodes 1020, each of which is characterized by an activation function S, an input weight U, a recurrent hidden node transition weight W, and an output transition weight V. The activation function S can be any nonlinear function known in the art and is not limited to (hyperbolic tangent (tanh) function. For example, the activation function S may be a Sigmoid or ReLu function. Unlike other types of neural networks, the RNN has a set of activation functions and weights across the layer. As shown in FIG. Fig. 10B As shown, the RNN can be considered as a series of nodes 1020 with the same activation function over time T and T + 1. Therefore, the RNN maintains historical information by feeding the results from the previous time T to the current time T + 1.

[0075] In some embodiments, a convolutional RNN can be used. Another type of RNN that can be used is a long short-term memory (LSTM) neural network, which adds a memory block to an RNN node with an input gate activation function, an output gate activation function, and a forget gate activation function, thereby generating a gated memory that allows the network to retain some information for a longer period of time, as described in Hochreiter and Schmidhuber, "Long Short-term memory" Neural Computation 9(8):1735-1780 (1997), which is incorporated herein by reference.

[0076] Fig. 10C An example layout of a convolutional neural network (such as a CRNN) that can be used, for example, in a trained model 805 in accordance with aspects of the present disclosure is depicted. In the depiction, a convolutional neural network is generated for an input 1032 having a size of 4 units in height and 4 units in width giving a total area of ​​16 units. The depicted convolutional neural network has a filter 1033 of size 2 units in height and 2 units in width with a skip value of 1 and a channel 136 of size 9. For clarity, the input 1032 is generated for a size of 4 units in height and 4 units in width giving a total area of ​​16 units. The depicted convolutional neural network has a filter 1033 of size 2 units in height and 2 units in width with a skip value of 1 and a channel 136 of size 9. Fig. 10C Only the connection 1034 between the first column of channels and their filter windows is depicted. However, aspects of the present disclosure are not limited to such implementations. According to aspects of the present disclosure, the convolutional neural network may have any number of additional neural network node layers 1031 and may include any size of layer types such as additional convolutional layers, fully connected layers, pooling layers, maximum pooling layers, local contrast normalization layers, etc.

[0077] As in Fig. 10D As seen in , training a neural network (NN) begins by initializing the weights of the NN at 1041. Generally, the initial weights should be randomly distributed. For example, a NN with a tanh activation function should have weights distributed in and A random value between , where n is the number of inputs to the node.

[0078] After initialization, the activation function and optimizer are defined. Then, at 1042, the NN is provided with a feature vector or input data set. Each of the different feature vectors can be generated by the NN from an input with a known label. Similarly, the NN can be provided with a feature vector corresponding to an input with a known label or classification. Then, at 1043, the NN predicts the label or classification of the feature or input. At 1044, the predicted label or classification is compared with the known label or classification (also called the ground truth), and the loss function measures the total error between the prediction and the ground truth on all training samples. As an example and not limitation, the loss function can be a cross entropy loss function, a quadratic cost function, a triple contrast function, an exponential cost function, etc. A plurality of different loss functions can be used according to the purpose. As an example and not limitation, for training a classifier, a cross entropy loss function can be used, while for learning a pre-trained embedding, a triple contrast function can be used. Then, the result of the loss function is used and the NN is optimized and trained using known methods for training a neural network (such as back propagation with adaptive gradient descent, etc.), as indicated at 1045. In each training epoch, the optimizer attempts to select model parameters (ie, weights) that minimize the training loss function (ie, total error). The data is divided into training, validation, and test samples.

[0079] During training, the optimizer minimizes the loss function on the training samples. After each training epoch, the model is evaluated on the validation samples by calculating the validation loss and accuracy. If there is no significant change, the training can be stopped and the resulting trained model can be used to predict the labels of the test data.

[0080] Thus, a neural network can be trained to recognize and classify inputs with known labels or classifications. Similarly, the described methods can be used to train a NN to generate feature vectors from inputs with known labels or classifications. Although the above discussion is related to RNNs and CRNNS, these discussions are applicable to NNs that do not include recurrent or hidden layers.

[0081] Hybrid Sensors

[0082] Fig.11A1 is a diagram depicting a hybrid DVS with multiple co-located sensor types in accordance with aspects of the present disclosure. In some embodiments, a hybrid DVS may be used to combine multiple sensor types into one device. The multiple sensor types may be, for example, but not limited to, DVS infrared light sensitive elements, DVS visible light sensitive elements, DVS wavelength specific light sensitive elements, visible light camera pixels, infrared camera pixels, DTOF camera pixels. A first sensor type 1102 may be interspersed with a second sensor type 1102 on the same array 1101. For example, but not limited to, a DVS visible light sensitive element 1103 may surround a visible light camera pixel 1102, or a DVS infrared sensitive element 1103 may surround a DVS visible light sensitive element 1102, or a visible light camera pixel 1103 may surround a DVS infrared sensitive element 1102, or any combination thereof. Although a single element 1102 is shown as being surrounded by other elements 1103, aspects of the present disclosure are not limited thereto. A single element may include a group of multiple DVS photosensitive elements or camera pixels, for example but not limited to a group of four camera pixels may be surrounded by eight DVS photosensitive elements, or a camera pixel may be surrounded by eight pairs, triplets, or quadruplets of DVS photosensitive elements.

[0083] In addition, if Fig. 11B As shown, a hybrid DVS may have multiple sensor types arranged in an array in a checkerboard pattern. Here, blocks of first sensor type 1102 are evenly distributed with blocks of second sensor type 1103. Each of first sensor type 1102 and second sensor type 1103 may be different. For example, but not limited to, first sensor type 1102 may be a DVS infrared light sensitive element, and second sensor type 1103 may be a DVS visible light sensitive element.

[0084] Alternatively, the sensor type distinction may be performed by filtering. In these embodiments, one or more filters selectively transmit light to a photosensitive element located behind the one or more filters. The one or more filters may selectively transmit, for example, light of a certain wavelength or certain wavelengths or a certain light polarization that is not limited to a certain wavelength or certain wavelengths. The one or more filters may also selectively block light of a certain wavelength or certain wavelengths or a certain light polarization. The photosensitive element behind the one or more filters may be configured to be used as a different sensor type. For example, but not limited to, an infrared pass filter 1102 that only allows infrared light to pass may cover one or more sensor elements in the array 1101, and other sensor elements may be unfiltered or may be an infrared cutoff filter 1103. In another alternative embodiment, one or more filters may be, for example, but not limited to, an optical notch filter that only allows light of a certain wavelength to pass 1102, while other filters may block the specific wavelength and allow other wavelengths to pass 1103. This filtering may allow the use of wavelengths of illuminator lights for DTOF and specific wavelengths for DVS photosensitive elements, thereby reducing the possibility of false light source detection. Here, the sensor element may be any type of DVS photosensitive element or any type of camera pixel.

[0085] Patterned sensor or filter elements can be incorporated into hybrid imaging units, e.g. Fig. 11C and Fig.11D shown. Fig. 11C An example of a hybrid DVS imaging unit 1112C is shown having one or more lens elements 1114, an optional microlens array 1116, a bandpass filter 1118, and a patterned hybrid sensor array 1120. The sensor array includes DVS sensor elements 1122 and conventional imaging sensor elements 1124, which may be arranged in a pattern, e.g. Fig.11A or Fig. 11B Such an imaging unit may be used in conjunction with an illumination unit (not shown) in a DTOF sensor. Fig.11D The hybrid DVS imaging unit 1112D shown has a DVS sensor array 1126 and a patterned filter element 1118, wherein the band pass regions 1118A and the band cut-off regions 1118B are arranged in a pattern, e.g. Fig.11A or Fig. 11B Depicted.

[0086] Fig.121 is a diagram illustrating a hybrid DVS including multiple sensor types with inputs separated by an optical splitter according to aspects of the present disclosure. In this embodiment, the optical splitter 1204 may filter light based on wavelength; the array 1201 may include multiple sensor element types that are physically separated based on the wavelength or polarization of the light desired to be detected. For example, but not limited to, unpolarized white light 1205 (which is a mixture of light of at least all visible wavelengths, and in most cases, includes some infrared wavelengths) may enter the optical splitter 1204, which may be a dispersion prism, a diffraction grating, or a dichroic mirror, etc. As shown, the white light 1205 entering the optical splitter 1204 may be separated by wavelength or polarization, with some light 1206 incident on a first portion 1202 of the array and other light 1207 incident on a second portion 1203 of the array. Here, the optical splitter 1204 may be considered a filter that changes the diffraction angle based on wavelength or polarization. Although the array shown depicts the array as a single unit with a hard separation line 1203, aspects of the present disclosure are not limited thereto. The first portion 1201 of the array may have up to one millimeter of separation from the second portion 1202 of the array, and although the array is shown as being separated top and bottom, other embodiments may have portions separated horizontally, diagonally, or circumferentially. Additionally, the optical splitters herein may be combined with different filtering or different sensor configurations, such as Fig.11A and Fig. 11B As shown, additional optical wavelength separation is provided for different sensor types.

[0087] Fig.13 is a diagram depicting a hybrid DVS with multiple sensor types including inputs separated by a microelectromechanical (MEMS) mirror according to aspects of the present disclosure. Here, the MEMS mirror 1304 can oscillate between different positions at a set time to reflect light to a first portion 1301 or a second portion 1302 of the array depending on the time when the light 1305 arrives at the MEMS mirror 1304. The first portion 1301 and the second portion 1302 of the array can be physically separated 1303 from each other based on the angle of incidence of the light reflected by the MEMS mirror 1304. In this way, light can be temporally filtered between different sensor types. Such temporal filtering can be timed according to a known flash pattern of a light source on a controller or headset. For example, but not limited to, a light source on a controller or headset can be turned on for a predetermined duration at a certain interval. For example, the light source can be turned on for 60 microseconds every 100 microseconds. In this case, the MEMS mirror 1304 can be synchronized to reflect light 1305 to the DVS portion 1301 of the array for more than 60 microseconds every 100 microseconds or less to capture changes in the light source. Another reflected light 1307 is detected by the second portion 1302 of the array during the time when the light source is off and can capture ambient light for image tracking or DTOF.

[0088] Alternatively, the MEMS mirror 1304 can filter the light 1305 based on wavelength. The MEMS mirror in these embodiments can be, for example, but not limited to, a MEMS Fabry-Perot filter or a diffraction grating. The MEMS mirror can diffract light 1306 of a first wavelength range to at least a first portion 1301 of the array or diffract light 1307 of a second wavelength range to a second portion 1302 of the array.

[0089] Fig.14 1401 is a diagram illustrating a hybrid DVS with multiple sensor types having inputs including temporal filtering according to aspects of the present disclosure. Here, the filter selectively allows light to pass through the array based on time. In some embodiments, the filter can be an optical waveguide. At a first time step, the filter can allow light of a first wavelength or polarization to pass to the array 1401 and can block light of a second wavelength or polarization or other wavelengths or polarizations. At a second time step, the filter can allow the second wavelength or polarization 1402 but block light of the first wavelength or polarization or other wavelength polarizations. In this way, light can be filtered temporally, which can be used for tracking with different sensor types. For example, but not limited to, the light source coupled to the controller or headset can be infrared light or have a specific wavelength. The light source can be configured to turn on and off at specific intervals. The specific intervals at which the light source turns on and off can be a sequence or coded pattern. Temporal filtering can be activated to allow specific wavelengths to pass to the sensor and block other wavelengths during the specific interval. In addition, the switching interval of the filtering can be longer than the specific interval of the light source to take into account the travel time of the light to the sensor.

[0090] Body Tracking

[0091] Fig.151 is a diagram depicting body tracking with a DVS according to aspects of the present disclosure. As shown, a user 1501 may wear a headset 1504 with a DVS 1503. Here, the DVS is described as having two arrays or DVS units or cameras and a DVS. The user 1501 is holding two controllers 1502 with two or more light sources 1505. The DVS 1503 has the controllers 1502 and their corresponding light sources 1505 within its field of view (FOV). In addition, the DVS 1503 may have the user's appendages, such as the user's hands or arms 1507 or legs or feet 1508, within its FOV. The DVS may also have the ground or other landmarks 1509 in its FOV. For example, but not limited to, when the user moves or the light changes, the light sensitive elements of the DVS may detect reflections of light corresponding to the user's appendages or the ground or other landmarks. The camera detects the light reflections from the field of view at the frame rate of the camera. Based on the detected light reflections, the user's appendages or the ground or landmarks 1506 may be determined. In some alternative embodiments, an external DVS 1510 may be used to track a user's appendage. The external DVS 1510 may be located at a distance from the selected user so that the user's appendage fits within the field of view of the external DVS 1510. For example, but not limitation, the external DVS may be located on top of or below a television or computer monitor or other freestanding or wall-mounted display.

[0092] The machine learning algorithm is trained to determine the user's body, appendages, ground or landmarks and their relative positions and orientations based on data (such as events or frames). The machine learning algorithm can be a neural network, and the training can be similar to the above in FIG. 10A to FIG. 10D The methods discussed in the general neural network training section of . A training set including labeled events or frames or both can be used to train a neural network. The labeled events or frames or both can include, for example, but not limited to, labels for a user's body, appendages, ground, landmarks, and the relative position and orientation of the user's body, appendages, ground, and landmarks. In addition to the position and orientation determination of the controller 1502 (e.g., using SLAM), the determination of the user's body, appendages, ground, or landmarks and their relative position and orientation can be performed. Alternatively, the position and orientation of the controller can be determined using the same trained neural network that determined the labels for the user's body, appendages, ground, or landmarks and their relative position and orientation. As shown in element 1506, the model can be fit to the determined user's body and appendages to improve the determination of relative position and positioning.

[0093] Safety baffle

[0094] Body tracking combined with determination of controller position and orientation as discussed above can be used to trigger safety shutters in a VR or AR headset. Fig.16AA headset with a safety shutter door in accordance with aspects of the present disclosure is depicted. The headset 1601 may include a headband 1604, an eyepiece 1603, and a display screen 1602. The eyepiece 1603 may include one or more lenses configured to focus on the display screen 1602. The one or more lenses may be, for example, Fresnel lenses or prescription lenses. The display screen 1602 may be transparent, and an aperture 1607 in the body of the headset may allow viewing through the display screen 1602 when the safety shutter is open.

[0095] In this embodiment, the safety shutter 1606 is a door that swings open from the hole 1607 when the security system is activated. Here, the system-operated buckle 1605 interacts with the buckle 1608 on the safety shutter door 1606 to ensure that the door is closed on the hole 1607. The spring-loaded hinge 1609 can ensure that the safety shutter door 1606 opens quickly when the system-operated buckle 1605 is opened. The spring-loaded hinge can, for example, but not limited to, have a bell spring wrapped around the hinge and fixed to the door, the spring being wound when the door is closed and unwound when the door is opened. Alternatively, when the door is closed, a leaf spring can push the safety shutter door 1606.

[0096] The system operated buckle 1605 may be configured to open the buckle when the security system is activated. The security system operated buckle 1605 may include an electric motor or a linear actuator that moves the buckle. The security system may be activated when the ground or one or more landmarks are detected near the system, the user, the user's body, or the user's appendage. The security system may use the user's body, appendage, the ground or landmarks and the determination of their relative position and orientation, as discussed above. When the security system is activated, a signal may be sent to the security system operated buckle 1605 to open the buckle. When the buckle is opened, the spring loaded hinge 1609 pushes the safety shutter door 1606 open, allowing the user to see through the display 1602 and avoid the danger that aroused the security system.

[0097] Other safety shield implementations may be used. For example, Fig. 16BAn alternative headset with a sliding safety shutter according to aspects of the present disclosure is shown. Here, when the security system is activated, the safety shutter slider 1616 slides out of the hole 1607. The safety shutter slider 1616 may travel on a spring-loaded rail 1619. Alternatively, the sliding safety shutter 1616 may include a tab that travels in a slot 1619 in the body of the headset, and a spring also in the slot may push the sliding safety shutter. In some additional alternative embodiments, the spring may be omitted and gravity may operate the sliding safety shutter 1616. The spring-loaded rail 1619 may push the sliding safety shutter 1616 when the safety shutter is closed and ensure that the safety shutter opens quickly when the security system is activated. The system-operated buckle 1605 interacts with the buckle 1618 on the sliding safety shutter 1616 to ensure that the slider is closed.

[0098] The security system may be activated when the ground or one or more landmarks are detected near the system, the user, the user's body, or the user's appendages. The security system may use the user's body, appendages, the ground, or landmarks and the determination of their relative position and orientation, as discussed above. Upon activation of the security system, a signal may be sent to the system-operated buckle 1605 to open the buckle. When the buckle is opened, the spring 1619 pushes the sliding safety shutter 1616 open, allowing the user to see through the display 1602 and avoid the danger that triggered the security system.

[0099] Fig. 16C A headset with a shuttered safety shield according to various aspects of the present disclosure is depicted. In this embodiment, the slats 1626 of the shuttered safety shield are longer in a first dimension than in a second dimension. In the closed position, the longer dimension of the slats is substantially parallel to the optical device, and each slat overlaps with another slat or the headset body, thereby blocking light from passing through the display screen 1602. In the open position, the slats change position so that the shorter dimension is parallel to the optical device 1603, thereby allowing light to pass through the slats to the display screen. An actuator rod 1628 can be hinged to each of the slats 1626. When the security system is activated, an actuator 1629 controlled by the security system can push or pull the actuator rod 1628 to open the shuttered safety shield. In some other embodiments, the actuator controlled by the security system can be spring-loaded, and the buckle connected to the actuator rod 1628 and the system-controlled actuator 1629 can include a buckle that docks with the buckle of the actuator rod. When the safety system is activated, the buckle of the system-controlled actuator may open, thereby allowing the actuator rod to move under spring pressure, thereby opening the shutter.

[0100] The security system may be activated when the ground or one or more landmarks are detected near the system, the user, the user's body, or the user's appendages. The security system may use the user's body, appendages, the ground, or landmarks and the determination of their relative position and orientation, as discussed above. Upon activation, a signal may be sent to an actuator operated by the system to move an actuator rod. When the actuator rod 1629 moves, the actuator rod pushes the slats 1626 of the louvered safety shield open, allowing the user to see through the display 1602 and avoid the danger that has aroused the security system.

[0101] Fig.16D A fabric safety shutter according to various aspects of the present disclosure is shown. In this embodiment, the safety shutter 1636 is composed of an opaque fabric, such as, but not limited to, a tightly woven cotton fabric, polyester, vinyl plastic, or a tightly woven wool fabric. The fabric safety shutter 1636 may be coupled to a fabric roller 1639. The fabric roller 1639 may be configured to roll up the fabric shutter when the security system is activated. The fabric roller may be, for example, but not limited to, a spring loaded using a spring so that when the safety shutter is closed, the spring is under tension, alternatively, an electric motor may be used to roll up the fabric safety shutter. The system-operated buckle 1605 may dock with the buckle 1638 coupled to the fabric safety shutter 1636 and ensure that the fabric safety shutter is not inadvertently opened.

[0102] During operation, the fabric safety shutter may be in a closed position. The security system may be activated when the ground or one or more landmarks are detected near the system, the user, the user's body, or the user's appendage. The security system may use the user's body, appendage, the ground, or landmarks and the determination of their relative position and orientation, as discussed above. Upon activation of the security system, a signal may be sent to the system-operated buckle 1605 to open the buckle. When the buckle is opened, the fabric roller 1619 rolls up the fabric safety shutter 1616, allowing the user to see through the display 1602 and avoid the danger that invoked the security system.

[0103] Fig.16E A headset with a liquid crystal security shield is shown in accordance with aspects of the present disclosure. In this embodiment, an LCD screen 1646 is integrated into the headset 1601, thereby covering the hole in the headset in an otherwise manner. An LCD screen driver 1649 controlled by a security system is communicatively coupled to the LCD screen 1646.

[0104] The liquid crystal screen 1646 can be, for example, but not limited to, a liquid crystal baffle having a first light polarizer and a second light polarizer, wherein the first polarizer has a light polarization difference of 90 degrees with the second light polarizer, and having a fluid-filled cavity. The fluid-filled cavity may include a liquid crystal configured to have a first orientation in the absence of an electric field that changes the polarization of light, thereby allowing light to pass from the first light polarizer to the second light polarizer. The liquid crystal may be further configured to align in a second orientation under an electric field. The second orientation of the liquid crystal does not change the polarization of the light, whereby light passing through the first light polarizer is blocked at the second light polarizer. Electrodes may be disposed along the surface of the liquid-filled cavity, thereby allowing control of the liquid crystal. The liquid crystal screen driver controlled by the security system may be communicatively coupled to the electrodes, thereby allowing control of the liquid crystal in the fluid-filled cavity. As used herein, communicatively coupled means being able to send and / or receive electrical signals representing messages or instructions from one coupled element to another coupled element, the signals may travel through intermediate elements, and their formats may change but the messages contained therein remain unchanged.

[0105] During operation, a driver 1649 controlled by the security system may send a signal to the LCD screen 1646, causing the LCD security shield 1646 to become opaque when the display screen 1602 is active. When the security system is activated, the driver 1649 may cause the LCD security shield 1646 to become transparent. For example, but not limited to, the driver may reduce the voltage supplied to the LCD security shield, thereby returning the liquid crystal to its first orientation, which causes a change in the polarization of the light, thereby allowing the light to pass through the second light polarizer. The security system may be activated when the ground or one or more landmarks are detected near the system, the user, the user's body, or the user's appendage. The security system may use the user's body, appendage, ground or landmarks and the determination of their relative position and orientation, as discussed above.

[0106] Finger position tracking

[0107] Aspects of the present disclosure may be applied to finger tracking. Fig.17 Finger tracking with a DVS and controller according to aspects of the present disclosure is depicted. Here, a controller 1701 includes two or more light sources and one or more buttons 1705. The two or more light sources include one or more light sources 1706 and two or more other tracking light sources 1703 near the one or more buttons 1705. As described above, the two or more other tracking light sources 1703 can generate events at the DVS 1702 for determining the position and orientation of the controller 1701. Two DVS 1702 or a DVS with two arrays and three other tracking light sources 1703 are shown here for determining the position and orientation of the controller.

[0108] One or more light sources 1706 near one or more buttons 1705 may be used for finger tracking. For example, but not limited to, finger tracking may be implemented with DVS 1702 using occlusion of one or more light sources 1706 near the button 1705. One or more light sources 1706 near the button may be turned off and on at predetermined intervals. DVS 1702 may generate an event at each flash. The event may be analyzed to determine occlusion of one or more light sources 1706 near the button 1705. The configuration of the one or more light sources near the button may be known, and thus when the light source is occluded by, for example, but not limited to, a finger or palm, the light pattern detected in the event generated by the DVS is different from when the light source is not occluded. As discussed above with respect to determining the position and orientation of the controller, here, the timing of the flash may be used to determine which light is occluded and thereby determine the corresponding finger or palm position. When the light source 1706 near the button 1705 has a reduced intensity or no intensity during the interval, it is known that the light source near the button is 'on'. The light source is determined to be occluded. Similarly, when the detected intensity of a light known to be 'on' changes; an event may be generated, and from that event, it may be determined that the user's finger or hand has moved and the button has become unpressed. Occlusion of one or more of the light sources near one or more buttons may be associated with a finger or palm position based on the position of the one or more light sources. For example, but not limited to, a light source located near the user's palm when grasping the controller 1701 may be used to determine the position of the user's hand 1704.

[0109] Light sources 1706 may be located around each button 1705, and the button configuration and design of the controller may be used to determine finger positions. For example, but not limited to, the controller 1701 may be designed so that when grasped, each finger of the user is in a position near the button 1705. Then, the light source occlusion pattern determined based on the DVS event may be used to determine when the user's finger is hovering over a button that has not yet been activated, and may also be used to determine when the user's finger moves past the button. This may be useful for providing the user with additional interaction options, such as half a button press or half a press or other button options. Multiple light sources may surround each button, allowing for improved determination of the position of the user's finger or palm. For example, but not limited to, in some embodiments, ten or more light sources may surround each button, and in other embodiments, a single light source may shine light through a translucent diffuser around the button, and interruptions in the diffuse light distribution may be used to determine finger positions.

[0110] In some embodiments, button 1705 itself may also be a light source. One or more buttons 1705 may be turned off and on at different intervals than one or more light sources 1706 or other light tracking light sources 1703 near the button. Alternatively, the light source of button 1705 may have a different wavelength or polarization than one or more light sources 1706 or other light tracking light sources 1703 near the button.

[0111] The button and tracking can also be used to enable power saving modes for the light sources. For example, when it is determined that a button is pressed, one or more light sources near the button can be dimmed or turned off. In addition, if it is determined that the controller 1701 is out of the field of view of the DVS 1702, one or more lights near the button can be dimmed or turned off. In some embodiments, data from the IMU can be used to determine whether the controller is grasped by the user, for example but not limited to, if no change in IMU data (such as acceleration, angular rate, etc.) is detected within a threshold time period, the light source can be dimmed or turned off. Once a change in the IMU data is detected, the light source can be turned back on.

[0112] In alternative embodiments, finger tracking may be performed without using one or more light sources. A machine learning model may be trained with a machine learning algorithm to detect finger positions from events generated by changes in ambient light caused by finger movement. The machine learning model may be a general machine learning model, such as a CNN, RNN, or DNN as discussed above. In some embodiments, a specialized machine learning algorithm may be used to train a specialized machine learning model (such as, for example, but not limited to, a spike (or spark) neural network (SNN)). SNNs mimic biological NNs by having an activation threshold and weights that are adjusted according to relative spike times within an interval (also known as spike timing dependent plasticity (STDP)). When the activation threshold is reached, the SNN is called a spike and transfers its weights to the next layer. SNNs may be trained via STDP and supervised or unsupervised learning techniques. More information about SNNs may be found in "Deep Learning in Spiking Neural Networks" Neural Networks (2018) arXiv:1804.08150 by Tavanaei, Amirhossein et al., the contents of which are incorporated herein by reference for all purposes.

[0113] Alternatively, high dynamic range (HDR) images may be constructed using aggregated events from environmental data. A machine learning model is trained to recognize hand positions or controller positions and orientations from the HDR images. The trained machine learning model may be applied to HDR images generated from the events to determine hand / finger positions or controller positions and orientations. The machine learning model may be a general machine learning model trained using supervised learning techniques as discussed in the general neural network training section.

[0114] Eye Tracking

[0115] Aspects of the present disclosure may be applied to eye tracking. Generally speaking, eye tracking image analysis utilizes characteristics distinct from how light reflects off of the eye to determine eye gaze direction from an image. For example, an image may be analyzed to identify eye position based on corneal reflections in the image data, and the image may be further analyzed to determine gaze direction based on the relative position of the pupil in the image.

[0116] Two common gaze tracking techniques for determining the direction of eye gaze based on pupil position are called bright pupil tracking and dark pupil tracking. Bright pupil tracking involves illuminating the eye with a light source that is substantially in line with the optical axis of the DVS, causing the emitted light to reflect off the retina and back through the pupil to the DVS. The pupil appears in the image as an identifiable bright spot at the location of the pupil, similar to the red eye effect that appears in an image during conventional flash photography. In this method of gaze tracking, if the contrast between the pupil and the iris is insufficient, the bright reflection from the pupil itself helps the system locate the pupil.

[0117] Dark pupil tracking involves illumination with a light source substantially offset from the optical axis of the DVS, causing light directed through the pupil to reflect off the optical axis of the DVS, thereby producing an identifiable dark spot in the event at the location of the pupil. In an alternative dark pupil tracking system, an infrared light source and camera pointed at the eye can observe the corneal reflection. This DVS-based system tracks the location of the pupil, and the corneal reflection, which provides parallax due to different reflection depths, gives additional accuracy.

[0118] Fig.18A An example of a dark pupil gaze tracking system 1800 that can be used in the context of the present disclosure is depicted. The gaze tracking system tracks the orientation of a user's eye E relative to a display screen 1801 presenting a visible image. Fig.18A In the example system of , a display screen is utilized, but some alternative embodiments may utilize an image projection system capable of projecting an image directly into the user's eye. In these embodiments, the user's eye E will be tracked relative to the image projected into the user's eye. Fig.18A In the example of FIG. 1 , the eye E collects light from the screen 1801 through the variable iris I; and the lens L projects an image on the retina R. The opening in the iris is called the pupil. Muscles control the rotation of the eye E in response to nerve impulses from the brain. The upper eyelid muscle ULM and the lower eyelid muscle LLM control the upper eyelid UL and the lower eyelid LL, respectively, in response to other nerve impulses.

[0119] The light-sensitive cells on the retina R generate electrical impulses that are sent to the user's brain (not shown) via the optic nerve ON. The visual cortex of the brain interprets the impulses. Not all parts of the retina R are equally sensitive to light. Specifically, the light-sensitive cells are concentrated in an area called the fovea.

[0120] The image tracking system shown includes one or more infrared light sources 1802, such as light emitting diodes (LEDs) that direct invisible light (e.g., infrared light) toward the eye E. Part of the invisible light is reflected from the cornea C of the eye, and part is reflected from the iris. The reflected invisible light is directed by a wavelength selective mirror 1806 to a DVS 1804 that is sensitive to infrared light. The mirror transmits visible light from the screen 1801 but reflects invisible light reflected from the eye.

[0121] The DVS 1804 generates events for the eye E, which can be analyzed to determine the gaze direction GD based on the relative position of the pupil. The event can be generated by the processor 1805. The DVS 1804 is advantageous in this embodiment because the extremely fast update rate of the event provides near real-time information about changes in the user's gaze.

[0122] like Fig.18B As shown, event 1811 showing the user's head H can be analyzed to determine the gaze direction GD based on the relative position of the pupil. For example, the analysis can determine the 2-dimensional offset of the pupil P from the center of the eye E in the image. The position of the pupil relative to the center can be converted to a gaze direction relative to the screen 1801 by direct geometric calculation of a 3D vector based on the known size and shape of the eyeball. The determined gaze direction GD can show the rotation and acceleration of the eye E as it moves relative to the screen 1801.

[0123] Also like Fig.18B As shown, the event may also include reflections 1807 and 1808 of invisible light from the cornea C and lens L, respectively. Since the cornea and lens are at different depths, the parallax and refractive index between the reflections can be used to provide additional accuracy in determining the gaze direction GD. An example of this type of eye tracking system is a dual Purkinje tracker, where the corneal reflection is the 1st Purkinje image and the lens reflection is the 4th Purkinje image. If the user is wearing the user's glasses 1809, there may also be reflections 1808 from the user's glasses.

[0124] The performance of an eye tracking system depends on a variety of factors, including: placement of the light source (IR light source, visible light source, etc.) and DVS, whether the user is wearing glasses or contact lenses, headset optics, tracking system latency, eye movement rate, eye shape (which changes during the course of the day or may change due to movement), eye condition (e.g., amblyopia), gaze stability, fixation on moving objects, the scene being presented to the user, and the user's head movement. The DVS provides an extremely fast update rate for events while reducing irrelevant information output to the processor. This allows for faster processing and faster determination of gaze tracking status and error parameters.

[0125] The error parameters that can be determined based on the gaze tracking data may include, but are not limited to, rotation speed and prediction error, error of fixation, confidence interval about current and / or future gaze position, and error of smooth tracking. State information about the user's gaze relates to discrete states of the user's eyes and / or gaze. Therefore, example state parameters that can be determined based on the gaze tracking data may include, but are not limited to, blink metrics, eye saccade metrics, depth of field response, color blindness, gaze stability, and eye movement as a precursor to head movement.

[0126] In some embodiments, gaze tracking error parameters may include confidence intervals about the current gaze position. The confidence intervals may be determined by checking the rotational velocity and acceleration of the user's eyes for the last change in position. In alternative embodiments, gaze tracking errors and / or state parameters may include predictions of future gaze positions. Future gaze positions may be determined by checking the rotational velocity and acceleration of the eyes and extrapolating possible future positions of the user's eyes. Generally speaking, for users with larger rotational velocity and acceleration values, the DVS update rate of the gaze tracking system may cause a smaller error between the determined future position and the actual future position because the update rate of the DVS is very high, making this small error much smaller than existing camera-based systems.

[0127] In yet other alternative embodiments, gaze tracking error parameters may include measurements of eye velocity (e.g., rotation rate). In certain alternative embodiments, the determined gaze tracking state parameters include metrics measuring the user's blinks. During a typical blink, typically a period of 150 milliseconds (ms) passes through which the user's vision is not focused on the presented image. Therefore, depending on the frame rate of the display device, the user's vision may not focus on the presented image for up to 20 to 30 frames. However, upon exiting the blink, the user's gaze direction may not correspond to the last measured gaze direction determined based on the obtained gaze tracking data. Therefore, metrics of the user's gaze may be determined based on the obtained gaze tracking data. These metrics may include, but are not limited to, the measured start and end times of the user's blinks and the predicted end time.

[0128] In another alternative embodiment, the gaze tracking state parameter determined includes a metric for measuring the user's eye pan. During a typical eye pan, typically through a period of 20ms to 200ms, the user's vision is not focused on the image presented. Therefore, according to the frame rate of the display device, the user's vision may not focus on any place on the image presented up to 40 frames. However, due to the nature of eye pans, when exiting eye pans, the user's gaze direction will be transferred to different regions of interest. Therefore, gaze tracking data can be used to establish a metric for the user's eye pan based on the actual or predicted time that will pass during the eye pan. These metrics may include but are not limited to the start and end time of the measurement of the user's eye pan and the predicted end time.

[0129] In some alternative embodiments, the determined gaze tracking state parameters include determining a transition of the user's gaze direction between regions of interest due to a change in depth of field between the presented images. Due to the transition between regions of interest provided in the presented images, the user will be caused to make an eye saccade.

[0130] In yet another alternative embodiment, the determined gaze tracking state parameters may accommodate color blindness. For example, regions of interest may be present in an image presented to a user such that a user with a particular form of color blindness will not notice these regions. The obtained gaze tracking data will determine whether the user's gaze is recognized or responsive to the region of interest (e.g., due to the user's changing gaze direction). Thus, as a gaze tracking error parameter, it may be determined whether the user is color blind to a particular color or spectrum.

[0131] In some alternative embodiments, the determined gaze tracking state parameter includes a measure of the user's gaze stability. Gaze stability can be determined by measuring the micro-saccade radius of the user's eyes; smaller gaze overshoot and undershoot equates to a more stable gaze of the user.

[0132] In yet another alternative embodiment, the gaze tracking error and / or state parameters determined include the user's ability to fixate on a moving object. These parameters may include a measure of the user's eye's ability to perform smooth tracking and the eye's maximum object tracking speed. Typically, a user with excellent smooth tracking ability experiences less jitter in the movement of the eye.

[0133] In some alternative embodiments, the gaze tracking error and / or state parameters determined include determining eye movement as a precursor to head movement. The offset between the head and eye orientation may affect certain errors and / or state parameters, as discussed above, for example, in smooth pursuit or fixation.

[0134] More information on gaze tracking and error parameter determination may be found in US Patent 10,192,528, the contents of which are incorporated herein by reference for all purposes.

[0135] system

[0136] Fig.19 19 is a system block diagram of a system for tracking with a DVS according to aspects of the present disclosure. By way of example and not limitation, according to aspects of the present disclosure, the system 1900 may be an embedded system, a mobile phone, a personal computer, a tablet computer, a portable gaming device, a workstation, a gaming console, etc.

[0137] The system 1900 typically includes a central processing unit (CPU) 1903 and memory 1904. The system 1900 may also include well-known support functions 1906, which may communicate with other components of the system, for example, via a data bus 1905. Such support functions may include, but are not limited to, input / output (I / O) elements 1907, power supply (P / S) 1911, clock (CLK) 1912, and cache 1913.

[0138] System 1900 may include a display device 1931 for presenting rendered graphics to a user. In alternative embodiments, the display device is a separate component that works in conjunction with system 1900. Display device 1931 may be in the form of a flat panel display, a head mounted display (HMD), a cathode ray tube (CRT) screen, a projector, or other device that can display visible text, numbers, graphic symbols, or images.

[0139] Here, the display device 1931 is coupled to the DVS 1901A, and the controller 1902 includes two or more light sources 1932A, which may be in any of the configurations described herein. In alternative embodiments, the DVS may be coupled to a game controller, and the display device may alternatively include two or more light sources. In still other alternative embodiments, the DVS is a separate unit decoupled from the display device or controller, in which case both the controller and the display device may include two or more light sources for tracking.

[0140] In some embodiments, for example, where the display device is part of a head mounted display (HMD), such HMD may include an inertial measurement unit (IMU), such as an accelerometer or gyroscope. Also as discussed above, such HMD may include a light source 1932B that may be tracked using a DVS separate from the display device 1901 and coupled to the CPU 1903. As an example, a separate DVS 1901B may be mounted to the controller 1902.

[0141] In some embodiments, DVS1901A or DVS1901B may be part of a hybrid sensor, for example, as described above with respect to Fig.11A , Fig. 11B , Fig. 11C or Fig.11D Such a hybrid sensor may include a depth sensor, for example a DTOF sensor, in which case the hybrid sensor may include an illumination unit (not shown).

[0142] Furthermore, where the display device 1931 is part of an HMD, the device may be equipped with an optional safety shield 1933 that is operably coupled to a processor, such as CPU 1903, and as described above with respect to Fig.16A , Fig. 16B , Fig. 16C , Fig.16D or Fig.16E Alternatively, the safety shield may be controlled by a separate processor mounted to the HMD.

[0143] The system 1900 may include a mass storage device 1915, such as a disk drive, a CD-ROM drive, a flash memory, a solid state drive (SSD), a tape drive, etc., to provide non-volatile storage for programs and / or data. The system 1900 may also optionally include a user interface unit 1916 for facilitating interaction between the system 1900 and a user. The user interface 1916 may include a keyboard, a mouse, a joystick, a light pen, or other devices that can be used in conjunction with a graphical user interface (GUI). The system 1900 may also include a network interface 1914 for enabling the device to communicate with other devices via a network 1920. The network 1920 may be, for example, a local area network (LAN), a wide area network (such as the Internet), a personal area network (such as a Bluetooth network), or other types of networks. These components may be implemented in hardware, software, or firmware, or some combination of two or more of these.

[0144] The CPUs 1903 may each include one or more processor cores, such as a single core, dual cores, quad cores, octa cores, or more. In some embodiments, the CPUs 1903 may include a GPU core or multiple cores of the same accelerated processing unit (APU). The memory 1904 may be in the form of an integrated circuit that provides addressable memory (e.g., random access memory (RAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.). The main memory 1904 may include application data 1923 used by the processor 1903 when processing. The main memory 1904 may also include event data 1909 received from the DVS 1901. A trained neural network (NN) 1910 may be loaded into the memory 1904 for use in determining position and orientation data, such as in Fig. 9 1910. In addition, the memory 1904 may include a machine learning algorithm 1921 for training or adjusting the NN 1910. A database 1922 may be included in the memory 1904. The database may contain information about the light source configuration, predetermined flash intervals for each one or more light sources, etc. The memory may also contain outputs from an IMU coupled to the controller 1902 or the display device 1931. In some embodiments, the memory 1904 may contain outputs from one or more light sources, such as timestamps when each of the light sources is turned on or off.

[0145] According to aspects of the present disclosure, the processor 1903 may perform a method for determining the position and orientation of a controller or user, such as in Figure 8 and Fig. 9 As discussed in , these methods may be loaded into memory 1904 as application 1923. Figure 8 and Fig. 9 Describing and concerning Fig.15 As a result of the method further described, the processor may generate one or more orientations and configurations of the controller, the headset, the user's body or appendages, the ground or landmarks. These positions and orientations may be stored in the database 1922 and may be used in Figure 8 and Fig. 9 In some embodiments, the processor 1903 may utilize such position and / or orientation in a machine learning algorithm trained to perform simultaneous localization and mapping (SLAM).

[0146] The mass storage device 1915 may contain an application or program 1917 that is loaded into the main memory 1904 when processing begins on the application 1923. In addition, the mass storage device 1915 may contain data 1918 used by the processor during processing of the application 1923, the NN 1910, the machine learning algorithm 1921, and the population of the database 1922.

[0147] As used herein and generally understood by those skilled in the art, an application specific integrated circuit (ASIC) is an integrated circuit that is customized for a specific purpose and is not intended for general use.

[0148] As used herein and generally understood by those skilled in the art, a field programmable gate array (FPGA) is an integrated circuit designed to be configured by a customer or designer after manufacture (hence "field programmable"). FPGA configuration is typically specified using a hardware description language (HDL) similar to that used for ASICs.

[0149] As used herein and generally understood by those skilled in the art, a system on a chip or system on a chip (SoC or SOC) is an integrated circuit (IC) that integrates all the components of a computer or other electronic system into a single chip. It can contain digital, analog, mixed signal and usually radio frequency functions - all on a single chip substrate. Typical applications are in the field of embedded systems.

[0150] A typical SoC includes the following hardware components:

[0151] One or more processor cores (e.g., microcontroller, microprocessor, or digital signal processor (DSP) cores.

[0152] Memory blocks such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), and flash memory.

[0153] A timing source, such as an oscillator or a phase-locked loop.

[0154] Peripherals such as counter-timers, real-time timers, or power-on reset generators.

[0155] External interfaces, for example, industry standards such as Universal Serial Bus (USB), FireWire, Ethernet, Universal Asynchronous Receiver / Transmitter (USART), Serial Peripheral Interface (SPI) bus.

[0156] Analog interfaces, including analog-to-digital converters (ADCs) and digital-to-analog converters (DACs).

[0157] Voltage regulators and power management circuits.

[0158] These components are connected by a proprietary or industry standard bus. A direct memory access (DMA) controller routes data directly between the external interface and the memory, bypassing the processor core and thereby increasing the data throughput of the SoC.

[0159] A typical SoC includes both the hardware components described above and executable instructions (eg, software or firmware) that control the processor core, peripherals, and interfaces.

[0160] Aspects of the present disclosure provide image-based tracking featuring a higher sampling rate than is possible with conventional image-based tracking systems, resulting in improved tracking fidelity. Additional advantages include reduced cost, reduced weight, reduced generation of extraneous data, and reduced processing requirements when using a DVS-based tracking system. These advantages allow for improved virtual reality (VR) and augmented reality (AR) systems, among other applications.

[0161] Although the above is a complete description of the preferred embodiments of the present invention, various alternatives, modifications and equivalents may be used. Therefore, the scope of the present invention should not be determined with reference to the above description, but should be determined with reference to the appended claims and their full range of equivalents. Any feature described herein (whether preferred or not) may be combined with any other feature described herein (whether preferred or not). In the appended claims, unless explicitly stated otherwise, No Definite article "a" or "an" Refers to the quantity of one or more of the items following the article. The appended claims should not be read as including means-plus-function limitations unless such a limitation is explicitly recited in a given claim using the phrase "means for..."

Claims

1. A tracking method, comprising: determining a position and orientation of the controller based on a known configuration of the two or more light sources relative to each other and to a body of the controller and based on signals from an output of a dynamic vision sensor (DVS) generated in response to changes in light output from the two or more light sources, wherein the output signals include information corresponding to times of two or more events at two or more corresponding light sensitive elements in an array of two or more light sensitive elements in the DVS and positions of the two or more corresponding light sensitive elements in the array; as well as The position and orientation of one or more objects is determined based on signals generated by the two or more photosensitive elements as a result of other light reaching the two or more photosensitive elements.

2. The tracking method of claim 1, wherein the one or more objects include the controller. 3 . The tracking method of claim 1 , wherein the one or more objects include one or more objects other than the controller. 4 . The tracking method of claim 1 , wherein the one or more objects include the controller and one or more objects other than the controller.

5. The tracking method of claim 1 , wherein determining the position and orientation of one or more objects based on signals generated by the two or more photosensitive elements due to other light reaching the two or more photosensitive elements comprises determining a map of the environment and the state of the one or more objects.

6. The tracking method of claim 1, further comprising triggering an optical safety shutter on a headset in response to the determined position of the one or more objects.

7. The tracking method of claim 6, wherein the optical safety shield comprises an opaque visor operably coupled to the headset, wherein the opaque visor is configured to selectively allow light to pass through to the eyes of a user wearing the headset. 8 . The tracking method of claim 7 , wherein triggering the optical safety shutter comprises triggering the optical safety shutter to change from an opaque state to a light-transmissive state in response to the determined position of the one or more objects.

9. The tracking method of claim 8, wherein a liquid crystal barrier is included in the optical security barrier.

10. The tracking method of claim 1 , wherein determining the position and orientation of one or more objects based on signals generated by the two or more photosensors due to other light reaching the two or more photosensors comprises determining a map of the environment and the state of the one or more objects based on signals generated by the two or more photosensors in a DVS located on the controller.

11. The tracking method of claim 1 , wherein determining the position and orientation of one or more objects comprises using a machine learning algorithm to fit the position and orientation of the one or more objects to the signals generated by the two or more photosensitive elements.

12. The tracking method of claim 1, further comprising receiving information corresponding to an angular rate of change or a specific force or a specific acceleration or a change in magnetic flux from an inertial measurement unit (IMU).

13. The tracking method of claim 12, wherein determining the position and the orientation of the controller further comprises determining the position and the orientation of the controller using the information corresponding to the angular rate of change or the specific force or the specific acceleration or the magnetic flux change.

14. The tracking method of claim 13, further comprising using a Kalman filter to fuse the angular rate of change or specific force / acceleration or magnetic flux change information with the two or more events.

15. The tracking method of claim 1, wherein determining the position and the orientation of the controller comprises using the change in output light and a known pattern of change in output light to determine the position of at least one of the two or more light sources in the known configuration.

16. The tracking method of claim 1, further comprising notifying a user of the location of the one or more objects based on the determined location and the determined orientation of the one or more objects.

17. The tracking method of claim 1 further comprising triggering an optical safety shutter on a headset in response to the determined position of the controller.

18. The tracking method of claim 17, wherein the optical safety shield comprises an opaque visor operably coupled to the headset, and wherein the opaque visor is configured to selectively allow light to pass through to the eyes of a user wearing the headset.

19. The tracking method of claim 17, wherein triggering the optical safety shutter comprises triggering the optical safety shutter to transition from an opaque state to a light-transmissive state in response to the determined positions of the one or more objects.

20. The tracking method of claim 19, wherein a liquid crystal barrier is included in the optical security barrier.

Images