Tracking system and method using event camera

Abstract

A method of tracking a position of a wearable worn by a user as an indication of a user's position, said wearable comprising a plurality of spatially-oriented markers, said method comprising: (a) causing said plurality of spatially-oriented markers to transmit strobed light of a certain wavelength at a certain strobe frequency in a light pattern indicative of the position and orientation of said wearable; (b) recording image data of said plurality of spatially orientated markers only upon a change in said light pattern, said image data being restricted to light of said certain wavelength at said strobe frequency in a changed light pattern; and (c) determining said position and orientation of said wearable based on said image data.

Description

TRACKING SYSTEM AND METHOD USING EVENT CAMERACROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present patent application claims the benefit of U.S. Provisional Patent Application 63 / 548,493, filed November 14, 2023, the entire disclosure of which is hereby incorporated herein by reference.FIELD OF INVENTION

[0002] Generally, the present disclosure relates to the field of mixed reality. More specifically, the present disclosure relates to tracking the position and orientation of a wearable in a mixed-reality experience.BACKGROUND

[0003] The ability to precisely determine the position and orientation of an object, such as a helmet mounted augmented reality (AR) device, in the cockpit of a plane or the cabin of a vehicle is often necessary to deliver augmented content to the pilot / driver in a realistic manner in an environment that is traveling through space.

[0004] For example, when presenting augmented reality content to a viewer, such as via a viewing device, it is desired to determine the precise location in space of the viewing device as well as the orientation of the viewing device. These two attributes, location and orientation, are generally sufficient to enable the display of augmented content. Once a location of the viewing apparatus is determined, it is necessary to know the precise orientation of the apparatus. For example, if a viewer is looking upwards or with a head tilted to the side, the augmented content must be adjusted accordingly or the content will fail to appear as properly merged with aspects and artifacts of the surrounding physical environment.

[0005] In the instance of a cockpit situated system, there exist various modalities for ascertaining the position and orientation of a helmet mounted AR system. In some embodiments, cockpit mounted cameras in known and calibrated positions continuously image markings on the helmet in order to determine the position and orientation of the helmet. In other embodiments, lidar may be employedto precisely determine the position and orientation of the helmet. While cameras work well in well-lit scenarios, the contrast between light and shadow as produced by sunlight inside a cockpit flying at high elevation is substantial. The inability of frame based cameras to adequately adjust to such stark contrasts often limits the ability of cameras to properly image the markings.

[0006] In response to this challenge, US20240053609A1, which is hereby incorporated by reference, discloses using an event camera. An event camera, for example, a neuromorphic camera, silicon retina or dynamic vision sensor, is an imaging sensor that responds to local changes in brightness. An event camera does not capture an image using a shutter as conventional cameras do. Instead, each pixel inside an event camera operates independently and asynchronously, independently reporting changes in brightness as they occur, and staying silent otherwise. The resulting event camera output is an asynchronous stream of events triggered by changes in scene illumination. The result is a virtually unlimited equivalent frame rate with a requisitely high resolution. Specifically, while the human eye is believed to have an equivalent frame rate of approximately 200-300 fps, and event cameras equivalent frame rate is on the order of 1,000,000 fps. Another advantage of event cameras is the ability to capture a greater dynamic range of intensities such that data is not drowned out in images containing relatively bright portions and dim portions. This aspect of event cameras is of special import in scenarios discussed below.

[0007] An event camera may be mounted in an environment to track movement of an object, or an event camera may be mounted on the object to track its movement in the environment. In US20240053609A1, Applicant describes how an event camera may be mounted within a cockpit of an airplane and positioned to monitor movements of an operator’s head for the tracking of the operator’s viewing direction. The tracking information may be used for the positioning of content within a mixed reality display (e.g., head worn augmented reality display, virtual reality display, pass through virtual / augmented reality system).

[0008] As described in US20240053609A1, event cameras exhibit a high equivalent frame rate, adequate resolution and a superior ability to gather images across an extended range of light intensities. As a result, the implementation of event cameras in a cockpit scenario allows for the ability to image location indicia, such ashelmet markings, in a dynamic cockpit environment in which light and shadow are abruptly and constantly changing. Thus, the tracking system described in US20240053609A1 provides unprecedented resolution and flexibility, yet Applicant recognizes a continuing need to improve tracking systems. The present invention fulfils this need among others.SUMMARY OF INVENTION

[0009] The following presents a simplified summary of the invention in order to provide a basic understanding of the invention. This summary is not an extensive overview of the invention. It is not intended to identify key / critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later.

[0010] Applicant’s tracking system disclosed in US20240053609A1 provides unprecedented resolution and flexibility, but Applicant recognizes that the immense data produced by an event camera can be challenging to process quickly.Accordingly, Applicant discloses herein a system and method for reducing the image data recorded by an event camera without compromising resolution. Specifically, in addition to limiting the image data by recording image data only upon a change in the imaged subject matter (which is how an event camera functions), the approach disclosed herein also restricts the image data to (1) light from a limited amount of targets, and / or (2) light having a known wavelength, and / or (3) light having a known strobe frequency. Limiting image data using one or more of these approaches significantly reduces the data that needs to be processed, thereby reducing latency and improving the user experience.

[0011] In one embodiment, the invention relates to a method of tracking a position of a wearable worn by a user as an indication of a user’s position, the wearable comprising a plurality of spatially-oriented markers, the method comprising: (a) causing the plurality of spatially-oriented markers to transmit strobed light of a certain wavelength at a certain strobe frequency in a light pattern indicative of the position and orientation of the wearable; (b) recording image data of the plurality ofspatially orientated markers only upon a change in the light pattern, the image data being restricted to light of the certain wavelength at the strobe frequency in a changed light pattern; and (c) determining the position and orientation of the wearable based on the image data.

[0012] In one embodiment, the invention relates to a system of tracking a position of a wearable worn by a user as an indication of a user’s position, the system comprising: (a) the wearable comprising a plurality of spatially-oriented markers; (b) a strobe light configured to cause the spatially-oriented markers to transmit strobed light having a certain strobe frequency and a certain wavelength in a light pattern indicative of the position and orientation of the wearable; (c) an event camera for recording image data of the plurality of spatially-oriented markers upon a change in the light pattern, the image data being restricted to light of the certain wavelength at the strobe frequency in a changed light pattern; and (d) a processor for processing the image data to determine the position and orientation of the wearable.BRIEF DESCRIPTION OF FIGURES

[0013] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various embodiments of the present disclosure. The drawings may contain text or captions that may explain certain embodiments of the present disclosure. This text is included for illustrative, non-limiting, explanatory purposes of certain embodiments detailed in the present disclosure.

[0014] Fig. 1 shows one embodiment of the system of the present invention.

[0015] Fig. 2 shows an image of a light pattern of the spatially-oriented markers.

[0016] Fig. 3 shows a schematic of determining orientation of a wearable based on the light pattern of the spatially-oriented markers.DETAIL DESCRIPTIONS

[0017] One embodiment of the present invention is a method of tracking a position of a wearable worn by a user as an indication of a user’s position. Thewearable comprising a plurality of spatially-oriented markers. In one embodiment, the method comprises: (a) causing the plurality of spatially-oriented markers to transmit strobed light of a certain wavelength at a certain strobe frequency in a light pattern indicative of the position and orientation of the wearable; (b) recording image data of the plurality of spatially orientated markers only upon a change in the light pattern, the image data being restricted to light of the certain wavelength at the strobe frequency in a changed light pattern; and (c) determining the position and orientation of the wearable based on the image data.

[0018] Referring to Fig. 1, one embodiment is shown of a system 100 of the present invention performing the aforementioned method. The system 100 comprises: (a) the wearable 101 comprising a plurality of spatially-oriented markers 102; (b) a strobe light 103 configured to cause the spatially-oriented markers 102 to transmit strobed light having a certain strobe frequency and a certain wavelength in a light pattern indicative of the position and orientation of the wearable; (c) an event camera 104 for recording image data of the plurality of spatially-oriented markers upon a change in the light pattern, the image data being restricted to light of the certain wavelength at the strobe frequency in a changed light pattern; and (d) a processor 105 for processing the image data to determine the position and orientation of the wearable.

[0019] Each of these claimed features are considered in greater detail below along with selected alternative embodiments.

[0020] As used herein, extended reality (XR) refers to a family of technologies also known as augmented reality (AR), virtual reality (VR), and mixed reality (MR). AR is an interactive experience that combines the real world and computer-generated content. VR generally replaces a user's real-world environment with a simulated one. MR, sometimes referred to as a hybrid of augmented reality and virtual reality, describes the merging of a real-world environment and a computer-generated one. IN some instances in the description below, “VR” may be used interchangeably with “AR.”

[0021] The system and method of the present invention involves causing the spatially-oriented markers of the wearable to transmit strobed light. This can be donein various ways. For example, in one embodiment, the spatially-oriented markers are passive / reflective. In such an embodiment, a discrete strobe light is used to emit the strobed light such that it reflects off of the spatially-oriented markers. Alternatively, the spatially-oriented markers are active, and are, in effect, strobe lights themselves such that the strobe light is emitted from the spatially-oriented markers. Still other alternatives will be obvious to those of ordinary skill in the art in light of this disclosure.

[0022] In one embodiment, the strobed light of a known frequency is emitted in an environment and an event camera positioned to image the environment may be programmed to filter light (e.g., through hardware and / or software) only to process light at the known frequency. In one embodiment, the strobed frequency is no less than 100Hz, 500Hz, 600Hz, 700Hz, 800Hz, 900Hz, or 1000HZ.

[0023] In one embodiment, the strobed light is not only strobed at a certain strobe frequency, but also has a certain wavelength. In such an embodiment, the event camera may further be configured to filter light (e.g., hardware, software and / or optical filtering) such that it only processes light having the certain wavelength emitted into the environment.

[0024] In one embodiment, the strobed light is not in the visual spectrum. In one embodiment, the strobed light is in the near infrared or infrared spectrum. In one particular embodiment, the strobed light has a wavelength of 850nm. Although not necessary, it is generally preferred to avoid wavelengths in the radio / microwave range as these may cause electromagnetic interference. Likewise, although not necessary, it is generally preferred to avoid wavelengths shorter than UV as such wavelengths can be harmful. In one embodiment, the strobed light has a narrow bandwidth. In one embodiment, the narrow bandwidth is no greater than + / - 25 or + / -10 nm of the certain wavelength. In one embodiment, the narrow bandwidth is produced by LEDs or lasers.

[0025] The tracking system using an event camera with strobe lighting is described below to operate within a specific environment, specifically, the cockpit of a fighter jet. It should be understood that the principles of the present inventions relate to other vehicles, situations, environments, etc. It should also be understood that theprinciples of the present inventions relate to outside-in, inside-out and other tracking techniques. For example, the event camera and / or light may be placed in an environment to track things in the environment and / or the event camera and / or light may be mounted on a person, object, etc. to track its movements relative to the environment.

[0026] For example, such an IR strobe light may be positioned next to such an event camera in a cockpit of an airplane. The image data captured by the event camera would be restricted to IR, the known strobing frequency and changing pixels, which reduces the amount of data processing while maintaining the fast capture rate of the event camera. By reducing the amount of data to be processed, the overall process is dramatically sped up, reduces the bandwidth for communicating the data, and increases the robustness of the tracking process.

[0027] In one embodiment, the light may be pulsed at 1000Hz, which results in the event camera imaging two events at every pulse, one corresponding to the light turning on and one to it turning off, or processing events at 2000Hz. The event camera may also be programmed with a ‘cool down’ period where it is programmed not to respond for a period of time after recognizing an event. The light may also be polarized in conjunction with filtering for the polarized light at the event camera to reduce stray light. The inventors have found that horizontally polarized light within a cockpit environment reduces stray light better than vertically polarized light. Tracking within different environments may benefit from different polarizations strategies.

[0028] An event camera set to receive strobed IR light may be mounted on the interior of an airplane cockpit and positioned to image a helmet worn by a pilot of the airplane. An IR light source set to strobe may be positioned to emit light towards the helmet from a similar perspective as the event camera. The helmet may be fitted with one or more IR reflectors. The emitted IR light may be reflected off of the reflectors and received by the event camera. The event camera may then image changes in the reflected light patterns as part of a process to determine the helmet’s position, and hence viewing angle of the pilot.

[0029] Fig. 1 shows one embodiment of the system of the present invention as it related to a helmet.

[0030] Depending on the placement of the reflectors, or other imaging sites, the expected pattern may not be the same shape from different perspectives. The pattern’s imaged perspective depends not only on the imaged points but the placement of the camera.

[0031] A sample image capture from an event camera imaging a helmet with reflectors described in Fig. 1 is illustrated in Fig. 2.

[0032] The point cloud visualized in the event camera image in figure 2 represents changes in the reflected light from the helmet reflectors, which can be interpreted as physical movements of the reflectors and hence helmet movements in that area. Not all areas move the same amount because of the perspective of the camera in relation to each of the reflectors. If the camera is behind the helmet and the helmet tips to the side, for example, the reflectors towards the edges, from the camera’s perspective, will move more than ones near the center.

[0033] The strobe light method for image capture drastically cuts the amount of data processed and as a result drastically increases the speed of the process. This is an important improvement because the more events detected close in time can be associated to detect small movements.

[0034] In some embodiments, one or more event cameras may capture one or more markings on an object within line if sight of the cameras. When three or more markings, which may form a triangle, are visible to only one camera the relative location of each marking to the other as captured by the camera is sufficient to ascertain an orientation of the object, such as a wearable, upon which the markings are arranged. When more than one camera captures the same one of the plurality of markings, it is possible to further ascertain the precise position of the marker. Combining this information, it is possible to precisely determine the position and orientation of the object.

[0035] The event camera detected events are collected and analyzed to determine helmet position and movement. Knowing the perspective of the camera as it relates to the reflectors on the helmet allows for a corrected shape to be calculated. Figure 3 illustrates how a corrected helmet shape may be generated from an imaged set of reflectors on a real helmet from a given camera perspective.

[0036] Figure 1c at Rw with the known relative orientations and perspective and the corrected shape Rim can be derived therefrom.

[0037] The corrected shape can then be used to accurately determine the position of the helmet and hence the viewing position of the pilot wearing the helmet.

[0038] Head tracking systems may identify the position of a person or persons’ heads within a known environment. Prior art use of conventional head tracking solutions is generally too slow, not accurate enough, and / or error prone. Working to locate a pilot’s head within a cockpit of an airplane, the inventors discovered that electromagnetic noise in the cockpit is difficult to manage and may cause significant errors when using electromagnetic location technologies. The inventors further discovered that using infrared light triangulation is also prone to errors due to the highly reflective nature of the environment.

[0039] As a result, exemplary and non-limiting embodiments relate to a data fusion computer process using Lidar and inertial measurement unit (IMU) data feeds for the estimation of a head position within a known environment. An IMU may be mounted on the helmet of a user (e.g., pilot) and the IMU may track the movements of the head such that the location of the head may be predicted. Simultaneously, Lidar may track the user’s head. The two data feeds may be fused to accurately track the head position and movement. The process involves tracking the head movements using the IMU and then correcting IMU drift by comparing the IMU predicted position with a Lidar determined position. The periodic calibration of the IMU prediction with the Lidar location is done throughout the tracking process leading to a very fast determination of the head position (e.g., less than 5ms). to the time-of-flight location and position calculation at a coincidental or near coincidental time(s) of data acquisition. The comparison may be used to re-calibrate the IMU location and position calculation. The re-calibration may be done each time the Lidar and the IMU have data acquisitions at coincidental or near coincidental times.

[0040] In some embodiments, data from more than one platform may be fused to increase accuracy. In some embodiments, mmWave information may be fused with GPS data to reduce error in determining the position of display systems.Likewise, in a cockpit / cabin implementation data from event cameras may be fused with lidar data to reduce error in determining the position of display systems.

[0041] As described herein elsewhere, an event camera system may be used to track the movements of an object such as a person, body part, head, helmet, head worn device, etc. In one embodiment, the object is a wearable. In one embodiment, the wearable is headgear. In one embodiment, the headgear is a helmet. In one embodiment, the helmet is a pilot’s helmet.

[0042] In some instances, the viewing device, such as a helmet or wearable display device may include markings or targets distributed about the device. One challenge with such a system is the necessity to capture video of an environment in which there is a viewing device. This need arises from the need to capture a series of images in rapid succession in an environment wherein the viewing device is moving. There is often an inverse correlation between the pixel resolution of video signals and the resolution of individual frames.

[0043] In instances where there is accessible a CAD description of the interior of a cockpit, a mounted camera (e.g., single or multiple cameras), such as an event camera, may serve to identify the position in space of aspects of a helmet. When the geometry of the interior of a cockpit is known, it may be possible to identify the position of one or more features of a helmet with respect to the position of other CAD mapped features in the cockpit on a two-dimensional image captured by the camera. Using this information, it may be possible to determine the position in three dimensional space of the helmet. As described above, when data including, but not limited to, Lidar, IMU measurements and camera imagery is fused, the resulting helmet position data may be more accurate and mire timely than position data derived from any sunset of such data sources.

[0044] In embodiments an event camera may be mounted on the interior of a vehicle (e.g., cockpit of an airplane), on a user (e.g., on a person’s body), on a user’s head (e.g., a head mounted device, head mounted display, helmet) or otherwise located to image surroundings in an effort to identify movements and / or viewing direction of a user in an XR environment.

[0045] These and other advantages maybe realized in accordance with the specific embodiments described as well as other variations. It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other embodiments and modifications within the spirit and scope of the claims will be apparent to those of skill in the art upon reviewing the above description. The scope of the invention should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

CLAIMSWhat is claimed is:

1. A method of tracking a position of a wearable worn by a user as an indication of a user’s position, said wearable comprising a plurality of spatially-oriented markers, said method comprising:(a) causing said plurality of spatially-oriented markers to transmit strobed light of a certain wavelength at a certain strobe frequency in a light pattern indicative of the position and orientation of said wearable;(b) recording image data of said plurality of spatially orientated markers only upon a change in said light pattern, said image data being restricted to light of said certain wavelength at said strobe frequency in a changed light pattern; and(c) determining said position and orientation of said wearable based on said image data.

2. The method of claim 1, wherein said spatially-oriented markers are active markers emitting said strobed light.

3. The method of claim 1, wherein said spatially-oriented markers are passive markers configured to reflect said strobed light at said certain wavelength, and step (a) comprises exposing said plurality of spatially-oriented markers to said strobed light having said certain strobe frequency4. The method of claim 3, wherein step (a) comprising using a strobe light to produce strobed light.

5. The method of claim 1, wherein step (b) comprises receiving received light and filtering received light to process just filtered light of said certain wavelength at said strobe frequency for a change of said light pattern.

6. The method of claim 1, wherein said strobed light has a certain polarization, and wherein said image data is further restricted to light having said certain polarization7. The method of claim 6, wherein said certain polarization is a horizonal polarization.

8. The method of claim 1, wherein said step (b) is performed by an event camera.

9. The method of claim 8, wherein said event camera detects said strobed light turning on and off as two separate events, and each event indicates a light pattern of said spatially-oriented markers.

10. The method of claim 8, wherein said event camera is configured with a certain perspective relative to said plurality of spatially-oriented markers.

11. The method of claim 10, wherein said strobe light is proximate the event camara12. The method of claim 11, wherein said strobe light has a perspective relative to said plurality of spatially-oriented markers similar to said certain perspective.

13. The method of claim 1, wherein in step (c), said light pattern determines the position and orientation of said wearable.

14. The method of claim 1, wherein said strobed frequency is no less than 100Hz, 500Hz, 600Hz, 700Hz, 800Hz, or 900 Hz, or 1000HZ.

15. The method of claim 1, wherein said strobed light is not in the visual spectrum.

16. The method of claim 15, wherein said strobed light is in the infrared spectrum.

17. The method of claim 16, wherein said strobed light has a wavelength of 850nm.

18. The method of claim 1, wherein said strobed light has a narrow bandwidth.

19. The method of claim 18, wherein said narrow bandwidth is produced by LEDs or lasers.

20. The method of claim 18, wherein said narrow bandwidth is no greater than + / - 25 or 10 nm of said certain wavelength.

21. A system of tracking a position of a wearable worn by a user as an indication of a user’s position, said wearable comprising a plurality of spatially-oriented markers, said system comprising:(a) a strobe light configured to cause said spatially-oriented markers to transmit strobed light having a certain strobe frequency and a certain wavelength in a light pattern indicative of the position and orientation of said wearable;(b) an event camera for recording image data of said plurality of spatially-oriented markers upon a change in said light pattern, said image data being restricted to light of said certain wavelength at said strobe frequency in a changed light pattern; and(c) a processor for processing said image data to determine said position and orientation of said wearable.

22. The system of claim 21, further comprising said wearable23. The system of claim 22, wherein said wearable is a helmet24. The system of claim 23, wherein said helmet is an airman’s helmet.

25. The system of claim 24, wherein said airman’s helmet is a fighter pilot’s helmet.

26. The system of claim 21, wherein said spatially-oriented markers are active and comprise said strobe light to emit said strobed light.

27. The system of claim 21, wherein said spatially-oriented markers are passive markers configured to reflect said strobed light at said certain wavelength, and said strobe light comprises a strobe light discrete from said wearable.

28. The system of claim 21, wherein said strobed light has a certain polarization, and wherein said image data is further restricted to light having said certain polarization29. The system of claim 28, wherein said certain polarization is a horizonal polarization.

30. The system of claim 21, wherein said event camera detects said strobed light turning on and off as two separate events, and each event indicates a light pattern of said spatially-oriented markers.

31. The system of claim 21, wherein said event camera is configured with a certain perspective relative to said plurality of spatially-oriented markers.

32. The system of claim 31, wherein said strobe light is proximate the event camara33. The system of claim 32, wherein said strobe light has a perspective relative to said plurality of spatially-oriented markers similar to said certain perspective.

34. The system of claim 21, wherein said even camera is a neuromorphic camera.

35. The system of claim 34, wherein said neuromorphic camera generates a model of the position of the helmet based on one or more pixels of the neuromorphic sensor indicating a light change from each of the plurality of spatially-oriented markers on the helmet.

36. A method of tracking a position of a helmet within an environment to establish a user’s field of view, comprising:a. strobing infrared light at a known frequency within the environment to produce strobed infrared light from a plurality of predefined discrete locations on the helmet; b. filtering light from the environment to pass a signal comprising the infrared light at the known frequency to enhance the detection of the strobed infrared light from the plurality of predefined discrete locations on the helmet; c. processing the filtered light with a neuromorphic sensor to generate a model of the position of the helmet based on one or more pixels of the neuromorphic sensor indicating a light change from each of the plurality of predefined discrete locations on the helmet.

37. The method of claim 36, wherein the predefined locations are active markers producing the IR light.

38. The method of claim 36, wherein the predefined locations are passive markers reflecting the IR light produced in the environment.

39. The method of claim 36, wherein the strobed infrared light is polarized and the filtering step passes polarized light.

40. The method of claim 36, wherein the neuromorphic sensor detects when the infrared light turns on and off as two separate events and each event indicates a position of the predefined locations.

41. The method of claim 36, wherein the neuromorphic sensor has a plurality of light detectors arranged in a matrix each of which detects when light received at the pixel changes, wherein each change is mapped into a helmet position model for the estimation of the position of the helmet.

Images