Augmented Reality Deep Gesture Network
By combining an electromagnetic tracking system with an inertial measurement unit, the accuracy and latency issues of head pose detection in augmented reality systems were resolved, achieving high-precision, low-latency head pose tracking and improving the user experience.
Patent Information
- Application Number
- CN201980038758.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-06-14
- Filing Date
- 2019-06-14
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-06-14
AI Technical Summary
Existing augmented reality systems struggle to achieve high-precision and low-latency head pose detection, causing virtual objects to appear unstable when the user's head moves, thus affecting the user experience.
An electromagnetic tracking system is used in conjunction with inertial measurement unit and camera data. The position and orientation of the sensor are determined by an electromagnetic field transmitter and sensor. Attitude tracking is performed by combining IMU data, and dynamic frequency tuning and noise cancellation techniques are used to optimize system performance.
It achieves high-precision, low-latency head pose tracking, improving the stability of virtual objects in the real world and user experience, and reducing the impact of electromagnetic interference.
Smart Images

Figure CN112292690B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 685,262, filed on June 14, 2018, entitled “Augmented Reality Deep Gesture Network,” which is incorporated herein by reference in its entirety. This application also incorporates by reference the following patent application in its entirety: U.S. Patent Application No. 15 / 683,664, filed on August 22, 2016, entitled “Augmented reality display device with deep learning sensors.” Technical Field
[0003] The present disclosure relates to augmented reality systems that use deep learning neural networks to combine multiple sensor inputs (e.g., inertial measurement units, cameras, depth sensors, microphones) into a unified path including shared layers and upper layers that perform multiple functions (e.g., facial recognition, location and mapping, object detection, depth estimation, etc.). Background Art
[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in such a way that they appear real or may be perceived as real. Virtual reality or "VR" scenarios generally involve presenting digital or virtual image information that is opaque to other actual real-world visual input; augmented reality or "AR" scenarios generally involve presenting digital or virtual image information as an enhancement to the visualization of the actual world around the user. Summary of the invention
[0005] In one aspect, a head-mounted augmented reality (AR) device may include a hardware processor programmed to receive different types of sensor data from multiple sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and use the different types of sensor data and a Hydra neural network to determine an event among multiple events (e.g., facial recognition, visual search, gesture recognition, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization). In another aspect, a system for training a Hydra neural network is also disclosed. In another aspect, a method for training a Hydra neural network or using a trained Hydra neural network to determine an event among multiple different types of events is disclosed.
[0006] In one embodiment, a computer-implemented method for identifying gestures using a random forest model includes training a random forest model. The method also includes acquiring image data. The method also includes clustering multiple pixels from the image data to generate multiple clusters. Moreover, the method includes analyzing the multiple clusters using a rejection cascade to generate multiple selected candidates. In addition, the method includes analyzing the multiple selected candidates using a classification decision tree from the random forest model. The method also includes skeletonizing the multiple selected candidates to generate a one dimension plus branches hand model. The method also includes analyzing the one dimension plus branches hand model using a regression decision tree from the random forest model.
[0007] In one or more embodiments, skeletonizing the plurality of selected candidates includes using a distance transform and filtering the results of the distance transform. Skeletonizing the plurality of selected candidates may include using a graphical tool. Skeletonizing the plurality of selected candidates may include marking portions of the plurality of selected candidates. The portions of the plurality of selected candidates may include joints of the hand. Skeletonizing the plurality of selected candidates may reduce the number of anchor points analyzed by the regression decision tree. Skeletonizing the plurality of selected candidates may increase the speed of the computer-implemented method and may increase the accuracy of the computer-implemented method.
[0008] In one or more embodiments, training the random forest model includes labeling and manual verification. Training the random forest model may include randomly selecting a plurality of image features and using the selected plurality of image features to build a decision tree. Training the random forest model may include using a thermal camera to acquire training images of the hand. Training the random forest model may also include using a temperature regulating substance. The thermal camera may be a hyperspectral camera.
[0009] In one or more embodiments, the image data includes amplitude data and depth data. Clustering multiple pixels from the image data to generate multiple clusters may include using a DB scan algorithm. Using a rejection cascade to analyze multiple clusters to generate multiple selected candidates may include determining the size of the selected candidates. Using a rejection cascade to analyze multiple clusters to generate multiple selected candidates may include determining the location of the selected candidates. Using a classification decision tree from a random forest model to analyze multiple selected candidates may include determining whether the selected candidate is a hand, a left hand or a right hand, or a back of hand view or a palm view. Using a regression decision tree from a random forest model to analyze a one-dimensional plus branch hand model may include determining the location of the center of the hand or the location of the joints of the hand.
[0010] In one or more embodiments, the method further comprises averaging multiple results from multiple decision trees of the random forest model. Averaging multiple results can reduce the variance of the results from the random forest model.
[0011] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects and advantages will become apparent from the specification, drawings and claims. Neither this summary nor the following detailed description is intended to define or restrict the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 An illustration of an augmented reality scene with certain virtual reality objects and certain physical objects viewed by a person is depicted.
[0013] Figures 2A-2D An example of a wearable system is schematically shown.
[0014] Figure 3 Coordination between cloud computing assets and local processing assets is schematically illustrated.
[0015] Figure 4 An example system diagram of an electromagnetic (EM) tracking system is schematically illustrated.
[0016] Figure 5 is a flow chart describing example functionality of an embodiment of an electromagnetic tracking system.
[0017] Figure 6 An example of an electromagnetic tracking system combined with an AR system is schematically shown.
[0018] Figure 7 is a flowchart describing the functionality of an example of an electromagnetic tracking system in the case of an AR device.
[0019] Figure 8 An example of components of an embodiment of an AR system is schematically shown.
[0020] Figures 9A-9F An example of a quick release module is schematically shown.
[0021] Fig.10 A head mounted display system is schematically shown.
[0022] Fig.11A and Fig. 11B An example of an electromagnetic sensing coil coupled to a head mounted display is schematically shown.
[0023] Figures 12A-12E An example configuration of a ferrite core that may be coupled to an electromagnetic sensor is schematically shown.
[0024] Fig.13A is a block diagram schematically illustrating an example of a frequency division multiplexing (FDM) EM transmitter circuit (EM transmitter).
[0025] Fig. 13B is a block diagram schematically illustrating an example of a frequency division multiplexed EM receiver circuit (EM sensor).
[0026] Fig. 13C is a block diagram schematically illustrating an example of a time division multiplexed (TDM) EM transmitter circuit.
[0027] Fig.13D is a block diagram schematically illustrating an example of a dynamically tunable circuit for an EM transmitter.
[0028] Fig.13E It is shown that the dynamic tuning Fig.13D Graph of an example of resonance achieved by the circuit shown in FIG.
[0029] Fig.13F An example of a timing diagram showing a time division multiplexed EM transmitter and receiver.
[0030] Figure 13G An example of scanning timing of a time-division multiplexed EM transmitter and receiver is shown.
[0031] Fig.13H is a block diagram schematically illustrating an example of a TDM receiver in an EM tracking system.
[0032] Fig.13I is a block diagram schematically illustrating an example of an EM receiver without automatic gain control (AGC).
[0033] Fig.13J is a block diagram schematically illustrating an example of an EM transmitter employing AGC.
[0034] Fig.14 and Fig.15 is a flow chart illustrating an example of gesture tracking employing an electromagnetic tracking system in a head-mounted AR system.
[0035] Fig.16A and Fig. 16B Examples of components of other embodiments of the AR system are schematically shown.
[0036] Fig.17A An example of a resonant circuit in a transmitter in an electromagnetic tracking system is schematically shown.
[0037] Fig. 17B It is shown Fig.17A An example graph of the resonance of a resonant circuit at 22kHz.
[0038] Fig. 17C is a graph showing an example of a current flowing through a resonance circuit.
[0039] Fig.17D and Fig.17E An example of a dynamically tunable configuration of a resonant circuit in an EM field transmitter for an electromagnetic tracking system is schematically shown.
[0040] Fig.17F By changing Fig.17E A graph illustrating an example of dynamically tuned resonance is shown with respect to the capacitance value of capacitor C4 in the example circuit shown.
[0041] Figure 17G is a graph showing examples of maximum current achieved at various resonant frequencies.
[0042] Fig.18A is a block diagram schematically illustrating an example of an electromagnetic field sensor adjacent to an audio speaker.
[0043] Fig.18B is a block diagram schematically illustrating an example of an electromagnetic field sensor with a noise cancellation system that receives input from both the sensor and an external audio speaker.
[0044] Fig. 18C is a graph showing an example of how to invert and add signals to cancel magnetic interference caused by an audio speaker.
[0045] Fig.18D is a flow chart illustrating an example method for canceling interference received by an EM sensor in an EM tracking system.
[0046] Fig.19 Schematically illustrating the use of light patterns to assist calibration of a vision system.
[0047] Figures 20A-20C is a block diagram of example circuits of subsystems or components that may be used in a wearable display device.
[0048] Fig.21 is a graph showing an example of fused output from an IMU, an electromagnetic tracking sensor, and an optical sensor.
[0049] Figures 22A-22C Additional examples of electromagnetic sensing coils coupled to a head mounted display are schematically shown.
[0050] Figures 23A-23C An example of recalibrating a head mounted display using electromagnetic and acoustic signals is schematically shown.
[0051] Figures 24A-24DAdditional examples of recalibrating a head mounted display using a camera or a depth sensor are schematically illustrated.
[0052] Fig.25A and Fig.25B Techniques for resolving position ambiguity that may be associated with electromagnetic tracking systems are schematically illustrated.
[0053] Fig.26 An example of feature extraction and generation of sparse 3-D map points is schematically shown.
[0054] Fig. 27 is a flow chart illustrating an example of a method for vision-based posture computation.
[0055] Figures 28A-28F An example of sensor fusion is schematically shown.
[0056] Fig.29 An example of a Hydra neural network architecture is schematically shown.
[0057] Reference numerals may be repeated throughout the drawings to indicate corresponding relationships between reference elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION
[0058] Overview of AR, VR and Localization Systems
[0059] exist Figure 1 , an augmented reality scene (4) is depicted in which a user of AR technology sees a real-world park-like setting (6) featuring people, trees, buildings in the background, and a concrete platform (1120). In addition to these items, the user of AR technology also perceives that he "sees" a robotic statue (1110) standing on the real-world platform (1120), and an anthropomorphic, flying, cartoon-like avatar character (2) that appears to be a bumblebee, even though these elements (2, 1110) do not exist in the real world. It turns out that the human visual perception system is very complex, and it is challenging to produce VR or AR technology that promotes comfortable, natural-feeling, rich presentation of virtual image elements among other virtual or real-world image elements.
[0060] For example, a head-mounted AR display (or helmet-mounted display or smart glasses) is typically at least loosely coupled to the user's head and therefore moves when the user's head moves. If the display system detects the user's head movement, the data being displayed can be updated to account for the change in head posture.
[0061] As an example, if a user wearing a head mounted display views a virtual representation of a three-dimensional (3D) object on the display and walks around the area where the 3D object appears, the 3D object can be re-rendered for each viewpoint, giving the user the feeling that he or she is walking around the object occupying real space. If the head mounted display is used to present multiple objects within a virtual space (e.g., a rich virtual world), measurements of head pose (e.g., the position and orientation of the user's head) can be used to re-render the scene to match the user's dynamically changing head position and orientation and provide increased immersion in the virtual space.
[0062] In an AR system, the detection or calculation of head posture can facilitate the display system to render virtual objects so that they appear to occupy space in the real world in a way that makes sense to the user. In addition, the detection of the position and / or orientation of real objects (such as handheld devices (also referred to as "totems"), tactile devices) or other real physical objects related to the user's head or AR system can also facilitate the display system to present display information to the user, so that the user can effectively interact with certain aspects of the AR system. When the user's head moves in the real world, the virtual object can be re-rendered as a function of the head posture so that the virtual object appears to remain stable relative to the real world. At least for AR applications, the placement of virtual objects in spatial relationship with physical objects (e.g., presented as being spatially close to physical objects in two or three dimensions) may be a non-trivial problem. For example, head movement may significantly complicate the placement of virtual objects in the view of the surrounding environment. This is true whether the view is captured as an image of the surrounding environment and then projected or displayed to the end user, or whether the end user directly perceives the view of the surrounding environment. For example, head movement will likely cause the end user's field of view to change, which may require updating the position of various virtual objects displayed in the end user's field of view. In addition, head motion can occur in a variety of ranges and speeds. The head motion speed can not only change between different head motions, but also change within the range of a single head motion or across the range of a single head motion. For example, the head motion speed can initially increase from a starting point (e.g., linearly or not linearly) and can decrease when reaching an end point, thereby obtaining a maximum speed somewhere between the starting point and the node of the head motion. Fast head motion may even exceed the capabilities of a specific display or projection technology to render an image that looks uniform and / or smoothly moving to an end user.
[0063] Head tracking accuracy and latency (e.g., the time that elapses between when a user moves his or her head and when the image is updated and displayed to the user) have become challenges for VR and AR systems. Particularly for display systems that fill a large portion of the user's field of view with virtual elements, it is advantageous if the accuracy of head tracking is high and the overall system latency from the first detection of head motion to the update of light delivered by the display to the user's visual system is very low. If the latency is high, the system may create a mismatch between the user's vestibular and visual sensory systems and produce a user-perceived scenario that may cause motion sickness or motion sickness. If the system latency is high, the apparent position of virtual objects will appear unstable during rapid head movements.
[0064] In addition to head mounted display systems, other display systems can benefit from accurate and low latency head pose detection. These include head tracking display systems, where the display is not worn on the user's body, but is mounted, for example, on a wall or other surface. The head tracking display acts like a window onto the scene, and as the user moves his head relative to the "window", the scene is re-rendered to match the user's changing viewpoint. Other systems include head mounted projection systems, where the head mounted display projects light into the real world.
[0065] In addition, in order to provide a realistic augmented reality experience, the AR system can be designed to interact with the user. For example, multiple users can play a ball game with a virtual ball and / or other virtual objects. One user can "catch" the virtual ball and throw it back to another user. In another embodiment, a totem (e.g., a real bat communicatively coupled to the AR system) can be provided to the first user to hit the virtual ball. In other embodiments, a virtual user interface can be presented to the AR user to allow the user to select one of many options. The user can use a totem, a haptic device, a wearable component, or simply touch the virtual screen to interact with the system.
[0066] Detecting the user's head pose and orientation, as well as detecting the physical position of real objects in space enables AR systems to display virtual content in an effective and enjoyable manner. However, although these functions are key to AR systems, they are difficult to implement. In other words, the AR system can identify the physical position of a real object (e.g., a user's head, a totem, a tactile device, a wearable component, a user's hand, etc.) and associate the physical coordinates of the real object with the virtual coordinates corresponding to one or more virtual objects being displayed to the user. This typically requires highly accurate sensors and sensor recognition systems that track the position and orientation of one or more objects at a fast rate. Current methods cannot perform positioning at a satisfactory speed or accuracy standard.
[0067] Therefore, better positioning systems are needed in the case of AR and VR devices.
[0068] Example AR and VR systems and components
[0069] refer to Figures 2A-2D , showing some common component options. Figures 2A-2D In the detailed description section following the discussion of the present invention, various systems, subsystems, and components are presented for addressing the goal of providing a high-quality, comfortable-perceiving display system for human VR and / or AR.
[0070] like Figure 2A As shown in , an AR system user (60) is depicted as wearing a head-mounted assembly (58) featuring a frame (64) structure coupled to a display system (62) located in front of the user's eyes. A speaker (66) is coupled to the frame (64) in the configuration shown and is located near the user's ear canal (in one embodiment, another speaker not shown is located near the user's other ear canal to provide stereo / shapeable sound control). The display (62) is operably coupled (68) (such as by a wired lead or a wireless connection) to a local processing and data module (70), which can be mounted in a variety of configurations, such as fixedly attached to the frame (64), fixedly attached to the display system (62), such as a wireless connection. Figure 2B The helmet or hat (80) shown in the embodiment is embedded in the earphones to Figure 2C The backpack configuration shown in the embodiment of FIG. 1 can be removably attached to the torso (82) of the user (60), or can be used as Figure 2D The belt-coupled configuration shown in the embodiment is removably attached to the hips (84) of the user (60).
[0071] The local processing and data module (70) may include a power efficient processor or controller, and digital memory such as flash memory, both of which may be used to assist in processing, caching and storing data that is: a) captured from a sensor (such as an image capture device (such as a camera), microphone, inertial measurement unit, accelerometer, compass, GPS unit, wireless device and / or gyroscope) that may be operably coupled to the frame (64); and / or b) acquired and / or processed using a remote processing module (72) and / or a remote data repository (74), possibly for delivery to the display (62) after such processing or retrieval. The local processing and data module (70) may be operably coupled (76, 78) to the remote processing module (72) and the remote data repository (74), such as via a wired or wireless communication link, so that these remote modules (72, 74) are operably coupled to each other and can be used as a resource for the local processing and data module (70).
[0072] In one embodiment, the remote processing module (72) may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository (74) may include a relatively large-scale digital data storage facility that is available via the Internet or other network configuration in a "cloud" resource configuration. In one embodiment, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from any remote module.
[0073] Reference now Figure 3 , schematic diagram illustrating coordination between cloud computing assets (46) and local processing assets, which may reside, for example, in a head-mounted component (58) coupled to a user's head (120) and a local processing and data module (70) coupled to a user's belt (308; thus component 70 may also be referred to as a "belt pack" 70), as shown Figure 3 In one embodiment, cloud (46) assets (such as one or more server systems (110)) are operably coupled (40, 42) directly (115) to one or both of the local computing assets, such as the processor and memory configuration coupled to the user's head (120) and belt (308) as described above, such as via a wired or wireless network (wireless is preferred for mobility, wired is preferred for certain high bandwidth or high data volume transmissions that may be required). These computing assets that are local to the user may also be configured (44) via a wired and / or wireless connection (such as described below with reference to Figure 8 In one embodiment, in order to maintain a low inertia and small size subsystem mounted to the user's head (120), the primary transmission between the user and the cloud (46) can be via a link between the subsystem mounted on the belt (308) and the cloud, where the head-mounted (120) subsystem primarily uses a wireless connection (such as an ultra-wideband ("UWB") connection) to the belt-based (308) subsystem for data communication, as currently employed in, for example, personal computing peripheral connection applications.
[0074] Through effective local and remote processing coordination, and appropriate display devices for users, such as Figure 2AThe user interface or user display system (62) shown, or a variation thereof, aspects of a world related to the user's current actual or virtual location can be transmitted or "delivered" to the user and updated in an efficient manner. In other words, a map of the world can be continuously updated at a storage location that may exist partially on the user's AR system and partially in a cloud resource. A map (also called a "navigable world model") can be a large database that includes raster images, 3-D and 2-D points, parametric information, and other information about the real world. As more and more AR users continue to capture information about their real environment (e.g., through cameras, sensors, IMUs, etc.), the map becomes more and more accurate and complete.
[0075] With a configuration as described above, where there is a model of the world that can reside on a cloud computing resource and be distributed from there, this world can be "streamed" to one or more users in a relatively low bandwidth form factor, preferably attempting to deliver real-time video data, etc. The augmented experience of a person standing near the statue (e.g., as Figure 1 ), a subset of which can be passed down to them and their local display device to complete the viewing. The person sitting at the remote display device (this could be as simple as a personal computer sitting on a desk) can effectively download the same portion of information from the cloud and render it on their display. In effect, a person physically present in a park near the statue might take a walk in that park with a friend located far away, who joins via virtual and augmented reality. The system would need to know where the street is, where the tree is, where the statue is - but with that information on the cloud, the friend who joins can download from the cloud aspect of the scene, and then begin walking as a local augmented reality relative to the person physically in the park.
[0076] Three-dimensional (3-D) points can be captured from the environment, and the pose (e.g., vector and / or origin position information relative to the world) of the camera that captured those images or points can be determined, so that the points or images can be "tagged" or associated with the pose information. The points captured by the second camera can then be used to determine the pose of the second camera. In other words, the second camera can be oriented and / or positioned based on a comparison with the tagged image from the first camera. This knowledge can then be used to extract textures, make maps, and create a virtual copy of the real world (because the two cameras are then registered around it).
[0077] Thus, at a basic level, in one embodiment, a system worn by a person may be utilized to capture 3-D points and produce 2-D images of the points, and these points and images may be sent to cloud storage and processing resources. They may also be cached locally with embedded pose information (e.g., cache tagged images); thus, the cloud may have ready (e.g., in available cache) tagged 2-D images (e.g., tagged with 3-D poses) as well as 3-D points. If the user is viewing something dynamic, he may also send additional information up to the cloud related to motion (e.g., if viewing another person's face, the user may obtain a texture map of the face and push it up at an optimized frequency, even if the surrounding world is otherwise substantially static). More information about object identifiers and navigable world models can be found in U.S. Patent Publication No. 2014 / 0306866, entitled “System and method for augmented and virtual reality,” which is incorporated herein by reference in its entirety along with the following additional disclosures relating to augmented and virtual reality systems such as those developed by Magic Leap, Inc. of Plantation, Florida: U.S. Patent Publication No. 2015 / 0178939; U.S. Patent Publication No. 2015 / 0205126; U.S. Patent Publication No. 2014 / 0267420; U.S. Patent Publication No. 2015 / 0302652; U.S. Patent Publication No. 2013 / 0117377; and U.S. Patent Publication No. 2013 / 0128230, each of which is incorporated herein by reference in its entirety.
[0078] GPS and other localization information may be used as input to such processing.Highly accurate positioning of the user's head, totems, gestures, haptic devices, etc. may be advantageous in order to display appropriate virtual content to the user.
[0079] The head mounted device (58) may include a display positionable in front of the eyes of a wearer of the device. The display may include a light field display. The display may be configured to present images to the wearer at multiple depth planes. The display may include a planar waveguide having a diffractive element. Examples of displays, head mounted devices, and other AR components that may be used with any of the embodiments disclosed herein are described in U.S. Patent Publication No. 2015 / 0016777. U.S. Patent Publication No. 2015 / 0016777 is incorporated herein by reference in its entirety.
[0080] Example of electromagnetic positioning
[0081] One method of achieving high-precision positioning may involve the use of an electromagnetic (EM) field coupled to an electromagnetic sensor strategically placed on a user's AR head device, belt pack, and / or other auxiliary device (e.g., totem, tactile device, gaming instrument, etc.). An electromagnetic tracking system typically includes at least an electromagnetic field transmitter and at least one electromagnetic field sensor. The electromagnetic field transmitter generates an electromagnetic field with a known spatial (and / or temporal) distribution in the environment of the wearer of the AR headset. The electromagnetic field sensor measures the electromagnetic field generated at the sensor location. Based on these measurements and knowledge of the distribution of the generated electromagnetic field, the pose (e.g., position and / or orientation) of the field sensor relative to the transmitter can be determined. Therefore, the pose of the object to which the sensor is attached can be determined.
[0082] Reference now Figure 4 , shows an example system diagram of an electromagnetic tracking system (e.g., such as those developed by the Biosense division of Johnson & Johnson Corporation, Polhemus, Inc. of Colchester, Vt., manufactured by Sixense Entertainment, Inc. of Los Gatos, Calif., and other tracking companies). In one or more embodiments, the electromagnetic tracking system includes an electromagnetic field transmitter 402 that is configured to emit a known magnetic field. Figure 4 As shown in , the electromagnetic field transmitter can be coupled to a power supply (eg, electrical current, battery, etc.) to provide power to the transmitter 402 .
[0083] In one or more embodiments, the electromagnetic field transmitter 402 includes several coils (e.g., at least three coils placed perpendicular to each other to produce fields in the X, Y, and Z directions) that generate a magnetic field. The magnetic field is used to establish a coordinate space (e.g., an XYZ Cartesian coordinate space). This allows the system to map the position of the sensor relative to a known magnetic field (e.g., (X, Y, Z) position) and helps determine the position and / or orientation of the sensor. In one or more embodiments, the electromagnetic sensors 404a, 404b, etc. can be attached to one or more real objects. The electromagnetic sensor 404 may include a smaller coil in which a current can be induced by the emitted electromagnetic field. In general, the "sensor" component (404) may include a small coil or loop, such as a set of three differently oriented (e.g., such as orthogonally oriented relative to each other) coils, the coils are coupled together within a small structure such as a cube or other container, the coils are positioned / oriented to capture the incident magnetic flux from the magnetic field emitted by the transmitter (402), and by comparing the current induced through these coils, and knowing the relative position and orientation of the coils relative to each other, the relative position and orientation of the sensor relative to the transmitter can be calculated.
[0084] One or more parameters related to the behavior of the coil and inertial measurement unit ("IMU") components operably coupled to the electromagnetic tracking sensor may be measured to detect the position and / or orientation of the sensor (and the object to which it is attached) relative to the coordinate system to which the electromagnetic field transmitter is coupled. In one or more embodiments, multiple sensors may be used relative to the electromagnetic transmitter to detect the position and orientation of each sensor within the coordinate space. The electromagnetic tracking system may provide a position in three directions (e.g., X, Y, and Z directions) and further in two or three orientation angles. In one or more embodiments, the measurements of the IMU may be compared with the measurements of the coil to determine the position and orientation of the sensor. In one or more embodiments, both electromagnetic (EM) data and IMU data, as well as various other data sources (such as cameras, depth sensors, and other sensors), may be combined to determine the position and orientation. This information may be sent (e.g., wireless communication, Bluetooth, etc.) to the controller 406. In one or more embodiments, the posture (or position and orientation) may be reported at a relatively high refresh rate in a conventional system. Traditionally, the electromagnetic field transmitter is coupled to a relatively stable and large object, such as a table, operating table, wall or ceiling, and one or more sensors are coupled to a smaller object, such as a medical device, a handheld game piece, etc. Alternatively, as described below with reference to Figure 6 As described, various features of an electromagnetic tracking system can be employed to produce a configuration in which a change or amount of change in position and / or orientation between two objects moving in space relative to a more stable global coordinate system can be tracked; in other words, Figure 6 , a configuration is shown in which a variation of an electromagnetic tracking system can be used to track position and orientation changes between a head-mounted component and a hand-held component while otherwise determining the head pose relative to a global coordinate system (e.g., a user's local room environment), such as by using simultaneous localization and mapping (“SLAM”) techniques using an outward-facing capture camera that can be coupled to the head-mounted component of the system.
[0085] Controller 406 can control electromagnetic field generator 402, and can also capture data from various electromagnetic sensors 404. It should be understood that the various components of the system can be coupled to each other by any electromechanical or wireless / Bluetooth means. Controller 406 can also include data about known magnetic fields, and a coordinate space associated with the magnetic field. This information is then used to detect the position and orientation of the sensor relative to the coordinate space corresponding to the known electromagnetic field.
[0086] One advantage of electromagnetic tracking systems is that they produce highly accurate tracking results with minimal latency and high resolution. Additionally, electromagnetic tracking systems do not necessarily rely on optical trackers and can easily track sensors / objects that are not in the user's line of sight.
[0087] It should be understood that the strength of the electromagnetic field v decreases as a cubic function of the distance r from the coil transmitter (e.g., electromagnetic field transmitter 402). Therefore, an algorithm can be used based on the distance away from the electromagnetic field transmitter. The controller 406 can be configured with such an algorithm to determine the position and orientation of the sensor / object at different distances away from the electromagnetic field transmitter. As the sensor moves farther and farther away from the electromagnetic transmitter, the electromagnetic field strength drops rapidly, and optimal results in terms of accuracy, efficiency, and low latency can be achieved at closer distances. In a typical electromagnetic tracking system, the electromagnetic field transmitter is powered by an electric current (e.g., a plug-in power supply) and has a sensor located within a radius of 20 feet away from the electromagnetic field transmitter. In many applications, including AR applications, a shorter radius between the sensor and the field transmitter may be more desirable.
[0088] Reference now Figure 5 , briefly describes an example flow chart describing the functionality of a typical electromagnetic tracking system. At 502, a known electromagnetic field is emitted. In one or more embodiments, a magnetic field transmitter can generate a magnetic field, and each coil can generate an electric field in one direction (e.g., X, Y, or Z). The magnetic field can be generated using an arbitrary waveform. In one or more embodiments, the magnetic field component along each axis can oscillate at a slightly different frequency than other magnetic field components along other directions. At 504, a coordinate space corresponding to the electromagnetic field can be determined. For example, Figure 4 The controller 406 can automatically determine the coordinate space around the transmitter based on the electromagnetic field. At 506, the behavior of the coil at the sensor (which can be attached to a known object) can be detected. For example, the current induced at the coil can be calculated. In other embodiments, the rotation of the coil or any other quantifiable behavior can be tracked and measured. At 508, the behavior can be used to detect the position or orientation of the sensor and / or the known object. For example, the controller 406 can consult a mapping table that associates the behavior of the coil at the sensor with various positions or orientations. Based on these calculations, the position in the coordinate space and the orientation of the sensor can be determined.
[0089] In the case of an AR system, one or more components of the electromagnetic tracking system may need to be modified to facilitate accurate tracking of moving parts. As described above, in many AR applications, it may be necessary to track the user's head posture and orientation. Accurately determining the user's head posture and orientation allows the AR system to display the correct virtual content to the user. For example, a virtual scene may include a monster hidden behind a real building. Depending on the posture and orientation of the user's head relative to the building, the view of the virtual monster may need to be modified so that a real AR experience is provided. Alternatively, the position and / or orientation of a totem, a tactile device, or some other tool for interacting with virtual content may be important for enabling an AR user to interact with the AR system. For example, in many gaming applications, an AR system may detect the position and orientation of a real object relative to virtual content. Alternatively, when a virtual interface is displayed, the position of a totem, a user's hand, a tactile device, or any other real object configured to interact with the AR system may be known relative to the displayed virtual interface so that the system understands commands, etc. Traditional positioning methods including optical tracking and other methods are often plagued by high latency and low resolution problems, which makes it challenging to render virtual content in many augmented reality applications.
[0090] In one or more embodiments, Figure 4 and Figure 5 The electromagnetic tracking systems discussed may be adapted for AR systems to detect the position and orientation of one or more objects relative to an emitted electromagnetic field. Typical electromagnetic systems tend to have large and bulky electromagnetic transmitters (e.g., Figure 4 402 in ), which is problematic for head-mounted AR devices. However, smaller electromagnetic transmitters (e.g., in the millimeter range) can be used to transmit known electromagnetic fields in the context of AR systems.
[0091] Reference now Figure 6, the electromagnetic tracking system can be combined with the AR system as shown, and the electromagnetic field transmitter 602 is combined as part of the handheld controller 606. The controller 606 can move independently relative to the AR headset (or belt bag 70). For example, the user can hold the controller 606 in his or her hand, or the controller can be mounted to the user's hand or arm (for example, as a ring or bracelet or as part of a glove worn by the user). In one or more embodiments, the handheld controller can be a totem used in a game scene (for example, a multi-degree-of-freedom controller) or provide a rich user experience in an AR environment or allow the user to interact with the AR system. In other embodiments, the handheld controller can be a tactile device. In other embodiments, the electromagnetic field transmitter can be simply combined as part of the belt bag 70. The handheld controller 606 can include a battery 610 or other power supply for powering the electromagnetic field transmitter 602. It should be understood that the electromagnetic field transmitter 602 can also include an IMU 650 component or be coupled to an IMU 650 component, which is configured to help determine the location and / or orientation of the electromagnetic field transmitter 602 relative to other components. This may be particularly advantageous where both the field emitter 602 and the sensor (604) are mobile. Figure 6 , placing the electromagnetic field transmitter 602 in the handheld controller rather than the belt pack helps ensure that the electromagnetic field transmitter does not compete for resources at the belt pack, but instead uses its own battery power at the handheld controller 606. In other embodiments, the electromagnetic field transmitter 602 can be provided on the AR headset, and the sensor 604 can be provided on the controller 606 or the belt pack 70.
[0092] In one or more embodiments, electromagnetic sensors 604 may be placed at one or more locations on the user's headset along with other sensing devices such as one or more IMUs or additional magnetic flux capture coils 608. For example, Figure 6As shown in , sensors (604, 608) can be placed on one or both sides of the headset (58). Since these sensors are designed to be quite small (and therefore may not be very sensitive in some cases), having multiple sensors can improve efficiency and accuracy. In one or more embodiments, one or more sensors can also be placed on the belt bag 70 or any other part of the user's body. The sensors (604, 608) can communicate wirelessly or via Bluetooth with a computing device that determines the posture and orientation of the sensor (and the AR headset to which it is attached). In one or more embodiments, the computing device can reside at the belt bag 70. In other embodiments, the computing device can reside at the headset itself, or even at the handheld controller 606. In one or more embodiments, the computing device can then include a mapping database (e.g., a traversable world model, a coordinate space, etc.) to detect postures, determine the coordinates of real objects and virtual objects, and even connect to cloud resources and traversable world models.
[0093] As mentioned above, conventional electromagnetic transmitters may be too bulky for AR devices. Therefore, electromagnetic field transmitters can be designed to be compact using smaller coils than conventional systems. However, assuming that the strength of the electromagnetic field decreases as a cubic function of the distance away from the field transmitter, then the same as Figure 4 A shorter radius (eg, approximately 3 to 3.5 feet) between electromagnetic sensor 604 and electromagnetic field transmitter 602 may reduce power consumption compared to conventional systems described in detail in FIG.
[0094] In one or more embodiments, this aspect can be used to extend the life of the battery 610 that can power the controller 606 and the electromagnetic field transmitter 602. Alternatively, in other embodiments, this aspect can be used to reduce the size of the coil that generates the magnetic field at the electromagnetic field transmitter 602. However, to obtain the same magnetic field strength, the power may need to be increased. This allows a compact electromagnetic field transmitter unit 602 that can fit compactly at the handheld controller 606.
[0095] When using an electromagnetic tracking system for an AR device, several other changes may be made. While this pose reporting rate is quite good, an AR system may require a more efficient pose reporting rate. To this end, IMU-based pose tracking may be used in the sensor (in addition or alternatively). Advantageously, the IMU can be kept as stable as possible in order to improve the efficiency of the pose detection process. The IMUs can be designed so that they remain stable for at most 50-100 milliseconds. It should be understood that some embodiments may utilize an external pose estimator module (e.g., the IMU may drift over time) that enables pose updates to be reported at a rate of 10 to 20 Hz. By keeping the IMU stable at a reasonable rate, the rate of pose updates can be significantly reduced to 10 to 20 Hz (compared to higher frequencies in traditional systems).
[0096] If the electromagnetic tracking system could run at, for example, a 10% duty cycle (e.g., only pinging the ground truth every 100 milliseconds), this would be another way to save power at the AR system. This would mean that the electromagnetic tracking system would wake up every 10 milliseconds out of every 100 milliseconds to generate a pose estimate. This translates directly into power savings, which in turn could impact the size, battery life, and cost of the AR device.
[0097] In one or more embodiments, this reduction in duty cycle can be strategically exploited by providing two handheld controllers (not shown) instead of just one. For example, a user may be playing a game that requires two totems, etc. Or, in a multi-user game, two users may have their own totems / handheld controllers to play the game. When two controllers (e.g., symmetrical controllers for each hand) are used instead of one controller, the controllers can operate at an offset duty cycle. For example, the same concept can also be applied to controllers used by two different users playing a multiplayer game.
[0098] Reference now Figure 7, describes an example flow chart describing an electromagnetic tracking system in the context of an AR device. At 702, a portable (e.g., handheld) controller emits a magnetic field. At 704, an electromagnetic sensor (placed on a headset, belt pack, etc.) detects the magnetic field. At 706, a pose (e.g., position or orientation) of the headset / belt is determined based on the behavior of the coil / IMU at the sensor. At 708, the pose information is transmitted to a computing device (e.g., at a belt pack or headset). At 710, optionally, a mapping database (e.g., a navigable world model) can be consulted to associate real-world coordinates (e.g., for determination of the pose of the headset / belt) with virtual world coordinates. At 712, virtual content can be delivered to and displayed to the user at the AR headset (e.g., via a light field display as described herein). It should be understood that the above flowchart is for illustrative purposes only and should not be construed as limiting.
[0099] Advantageously, using something like Figure 6 An electromagnetic tracking system as outlined in enables gesture tracking (e.g., head position and orientation, position and orientation of totems, and position and orientation of other controllers). This allows an AR system to project virtual content (based at least in part on determined gestures) with greater accuracy and very low latency than optical tracking techniques.
[0100] refer to Figure 8 , showing a system configuration featuring a number of sensing components. A head mounted wearable component (58) is shown operatively coupled (68) to a local processing and data module (70), such as a belt pack, using a physical multi-core lead, which is also referred to below. Figures 9A-9F The control and quick release module (86) is characterized. The local processing and data module (70) is operably coupled (100) to the handheld component (606), here via a wireless connection such as low-power Bluetooth; the handheld component (606) can also be operably coupled (94) directly to the head-mounted wearable component (58), such as via a wireless connection such as low-power Bluetooth. Typically, high-frequency connections are required in the case of transmitting IMU data to coordinate posture detection of various components, such as in the range of hundreds or thousands of cycles / second or higher; such as through pairing of sensors (604) and transmitters (602), tens of cycles per second may be sufficient for electromagnetic positioning sensing. A global coordinate system (10) is also shown, which represents fixed objects in the real world around the user, such as walls (8).
[0101] The cloud resources (46) may also be operably coupled (42, 40, 88, 90) to the local processing and data module (70), to the head mounted wearable component (58), to resources that may be coupled to the wall (8) or other items that are fixed relative to the global coordinate system (10). Resources coupled to the wall (8) or having a known position and / or orientation relative to the global coordinate system (10) may include a wireless transceiver (114), an electromagnetic transmitter (602) and / or receiver (604), a beacon or reflector (112) configured to emit or reflect a given type of radiation (such as an infrared LED beacon), a cellular network transceiver (110), a RADAR transmitter or detector (108), a LIDAR transmitter or detector (106), a GPS transceiver (118), a poster or marker (122) having a known detectable pattern, and a camera (124).
[0102] The head-mounted wearable component (58) features components similar to those shown, except that the illumination emitter (130) is configured as an auxiliary camera (124) detector (such as an infrared emitter (130) for an infrared camera (124); and one or more strain gauges (116) are also present on the head-mounted wearable component (58) which can be fixedly coupled to a frame or mechanical platform of the head-mounted wearable component (58) and are configured to determine deflection of such platform between components such as electromagnetic receiver sensors (604) or display elements (62), where it can be valuable to understand whether the platform has bent, such as at a thinned portion of the platform, such as Figure 8 The part above the nose on the glasses-like platform shown in .
[0103] The head mounted wearable component (58) also features a processor (128) and one or more IMUs (102). Each component is preferably operably coupled to the processor (128). A handheld component (606) and a local processing and data module (70) are shown, which feature similar components. Figure 8As shown in , with so many sensing and connection tools, such a system may be heavy, power-hungry, large, and relatively expensive. However, for illustrative purposes, such a system can be used to provide a very high level of connectivity, system component integration, and position / orientation tracking. For example, with this configuration, various major mobile components (58, 70, 606) can be located in terms of their position relative to a global coordinate system using WiFi, GPS, or cellular signal triangulation; beacons, electromagnetic tracking (as described herein), RADAR, and LIDAR systems can provide further position and / or orientation information and feedback. Markers and cameras can also be used to provide further information about relative and absolute position and orientation. For example, various camera components (124), such as those shown as coupled to a head-mounted wearable component (58), can be used to capture data that can be used in a simultaneous localization and mapping protocol or "SLAM" to determine where the component (58) is located and how it is oriented relative to other components.
[0104] refer to Figures 9A-9F , depicting various aspects of the control and quick release module (86). Fig.9A The two housing components (132, 134) are coupled together using a magnetic coupling arrangement that may be enhanced with a mechanical latch. Buttons (136) for operation of the associated system may be included, such as an on / off button (round button) and up / down volume control (triangular button). The opposite end of the module 86 may be connected to electrical leads running between the local processing and data module (70) and the display (62), such as Figure 8 as shown in .
[0105] Fig. 9B A partial cutaway view is shown with the housing (132) removed, showing the button (136) and top printed circuit board (138) underneath. Fig. 9C , with the button (136) and the top printed circuit board (138) below removed, the female contact pin array (140) is visible. Fig.9D , with the corresponding portion of the housing (134) removed, the lower printed circuit board (142) is visible. With the lower printed circuit board (142) removed, as shown Fig.9E As shown in FIG. 1 , the male contact pin array ( 144 ) is visible.
[0106] refer to Fig.9FA cross-sectional view of the pins, at least one of the male or female pins is configured to be spring loaded so that they can be depressed along the longitudinal axis of each pin; the pins can be referred to as "spring pins" and typically include a highly conductive material such as copper or gold. The conductive material can be plated onto the pins (e.g., dip-plated or electroplated), and the width of the conductive material can be, for example, at least 25 μm of gold in some cases. When assembled, the configuration shown pairs 46 male pins with 46 corresponding female pins, and the entire assembly can be quickly released and decoupled by manually pulling the two housings (132, 134) apart and overcoming a magnetic interface (146) load developed using north and south pole magnets oriented around the perimeter of the pin array (140, 144). In one embodiment, a closure retention force of approximately 4 kg provided by the magnetic interface (146) offsets a load of approximately 2 kg compressing the 46 spring pins. The pins in the array can be approximately 1.3 mm apart, and the pins can be operably coupled to various types of conductors, such as twisted pair or other combinations, to support interfaces such as USB 3.0, HDMI 2.0 (for digital video), and I2S (for digital audio), transition minimized differential signaling (TMDS) for high-speed serial data, general purpose input / output (GPIO), and mobile interface (e.g., MIPI) configurations, battery / power connections, and high current analog lines and ground configured in one embodiment to up to approximately 4 amps and 5 volts.
[0107] In one embodiment, the magnetic interface (146) is generally rectangular and surrounds the pin array (140, 144) and is approximately 1 mm wide and 4.8 mm high. The inner diameter of the rectangular magnet is approximately 14.6 mm. The magnets surrounding the male pin array (144) can have a first polarity (e.g., north) and the magnets surrounding the female pin array (140) can have a second (opposite) polarity (e.g., south). In some cases, each magnet includes a mix of north and south poles, where opposing magnets have corresponding opposite polarities to provide magnetic attraction to help hold the housings (132, 134) together.
[0108] The spring pins in the array (140, 144) have a height in the range of 4.0 to 4.6 mm and a diameter in the range of 0.6 to 0.8 mm. Different pins in the array can have different heights, diameters, and spacings. For example, in one embodiment, the pin array (140, 144) has a length of about 42 to 50 mm, a width of about 7 to 10 mm, and a height of about 5 mm. The spacing of the pin arrays for USB 2.0 and other signals is about 1.3 mm, and the spacing of the pin arrays for high-speed signals is about 2.0 to 2.5 mm.
[0109] refer to Fig.10, having a minimized set of components / features to be able to reduce or minimize the weight or volume of various components and achieve a relatively slim head-mounted component (e.g., such as Fig.10 The head-mounted component (58) characterized in FIG. 5 may be helpful. Figure 8 Various arrangements and combinations of the various components shown in .
[0110] Example Electromagnetic Sensing Components in an AR System
[0111] refer to Fig.11A , shows an electromagnetic sensing coil assembly (604, e.g., 3 separate coils coupled to a housing) coupled to a head-mounted component (58); this configuration adds additional geometry to the overall assembly, which may be undesirable. Reference Fig. 11B , not as Fig.11A Rather than housing the coils in a box or single housing 604 as in the configuration of the head-mounted component (58), the individual coils may be integrated into various structures of the head-mounted component (58), such as Fig. 11B as shown in . Fig. 11B Examples of locations for the X-axis coil (148), Y-axis coil (150), and Z-axis coil (152) on the head mounted display 58 are shown. Thus, the sensing coils may be spatially distributed on or around the head mounted display (58) to provide a desired spatial resolution or accuracy in the positioning and / or orientation of the display (58) via an electromagnetic tracking system.
[0112] Figures 12A-12E Various configurations are shown using ferrite cores 1200a-1200e coupled to electromagnetic sensors to increase field sensitivity. Fig. 12A A solid ferrite core 1200a in the shape of a cube is shown, Fig. 12B shows a ferrite core 1200b configured as a plurality of rectangular magnetic disks spaced apart from each other, Fig. 12C A ferrite core 1200c having a uniaxial air core is shown, Fig.12D A ferrite core 1200d having a triaxial air core is shown, and Fig.12E A ferrite core 1200e is shown comprising a plurality of ferrite rods in a housing (which may be made of plastic). Figures 12B-12E The weight ratio of Examples 1200b-1200e is Fig. 12A The solid core embodiment 1200a is lightweight and can be used to save mass. Figures 12A-12E 1 is shown as a cube, but in other embodiments the shape of the ferrite core may be different.
[0113] Frequency Division Multiplexing, Time Division Multiplexing and Gain Control of EM Tracking System
[0114] Conventional EM tracking solutions typically utilize either a frequency division multiplexing (FDM) circuit design or a time division multiplexing (TDM) circuit design. However, FDM designs typically use more current, while TDM designs typically only support a limited number of users. As further described below, a circuit design that incorporates FDM and TDM designs can achieve the benefits of both. Advantages of such a design may include savings in printed circuit board (PCB) area, material cost, number of components used, and / or current consumption compared to conventional designs. The design may also allow multiple users to obtain improved or optimal performance.
[0115] Fig.13A 1 is a block diagram schematically illustrating an example of a frequency division multiplexed EM transmitter (TX) circuit 1302. The EM transmitter circuit can drive three tuned orthogonal coils in an EM tracking system. The time-varying EM field generated by the EM TX can be sensed by an EM receiver (e.g., reference Fig. 13B Description). The circuit uses a main control unit (MCU) to control three different synthesizers at three different radio frequency (RF) frequencies (f1, f2 and f3), whose outputs are filtered (e.g., at a band pass filter (BPF) and optional ferrite bead (FB)) and amplified (e.g., via a preamplifier (PA)) and fed to the corresponding X, Y, Z coils. The circuit also utilizes current sensing control circuits (R-sense (R-sense) and current control (Current Ctrl)) to ensure that the current entering each coil remains constant. The circuit also has an RF wireless communication interface (e.g., Bluetooth Low Energy (BLE)) connected to the MCU, which is connected to a reference Fig. 13B Describes the EM receiver unit communications.
[0116] Fig. 13B 1 is a block diagram schematically illustrating an example of a frequency division multiplexed EM receiver (RX) circuit 1304. The EM receiver uses three orthogonal coils (X coil operating at frequency f1, Y coil operating at frequency f2, and Z coil operating at frequency f3) to receive the time-varying EM signal generated by the EM TX circuit 1302 (see, e.g., Fig.13A ). The three signals are amplified individually (e.g., via a preamplifier (PA)) and filtered in parallel (e.g., by a bandpass filter (BPF)). Optionally, the filter output can be further amplified. The amplified output is then fed into an analog-to-digital (ADC), and the digital signal is processed by a digital signal processor (DSP). The DSP can control the gain of the preamplifier to prevent ADC saturation. The receiver design also has a radio frequency (RF) communication link connected to the DSP (or MCU) (e.g., reference Fig. 13BDescription), a radio frequency (RF) communication link communicates with the EM transmitter. The RF link can be configured to support any suitable wireless standard, including Bluetooth Low Energy (BLE).
[0117] Fig.13A and Fig. 13B EM TX and RX circuits 1302, 1304 shown in FIG. Figures 13C-13J 1302 can be used in EM field transmitter 402, and EM RX circuit 1304 can be used in EM field sensor 404, see Figure 4 Additional embodiments of EM TX and RX circuits will be described that provide advantages such as reduced component count, reduced PCB area, lower material cost, and can allow multiple users to obtain optimal performance.
[0118] Fig. 13C 1 is a block diagram schematically illustrating an example of a time-division multiplexed EM transmitter circuit 1302. In this embodiment, Fig.13A The FDM circuit has been changed to a time division multiplexing circuit. The TDM circuit uses only one path, which is divided into 3 orthogonal coils. The X, Y and Z coils are operated at frequencies f1, f2 and f3, respectively, to generate a time-varying EM field received by the EM receiver circuit. The TDM circuit can operate these coils at corresponding times t1, t2 and t3 according to the TDM timing protocol (see, e.g. Fig.13F and Figure 13G ). Automatic gain control (AGC) may be included in the transmitter circuit (see below). Fig.13I and 13J Further described). Each coil can be dynamically frequency tuned to the desired frequency assigned by the MCU.
[0119] Dynamic frequency tuning
[0120] Dynamic frequency tuning can be used to achieve resonance on each coil to obtain increased or maximum current in the EM TX circuit. Dynamic frequency tuning can be used to accommodate multiple users. Fig.13D is a block diagram schematically illustrating an example of a dynamically tunable circuit 1306. Figures 17D-17G Other embodiments of dynamically tunable circuit 1306 are described. Fig.13D In the circuit shown in , the transmitting coil is represented by an inductor L1. An electrostatic capacitor (C2) is connected in parallel with a tunable capacitor (C1). In this example, the frequency generated by the coil covers the frequency range of 16kHz to 30kHz by tuning capacitor C1. Fig.13E It is shown that the dynamic tuning Fig.13D1306 to achieve various frequencies (from 16kHz to 30kHz). To accommodate multiple users, the example dynamic frequency tuning circuit can utilize one transmit (TX) frequency per user. An example of frequency allocation is shown in Table 1.
[0121]
[0122] Table 1
[0123] Time Division Multiplexing
[0124] In some embodiments, to implement time division multiplexing at the transmitter, synchronization between the transmitter and receiver circuits may be utilized. Two possible synchronization scenarios are discussed below.
[0125] The first scenario uses synchronization through the RF wireless interface (e.g., BLE) of both the receiver and the transmitter. The wireless RF link can be used to synchronize the clocks of both the transmitter and the receiver. After synchronization is achieved, time division multiplexing can reference the onboard real-time clock (RTC).
[0126] The second scenario uses synchronization via electromagnetic pulses. The flight time of the EM pulse will be significantly shorter than the tolerance typically used in TDM circuits and can be ignored. The TX EM pulse is sent by the transmitter to the receiver, which calculates the time difference between the receiver clock and the transmitter clock. This time difference is transmitted over the RF wireless link as a known offset or used to adjust the reference on the wireless interface (e.g., BLE) clock.
[0127] In some embodiments, one or both of these synchronization scenarios may be implemented.After synchronization is completed, the time sequence of TDM for the transmitter and receiver may be established. Fig.13F An example of a TDM timing diagram 1308 is shown. The TX on the X coil will remain on for a first time period that allows the X, Y, and Z coils of the receiver to receive the magnetic flux generated by the X coil. During the first time period, the TX on the Y coil and Z coil are substantially off (e.g., the coils are completely off or operating at a voltage much lower than normal operating voltage (e.g., <10%, <5%, <1%, etc.). After the X coil transmits, the TX on the Y coil will turn on (and the X coil will be substantially off while the Z coil remains substantially off), and the X, Y, and Z coils of the receiver will receive the magnetic flux generated by the TX Y coil. After the Y coil transmits, the TX on the Z coil will turn on (and the Y coil will be substantially off while the X coil remains substantially off), and the X, Y, and Z coils of the receiver will receive the magnetic flux generated by the TX Z coil. This timing sequence is then repeated continuously as the EM transmitter operates.
[0128] A non-limiting illustrative example of accommodating multiple users is described below. For example, to accommodate four users with up to two transmitters, eight TX frequencies are required for each transmitter. It is generally advantageous if these frequencies are not repeated. In such an embodiment, a scanning process may be implemented by the EM receiver to determine whether a particular frequency is being used closely. Figure 13G An example of a scan timing 1310 is shown. This scan can be done by the EM receiver 1304 at initialization and periodically during a user session. The scan can be performed by intentionally turning off the TX in the transmitter 1302 and cycling through the RX (in the receiver 1304) to measure the presence of unintentional interference. If it is determined that energy is present at that frequency, an alternate frequency can be selected. The scan can also be shortened by monitoring one or two (rather than all three) of the three orthogonal coils, as the position and orientation (Pn0) is not required in this slot.
[0129] Fig.13H is a block diagram schematically illustrating another example of a receiver 1304 in an EM tracking system. Fig. 13B Compared to the example FDM receiver shown in , the TDM switch has replaced the individual paths from the three orthogonal coils. The TDM switch can be controlled by an RF wireless interface (e.g., BLE). The TDM switch can utilize Fig.13F The timing protocol 1308 shown in .
[0130] In various embodiments, reference Figures 13C-13H The described time-division multiplexing of the TX and / or RX circuits may provide one or more of the following advantages. (A) Current consumption and battery life. By time-multiplexing the transmitter and receiver, the amount of current used may be reduced. This reduction comes from the fact that high current circuits such as the transmitter are no longer utilized 100% of the time. Fig.13A and Fig. 13B Compared to the FDM circuit shown in , the current consumption of the system can be reduced to slightly more than 1 / 3. (B) Bill of Materials Cost. In the TDM embodiment described above, the number of components used to achieve the same result has been reduced (compared to Fig.13A and Fig. 13B (C) PCB area. Another benefit of the reduction in components is the savings gained in PCB area. The number of components is reduced by nearly 2 / 3, and therefore the space required on the PCB is reduced.
[0131] Other possible advantages may be the reduction in mass of the TX and RX circuits. For example, Fig.13A and Fig. 13BThe FDM TX and RX circuits shown in utilize separate filter and amplifier paths for each of the three orthogonal coils. In contrast, Fig. 13C and Fig.13H The TDM TX and RX circuits shown in share filter and amplifier paths.
[0132] In addition to removing sensor housings and multiplexing to save hardware overhead, the signal-to-noise ratio can be increased by having more than one set of electromagnetic sensors, each set of electromagnetic sensors being relatively small relative to a single larger coil set. In addition, the low-side frequency limitations that typically require multiple sensing coils in close proximity can be improved to facilitate improved bandwidth requirements. There is typically a tradeoff with TD multiplexing because multiplexing typically spreads the reception of the RF signal in time, which results in a typically noisier signal; therefore, larger coil diameters can be used for multiplexed systems. For example, where a multiplexed system can utilize a 9mm side-sized cubic coil sensor box, a non-multiplexed system may only utilize a 7mm side-sized cubic coil box to obtain similar performance; therefore, there may be a tradeoff in minimizing geometry and mass and choosing between embodiments of FDM and TDM circuits.
[0133] Example of Automatic Gain Control for an Electromagnetic Tracking System
[0134] refer to Fig.13A and Fig. 13B , FDM receiver ( Fig. 13B ) implements closed-loop gain control, while the FDM transmitter ( Fig.13A ) does not implement gain control and is left to transmit at its maximum output power regardless of the received level. The gain of the receiver can be set by the DSP. For example, the received voltage on the receiver coil is fed directly into the first stage, which has gain control. The maximum voltage can be determined in the DSP, and the DSP can automatically adjust the gain of the first stage. Placing the gain control in the receiver can utilize more power in the transmitter, even when it is not needed. Therefore, it may be advantageous to employ automatic gain control (AGC, sometimes also called adaptive gain control) on the transmitter side (rather than the receiver side), which can save space in the receiver system (which would otherwise be used for AGC), allowing for a smaller and more portable receiver.
[0135] Fig.13I is a block diagram schematically illustrating an example of an EM receiver 1304 that does not utilize automatic gain control (AGC). The first stage is no longer an AGC circuit (with Fig. 13BThe level of the received voltage on the coil is determined by the DSP, and the DSP provides this information to the wireless (e.g., BLE) link. The BLE link can provide this information to the transmitter (see Fig.13J ) to control the TX level.
[0136] Fig.13J is a block diagram schematically illustrating an example of an EM transmitter 1302 employing AGC. Fig.13J The EM transmitter 1302 can be used with Fig.13I The wireless link (e.g., BLE) communicates the received voltage level (from the BLE link on the receiver) to the MCU. The amplifier stage can have an adjustable gain controlled by the MCU. This can allow current to be saved on the transmitter when the required receive voltage is small.
[0137] therefore, Fig.13I and Fig.13J The RX and TX circuit examples in FIG. 1 employ AGC in the EM transmitter 1302 rather than the EM receiver 1304 . Fig.13A and Fig. 13B This change in the RX and TX circuit examples in may allow for a smaller RX design as well as a more power efficient design as it would allow the TX power to be reduced when necessary.
[0138] Example of EM tracking of user's head pose or hand pose
[0139] refer to Fig.14In one embodiment, after a user powers up his or her wearable computing system (160), the head-mounted component assembly can capture a combination of IMU and camera data (e.g., camera data for SLAM analysis, such as at a belt-pack processor where more raw processing horsepower may be available) to determine and update a head pose (e.g., position or orientation) relative to a real-world global coordinate system (162). The user can also activate a hand-held component to, for example, play an augmented reality game (164), and the hand-held component can include an electromagnetic transmitter (166) operably coupled to one or both of the belt-pack and the head-mounted component. One or more electromagnetic field coil receiver groups (e.g., a group of 3 individual coils in different orientations) are coupled to the head-mounted component to capture magnetic flux from the transmitter, which can be used to determine a position or orientation difference (or "delta") between the head-mounted component and the hand-held component (168). The combination of a head mounted component that helps determine pose relative to a global coordinate system, and a hand held component that helps determine the relative position and orientation of the hand held component relative to the head mounted component, allows the system to generally determine where each component is relative to the global coordinate system, and thus preferably can track the user's head pose and hand held pose with relatively low latency for presenting augmented reality image features and interactions using movement and rotation of the hand held component (170).
[0140] refer to Fig.15 , showing the Fig.14 Embodiments of the present invention are somewhat similar to the embodiments of the present invention, except that the system has more available sensing devices and configurations to help determine the posture (176, 178) of the head-mounted component (172) and the hand-held component, so that the user's head posture and hand-held posture can be tracked preferably with relatively low latency for presenting augmented reality image features and interactions (180) using the movement and rotation of the hand-held component.
[0141] Example Stereo and Time-of-Flight Depth Sensing
[0142] refer to Fig.16A and Fig. 16B , showing the Figure 8 The configuration is similar to all aspects of the configuration. Fig.16A Configuration and Figure 8 The configuration is different in that, in addition to a depth sensor of the LIDAR (106) type, Fig.16A The configuration also features, for illustrative purposes, a general purpose depth camera or depth sensor (154), which may be, for example, a stereo triangulation depth sensor (such as a passive stereo depth sensor, a texture projection stereo depth sensor, or a structured light stereo depth sensor) or a time or flight depth sensor (such as a LIDAR depth sensor or a modulated emission depth sensor); furthermore, Fig.16AThe configuration has an additional forward-facing "world" camera (124, which may be a grayscale camera having a sensor capable of 720p range resolution) and a relatively high resolution "picture camera" (156, which may be a panchromatic camera having a sensor capable of two megapixels or more resolution, for example). Fig. 16B Shown for illustration purposes Fig.16A A partial orthogonal view of the configuration is shown below. Fig. 16B Further described.
[0143] Return to reference Fig.16A As well as the stereo and time-of-flight depth sensors mentioned above, each of these depth sensor types can be used with the wearable computing solutions disclosed herein, although each has various advantages and disadvantages. For example, many depth sensors have challenges with black surfaces and shiny or reflective surfaces. Passive stereo depth sensing is a relatively simple way to calculate depth using a depth camera or sensor for triangulation, but it may be challenging if a wide field of view ("FOV") is required and relatively significant computing resources may be required; in addition, this sensor type may have challenges with edge detection, which may be important for the specific use case at hand. Passive stereo may pose challenges to untextured walls, low light situations, and repeating patterns. Passive stereo depth sensors are available from manufacturers such as Intel and Aquifi. Stereo with texture projection (also called "active stereo") is similar to passive stereo, but the texture projector broadcasts the projected pattern onto the environment, and the more textures broadcast, the higher the accuracy that can be obtained in the triangulation for depth calculation. Active stereo can also require relatively high computational resources, presents challenges when a wide FOV is required, and is somewhat subpar at detecting edges, but it does address some of the challenges of passive stereo in that it works well on textureless walls, performs well in low light, and generally does not have issues with repeating patterns. Active stereo depth sensors are available from manufacturers such as Intel and Aquifi.
[0144] Stereo with structured light, such as systems developed by Primesense and available under the trade name Kinect, and systems available from Mantis Vision, typically utilize a single camera / projector pairing, and the projector is specialized in that it is configured to broadcast a pattern of dots that is known a priori. Essentially, the system knows the pattern that is broadcast, and it knows that the variable to be determined is depth. This configuration can be relatively efficient in computational load, and can be challenged in wide FOV demand scenarios and in scenarios with ambient light and patterns broadcast from other nearby devices, but can be very effective and efficient in many scenarios. Using modulated time-of-flight depth sensors, such as those available from PMD Technologies, AG, and SoftKinetic, the emitter can be configured to emit a wave of amplitude modulated light (such as a sine wave); a camera component, which can be located nearby or even overlapping in some configurations, receives a return signal at each pixel of the camera component, and a depth map can be determined / calculated. This configuration can be relatively compact in geometry, high in accuracy and low in computational load, but can have challenges in image resolution (such as at the edges of objects), multipath errors (such as where the sensor is aimed at a reflective or shiny corner and the detector ends up receiving more than one return path), resulting in some depth detection aliasing.
[0145] Direct time-of-flight sensors (also referred to as aforementioned LIDAR) are available from suppliers such as LuminAR and Advanced Scientific Concepts. With these time-of-flight configurations, typically a pulse of light (such as long pulses of light of picoseconds, nanoseconds, or femtoseconds) is sent out to bathe the world oriented around it with that pulse of light; each pixel on the camera sensor then waits for that pulse to return, and knowing the speed of light, the distance at each pixel can be calculated. This configuration can have many of the advantages of a modulated time-of-flight sensor configuration (no baseline, relatively wide FOV, high accuracy, relatively low computational load, etc.) as well as relatively high frame rates (such as tens of thousands of hertz). They can also be relatively expensive, have relatively low resolution, be sensitive to bright light, and be susceptible to multipath errors; they can also be relatively large and heavy.
[0146] refer to Fig. 16B, a partial top view is shown for illustration purposes, featuring a user's eye (12) and a camera (14, such as an infrared camera) with a field of view (28, 30) and a light source or radiation source (16, such as infrared) directed toward the eye (12) to facilitate eye tracking, observation and / or image capture. Three outward-facing world capture cameras (124) are shown with their FOVs (18, 20, 22), as are a depth camera (154) and its FOV (24) and an image camera (156) and its FOV (26). Depth information obtained from the depth camera (154) can be supported by using overlapping FOVs and data from other forward-facing cameras. For example, the system may end up with something like a sub-VGA image from the depth sensor (154), a 720p image from the world camera (124), and an occasional 2 megapixel color image from the image camera (156). This configuration has four cameras sharing a common FOV, two of which have heterogeneous visible spectrum images, one has color, and the third has relatively low resolution depth. The system can be configured to segment in grayscale and color images, fuse the two and make a relatively high resolution image from them, get some stereo correspondences, use the depth sensor to provide an assumption about stereo depth, and use the stereo correspondences to get a more accurate depth map that may be significantly better than the depth map obtained from the depth sensor alone. This process can be run on local mobile processing hardware, or it can be run using cloud computing resources, or it can be run with data from other people in the area (such as two people sitting across a table from each other), and end up with a fairly detailed map. In another embodiment, all of the above sensors can be combined into one integrated sensor to achieve this functionality.
[0147] Example of dynamic tuning of a transmission coil for EM tracking
[0148] refer to Figures 17A-17G , shows aspects of a dynamic transmit coil tuning configuration for electromagnetic tracking so that the transmit coils operate optimally at multiple frequencies in each orthogonal axis, which allows multiple users to operate on the same system. Typically, an electromagnetic tracking transmitter will be designed to operate at a fixed frequency in each orthogonal axis. With this approach, each transmit coil is tuned using a static series capacitor that resonates only at the operating frequency. This resonance allows the maximum possible current to flow through the coil, which in turn maximizes the magnetic flux generated. Fig.17A A typical resonant circuit 1305 for generating resonance at a fixed operating frequency is shown. Inductor "L" represents a single-axis transmission coil with an inductance of 1mH, and the capacitance is set to 52nF, generating resonance at 22kHz, as shown in FIG. Fig. 17B as shown in . Fig. 17C shows the frequency plotted against the Fig.17AThe current of circuit 1305 is shown in FIG. 13 and it can be seen that the current is maximum at the resonant frequency. If the system were expected to operate at any other frequency, the operating circuit would not be at the maximum current possible (which occurs at the resonant frequency of 22 kHz).
[0149] Fig.17D An embodiment of a dynamically tunable configuration of a transmitter circuit 1306 of a transmitter 1302 for an electromagnetic tracking system is shown. Fig.17D The example circuit 1306 shown in FIG. 1 may be used in the embodiments of the EM field transmitters 402 , 602 , 1302 described herein. Fig.17D The circuit in includes an oscillating voltage source 1702, a transmit (TX) coil, a high voltage (HV) capacitor, and a plurality of low voltage (LV) capacitors in a capacitor bank 1704, which can be selected to provide tuning for a desired resonant frequency. Dynamic frequency tuning can be set to achieve resonance on the coil (at a desired dynamically adjustable frequency) for maximum current. Fig.17E Another example of a dynamically tunable circuit 1306 is shown in FIG. 1 , where a tunable capacitor 1706 (“C4”) can be tuned to produce resonance at different frequencies, such as Fig.17F Tuning the tunable capacitor may include switching between a plurality of different capacitors, such as Fig.17D The circuit shown in FIG. Fig.17E In the embodiment shown in FIG. 1 , one of the orthogonal coils of the electromagnetic tracker is modeled as an inductor "L", while the static capacitor ("C5") is a fixed high voltage capacitor. This high voltage capacitor will see a higher voltage due to resonance, and therefore its package size will typically be larger. Capacitor C4 will be a capacitor that is dynamically switched at different values, and therefore will see a lower maximum voltage, and will typically be a smaller geometry package to save placement space. Inductor L3 can also be used to fine tune the resonant frequency.
[0150] Fig.17F Show that it can be Fig.17E An example of resonance achieved by circuit 1306. Fig.17F , the higher curve (248) shows the voltage Vmid-Vout across capacitor C5, while the lower curve (250) shows the voltage Vout across capacitor C4. As the capacitance of C4 changes, the resonant frequency changes (between about 22kHz and 30kHz in this example), and it is noteworthy that the voltage across C5 (Vmid-Vout; curve 248) is higher than the voltage across C4 (Vout; curve 250). This will generally allow for a smaller package component on C4, as multiple capacitors of this capacitor will generally be used in a system, for example, one capacitor per resonant operating frequency (see, e.g., Fig.17D1704). Figure 17G is a graph of current versus frequency showing that the maximum current achieved follows resonance regardless of the voltage across the capacitor. Thus, embodiments of dynamically tunable circuits can provide increased or maximum current in the transmitter coil across multiple frequencies, thereby allowing improved or optimized performance for multiple users of a single EM tracking system.
[0151] Example of audio noise cancellation for an EM tracking system
[0152] An audio speaker (or any external magnet) can generate a magnetic field that can unintentionally interfere with the magnetic field generated by the EM field transmitter of the EM tracking system. Such interference can degrade the accuracy or reliability of the position estimate provided by the EM tracking system.
[0153] As AR devices evolve, they become more complex and integrate more technologies that must coexist and perform independently. EM tracking systems rely on tiny changes in magnetic flux (generated by EM field transmitters) received (by EM sensors) to determine the 3-D position of the EM sensor (and thereby the 3-D position of the object to which the sensor is attached or coupled). Audio speakers residing near the EM tracking sensor coils can emit magnetic flux that can interfere with the EM tracking system's ability to calculate true position.
[0154] refer to Figures 18A-18C , electromagnetic tracking systems can be limited to operation below about 30kHz, which is slightly above the audible range of human hearing. Fig.18A A configuration is shown in which an audio speaker 1820 is in close proximity to the EM sensor 604. The audio speaker 1820 is driven by a voltage source 1822 that varies over time and an amplifier 1824. The magnetic field of the speaker 1820 may cause unintentional magnetic field interference to the EM tracking system because the speaker generates noise in the magnetic field sensed by the coil of the EM sensor 604. In some embodiments, the distance between the audio speaker 1820 and the EM sensor 604 may be increased to reduce received interference. However, because the magnetic flux from the speaker is proportional to the cube of the distance from the sensor (1 / r 3 ) attenuation, so there will be points at large distances that provide very little attenuation in the interference. Audio speakers (e.g., Figures 2A-2D The speaker 66 shown in FIG. 1 will typically be used in an AR device to provide an audio experience to a wearer of the AR device; therefore, it may be common for the audio speaker to be relatively close to the EM sensor also disposed on the AR device (see, for example, FIG. 1 ). Fig.11A ). The EM sensor 604 is disposed near the speaker 66 in the example wearable display device 58 shown in FIG. 4 . The magnetic field from the audio speaker may interfere with the EM field sensed by the EM sensor of the EM tracking system.
[0155] refer to Fig.18A , there may be some audio systems that generate noise in the usable frequencies for such electromagnetic tracking systems. In addition, audio speakers typically have a magnetic field and one or more coils, which may also interfere with the electromagnetic tracking system. Fig.18B , a block diagram of an example of a noise cancellation system 1830 for an electromagnetic tracking system is shown. Since the unintentional EM interference is a known entity (because the signal provided to the audio speaker 1820 by the voltage source 1822 is known or can be measured), this knowledge can be used to cancel the EM interference from the audio speaker 1820 and improve the performance of the EM tracking system. In other words, the audio signal generated by the system can be used to cancel the magnetic interference from the speaker received by the coil of the EM sensor 604. Fig.18B As schematically shown in , the noise cancellation circuit 1830 may be configured to accept a corrupted signal 1850a from the EM sensor 604 and a signal 1850b from the audio system. The noise cancellation system may combine the signals 1850a, 1850b to cancel interference received from the audio speaker 1820 and provide an uncorrupted sensor signal 1850c.
[0156] Fig. 18C1850a to cancel interference and provide a substantially uncorrupted sensor signal 1850c. The top curve V(noise) is the noise signal 1850b added to the EM tracking system by the audio speaker 1820. The bottom curve V(cancellation) is the inverted audio signal (e.g., -V(noise)), and when these are added together, the effect is that there is no noise reduction from the audio. In other words, the noise cancellation system receives the corrupted signal 1850a, which is the sum of the true EM sensor signal, V(sensor) representing the signal from the EM transmitter coil, and the noise signal: V(sensor)+V(noise). By adding the inverted audio signal -V(noise) to the corrupted signal 1850a, the uncorrupted signal V(sensor) 1850c is restored. The uncorrupted signal 1850c reflects the response of the sensor 604 as if the audio speaker 604 were not present and therefore reflects the EM transmitter field at the location of the sensor 604. Equivalently, the noise signal 1850b may be subtracted from the corrupted signal 1850a to recover the uncorrupted signal V(sensor) 1850c. Noise cancellation may result in substantially all (e.g., >80%, >90%, >95%, or more) of the noise signal (e.g., from an audio speaker) being cancelled. The noise cancellation technique is not limited to canceling audio speaker noise, but may be applied to other noise sources that interfere with the EM sensor signal if a measure (or estimate) of the noise signal can be determined (such that it can be removed from the EM sensor signal as described above).
[0157] Fig.18D 18 is a flow chart illustrating an example method 1800 for canceling interference received by an EM sensor in an EM tracking system. The method 1800 may be performed by a hardware processor in an AR device (such as, for example, the local processing and data module 70) or by a hardware processor in the EM tracking system. At block 1802, the method receives a noise signal from an electromagnetic sensor. As described above, the noise signal may be caused by interference from a nearby audio speaker that generates electromagnetic interference. At block 1804, the method receives a signal from a source of EM interference. For example, the signal may be a signal 1850b for driving an audio speaker (see, e.g., Fig.18B ). At block 1806, the noisy signal and the interference signal are combined to obtain a de-noised EM signal. For example, the interference signal may be inverted and added to the noisy signal, or the interference signal may be subtracted from the noisy signal. At block 1808, the de-noised signal may be used to determine the position of the EM sensor. The position obtained using the de-noised signal (compared to using the noisy signal) is more accurate and reliable.
[0158] Thus, the foregoing provides a method to remove unintentional noise generated by an audio speaker near an EM tracker sensor. The method employs a noise cancellation method that uses known information about the audio to remove it from the EM tracking signal. The system can be used when sufficient physical separation of the audio speaker and the EM sensor coil cannot be achieved (so that the interference is sufficiently low). Although in the above, the interference noise has been described as being generated by the audio speaker, this is for illustration only and not limitation. The foregoing embodiments can be applied to any interference signal that can be measured and then subtracted from the corrupted sensor signal.
[0159] Example Calibration of a Vision System
[0160] refer to Fig.19 In one embodiment, a known pattern 1900 of light or other emitters (such as a circular pattern) can be utilized to assist in the calibration of the vision system. For example, the circular pattern can be used as a reference; when a camera or other capture device with a known orientation captures the shape of the pattern while reorienting an object coupled to the pattern, the orientation of the object (such as a handheld totem device 606) can be determined; this orientation can be compared to the orientation from an associated IMU on the object (e.g., a totem) for use in error determination and calibration. Further references Fig.19 , the pattern of light 1900 can be formed by holding a totem 606 (in Fig.19 The light is generated by a light emitter (e.g., a plurality of LEDs) on a cylindrical structure (schematically represented as a cylinder in FIG. 1 ). Fig.19 As shown in , the pattern of light 1900 appears circular when the totem is viewed head-on by the camera on the AR headset 58. When the totem 606 is tilted in other orientations, the pattern 1900 appears elliptical. Computer vision techniques can be used to identify the pattern of light 1900, and the orientation of the totem 606 can be determined.
[0161] In various implementations, an augmented reality device may include a computer vision system configured to implement one or more computer vision techniques to recognize patterns of light (or perform other computer vision processes used or described herein). Non-limiting examples of computer vision techniques include: Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces method, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) technology, sequential Bayesian estimation (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative closest point (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms (such as, for example, support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), and the like.
[0162] Example circuit for a subsystem of a wearable display device
[0163] refer to Figures 20A-20C , shows a configuration with a summing amplifier 2002 to simplify the circuit between two subsystems or components (such as a head-mounted component and a belt-pack component) of a wearable computing configuration. In a conventional configuration, each coil 2004 of the electromagnetic tracking sensor 604 (in Fig. 20A ) is associated with amplifier 2006, and the three different amplified signals can be sent to another component (e.g., such as Fig. 20B In the embodiment shown, three different amplified signals may be directed to a summing amplifier 2002, which produces one amplified signal directed down to an advantageously simplified cable 2008, and each signal may be at a different frequency. The summing amplifier 2002 may be configured to amplify all three signals received by the amplifier; then (as shown in FIG. Fig. 20B After analog-to-digital conversion, the receiving digital signal processor separates the signal at the other end. Gain control can be used. Fig. 20C Filters for each frequency (F1, F2, and F3) are shown - so the signals can be separated back at this stage. These three signals can be analyzed by computational algorithms (e.g., to determine sensor pose), and the AR system can use the position or orientation results (e.g., to correctly display virtual content to the user based on the user's instantaneous head pose).
[0164] EM Tracking System Update Example
[0165] refer to Fig.21 Electromagnetic ("EM") tracking updates can be relatively "expensive" in terms of power for portable systems, and may not be capable of very high frequency updates. In a "sensor fusion" configuration, more frequently updated positioning information from another sensor such as an IMU can be combined with data from another sensor such as an optical sensor (e.g., a camera or depth camera) that may or may not be at a relatively high frequency; the network that fuses all of these inputs places lower demands on the EM system and provides faster updates.
[0166] Return to reference Fig. 11B , shows a distributed sensor coil configuration for an AR device 58. Fig.22A As described above, an AR device 58 having a single electromagnetic sensor device (604), such as a housing containing three orthogonal sensing coils (one for each of the X, Y, and Z directions) can be coupled to a wearable component (58) for 6-DOF tracking. Also as described above, such a device can be split, with three sub-parts (e.g., coils) attached at different locations on the wearable component (58), such as Fig. 22B and Fig. 22C As shown in . Fig. 22C To provide further design alternatives, each individual sensor coil can be replaced with a group of similarly oriented coils such that the total magnetic flux for any given orthogonal direction is captured by a group (148, 150, 152) rather than a single coil for each orthogonal direction. In other words, instead of one coil for each orthogonal direction, a group of smaller coils can be utilized and their signals aggregated to form a signal for that orthogonal direction. In another embodiment, where a particular system component, such as a head-mounted component (58), features two or more electromagnetic coil sensor groups, the system can be configured to selectively utilize sensor and transmitter pairings that are closest to each other (e.g., within 1 cm, 2 cm, 3 cm, 4 cm, 5 cm, or 10 cm) to improve or optimize system performance.
[0167] Example of recalibrating a wearable display system
[0168] refer to Figures 23A-23C, recalibrating a wearable computing system such as that discussed herein may be useful, and in one embodiment, an acoustic (e.g., ultrasonic) signal generated at a transmitter and an acoustic sensor (e.g., microphone) and acoustic time-of-flight calculation at a receiver can be used to determine the sound propagation delay between the transmitter and receiver, and thereby determine the distance between the transmitter and receiver (since the speed of sound is known). Fig.23A It is shown that in one embodiment, the three coils on the transmitter are excited with a burst sine wave, while the ultrasound transducer can be excited with a burst sine wave, preferably at the same frequency as one of the coils. Fig. 23B The EM receiver is shown to be configured to receive three EM waves using the X, Y, Z sensor coils and to receive the acoustic ultrasound wave using a microphone (MIC). The total distance can be calculated based on the amplitude of the three EM signals. The flight time (sound propagation delay time 2300) can be calculated by comparing the timing 2302 of the acoustic (microphone) response with the response 2304 of the EM coil (see, e.g. Fig.23C ). This can also be used to calculate distance. Comparing the electromagnetically calculated distance to the acoustic delay time 2300 can be used to calibrate the EM TX or RX circuits (e.g., via a correction factor).
[0169] refer to Fig.24A In another embodiment, in an augmented reality system featuring a camera, the known size alignment feature (in Fig.24A The distance is calculated based on the pixel size of the image (depicted as an arrow in the figure).
[0170] refer to Fig. 24B In another embodiment, in an augmented reality system featuring a depth sensor, such as an infrared ("IR") depth sensor, the distance may be calculated by such a depth sensor and reported directly to the controller.
[0171] refer to Fig.24C and Fig.24D , once the total distance is known, a camera or depth sensor can be used to determine the position in space. The augmented reality system can be configured to project one or more virtual alignment targets to the user. The user can align the controller with the target, and the system can calculate the position based on both the EM response and the direction of the virtual target plus the previously calculated distance. Roll angle calibration can be done by aligning a known feature on the controller with the virtual target projected to the user; yaw and pitch angles can be calibrated by presenting a virtual target to the user and having the user align two features on the controller with the target (much like aiming a rifle).
[0172] refer to Fig.25A and Fig.25B , there may be an inherent ambiguity associated with EM tracking systems: the receiver will generate similar responses in two diagonally opposite locations around the transmitter. For example, Fig.25A Shown is a handheld device 606 and a ghost device 606a that generates a similar response.This challenge is particularly relevant in systems where both the transmitter and receiver can move relative to each other.
[0173] In one embodiment, the system can use an IMU sensor to determine whether the user is on the positive or negative side of a reference (e.g., symmetry) axis. In embodiments such as those described above that feature a world camera and a depth camera, the system can use this information to detect handheld components (e.g., Fig.25B If the handheld device 2500 is outside the field of view of the camera and / or depth sensor, the system can be configured to determine (or the user can determine) that the handheld device 2500 is located in a 180 degree area directly behind the user, for example, Fig.25B The ghost position shown in is 2500a.
[0174] Referring back to the above-described embodiments, in which outward-facing camera devices (124, 154, 156) are coupled to a system component such as a head-mounted component (58), information collected from these camera devices can be used to determine the position and orientation of a head coupled to such head-mounted component (58) using techniques such as simultaneous localization and mapping, or "SLAM" techniques (also known as parallel tracking and mapping, or "PTAM" techniques). Understanding the position and orientation of a user's head (also known as the user's "head pose") in real time or near real time (e.g., preferably with low latency determination and updating) is valuable in determining the user's position within the actual environment around him or her and how to place and present virtual content relative to the user and the environment in connection with the user's augmented or mixed reality experience. A typical SLAM or PTAM configuration involves extracting features from the input image information and using it to triangulate 3-D map points, and then tracking these 3-D map points. SLAM technology has been utilized in many implementations, such as in autonomous vehicles, where computing, power, and sensing resources may be relatively abundant compared to those available onboard wearable computing devices (such as head-mounted components (58)).
[0175] Example of pose calculation and position mapping via camera feature extraction
[0176] refer to Fig.26In one embodiment, a wearable computing device (such as a head mounted component (58)) may include two outward facing cameras, producing two camera images (left - 204, right - 206). In one embodiment, a relatively lightweight, portable, and power efficient embedded processor (such as a or An embedded processor (such as those sold as such) may comprise a portion of the head mounted component (58) and be operably coupled to the camera arrangement. The embedded processor may be configured to first extract features (210, 212) from the camera images (204, 206). If the calibration between the two cameras is known, the system may triangulate the 3-D mapping points of these features (214), thereby producing a sparse set of 3-D mapping points (202). This may be stored as a "map", and these first frames may be used to establish a "world" coordinate system origin (208). As subsequent image information enters the embedded processor from the camera, the system may be configured to project the 3-D mapping points into the new image information and compare to the locations of the 2-D features detected in the image information. Thus, the system may be configured to attempt to establish 2-D to 3-D correspondences, and using a set of such correspondences, such as about six of them, the pose of the user's head (which, of course, is coupled to the head mounted device 58) may be estimated. More correspondences (such as more than six) generally mean a better estimate of the pose. Of course, this analysis relies on having some sense of where the user's head was (e.g., in terms of position and orientation) before examining the current image. As long as the system is able to track without too much latency, the system can use the pose estimate from the most recent time to estimate the head position for the most current data. Thus, the last frame is the origin, and the system can be configured to estimate that the user's head is not far from it in terms of position and / or orientation, and can search around it to find a correspondence for the current time interval. This is the basis of one embodiment of a tracking configuration.
[0177] After moving sufficiently far away from the original set of mapped points (202), one or both camera images (204, 206) may begin to lose mapped points in the newly incoming image (e.g., if the user's head rotates to the right in space, the original mapped points may begin to disappear to the left, and may only appear in the left image, and then there is no more rotation at all). Once the user rotates too far away from the original set of mapped points, the system can be configured to create new mapped points, such as by using a process similar to the process described above (detect features, create new mapped points) - this is an example of how the system can be configured to keep populating the map. In one embodiment, this process can be repeated every 10 to 20 frames, depending on how much the user translates and / or rotates his head relative to his environment, and thereby translates and / or rotates the associated camera. The frames associated with the newly created mapped points can be considered "key frames", and the system can be configured to delay the feature detection process with key frames, or alternatively, feature detection can be performed on each frame to try to establish a match, and then when the system is ready to create a new key frame, the system has already completed the associated feature detection. Thus, in one embodiment, the basic paradigm is to start creating a map, and then track, track, track until the system needs to create another map, or additional portions thereof.
[0178] refer to Fig. 27 In one embodiment, vision-based gesture calculation can be divided into five stages (e.g., pre-tracking 216, tracking 218, low latency mapping 220, latency tolerant mapping 222, post-mapping / cleanup 224) to help with accuracy and optimization for embedded processor configurations, which may have limited computing, power, and sensing resources. Vision-based gesture calculation can be performed by the local processing and data module 70 or the remote processing and data modules 72, 74.
[0179] With respect to pre-tracking (216), the system may be configured to identify which mapping points are projected into the image before the image information arrives. In other words, the system may be configured to identify which mapping points will be projected into the image, assuming that the system knows where the user was previously and has a sense of where the user is going. The concept of "sensor fusion" is discussed further below, but it is worth noting here that one of the inputs that the system may obtain from a sensor fusion module or function may be "post-estimation" information at a relatively fast rate, such as 250 Hz from an inertial measurement unit ("IMU") or other sensor or device (this is relative to a high rate of, for example, 30 Hz, at which vision-based posture calculation operations may provide updates). Thus, there may be a finer temporal resolution of posture information derived from an IMU or other device relative to vision-based posture calculations; however, it is worth noting that data from devices such as IMUs tend to be somewhat noisy and susceptible to posture estimate drift, as described below. For relatively short time windows, such as 10-15 milliseconds, IMU data may be very useful in predicting posture, and again, when combined with other data in a sensor fusion configuration, an optimized overall result may be determined.
[0180] The pose information from the sensor fusion module or function can be referred to as a "pose prior," and the system can use this pose prior to estimate which groups of points will be projected into the current image. Therefore, in one embodiment, the system is configured in a "pre-tracking" step (216) to pre-extract these mapping points and perform some pre-processing that helps reduce latency in the overall processing. Each 3-D mapping point can be associated with a descriptor so that the system can uniquely identify them and match them to regions in the image. For example, if a given mapping point is created using a feature that has a patch around it, the system can be configured to maintain some appearance of that patch along with the mapping point so that when the mapping point is seen projected onto other images, the system can look back at the original image used to create the map, check the patch correlations, and determine if they are the same points. Therefore, in pre-processing, the system can be configured to perform some amount of extraction of mapping points, as well as some amount of pre-processing associated with the patches associated with those mapping points. Thus, in pre-tracking (216), the system can be configured to pre-extract mapping points, and pre-warp image patches (“warping” of the image can be done to ensure that the system can match patches associated with mapping points with the current image; warping is an example of a method of ensuring that the compared data is compatible).
[0181] Return to reference Fig. 27, the tracking phase can include several components, such as feature detection, optical flow analysis, feature matching, and pose estimation. When detecting features in the input image data, the system can be configured to utilize optical flow analysis to save computational time in feature detection by attempting to follow features from one or more previous images. Once features are identified in the current image, the system can be configured to attempt to match the features to projected mapping points - this can be considered the "feature matching" part of the configuration. In the pre-tracking phase (216), the system preferably has already identified which mapping points are of interest and extracted them; in feature mapping, they are projected into the current image and the system attempts to match them to features. The output of feature mapping is a set of 2-D to 3-D correspondences, and in hand, the system is configured to estimate pose.
[0182] As the user tracks his head coupled to the head mounted assembly (58), the system is preferably configured to identify whether the user is looking at a new area of the environment to determine whether a new keyframe is needed. In one embodiment, this analysis of whether a new keyframe is needed can be based almost purely on geometry; for example, the system can be configured to look at the distance from the current frame to the remaining keyframes (the translation distance; also the field of view capture redirection - the user's head can be close to translation but redirected so that, for example, a completely new mapping point may be needed). Once the system determines that a new keyframe should be inserted, the mapping phase can begin. As described above, the system can be configured to split the mapping operation into three different operations (low latency mapping, latency tolerant mapping, post / mapping, or clearing), rather than the single mapping operation more likely seen in traditional SLAM or PTAM operations.
[0183] Low latency mapping (220), which can be viewed in a simple form as triangulation and creation of new map points, is a critical stage where the system is preferably configured to perform such a stage immediately, since the tracking paradigm discussed herein relies on map points where the system only finds a location when there are map points available for tracking. The "low latency" designation refers to the concept of no tolerance for undue delays (in other words, this part of the mapping needs to be performed as quickly as possible or the system has tracking problems).
[0184] In a simple form, the delay tolerant mapping (222) can be considered as an optimization stage. The entire process does not absolutely require low latency to perform an operation called "bundle adjustment", which provides a globally optimized result. The system can be configured to check the location of the 3-D points and from where they are observed. In the process of creating the mapped points, there are many errors that can be linked together. The bundle adjustment process can take, for example, a specific point observed from two different view positions and use all of this information to get a better sense of the actual 3-D geometry. The result can be that the 3-D points as well as the calculated trajectory (e.g., capture camera position, path) can be adjusted by a small amount. It is desirable to perform these types of processes so that errors are not accumulated through the mapping / tracking process.
[0185] The post-mapping / cleanup (224) phase is where the system can be configured to remove points on the map that do not provide valuable information in the mapping and tracking analysis. In this phase, these points that do not provide useful information about the scene are removed, and this analysis helps keep the entire mapping and tracking process scalable.
[0186] During the visual pose computation process, there is an assumption that the features that the outward-facing camera is looking at are static features (e.g., not moving from frame to frame relative to the global coordinate system). In various embodiments, semantic segmentation and / or object detection techniques may be utilized to remove moving objects, such as people, moving vehicles, etc., from the field of interest so that features for mapping and tracking are not extracted from these areas of the various images. In one embodiment, deep learning techniques, such as those described below, may be utilized to segment out these non-static objects.
[0187] Sensor Fusion Example
[0188] refer to Figures 28A-28F , a sensor fusion configuration can be used to benefit from one source of information from a sensor with a relatively high update frequency (such as an IMU updating gyroscope, accelerometer, and / or magnetometer data related to head pose at a frequency such as 250 Hz) and another source of information updated at a lower frequency (such as a vision-based head pose measurement process updated at a frequency such as 30 Hz). In various embodiments, the higher frequency sensor data is at a frequency above 100 Hz and the lower frequency sensor data is at a frequency below 100 Hz. In some embodiments, the higher frequency sensor data is at a frequency greater than 3 times, 5 times, 10 times, 25 times, 100 times, or greater than the frequency at which the lower frequency sensor acquires data.
[0189] refer to Fig.28AIn one embodiment, the system can be configured to use an extended Kalman filter (EKF, 232) and track a large amount of information about the device. For example, in one embodiment, it can consider 32 states, such as angular velocity (e.g., from the IMU gyroscope), translational acceleration (e.g., from the IMU accelerometer), calibration information for the IMU itself (e.g., coordinate systems and calibration factors for the gyroscope and accelerometer; the IMU may also include one or more magnetometers). Thus, the system can be configured to receive IMU measurements at a relatively high update frequency (226) such as 250 Hz, and to receive data from some other source at a lower update frequency (e.g., calculated visual posture measurements, odometry data, etc.), such as visual posture measurements (228) at an update frequency such as 30 Hz.
[0190] Each time the EKF obtains a round of IMU measurements, the system can be configured to integrate the angular velocity information to obtain rotation information (e.g., the integral of angular velocity (change in rotational position over time) is angular position (change in angular position)); similarly for translation information (in other words, by doubly integrating the translational acceleration, the system will obtain position data). Through this calculation, the system can be configured to obtain 6 degrees of freedom (DOF) pose information (translation in X, Y, Z; orientation of the three rotational axes) from the head at a high frequency (e.g., 250 Hz in one embodiment) from the IMU. Each time an integration is completed, noise accumulates in the data; doubly integrating the translational or rotational acceleration can propagate the noise. Typically, the system is configured not to rely on such data, which is prone to "drift" due to noise over long time windows (such as anything longer than about 100 milliseconds in one embodiment). The input lower frequency (eg, updated at approximately 30 Hz in one embodiment) data from the visual gesture measurement (228) may be used to operate with the EKF (232) as a correction factor to produce a corrected output (230).
[0191] refer to Figures 28B-28F , to illustrate how data from one source with a higher update frequency can be combined with data from another source with a lower update frequency, a first set of points (234) from an IMU with a higher frequency (such as 250 Hz) is shown, with points (238) coming in from the visual pose calculation process at a lower frequency (such as 30 Hz). The system can be configured to make corrections (242) to the visual pose calculation points as such information becomes available, and then proceed with more points from the IMU data (236), and proceed with another correction (244) from another point (240) available in the visual pose calculation process. The "update" of the visual pose data can be applied to the "propagation" of the data from the IMU using an EKF.
[0192] It is worth noting that in some embodiments, data from a second source (e.g., such as visual gesture data) may not only come in at a lower update frequency, but also have some latency - meaning that the system is preferably configured to navigate temporal adjustments as the information from the IMU and the visual gesture calculation are integrated. In one embodiment, to ensure that the system converges on the visual gesture calculation input at the correct temporal location in the IMU data, a buffer of the IMU data may be maintained to converge and calculate an "update" or adjustment to the time associated with the input to the visual gesture calculation, and then consider that "update" or adjustment propagated forward to the current time (e.g., "Tcurrent"), which leaves a gap between the adjusted position and / or orientation data and the latest data from the IMU. To ensure that there is not too much "jumping" or "jittering" when presented to the user, the system may be configured to use smoothing techniques. One way to solve this problem is to use a weighted averaging technique, which can be linear, nonlinear, exponential, etc., to ultimately drive the fused data stream down to the adjustment path.
[0193] refer to Fig.28C , for example, a weighted averaging technique may be used in the time domain between T0 and T1 to drive a signal from an unregulated path (252; e.g., directly from an IMU) to a regulated path (254; e.g., based on data from a visual posture calculation process); Fig.28D An example is shown in FIG. 2 , which shows a fused result ( 260 ) starting from an unregulated path ( 252 ) and time T 0 and moving exponentially to a regulated path ( 254 ) by T 1 . Fig.28E , showing a sequence of correction opportunities, with exponential temporal corrections of the fused results (260) from an upper path toward a lower path in each sequence (the first correction is from a first path 252, e.g., from an IMU, to a second path 254, e.g., from a vision-based pose calculation; then a similar pattern continues forward using successive IMU data, while using each incoming vision-based pose calculation point to correct based on successive points from the vision pose, in this example downward toward a lower path 256, 258 of successive corrections). Reference Fig.28F , with a sufficiently short time window between "updates" or corrections, the overall fused result (260) can be functionally perceived as a relatively smooth patterned result (262).
[0194] In other embodiments, rather than relying directly on visual pose measurements, the system can be configured to examine a derivative EKF; in other words, rather than using the visual pose calculations directly, the system uses the change in visual pose from the current time to a previous time. This configuration may be pursued, for example, if the amount of noise in the visual pose differences is much less than the amount of noise in the absolute visual pose measurements. It is preferable to not have transient errors that throw off the fusion results, since the output of all of these is the pose, which is sent back to the vision system as a "pose prior" value.
[0195] Although some embodiments use an EKF, other embodiments may use different estimation algorithms such as, for example, an unscented Kalman filter, a linear Kalman filter, a Bayesian model, a hidden Markov model, a particle filter, a sequential Monte Carlo model, or other estimation techniques.
[0196] Posture Service Example
[0197] The external system-based "consumer" of gesture results can be referred to as a "gesture service", and the system can be configured so that all other system components enter the gesture service when a gesture is requested at any given time. The gesture service can be configured as a queue or stack (e.g., a buffer) with data for a time slice sequence, with the most recent data at one end. If the request to the gesture service is the current gesture or some other gesture in the buffer, it can be output immediately; in some configurations, the gesture service will receive the following request: What is the gesture 20 milliseconds from now (for example, in a video game content rendering scenario-the relevant service may need to know that from now on, it needs to render something in a certain position and / or orientation slightly in the future). In one model for generating future gesture values, the system can be configured to use a constant velocity prediction model (e.g., assuming that the user's head moves at a constant velocity and / or angular velocity); in another model for generating future gesture values, the system can be configured to use a constant acceleration prediction model (e.g., assuming that the user's head translates and / or rotates at a constant acceleration). The data in the data buffer can be used to infer the position where the gesture will use such a model. The constant acceleration model uses a slightly longer tail of data entering the buffer for prediction than the constant velocity model, and we have found that the system can predict up to 20 milliseconds into the future without significant degradation. Therefore, the gesture service can be configured to have a data buffer that goes back in time and forward about 20 milliseconds or more in terms of data available for outputting gestures. Operationally, the content operation will typically be configured to identify when the next frame drawing is due in time (e.g., it will attempt to draw at time T or at time T+N, where N is the next update data interval provided by the gesture service).
[0198] A user-facing (e.g., inward-facing, such as toward the user's eyes) camera (such as Fig. 16B ) to perform eye tracking as described, for example, in U.S. Patent Application Serial Nos. 14 / 707,000 and 15 / 238,516, the entire contents of which are incorporated herein by reference. The system can be configured to perform several steps in eye tracking, such as first taking an image of the user's eyes; then using segmentation analysis to segment the anatomy of the eye (e.g., separating the pupil from the iris, sclera, and surrounding skin); then the system can be configured to estimate the pupil center using the location of glints identified in the image of the eye, the glints being produced by a small illumination source (16) (such as an LED) that can be placed around the inward-facing side of the head-mounted component (58); from these steps, the system can be configured to use geometric relationships to determine an accurate estimate of the spatial location of a particular eye's gaze. For both eyes, this process is quite computationally intensive, particularly given the resources available on a portable system (such as a head-mounted component (58) that features an onboard embedded processor and limited power).
[0199] Deep learning techniques can be trained and utilized to address these and other computational challenges. For example, in one embodiment, a deep learning network can be utilized to perform the segmentation portion of the eye tracking paradigm described above (e.g., a deep convolutional network can be utilized to robustly segment the left and right eye images into iris, pupil, sclera, and other classes on a pixel-by-pixel basis), with everything else remaining the same; this configuration takes one of the heavily computationally intensive parts of the process and makes it significantly more efficient. In another embodiment, a joint deep learning model can be trained and utilized to perform segmentation, pupil detection, and glint detection (e.g., a deep convolutional network can be utilized to robustly segment the left and right eye images into iris, pupil, sclera, and other classes on a pixel-by-pixel basis; eye segmentation can then be utilized to narrow down the 2-D glint location of the active inward-facing LED illumination source); geometric calculations can then be performed to determine gaze. This paradigm also simplifies computations. In a third embodiment, a deep learning model can be trained and utilized to estimate gaze directly based on two images of the eyes from an inward-facing camera (e.g., in such an embodiment, a deep learning model using only pictures of the user's eyes can be configured to tell the system where the user is looking in three-dimensional space; a deep convolutional network can be used to robustly segment the left and right eye images pixel by pixel into iris, pupil, sclera, and other classes; the eye segmentation can then be used to narrow down the 2-D glint location of the active inward-facing LED illumination source; the 2-D glint location as well as the 3-D LED location can be used to detect the corneal center in 3-D; note that all 3-D locations may be in the corresponding camera coordinate system; the eye segmentation can then also be used to detect the pupil center in the 2-D image using ellipse fitting; using offline calibration information, the 2-D pupil center can be mapped to a 3-D gaze point, with depth determined during calibration; the line connecting the corneal 3-D location and the 3-D gaze point location is the gaze vector for that eye); this paradigm also simplifies computation, and the associated deep network can be trained to directly predict the 3-D gaze point given the left and right images. The loss function used to perform this training of the deep network can be a simple Euclidean loss, or it can also include well-known geometric constraints of the eye model.
[0200] In addition, a deep learning model can be included for biometric identification using images of the user's irises from the inward-facing camera. Such a model can also be used to determine if the user is wearing contact lenses - since the model will jump out of the Fourier transform of the image data from the inward-facing camera.
[0201] Using an outward-facing camera such as Fig.16AUsing the cameras (124, 154, 156) shown in , SLAM or PTAM analysis is performed to determine poses, such as the pose of a user's head relative to the environment in which he is currently wearing the head-mounted component (58), as described above. Most SLAM techniques depend on the tracking and matching of geometric features, as described in the above embodiments. Generally, it is helpful to be in a "textured" world, where outward-facing cameras are able to detect corners, edges, and other features; in addition, certain assumptions can be made about the persistence / static nature of features detected in the scene, and it is helpful to employ a SLAM or PTAM process to provide significant computational and power resources for all such mapping and tracking analysis; such resources may be in short supply in certain systems (such as some systems that are portable or wearable) and may have limited embedded processing capabilities and available power.
[0202] Example DeepSLAM Network
[0203] Deep learning networks can be incorporated into various embodiments to observe differences in image data and, based on training and configuration, play a key role in SLAM analysis of variations of the present system (in the context of SLAM, the deep networks herein can be considered "DeepSLAM" networks).
[0204] In one embodiment, a DeepSLAM network may be utilized to estimate a pose between a pair of frames captured from a camera coupled to a component to be tracked, such as a head mounted component (58) of an augmented reality system. The system may include a convolutional neural network configured to learn a transformation of a pose (e.g., the pose of the head mounted component 58) and apply it in a tracking manner. The system may be configured to start looking at a particular vector and orientation, such as straight ahead at a known origin (thus 0,0,0 as X, Y, Z). The user's head may then move, for example, a little to the right and then a little to the left between frame 0 and frame 1, with the goal of finding a pose transformation or a relative pose transformation. An associated deep network may be trained on a pair of images, for example, where we know pose A and pose B, as well as image A and image B; this results in a certain pose transformation. By determining the pose transformation, the associated IMU data (from accelerometers, gyroscopes, etc. - as described above) may then be integrated into the pose transformation and tracking continued as the user moves away from the origin, around the room, and any trajectory. Such a system may be referred to as a "relative pose net" and, as described above, is trained based on pairs of frames where known pose information is available (the transformation is determined from one frame to another, and based on the changes in the actual image, the system learns the pose transformation in terms of translation and rotation). Deep homography estimation or relative pose estimation has been discussed, for example, in U.S. Patent Application Serial No. 62 / 339,799, which is incorporated herein by reference in its entirety.
[0205] When using this configuration to perform pose estimation from frame 0 to frame 1, the results are generally not perfect, and the system can implement methods for dealing with drift. As the system moves forward from frame 1 to 2, to 3, to 4, and estimates the relative pose, a small amount of error is introduced between each pair of frames. This error typically accumulates and becomes a problem (e.g., without addressing this error-based drift, the system may end up placing the user and his or her associated system components in the wrong position and orientation with a pose estimate. In one embodiment, the concept of "loop closure" can be applied to address the so-called "relocalization" problem. In other words, the system can be configured to determine whether it has been in a previous specific position - if so, then the predicted pose information should make sense given the previous pose information at the same position. For example, the system can be configured so that when it sees a frame on a map that it has seen before, it relocalizes; if the translation is off, say 5 mm in the X direction, and the rotation is off, say 5 degrees in the θ direction, then the system fixes that difference along with the differences of the other associated frames; thus, the trajectory becomes a true trajectory, rather than an erroneous trajectory. Relocalization is discussed in U.S. patent application serial number 62 / 263,529, the entire contents of which are incorporated herein by reference.
[0206] It has also been found that when estimating pose, particularly by using IMU information (e.g., such as data from associated accelerometers, gyroscopes, etc., as described above), there is noise in the determined position and orientation data. For example, if the system directly utilizes this data without further processing to render an image, the user may experience undesirable jitter and instability; this is why in certain techniques, such as those described above, Kalman filters, sensor fusion techniques, and smoothing functions can be utilized. Employing deep network solutions such as the above-described use of convolutional neural networks to estimate pose, the smoothing problem can be solved using recurrent neural networks (RNNs) similar to long short-term memory networks. In other words, the system can be configured to build convolutional neural networks and, most importantly, place RNNs. Traditional neural networks are feed-forward by design and static in time; given an image or image pair, they give you an answer. With RNNs, the output of a layer is added to the next input and fed back into the same layer again - this is usually the only layer in the network; this can be thought of as "traveling through time" - at each point in time, the same network layer reconsiders the input slightly adjusted in time, and the cycle repeats. Furthermore, unlike a feed-forward network, an RNN can receive a sequence of values as input (e.g., sequenced over time) - and can also produce a sequence of values as output. The simple structure of an RNN with a built-in feedback loop allows it to behave like a prediction engine, and the result when combined with a convolutional neural network in this embodiment is that the system can take relatively noisy trajectory data from a convolutional neural network, push it through the RNN, and it will output a smoother trajectory that is more like human motion, such as the motion of a user's head, which can be coupled to a head-mounted component of a wearable computing system (58).
[0207] The system can also be configured to determine the depth of an object from a stereo image pair, where you have a deep network and input a left image and a right image. The convolutional neural network can be configured to output the disparity between the left and right cameras (such as between the left eye camera and the right eye camera on the head mounted component 58); if the focal length of the camera is known, the determined disparity is the inverse of the depth, so the system can be configured to efficiently calculate the depth with the disparity information; meshing and other processes can then be performed without involving alternative components for sensing depth, such as a depth sensor, which may require a relatively high load of computational and power resources.
[0208] With respect to the application of semantic analysis and deep networks to various embodiments of the present augmented reality configuration, several areas are of particular interest and applicability, including but not limited to detecting gestures and key points, facial recognition, and 3-D object recognition.
[0209] Regarding gesture recognition, in various embodiments, the system is configured to recognize certain gestures by the user's hand to control the system. In one embodiment, the embedded processor can be configured to utilize the so-called "random forest" and the sensed depth information to recognize certain gestures of the user. The random (decision) forest model is a non-deterministic model (during training) that can use a relatively large parameter library and can use relatively large processing power, system resources, and therefore power requirements. After training, the random (decision) forest model is deterministic. The random (decision) forest model includes one or more decision trees that vote on various characteristics of the image data (e.g., hand / non-hand, left / right, center / non-center, back of hand / palm, etc.). Random (decision) forests include classification forests (X possibilities are described in the image) and regression forests (numerical values of image characteristics; for example, where in space are fingertips, joints, etc.). Repeatedly reading and writing these decision trees to limited function / working memory (e.g., memory slices) also increases the burden on system resources. Additionally, depth sensors may not always be best suited for reading gestures with certain backgrounds (such as a table or tabletop or wall near the depth of the subject's hand) due to noise limitations of certain depth sensors and the inability to accurately determine the difference between, for example, 1 or 2 cm depth. In certain embodiments, random forest type gesture recognition may be replaced with a deep learning network. One of the challenges of utilizing a deep network for such a configuration is labeling portions of the image information (such as pixels) as "hand" or "non-hand"; training and utilizing a deep network with such segmentation challenges may require segmentation with millions of images, which is very expensive and time consuming. To address this issue, in one embodiment, during training time, a thermal camera (such as a thermal camera that may be used for military or security purposes) may be coupled to a conventional outward-facing camera such that the thermal camera significantly performs the segmentation of "hand" and "non-hand" by showing which portions of the image are hot enough to be a human hand and which are not. During training, the thermal camera may be used to collect image data (e.g., more accurately label key points) to improve thermal contrast with the use of temperature regulating substances (e.g., cooling gel, hot water, etc.) on the hand or background.
[0210] In some embodiments, a hyperspectral camera (e.g., RGB / IR camera) is used to train a random (decision) forest model. Since a hyperspectral camera detects image data at both visible and invisible wavelengths, an invisible wavelength (e.g., IR) marking scheme can be used to mark the hands during training of the random (decision) forest model.
[0211] In some embodiments, the image data includes amplitude data and / or depth data. In some embodiments, the amplitude data may have a higher resolution than the depth data.
[0212] In some embodiments, a random (decision) forest model is trained using: (1) automatic and / or manual labeling; (2) manual validation. During training, multiple (e.g., 20) image features are randomly selected and used to build a trained random decision tree. The decision tree can have a certain amount of variation (proportional to the size of the tree). To reduce this variation, the random (decision) forest model can include a larger number of trees to average out the variation. In some embodiments, each decision tree is approximately 100kb in size. The working memory size is in the 100kb range, and only one decision tree is loaded and processed at a time. Training generates a selection of the "best" features to determine the characteristics. In some embodiments, the entire random (decision) forest model is approximately 13MB.
[0213] In some embodiments, graphical tools may be used during segmentation of hand image data during training of a random (decision) forest model.
[0214] During training and testing, labeling the data can include labeling various features of the images: all pixels; key points (e.g., 25 per hand); categorical attributes (left / right, back of hand / palm / etc.); joints, and so on.
[0215] In some embodiments, the decision trees in the random (decision) forest will each apply features selected from a set of randomly selected tests during training. In the first step, all pixels of the image data are clustered (segmented) using a DB scanning algorithm (seed (flood) filling) to determine connected components, which are a set of clusters. This set of clusters / connected components is then passed through a rejection cascade. The cascade may include questions such as: (1) Too small to be a hand? (2) Too large to be a hand? (3) Too far to be the user's hand? (4) Too close to be within the operating range of the augmented reality device? This cascade will eliminate many clusters / connected components while retaining most (if not all) possible clusters / connected components as selected candidates for further analysis. The selected candidates can be modified (filtered to remove artifacts and / or morphological filling is applied in the holes) to clean up noise from the camera / depth sensor / image source. The modified candidates are classified as "hand" or "non-hand" (a binary classification problem) by the decision tree in the random (decision) forest for that feature. These hands are then classified as left or right by each decision tree in the random (decision) forest for that feature. Each of these classification decision trees generates a "vote" about the corresponding feature (e.g., hand / non-hand, left / right, etc.), and the "votes" are averaged to reduce variation.
[0216] These hands are also analyzed using each regression decision tree for this feature in the random (decision) forest to identify the center of the hand. First, the hand is skeletonized using a distance transform / medial axis transform. Using this transform, points on the contour / surface of the hand are assigned a zero value, and points inside the hand are assigned a value proportional to its distance from the surface of the hand. The values of these points are the largest on the skeleton of the hand, and these points are farthest from the surface of the hand. The hand is then filtered into a skeleton value, which reduces the two-dimensional image of the hand to a "one-dimensional" skeleton (i.e., a line drawing of the skeleton of the hand). This reduces the number of anchor points used to calculate any geometric features. The hand is skeletonized to simplify the two-dimensional analysis problem to a one-dimensional plus branch analysis problem, reducing system requirements and increasing the speed of the random forest model, while improving the robustness and accuracy of gesture classification. Each of these regression decision trees generates a "vote" about the corresponding feature (the location of the center of the hand), and the "vote" is averaged to reduce variation. Although skeletonization produces fewer "votes", the votes are more accurate. Skeletonization can also increase the speed of the random forest model by an order of magnitude.
[0217] Similar techniques can be applied to regress / identify the locations of the various joints of the hand (e.g., which joint is closest to the camera?). After the hand is skeletonized and the locations of the hand centers and joints are identified, the skeletonized hand can be classified (yes or no) for each gesture identified by the random (decision) forest using a classification decision tree for that gesture.
[0218] In some embodiments, gesture recognition may be performed on image data using a deep neural network.
[0219] With respect to facial recognition, and given that the present augmented reality system is configured to be worn in a social environment with other people, understanding the people around the user may be of relatively high value - not just for simply identifying other nearby people, but also for adjusting the information presented (e.g., if the system identifies the nearby person as an adult friend, it may suggest that you play chess and assist; if the system identifies the nearby person as your child, it may suggest that you go play soccer and assist; if the system fails to identify the nearby person, or identifies them as a known danger, the user may tend to avoid approaching such a person). In some embodiments, a deep neural network configuration may be utilized to assist with facial recognition in a manner similar to that discussed above with respect to deep relocalization. The model may be trained with multiple different faces associated with the user's life, and then when a face is close to the system, such as near the head mounted component (58), the system may take an image of the face in pixel space, translate it, for example into a 128-dimensional vector, and then use the vector as a point in a high-dimensional space to determine whether the person exists in your list of known people. Essentially, the system can be configured to do a "nearest neighbor" search in this space, and when results come in, this configuration can be very accurate, with false positive rates running in the 1 in 1000 range.
[0220] With respect to 3-D object detection, in some embodiments it is useful to incorporate a deep neural network that will tell the user from a three-dimensional perspective what space they are in (e.g., not just the walls, floor, ceiling, and objects that fill the room, such as couches, chairs, cabinets, etc. - not just the traditional two-dimensional sense, but a true three-dimensional sense). For example, in one embodiment, it is desirable for the user to have a model that understands the true volumetric extent of a couch in a room - so that, for example, in the case of a virtual ball or other object to be thrown, the user knows what volume the volume of the couch occupies. A deep neural network model can be utilized to form a cube model with a high degree of complexity.
[0221] In some embodiments, deep reinforcement networks or deep reinforcement learning can be used to effectively learn what an agent should do in a particular environment without the user having to directly tell the agent. For example, if a user wants to always have a virtual representation of his dog walking around the room he occupies, but he wants the representation of the dog to always be visible (e.g., not hidden behind a wall or cabinet), the deep reinforcement method can turn the scene into a game of sorts, in which the virtual agent (here, a virtual dog) is allowed to roam in the physical space near the user, but during training time, if the dog stays in an acceptable position from, say, T0 to T1, then a reward is given, and if the user's view of the dog is blocked, lost, or the dog hits a wall or object, then a penalty is given. With this embodiment, the deep network begins to learn what it needs to do to win points instead of losing points, and soon it knows what it needs to know to provide the desired functionality.
[0222] The system can also be configured to address the lighting of the virtual world in a manner that approximates or matches the lighting of the real world surrounding the user. For example, in order for the virtual perception to blend as optimally as possible with the actual perception in the augmented reality, the virtual objects are used to reproduce the lighting color, shadows, and lighting vectorization as realistically as possible. In other words, if a virtual opaque coffee cup is to be placed on an actual tabletop in a room, with a yellow colored light emitting from a particular corner of the room, casting shadows from real world objects on the real world table, then the light shading and shadows of the virtual coffee cup are optimally matched to the actual scene. In some embodiments, a deep learning model can be used to learn the lighting of the actual environment in which the system components are placed. For example, a model can be used that, given an image or image sequence from the actual environment, learns the lighting of the room to determine factors such as brightness, hue, and vectorization by one or more light sources. Such a model can be trained from synthetic data and images captured from a user device (such as from a user's head-mounted component (58)).
[0223] Example Hydra Architecture
[0224] refer to Fig.29, shows a deep learning network architecture that may be referred to as a "Hydra" architecture (272). With this configuration, various inputs (270) such as IMU data (from accelerometers, gyroscopes, magnetometers), outward-facing camera data, depth-sensing camera data, and / or sound or voice data may be directed to a multi-layered centralized processing resource having one or more lower layers (268) that perform a significant portion of the overall processing, pass their results to one or more middle layers (266), and ultimately to one or more of a plurality of associated "heads" (264) representing various processing functions such as facial recognition, visual search, gesture recognition, semantic segmentation, object detection, lighting detection / determination, SLAM, relocalization, and / or depth estimation (such as from stereo image information, as described above). The occurrence, determination, or recognition of a gesture, object, relocalization, or depth (or any state associated with any function) may be referred to as an event associated with a particular function. In the wearable display system, in various embodiments, the Hydra architecture may be implemented on and executed by the local processing and data module 70 or the remote processing modules and data repositories 72, 74. The plurality of lower layers (268) and the intermediate layers (266) may be referred to as a plurality of mid-level layers.
[0225] Traditionally, when deep networks are used to implement various tasks, algorithms are built for each task. So if it wants to recognize cars, an algorithm will be built for that; if it needs to recognize faces, an algorithm will be built for that; and these algorithms can run at the same time. If unlimited or high levels of power and computing resources are available, this configuration will work well and get results; but in many scenarios, such as scenarios with limited power supplies in embedded processors and portable augmented reality systems with limited processing power, computing and power resources may be relatively limited, and some aspects of the task may need to be processed together. In addition, if one algorithm has knowledge from another algorithm, it can make the second algorithm better in some embodiments. For example, if one deep network algorithm knows about dogs and cats, knowledge transfer (also known as "domain adaptation") from that algorithm can help another algorithm better recognize shoes. Therefore, there is a reason to have some crosstalk between algorithms during training and inference.
[0226] Additionally, there are considerations related to algorithm design and modification. Preferably, if further capabilities are needed relative to the initial version of the algorithm, new capabilities do not need to be completely rebuilt from scratch. The depicted Hydra architecture (272) can be used to address these challenges, as well as computational and power efficiency challenges, because, as described above, there are cases where common aspects of certain computational processes can be shared. For example, in the depicted Hydra architecture (272), the input (270) (such as image information from one or more cameras) can be brought into the lower layer (268) where relatively low-level feature extraction can be performed. For example, Gabor functions, Gaussian derivatives, things that basically affect lines, edges, corners, colors - these are uniform to many problems at the low level. Therefore, regardless of the task changes, the low-level feature extraction can be the same, whether the goal is to extract a cat, a car, or a cow - and the computations associated with it can therefore be shared. The Hydra architecture (272) is a high-level paradigm that allows knowledge sharing across algorithms to make each better, it allows feature sharing so that computation can be shared and reduced rather than redundant, and it allows one to extend the suite of functionality without having to rewrite everything—instead, new functionality can be stacked on top of existing functionality.
[0227] Thus, as described above, in the depicted embodiment, the Hydra architecture represents a deep neural network with one unified path. The bottom layers (268) of the network are shared, and they extract the basic units of visual primitives from the input image and other inputs (270). The system can be configured to go through several layers of convolution to extract edges, lines, contours, connection points, etc. The basic components that programmers use for feature engineering have now been learned by the deep network. It turns out that these features are useful for many algorithms, whether the algorithm is facial recognition, tracking, etc. Therefore, once the lower computational work is done and there is a shared representation from the image or other input to all other algorithms, then there can be individual paths, one path per problem or function. Thus, in addition to this shared representation, there is a path to facial recognition that is very specific to faces, there is a path to tracking that is very specific to SLAM, and paths for other "heads" (264) of the architecture (272), and so on. With this embodiment, one has all the shared computations that allow basically multiplication and addition, and on the other hand one has very specific paths that are on top of the general knowledge and allow one to fine tune and find answers to very specific problems.
[0228] Also valuable with this configuration is the fact that this neural network is designed so that the lower layers (268) that are closer to the input (270) utilize more compute, because at each computational layer, the system takes the raw input and transforms it into some other dimensional space, typically where the dimensionality of things is reduced. Thus, once the fifth layer of the network from the bottom is implemented, the amount of compute can be in the range of 5, 10, 20, 100 times (or more) less than the amount used in the lowest levels (e.g., because the input is much larger and larger matrix multiplications are used). In one embodiment, by the time the system extracts shared compute, it is fairly agnostic about the problem that needs to be solved. Most of the computation for almost all algorithms is already done in the lower layers, so when new paths are added for facial recognition, tracking, depth, lighting, etc., these contribute relatively little to the computational constraints - and so this architecture provides ample scalability. In one embodiment, for the first few layers, there may be no pooling to retain the highest resolution data; the intermediate layers may have a pooling process because at that point, high resolution is not needed (e.g., high resolution is not needed to know where the wheels of a car are at the intermediate layers; the network typically needs to know where the nuts and bolts are at high resolution from the higher levels, and then the image data can be significantly reduced as it is passed to the intermediate layers for the location of the wheels). For example, the features generated in the lower levels include features with a first resolution, and the features generated in the intermediate layers include features with a second resolution that is less than the first resolution. In addition, once the network has all the learned connections, everything is loosely wired and the connections are advantageously learned through data. The intermediate layers (266) can be configured to start learning parts, such as - object parts, facial features, etc.; thus, rather than simple Gabor functions, the intermediate layers are dealing with more complex structures or higher level features (e.g., wave shapes, shadows, etc.). Then, as the process moves up to the top, separation occurs into distinct head components (264), some of which may have many layers and some of which may have few layers. The layers of head components (264) may be referred to as head component layers. Likewise, scalability and efficiency are largely due to the fact that most (such as 90%) of the processing power (e.g., measured in floating point operations per second (flops)) is within the lower tier (268), with a small portion (such as 5% of the flop) located at the middle tier (266), and another 5% located in the head (264).
[0229] Such a network can be pre-trained using already existing information. For example, in one embodiment (ImageNet), all classes can be trained using a large set (in the range of 10 million) from a large set of classes (in the range of 1000). In one embodiment, once trained, the top layers of the distinguishing classes can be thrown out, but all weights learned during training are retained.
[0230] The process of training a neural network having a hydra architecture (272) involves presenting input data and corresponding target output data to the network. This data, including both example inputs and target outputs, may be referred to as a training set. Through the training process, the weights of the network, including weights associated with lower layers (268), intermediate layers (266), and head components (264), may be incrementally or iteratively adapted so that the output of the network, given specific input data from the training set, matches (e.g., as closely as possible) the target output corresponding to the specific input data.
[0231] In some embodiments for gesture tracking, separate heads of the hydra architecture are built on top of a multi-channel main encoder to address selected components of gesture tracking. By passing through the main encoder, the image input is converted into a latent representation of the original image for each hydra head to use for discrete predictions.
[0232] For example, the first head is configured to recognize discrete hand gestures for the left and right hands, such as key gestures (e.g., "ok", pinch, point, thumbs up, fist, open hand, L-shape), as well as "unknown gestures." In an embodiment of eight known hand gestures, there are nine predictions to make (eight known gestures plus the "unknown gesture"). If the image includes both hands of the user, this results in a total of 18 possible outputs, corresponding to an equal number of channels in the head.
[0233] In some embodiments, the Hydra architecture includes binary prediction heads for the presence of multiple hands (left, right, none, both) with a confidence measure (e.g., 50% prediction of the presence of either hand). If this initial predictor returns a low confidence score indicating absence, then no additional predictions need to be made by the parallel heads, unlike conventional systems where structures are measured for parallel outputs even if the data API is not present. In some embodiments, different weights of the confidence scores may be applied depending on the desire for a preference for false positives (resulting in a high confidence barrier, such as 90%) or false negatives. This greatly increases the computational burden by determining whether additional computation is needed at all, rather than to what extent additional computation is considered.
[0234] In some embodiments, the Hydra architecture includes a keypoint recognition head. The keypoint recognition head inverts the spatial resolution performed by the main encoder and generates multiple layers of heatmaps based on the target number of binary predictions (i.e., keypoints) requested. For example, a pointing gesture keypoint indication layer will predict where the index finger is located with a heatmap; similarly, a thumbs-up keypoint indication layer will predict where the thumb is located with a heatmap. In some embodiments, the binary flags of a first Hydra head inform the keypoint predictor of a third Hydra head to uniformly compute such heatmaps. For example, if the binary discrete posture predictor of the first head predicts that there is no left hand, or there are no known gestures for the left hand, then half of the layers in the keypoint classification (those dedicated to the left hand) can be ignored. At the API level, they can be associated using the argmax function.
[0235] By replacing the random forest of keypoints conventionally performed in the art (which requires explicit and discrete forests to determine left / right, expected gestures to derive keypoints, thereby doubling the number of computational resources per forest), a single computation is performed by the Deep Gesture Hydra Network.
[0236] A neural network (NN) layer such as a deep neural network (DNN) can apply a linear or nonlinear transformation to its input to generate its output. The deep neural network layer can be a normalization layer, a convolution layer, a softsign layer, a corrected linear layer, a cascade layer, a pooling layer, a recurrent layer, a similar initial layer, or any combination thereof. The normalization layer can normalize the brightness of its input to generate its output by, for example, L2 normalization. For example, the normalization layer can normalize the brightness of multiple images relative to each other at once to generate multiple normalized images as its output. Non-limiting examples of methods for normalizing brightness include local contrast normalization (LCN) or local response normalization (LRN). Local contrast normalization can nonlinearly normalize the contrast of an image by normalizing a local area of the image on a per-pixel basis to have a mean of zero and a variance of 1 (or other values of the mean and variance). Local response normalization can normalize an image on a local input area to have a mean of zero and a variance of 1 (or other values of the mean and variance). Normalization layers can speed up the training process.
[0237] A convolutional layer may apply a set of kernels that convolve its input to generate its output. A softsign layer may apply a softsign function to its input. The softsign function (softsign(x)) may be, for example, (x / (1+|x|)). The softsign layer may ignore the effects of outliers at each element. A rectified linear layer may be a rectified linear layer unit (ReLU) or a parameterized rectified linear layer unit (PReLU). A ReLU layer may apply a ReLU function to its input to generate its output. The ReLU function ReLU(x) may be, for example, max(0, x). A PReLU layer may apply a PReLU function to its input to generate its output. For example, if x≥0, the PReLU function PReLU(x) may be x, and if x<0, it may be ax, where a is a positive number. A cascade layer may cascade its input to generate its output. For example, a cascade layer may cascade four 5×5 images to generate a 20×20 image. A pooling layer may apply a pooling function that downsamples its input to generate its output. For example, a pooling layer may downsample a 20×20 image to a 10×10 image. Non-limiting examples of pooling functions include max pooling, average pooling, or min pooling.
[0238] At time point t, the recurrent layer can calculate the hidden state s(t), and the recurrent connection can provide the hidden state s(t) to the recurrent layer at time t as an input for a subsequent time point t+1. The recurrent layer can calculate its output at time t+1 based on the hidden state s(t) at time t. For example, the recurrent layer can apply a softsign function to the hidden state s(t) at time t to calculate its output at time t+1. The hidden state of the recurrent layer at time t+1 has the hidden state s(t) of the recurrent layer at time t as its input. The recurrent layer can calculate the hidden state s(t+1) by applying, for example, a ReLU function to its input. Similar initial layers may include one or more of a normalization layer, a convolutional layer, a soft design layer, a rectified linear layer (e.g., a ReLU layer and a PReLU layer), a cascade layer, a pooling layer, or any combination thereof.
[0239] In different embodiments, the number of layers in the NN can be different. For example, the number of layers in the lower layer (268) or the middle layer (266) can be 50, 100, 200 or more. The input type of the deep neural network layer can be different in different embodiments. For example, a layer can receive the output of multiple layers as its input. The input of a layer can include the output of five layers. As another example, the input of a layer can include 1% of the NN layers. The output of a layer can be the input of multiple layers. For example, the output of a layer can be used as the input of five layers. As another example, the output of a layer can be used as the input of 1% of the NN layer.
[0240] The input size or output size of a layer may be very large. The input size or output size of a layer may be n×m, where n represents the width and m represents the height of the input or output. For example, n or m may be 11, 21, 31, or more. In different embodiments, the channel size of the input or output of a layer may be different. For example, the channel size of the input or output of a layer may be 4, 16, 32, 64, 128, or more. In different embodiments, the kernel size of a layer may be different. For example, the kernel size may be n×m, where n represents the width and m represents the height of the kernel. For example, n or m may be 5, 7, 9, or more. In different embodiments, the stride size of a layer may be different. For example, the stride size of a deep neural network layer may be 3, 5, 7, or more.
[0241] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying drawings can be embodied in a code module executed by one or more physical computing systems, hardware computer processors, dedicated circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and fully or partially automated by the code module. For example, a computing system may include a general-purpose computer (e.g., a server) programmed with specific computer instructions or a dedicated computer, dedicated circuit, etc. The code module can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some embodiments, specific operations and methods can be performed by circuits specific to a given function.
[0242] Furthermore, certain implementations of the functionality of the present disclosure may be mathematically, computationally or technically complex enough that application-specific hardware or one or more physical computing devices (with appropriate dedicated executable instructions) may be necessary to perform the functionality, e.g., due to the amount or complexity of the computations involved or to provide results in substantially real time. For example, a video may include many frames, each with millions of pixels, and specially programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.
[0243] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disk, volatile or non-volatile storage devices, combinations thereof, and the like. Methods and modules (or data) may also be sent as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based and wired / cable-based media), and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or may be transmitted via a computer-readable transmission medium.
[0244] Any process, frame, state, step or function in the flowchart described herein and / or depicted in the accompanying drawings should be understood to represent a code module, code segment or code portion, which includes one or more executable instructions for implementing a specific function (e.g., logic or arithmetic) or step in a process. Various processes, blocks, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed with the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps or states associated therewith can be performed in other appropriate sequences, for example, in serial, parallel or in some other manner. Tasks or events can be added to or deleted from the disclosed example embodiments. In addition, the separation of various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.
[0245] The processes, methods and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.
[0246] The present invention includes methods that can be performed using the present apparatus. The method can include the act of providing such suitable apparatus. Such providing can be performed by the end user. In other words, the act of "providing" only requires the end user to obtain, access, approach, locate, set up, activate, power on or otherwise act to provide the necessary apparatus in the present method. The methods described herein can be performed in any order of the events described that is logically possible and arranged in the order of the events described.
[0247] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or requires the desired attributes disclosed herein. The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications to the embodiments described in the present disclosure will be apparent, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the embodiments shown herein, but are consistent with the widest range consistent with the present disclosure, the principles disclosed herein, and the novel features.
[0248] Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. In addition, although the above features may be described as functioning in certain combinations and even initially claimed as such, in some cases one or more features from the claimed combination may be excised from the combination, and the claimed combination may be directed to a subcombination or variations of the subcombination. It is not necessary or essential for each and all embodiments to require a single feature or group of features.
[0249] Unless otherwise specified, or understood in the context of use in other ways, conditional language used herein, such as "can", "may", "possibly", "may", "for example", etc., is generally intended to convey that certain embodiments include certain features, elements and / or steps, while other embodiments do not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements and / or steps in any way, or that one or more embodiments must include logic for determining whether these features, elements and / or steps are included or to be performed in any particular embodiment (regardless of whether there is author input or prompt). The terms "comprise", "include", "have", etc. are synonymous and are used inclusively in an open-ended manner, and do not exclude additional elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so when used, for example, to connect a list of elements, the term "or" means one, some or all of the elements in the list. In addition, unless otherwise specified, the articles "a", "an" and "the" used in this application and the appended claims should be understood to mean "one or more" or "at least one". Unless specifically defined herein, all technical and scientific terms used herein are given the broadest possible commonly understood meaning while maintaining claim validity. It should also be noted that the claims may be drafted to exclude any optional element.
[0250] As used herein, phrases referring to "at least one" of a list of items refer to any combination of those items, including single members. For example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Unless otherwise specifically stated, conjunctive language such as the phrase "at least one of X, Y, and Z" is otherwise understood by context as being generally used to convey that an item, term, etc. may be at least one of X, Y, or Z. Thus, such conjunctive language is generally not intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z, respectively.
[0251] Similarly, although operations can be depicted in the drawings in a particular order, it should be appreciated that it is not necessary to perform these operations in the particular order shown or in sequence, or to perform all the operations shown, to achieve the desired result. In addition, the drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown may be included in the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously, or between any of the illustrated operations. In addition, operations may be rearranged or reordered in other embodiments. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. In addition, other embodiments are within the scope of the following claims. In some cases, the actions recorded in the claims may be performed in different orders and still achieve the desired results.
Claims
1. A computer-implemented method for recognizing hand gestures using a random forest model, comprising: Training the random forest model; Get image data; clustering a plurality of pixels from the image data to generate a plurality of clusters; analyzing the plurality of clusters using a plurality of questions to filter the plurality of clusters and generate a plurality of selected candidates; analyzing the plurality of selected candidates using a classification decision tree from a trained random forest model to identify a hand among a selected candidate among the plurality of selected candidates; skeletonizing the plurality of selected candidates by applying a distance transform / medial axis transform and filtering the transformed results to generate a one-dimensional hand model, wherein the one-dimensional hand model depicts a graph of the skeleton of the hand, determining the position of the center of the hand or the positions of the joints of the hand by analyzing the one-dimensional hand model using a regression decision tree from the random forest model; and recognizing a gesture using a plurality of classification decision trees based at least in part on a first position of the center of the hand and a second position of the joint of the hand, Wherein, applying the distance transformation / medial axis transformation comprises: assigning a first value to a point on the contour of the hand in the selected candidate; A second value is assigned to the additional point based at least in part on a distance between a corresponding one of the additional points within the outline of the hand and the surface of the hand.
2. The method according to claim 1, wherein: Skeletonizing the plurality of selected candidates includes using a graphical tool.
3. The method according to claim 1, wherein: Skeletonizing the plurality of selected candidates includes providing labels to portions of the plurality of selected candidates, wherein the labels indicate whether the selected candidate is a hand, a left hand or a right hand, or a back-of-hand view or a palm-of-hand view.
4. The method according to claim 3, wherein: The portions of the plurality of selected candidates include joints of a hand.
5. The method according to claim 1, wherein: Skeletonizing the plurality of selected candidates reduces the number of anchor points analyzed by the regression decision tree.
6. The method according to claim 1, wherein: Skeletonizing the plurality of selected candidates increases the speed of the computer-implemented method and increases the accuracy of the computer-implemented method.
7. The method according to claim 1, wherein: Training the random forest model includes providing labels to the plurality of selected candidates, wherein the labels indicate whether the selected candidate is a hand, a left hand or a right hand, or a back of hand view or a palm view.
8. The method according to claim 1, wherein: Training the random forest model includes randomly selecting a plurality of image features and constructing a decision tree using the selected plurality of image features.
9. The method according to claim 1, wherein: Training the random forest model includes using a thermal camera to acquire training images of a hand.
10. The method according to claim 9, wherein: Training the random forest model also includes: using a temperature regulating substance.
11. The method according to claim 9, wherein: The thermal camera is a hyperspectral camera.
12. The method according to claim 1, wherein: The image data includes amplitude data and depth data.
13. The method according to claim 1, wherein: Clustering the plurality of pixels from the image data to generate the plurality of clusters includes using a seed filling algorithm.
14. The method according to claim 1, wherein: Analyzing the plurality of clusters using the plurality of questions to generate the plurality of selected candidates includes determining a size of the selected candidates.
15. The method according to claim 1, wherein: Analyzing the plurality of clusters using the plurality of questions to generate the plurality of selected candidates includes determining positions of the selected candidates.
16. The method according to claim 1, wherein: Analyzing the plurality of selected candidates using the classification decision tree from the random forest model includes determining whether the selected candidate is a hand, a left hand or a right hand, or a back of hand view or a palm view.
17. The method according to claim 1, wherein: Analyzing the one-dimensional hand model using the regression decision tree from the random forest model includes determining the location of a center of a hand or locations of joints of the hand.
18. The method of claim 1, further comprising averaging multiple results from multiple decision trees of the random forest model.
19. The method according to claim 18, wherein: Averaging the multiple results reduces the variation in results from the random forest model.
