Systems and methods for augmented reality

Abstract

The present invention relates to systems and methods for augmented reality. A configuration for presenting virtual reality and augmented reality experiences to a user is disclosed. The augmented reality display system includes: a handheld component that houses an electromagnetic field transmitter; the electromagnetic field transmitter that emits a known magnetic field; a head-mounted component that is coupled to one or more electromagnetic sensors that detect the magnetic field emitted by the electromagnetic field transmitter housed in the handheld component, wherein the head posture is known; and a controller that is communicatively coupled to the handheld component and the head-mounted component, the controller receiving magnetic field data from the handheld component and receiving sensor data from the head-mounted component, wherein the controller determines the posture of the hand based at least in part on the received magnetic field data and the received sensor data.

Description

[0001] This application is a divisional application of the PCT application with an application date of March 5, 2016, an international application number of PCT / US2016 / 021095, a Chinese national phase application number of 201680025479.1, and an invention name of “System and method for augmented reality”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Application Serial No. 62 / 128,993, filed on March 5, 2015, “ELECTROMAGNETIC TRACKING SYSTEM AND METHOD FOR AUGMENTED REALITY” (Docket No. ML 30031.00) and U.S. Provisional Application Serial No. 62 / 292,185, filed on February 5, 2016, “SYSTEMS AND METHODS FOR AUGMENTED REALITY” (Docket No. ML 30062.00). Background Art

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or may be perceived as real. Virtual reality "VR" scenarios generally involve the presentation of digital or virtual image information without transparency to other actual real-world visual input. Augmented reality "AR" scenarios generally involve the presentation of digital or virtual image information as an enhancement to the visualization of the actual world around the user.

[0005] For example, see Figure 1 , depicting an augmented reality scene 4 in which the user of AR technology sees a setting 6 resembling a real-world park featuring people, trees, buildings, and a concrete platform 1120 in the background. In addition to these items, the user of AR technology can also perceive a robotic statue 1110 standing on the real-world platform 1120 and a cartoon-like avatar character 2 flying around the park. Of course, the virtual elements 2 and 1110 do not exist in the real world, but the user perceives these virtual objects as part of the real world and interacts with real-world objects (e.g., 6, 1120, etc.). It should be understood that the human visual perception system is very complex, and it is challenging to produce such AR scenes that facilitate a comfortable, natural feel, and rich presentation of virtual image elements among other virtual or real-world image elements.

[0006] For example, a head-mounted AR display (e.g., a helmet display or smart glasses) can be coupled to a user's head and, therefore, can move when the user's head moves. If the display system detects the user's head movement, the data being displayed can be updated to account for the change in head posture. The head posture can be used to render virtual content appropriately to the user. Therefore, any small changes may interrupt and / or reduce the delivery or timing of virtual content delivered to the user's AR display.

[0007] As an example, if a user wearing a head-mounted display views a virtual representation of a three-dimensional (3D) object on the display and walks around the area where the 3D object appears, the 3D object can be re-rendered for each perspective, providing the user with the perception that he or she is walking around an object that occupies real space. If the head-mounted display is used to present multiple objects in a virtual space (e.g., a rich virtual world), measurements of head gestures (i.e., the position and orientation of the user's head) can be used to re-render the scene to match the user's dynamically changing head position and orientation, and provide an increased sense of immersion in the virtual space. Similarly, when a user of AR technology interacts with the virtual world, he or she can use an object or his / her hand to point to an object or select an option. For this interaction to occur, it is also important that the positioning of the object or the user's hand is accurate. Therefore, both head gestures and "hand gestures" are critical, and positioning technology must be used to accurately portray virtual content to the user.

[0008] In an AR system, detection and / or calculation of head pose and / or hand pose can facilitate the AR display system to render virtual objects so that they appear to occupy space in the real world in a manner consistent with real-world objects. Realistically presenting an AR scene so that the virtual content does not appear discordant / chaotic relative to one or more real objects improves the user's enjoyment of the AR experience. In addition, detection of the position and / or orientation of real objects such as handheld devices (which may also be referred to as "totems"), haptic devices, or other real physical objects relative to the user's head or AR system can also facilitate the display system to present display information to the user so that the user can effectively interact with certain aspects of the AR system.

[0009] It should be understood that in AR applications, the placement of virtual objects in space relative to physical objects (e.g., rendered to appear spatially close to physical objects in two or three dimensions) is an important issue. For example, head movement can significantly complicate the placement of virtual objects in the view of the surrounding environment. This may be true whether the view is captured as an image of the surrounding environment and then projected or displayed to the end user, or the end user directly perceives the view of the surrounding environment. For example, head movement may cause the user's field of view to change. In turn, this may require updating the positions of various virtual objects displayed in the field of view of the end user. Similarly, hand movements (in the case of handheld objects) pose the same challenges when used to interact with the system. These movements can be very fast and often need to be accurately detected and located at high refresh rates and low latency.

[0010] In addition, the movement of the head and / or hand can occur at a variety of ranges and speeds. The speed can vary not only between different types of head movements, but also within or over the range of a single movement. For example, the speed of the head movement can initially increase from a starting point (e.g., linearly, etc.) and can decrease when reaching an end point, with a maximum speed being obtained somewhere between the starting point and the end point of the head movement. Rapid movement may even exceed the ability of a particular display or projection technology to render an image that appears to be uniform and / or smoothly moving to an end user.

[0011] Head or hand tracking accuracy and latency (i.e., the time that elapses between when a user moves his or her head / hand and when the image gets updated and displayed to the user) has become a challenge for VR and AR systems. Especially for display systems that fill a large portion of the user's field of view with virtual elements, it is critical that the tracking accuracy is high and the total system latency from the first motion detection to the update of the light delivered by the display to the user's visual system is very low. If the latency is high, the system can cause a mismatch between the user's vestibular and visual sensory systems and produce a user-perceived scene that can cause motion sickness or simulator syndrome. If the system latency is high, the apparent position of virtual objects can appear unstable during rapid head movements.

[0012] In addition to head mounted display systems, other display systems may also benefit from accurate and low latency head pose detection. These may include head tracking display systems, where the display is not worn on the user's body, but is mounted, for example, on a wall or other surface. The head tracking display may act like a window on the scene, and as the user moves his head relative to the "window", the scene is re-rendered to match the user's changing viewpoint. Other systems may include head mounted projection systems, where the head mounted display projects light into the real world.

[0013] In addition, in order to provide a realistic AR experience, the AR system may be designed to interact with the user. For example, multiple users can play a ball game with a virtual ball and / or other virtual objects. One user can "catch" the virtual ball and throw it back to another user. In another embodiment, the first user may be equipped with a totem device (e.g., a physical "bat" communicatively coupled to the AR system) to hit the virtual ball. In other embodiments, a virtual user interface may be presented to the AR user to allow the user to select one of multiple options. The user may use a totem device, a haptic device, a wearable component, or simply touch the virtual screen to interact with the system.

[0014] Detecting the posture and orientation of a user (e.g., the user's head and hands) and detecting the physical position of real objects in space can enable an AR system to display virtual content in an effective and enjoyable manner. However, accurate detection of such head and hand postures can be difficult to achieve. In other words, the AR system must identify the physical position of real objects (e.g., the user's head, totem devices, haptic devices, wearable components, user's hands, etc.) and associate the physical coordinates of the real objects with the virtual coordinates corresponding to one or more virtual objects displayed to the user. This process can be improved by high-precision sensors and sensor recognition systems that quickly track the position and orientation of one or more objects. Current methods cannot perform positioning at a satisfactory speed or accuracy standard.

[0015] Therefore, there is a need for better positioning systems in the context of AR and VR devices. Summary of the invention

[0016] Embodiments of the present invention relate to apparatus, systems, and methods for facilitating virtual reality and / or augmented reality interaction for one or more users.

[0017] In one aspect, an augmented reality (AR) display system includes: an electromagnetic field transmitter for transmitting a known magnetic field; an electromagnetic sensor for measuring a parameter related to a magnetic flux measured at the electromagnetic sensor as a result of the transmitted known magnetic field, wherein the world coordinates of the electromagnetic sensor are known; a controller for determining posture information relative to the electromagnetic field transmitter based at least in part on the measured parameter related to the magnetic flux measured at the electromagnetic sensor; and a display system for displaying virtual content to a user based at least in part on the determined posture information relative to the electromagnetic field transmitter.

[0018] In one or more embodiments, the electromagnetic field transmitter is located in a mobile component of the AR display system. In one or more embodiments, the mobile component is a handheld component. In one or more embodiments, the mobile component is a totem device.

[0019] In one or more embodiments, the mobile component is a head mounted component of the AR display system. In one or more embodiments, the AR display system further includes a head mounted component that houses the display system, wherein the electromagnetic sensor is operably coupled to the head mounted component. In one or more embodiments, the world coordinates of the electromagnetic sensor are known at least in part based on a SLAM analysis performed to determine head pose information, wherein the electromagnetic sensor is operably coupled to the head mounted component that houses the display system.

[0020] In one or more embodiments, the AR display system further includes one or more cameras operably coupled to the head-mounted assembly, wherein the SLAM analysis is performed based at least on data captured by the one or more cameras. In one or more embodiments, the electromagnetic sensor includes one or more inertial measurement units (IMUs).

[0021] In one or more embodiments, the gesture information corresponds to at least a position and orientation of the electromagnetic field transmitter relative to the world. In one or more embodiments, the gesture information is analyzed to determine world coordinates corresponding to the electromagnetic field transmitter. In one or more embodiments, the controller detects interaction with one or more virtual contents based at least in part on the gesture information corresponding to the electromagnetic field transmitter.

[0022] In one or more embodiments, the display system displays virtual content to the user based at least in part on the detected interaction. In one or more embodiments, the electromagnetic sensor includes at least three coils to measure magnetic flux in three directions. In one or more embodiments, the at least three coils are housed together in substantially the same location, and the electromagnetic sensor is coupled to a head-mounted component of the AR display system.

[0023] In one or more embodiments, at least three coils are housed at different locations of a head-mounted assembly of the AR display system.

[0024] The AR display system according to claim 1 further comprises a control and quick release module for decoupling the magnetic field emitted by the electromagnetic field transmitter. In one or more embodiments, the AR display system further comprises an additional positioning resource for determining the world coordinates of the electromagnetic field transmitter. In one or more embodiments, the additional positioning resource comprises a GPS receiver. In one or more embodiments, the additional positioning resource comprises a beacon.

[0025] In one or more embodiments, the electromagnetic sensor comprises a non-solid ferrite cube. In one or more embodiments, the electromagnetic sensor comprises a stack of ferrite disks. In one or more embodiments, the electromagnetic sensor comprises a plurality of ferrite rods, each of which has a polymer coating. In one or more embodiments, the electromagnetic sensor comprises a time division multiplexing switch.

[0026] In another aspect, a method for displaying augmented reality includes: emitting a known magnetic field through an electromagnetic field transmitter; measuring, through an electromagnetic sensor, a parameter related to a magnetic flux measured at the electromagnetic sensor as a result of the emitted known magnetic field, wherein a world coordinate of the electromagnetic sensor is known; determining posture information relative to the electromagnetic field transmitter based at least in part on the measured parameter related to the magnetic flux measured at the electromagnetic sensor; and displaying virtual content to a user based at least in part on the determined posture information relative to the electromagnetic field transmitter.

[0027] In one or more embodiments, the electromagnetic field transmitter is located in a mobile component of the AR display system. In one or more embodiments, the mobile component is a handheld component. In one or more embodiments, the mobile component is a totem device. In one or more embodiments, the mobile component is a head-mounted component of the AR display system.

[0028] In one or more embodiments, the method further includes housing the display system in a head mounted assembly, wherein the electromagnetic sensor is operably coupled to the head mounted assembly. In one or more embodiments, the world coordinates of the electromagnetic sensor are known based at least in part on a SLAM analysis performed to determine head pose information, wherein the electromagnetic sensor is operably coupled to the head mounted assembly housing the display system.

[0029] In one or more embodiments, further comprising capturing image data via one or more cameras operably coupled to the head mounted assembly, wherein the SLAM analysis is performed based at least on the data captured by the one or more cameras. In one or more embodiments, the electromagnetic sensor comprises one or more inertial measurement units (IMUs).

[0030] In one or more embodiments, the gesture information corresponds to at least a position and an orientation of the electromagnetic field transmitter relative to the world. In one or more embodiments, the gesture information is analyzed to determine world coordinates corresponding to the electromagnetic field transmitter. In one or more embodiments, the method further includes detecting an interaction with one or more virtual contents based at least in part on the gesture information corresponding to the electromagnetic field transmitter.

[0031] In one or more embodiments, the method further includes displaying virtual content to the user based at least in part on the detected interaction. In one or more embodiments, the electromagnetic sensor includes at least three coils to measure magnetic flux in three directions. In one or more embodiments, the at least three coils are housed together in substantially the same position, and the electromagnetic sensor is coupled to a head-mounted assembly of the AR display system. In one or more embodiments, the at least three coils are housed at different positions of the head-mounted assembly of the AR display system.

[0032] In one or more embodiments, the method further comprises decoupling the magnetic field emitted by the electromagnetic field transmitter by the control and quick release module. In one or more embodiments, the method further comprises determining the world coordinates of the electromagnetic field transmitter by an additional positioning resource. In one or more embodiments, the additional positioning resource comprises a GPS receiver. In one or more embodiments, the additional positioning resource comprises a beacon.

[0033] In yet another aspect, an augmented reality display system includes: a handheld component housing an electromagnetic field transmitter; an electromagnetic field transmitter that emits a known magnetic field; a head-mounted component having a display system that displays virtual content to a user, the head-mounted component coupled to one or more electromagnetic sensors that detect the magnetic field emitted by the electromagnetic field transmitter housed in the handheld component, wherein a head posture is known; and a controller communicatively coupled to the handheld component and the head-mounted component, the controller receiving magnetic field data from the handheld component and receiving sensor data from the head-mounted component, wherein the controller determines a hand posture based at least in part on the received magnetic field data and the received sensor data, and wherein the display system modifies the virtual content displayed to the user based at least in part on the determined hand posture.

[0034] In one or more embodiments, the handheld component is mobile. In one or more embodiments, the handheld component is a totem device. In one or more embodiments, the handheld component is a gaming component. In one or more embodiments, the head pose is known based at least in part on SLAM analysis.

[0035] In one or more embodiments, the AR display system further includes one or more cameras operably coupled to the head-mounted assembly, wherein the SLAM analysis is performed based at least on data captured by the one or more cameras. In one or more embodiments, the electromagnetic sensor includes one or more inertial measurement units (IMUs).

[0036] In one or more embodiments, the head gesture corresponds to at least a position and orientation of the electromagnetic sensor relative to the world. In one or more embodiments, the hand gesture is analyzed to determine world coordinates corresponding to the handheld component. In one or more embodiments, the controller detects interaction with one or more virtual contents based at least in part on the determined hand gesture.

[0037] In one or more embodiments, the display system displays virtual content to the user based at least in part on the detected interaction. In one or more embodiments, the electromagnetic sensor includes at least three coils to measure magnetic flux in three directions. In one or more embodiments, the at least three coils are housed together in substantially the same position. In one or more embodiments, the at least three coils are housed at different positions of the head mounted assembly.

[0038] In one or more embodiments, the AR display system further includes a control and quick release module for decoupling the magnetic field emitted by the electromagnetic field transmitter. In one or more embodiments, the AR display system further includes an additional positioning resource for determining the posture of the hand. In one or more embodiments, the additional positioning resource includes a GPS receiver. In one or more embodiments, the additional positioning resource includes a beacon.

[0039] Additional and other objects, features and advantages of the present invention are described in the detailed description, drawings and claims.

[0040] Additional and other objects, features and advantages of the present invention are described in the detailed description, drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings illustrate the design and use of various embodiments of the invention. It should be noted that the drawings are not drawn to scale and that elements of similar structure or function are represented by the same reference numerals throughout the drawings. In order to better understand how the above and other advantages and objects of the various embodiments of the invention are obtained, the invention briefly described above will be described in more detail by reference to specific embodiments shown in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the invention and are not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0042] Figure 1 shows a plan view of an AR scene displayed to a user of an AR system according to one embodiment;

[0043] Figures 2A-2D Various embodiments of wearable AR devices are shown;

[0044] Figure 3 An example embodiment of a user of a wearable AR device interacting with one or more cloud servers of an AR system is shown;

[0045] Figure 4 An example embodiment of an electromagnetic tracking system is shown;

[0046] Figure 5 An example method of determining the position and orientation of a sensor according to an example embodiment is shown;

[0047] Figure 6 An example diagram showing the use of an electromagnetic tracking system to determine head posture;

[0048] Figure 7 An example method of delivering virtual content to a user based on a detected head gesture is shown;

[0049] Figure 8 A schematic diagram showing various components of an AR system having an electromagnetic transmitter and an electromagnetic sensor according to one embodiment;

[0050] Figures 9A-9F Various embodiments of control and quick release modules are shown;

[0051] Fig.10 A simplified embodiment of an AR device is shown;

[0052] Fig.11A and Fig. 11B Various embodiments of arrangements of electromagnetic sensors on a head-mounted AR system are shown;

[0053] Figures 12A-12E Various embodiments of ferrite cubes to be coupled to electromagnetic sensors are shown;

[0054] Figures 13A-13C Various embodiments of circuits for electromagnetic sensors are shown;

[0055] Fig.14 An example method of using an electromagnetic tracking system to detect head gestures and hand gestures is shown;

[0056] Fig.15 Another example method of using an electromagnetic tracking system to detect head gestures and hand gestures is shown. DETAILED DESCRIPTION

[0057] refer to Figures 2A-2D , showing some common component options. Figures 2A-2D The detailed description section that follows the description provides various systems, subsystems, and components for achieving the purpose of providing a high-quality, perceptually comfortable display system for human VR and / or AR.

[0058] like Figure 2A , an AR system user 60 is depicted wearing a head mounted assembly 58 featuring a frame 64 structure connected to a display system 62 located in front of the user's eyes. In the depicted configuration, a speaker 66 is connected to the frame 64 and is located near the user's ear canal (in one embodiment, another speaker (not shown) is located near the user's other ear canal to provide stereo / shapeable sound control). The display 62 can be operably connected 68, such as by a wired lead or a wireless connection, to a local processing and data module 70, which can be mounted in a variety of configurations, such as fixedly attached to the frame 64, fixedly attached to a display system 62, such as a display system 62 located in front of the user's eyes. Figure 2B The helmet or hat 80 shown in the embodiment is embedded in the headset, such as Figure 2CThe backpack configuration shown in the embodiment of FIG. 1 can be removably attached to the torso 82 of the user 60, or in a manner such as Figure 2D The belt-coupled configuration shown in the embodiment is removably attached to the hip 84 of the user 60.

[0059] The local processing and data module 70 may include a low power processor or controller and digital memory such as flash memory, both of which may be used to facilitate processing, caching and storage of data, where the data may be: (a) captured from a sensor operatively coupled to the frame 64, such as an image capture device (e.g., a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device and / or a gyroscope; and / or (b) acquired and / or processed using a remote processing module 72 and / or a remote data repository 74, and may be used for transmission to the display 62 after such processing or retrieval. The local processing and data module 70 may be operatively coupled (76, 78) to the remote processing module 72 and the remote data repository 74, such as via a wired or wireless communication link, so that these remote modules (72, 74) are operatively coupled to each other and may serve as resources for the local processing and data module 70.

[0060] In one embodiment, the remote processing module 72 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository 74 may include a relatively large-scale digital data storage facility that may be available via the Internet or other networking configurations that employ a "cloud" resource configuration. In one embodiment, all data may be stored and all computations performed in the local processing and data module, allowing for fully autonomous use from any remote module.

[0061] Reference now Figure 3 , the schematic diagram shows the collaboration between cloud computing assets 46 and local processing assets, which may be located, for example, in a head-mounted assembly 58 coupled to the user's head 120 and a local processing and data module 70 coupled to the user's belt 308; therefore, the local processing and data module 70 may also be referred to as a "belt pack", such as Figure 3 In one embodiment, cloud assets 46 (such as one or more cloud server systems 110 operatively coupled 115) are directly coupled (40, 42) to one or both of the local computing assets, such as the processor and memory configuration coupled to the user's head 120 and waistband 308 as described above, for example, via a wired or wireless network (wireless is preferred for mobility, wired is preferred for certain high-bandwidth or high-data volume transmissions that may be required). These computing assets local to the user may also be configured 44 via wired and / or wireless connections, such as those described below with reference to Figure 8The wired coupling 68 is operatively coupled to each other. In one embodiment, in order to maintain a low inertia and small size subsystem mounted on the user's head 120, the primary transmission between the user and the cloud 46 can be via a link between the subsystem mounted on the belt 308 and the cloud, while the head-mounted subsystem 120 primarily uses a wireless connection such as an ultra-wideband (UWB) connection as currently employed in, for example, personal computing peripheral connection applications to bind data to the belt-based subsystem 308.

[0062] Using effective local and remote processing collaboration and appropriate display devices for users, such as Figure 2A , or variations thereof, aspects of a world related to the user's current actual or virtual location can be transmitted or "delivered" to the user and updated in an efficient manner. In other words, a map of the world can be continuously updated at a storage location that may be located partially on the user's AR system and partially in a cloud resource. The map (also called a "deliverable world model") can be a large database that includes raster images, 3D points and 2D points, parametric information, and other information about the real world. As more and more AR users continue to capture information about their real environment (e.g., through cameras, sensors, IMUs, etc.), the map becomes more and more accurate and complete.

[0063] With the configuration described above, there is a "model" of the world that can be located on cloud computing resources and distributed from a cloud server, which can be delivered to one or more users in a relatively low bandwidth form factor. This may be preferred for transmitting real-time video data or similar complex information from one AR system to another. A person standing near the statue (i.e., Figure 1 The augmented experience of a user (shown in FIG. 1 ) can be informed by a cloud-based world model, a subset of which can be downloaded to the person's local display device to complete the view. The person sitting at the remote display device (e.g., a personal computer on a desk) can effectively download that same portion of information from the cloud and present it on the personal computer display. In another embodiment, another user can be present in real time at the park and can walk in the park with a friend (e.g., a person at the personal computer) who joins the user through a shared AR and / or VR experience. In order to render the park scene to the friend, the AR system can detect the location of the streets, the location of the trees in the park, the location of the statues, etc. This location can be uploaded to the transferable world model in the cloud, and the friend (at the personal computer) can download that portion of the transferable world from the cloud and then begin a "walk" with the AR user in the park. Of course, in some embodiments, for the AR user in the park, the friend can be rendered as an avatar in the transferable world model so that the AR user can walk in the park with the virtual friend.

[0064] More specifically, in order to capture the details of the world so that it can be delivered to the cloud (and subsequently to other AR users), 3D points associated with various objects can be captured from the environment, and the pose (i.e., vector and / or initial position information relative to the world) of the camera that captured those images or points can be determined. These 3D points can be "tagged" or associated with this pose information. It should be understood that there may be a large number of AR systems that capture the same points in any given environment. For example, points captured by a second camera (of a second AR system) can be used to determine the head pose of the second camera. In other words, the method of the second camera and / or the positioning of the second camera can be determined based on a comparison with the tagged image from the first camera. This information can then be used to extract textures, make maps, and create one or more virtual copies of the real world.

[0065] In one or more embodiments, the AR system may be used to capture 3D points and generate 2D images of the points. As described above, in some embodiments, these points and images may be sent to a cloud storage and processing resource (e.g., Figure 3 In other embodiments, this information may be cached locally with the embedded pose information (i.e., the labeled images) so that the labeled 2D images can be sent to the cloud along with the 3D points. If the user is observing a dynamic scene, the user may also send additional information to the cloud server. In one or more embodiments, an object identifier may be run (on the cloud resources or on the local system) to identify one or more objects in the captured points. More information about object identifiers and transferable world models can be found in U.S. Patent Application No. 14 / 205,126, "SYSTEM AND METHOD FOR AUTMENTED AND VIRTUALREALITY." Further information related to augmented and virtual reality systems such as those developed by Magic Leap, Inc. of Fort Lauderdale, Florida, is disclosed in U.S. Patent Application No. 14 / 641,376, U.S. Patent Application No. 14 / 555,585, U.S. Patent Application No. 14 / 212,961; U.S. Patent Application No. 14 / 690,401, U.S. Patent Application No. 13 / 663,466 and U.S. Patent Application No. 13 / 684,489.

[0066] In order to capture points that can be used to create a "transferable world model", it is helpful to accurately know the user's position, posture, and orientation relative to the world. More specifically, the user's position must be located to a granular level, as it is important to know the user's head posture as well as the posture of the hands (if the user is grasping a handheld component, making gestures, etc.). In one or more embodiments, GPS and other positioning information can be used as input to this processing. When processing images and points derived from a particular AR system, and also in order to display appropriate virtual content to the user, high-precision positioning of the user's head, totem devices, gestures, haptic devices, etc. is desired.

[0067] One way to achieve high-precision positioning can include using an electromagnetic field coupled to an electromagnetic sensor that is strategically placed on the user's AR headset, belt bag, and / or other auxiliary devices (e.g., totem devices, tactile devices, gaming tools, etc.). An electromagnetic tracking system typically includes at least an electromagnetic field transmitter and at least one electromagnetic field sensor. The electromagnetic sensor can measure an electromagnetic field with a known distribution. Based on these measurements, the position and orientation of the field sensor relative to the transmitter is determined.

[0068] Reference now Figure 4 , shows an example system of an electromagnetic tracking system (e.g., those developed by organizations such as the Biosense (RTM) division of Johnson & Johnson, Polhemus (RTM) of Colchester, Vermont, and manufactured by Sixense (RTM) Entertainment of Los Gatos, California, and other tracking companies). In one or more embodiments, the electromagnetic tracking system includes an electromagnetic field transmitter 402 configured to emit a known magnetic field. Figure 4 As shown, electromagnetic field transmitter 402 may be coupled to a power source 410 (eg, an electric current, a battery, etc.) to provide power to electromagnetic field transmitter 402 .

[0069] In one or more embodiments, the electromagnetic field transmitter 402 includes several coils (e.g., at least three coils that are perpendicular to each other to generate fields in the x, y, and z directions) that generate magnetic fields. These magnetic fields are used to establish a coordinate space. This allows the system to map the position of the sensor 404 relative to a known magnetic field, which helps determine the position and / or orientation of the sensor 404. In one or more embodiments, the electromagnetic sensors 404a, 404b, etc. can be attached to one or more real objects. The electromagnetic sensor 404 can include smaller coils in which currents can be induced by the emitted electromagnetic field. Typically, the "sensor" component 404 can include a small coil or loop positioned / oriented to capture the incoming magnetic flux from the magnetic field emitted by the electromagnetic transmitter 402, such as a set of three coils with different orientations (i.e., such as orthogonal orientations relative to each other), which are coupled together in a small structure such as a cube or other container. By comparing the currents induced through these coils, and by knowing the relative position and orientation of the coils relative to each other, the relative position and orientation of the sensor 404 relative to the electromagnetic transmitter 402 can be calculated.

[0070] One or more parameters related to the behavior of the coils in the electromagnetic tracking sensor 404 and the inertial measurement unit (IMU) components operatively coupled to the electromagnetic tracking sensor 404 may be measured to detect the position and / or orientation of the sensor 404 (and the object to which it is attached) relative to the coordinate system to which the electromagnetic field transmitter 402 is coupled. Of course, the coordinate system may be converted to a world coordinate system to determine the position or attitude of the electromagnetic field transmitter in the real world. In one or more embodiments, multiple sensors 404 may be used in association with the electromagnetic transmitter 402 to detect the position and orientation of each sensor 404 within the coordinate space associated with the electromagnetic field transmitter 402.

[0071] It should be understood that in some embodiments, the head pose may already be known based on sensors on the head-mounted assembly of the AR system and based on SLAM analysis performed on sensor data and image data captured by the head-mounted AR system. However, it is important to know the position of the user's hands (e.g., a handheld assembly like a totem device, etc.) relative to the known head pose. In other words, it is important to know the pose of the hands relative to the head pose. Once the relationship between the head (assuming the sensors are placed on the head-mounted assembly) and the hands is known, the position of the hands relative to the world (e.g., world coordinates) can be easily calculated.

[0072] In one or more embodiments, the electromagnetic tracking system can provide the 3D position (i.e., X, Y, and Z directions) of the sensor 404, and can further provide position information of the sensor 404 in two or three azimuths. In one or more embodiments, the measurements of the IMU can be compared with the measurements of the coil to determine the position and orientation of the sensor 404. In one or more embodiments, both electromagnetic (EM) data and IMU data, as well as various other data sources (such as cameras, depth sensors, and other sensors), can be combined to determine the position and orientation of the electromagnetic sensor 404.

[0073] In one or more embodiments, the information can be sent (e.g., wireless communication, Bluetooth, etc.) to the controller 406. In one or more embodiments, the posture information (e.g., position and orientation) corresponding to the sensor 404 can be reported to the controller 406 at a relatively high refresh rate. Typically, the electromagnetic transmitter 402 can be coupled to a relatively stable and large object, such as a table, a console, a wall or a ceiling, etc., and one or more sensors 404 can be coupled to a smaller object, such as a medical device, a handheld game component, a totem device, a frame of a head-mounted AR system, etc.

[0074] Alternatively, refer to Figure 6 As described above, various features of an electromagnetic tracking system can be employed to produce a configuration in which changes or deltas in position and / or orientation between two objects moving in space relative to a more stable global coordinate system can be tracked. In other words, Figure 6 A configuration is shown in which changes in an electromagnetic tracking system can be used to track position changes and orientation changes between a head-mounted component and a handheld component, while the head pose relative to a global coordinate system (e.g., the user's local room environment) is additionally determined, such as by using simultaneous localization and mapping (SLAM) techniques using an outward-facing camera that can be coupled to the head-mounted component of the AR system.

[0075] Return to reference Figure 4 , controller 406 can control electromagnetic field transmitter 402 and can also capture measurement data from various electromagnetic sensors 404. It should be understood that the various components of the system can be coupled to each other by any electromechanical or wireless / Bluetooth means. Controller 406 can also store data about known magnetic fields and coordinate spaces related to magnetic fields. This information can then be used to detect the position and orientation of sensor 404 relative to the coordinate space corresponding to the known electromagnetic field, which can then be used to determine the world coordinates of the user's hand (e.g., the position of the electromagnetic transmitter).

[0076] One advantage of an electromagnetic tracking system is that it can produce highly accurate tracking results with minimal latency and high resolution. In addition, an electromagnetic tracking system does not necessarily rely on an optical tracker, making it easier to track sensors / objects that are not within the user's line of sight.

[0077] It should be understood that the strength of the electromagnetic field (v) decreases as a cubic function of the distance (r) from the coil transmitter (e.g., electromagnetic field transmitter 402). One or more algorithms can be formulated based on the distance of the sensor from the electromagnetic field transmitter. The controller 406 can be configured with such an algorithm to determine the position and orientation of the sensor / object at different distances from the electromagnetic field transmitter. Assuming that the strength of the electromagnetic field drops sharply as one moves further away from the electromagnetic transmitter, improved results in terms of accuracy, efficiency, and low latency can be achieved at closer distances. In a typical electromagnetic tracking system, the electromagnetic field transmitter is powered by an electric current (e.g., a plug-in power supply) and has a sensor located within a 20-foot radius of the electromagnetic field transmitter. In many applications, including AR applications, a shorter radius between the sensor and the field transmitter is more desirable.

[0078] Reference now Figure 5 , briefly illustrating an example flow chart describing the functionality of a typical electromagnetic tracking system. At 502, a known electromagnetic field is transmitted. In one or more embodiments, the electromagnetic field transmitter can generate a magnetic field. In other words, each coil of the transmitter can generate an electric field in one direction (e.g., x, y, or z). A magnetic field with an arbitrary waveform can be generated. In one or more embodiments, each axis can oscillate at a slightly different frequency.

[0079] At 504, a coordinate space corresponding to the electromagnetic field may be determined. For example, Figure 4 The controller 406 can automatically determine the coordinate space around the electromagnetic transmitter based on the parameters of the electromagnetic field. At 506, the behavior of the coil at the sensor (which can be attached to a known object) can be detected / measured. For example, the current induced at the coil can be measured. In other embodiments, the rotation of the coil or other quantifiable behavior can be tracked and measured. At 508, the measurement can be used to determine / calculate the position and orientation of the sensor and / or the known object. For example, the controller can refer to a mapping table that relates the behavior of the coil at the sensor to various positions or orientations. Based on these calculations, the position and orientation of the sensor (or the object attached thereto) in the coordinate space can be determined. In some embodiments, posture / position information can be determined at the sensor. In another embodiment, the sensor transmits the data detected at the sensor to the controller, and the controller can refer to the mapping table to determine the posture information relative to the known magnetic field (for example, relative to the coordinates of the handheld component).

[0080] In the context of an AR system, it may be necessary to modify one or more components of the electromagnetic tracking system to facilitate accurate tracking of the moving components. As described above, tracking the user's head posture and orientation is helpful in many AR applications. Accurately determining the user's head posture and orientation allows the AR system to display the correct virtual content to the user at the appropriate location in the AR display. For example, a virtual scene may include a monster hidden behind a real building. Depending on the posture and orientation of the user's head relative to the building, the view of the virtual monster may need to be modified to provide a realistic AR experience.

[0081] In other embodiments, the position and / or orientation of a totem device, a haptic device, or some other device for interacting with virtual content is important for enabling an AR user to interact with the AR system. For example, in many gaming applications, the AR system must detect the position and orientation of real objects relative to virtual content. Alternatively, when a virtual interface is displayed, the position of a totem device, a user's hand, a haptic device, or any other real object configured to interact with the AR system relative to the displayed virtual interface must be known so that the system understands commands, etc. Traditional positioning methods including optical tracking and other methods are often plagued by high latency and low resolution issues, which makes rendering virtual content challenging in many AR applications.

[0082] In one or more embodiments, the electromagnetic tracking systems discussed above may be adapted for use in AR systems to detect the position and orientation of one or more objects relative to an emitted electromagnetic field. Typical electromagnetic systems tend to have large and bulky electromagnetic transmitters (e.g., Figure 4 402 in ), which may make them less than ideal for use in AR applications. However, smaller electromagnetic transmitters (e.g., in the millimeter range) may be used to transmit known electromagnetic fields in the context of an AR system.

[0083] Reference now Figure 6 , the electromagnetic tracking system can be incorporated into an AR system as shown, where the electromagnetic field transmitter 602 is incorporated as part of a handheld controller 606. In one or more embodiments, the handheld controller can be a totem device used in a gaming application. In other embodiments, the handheld controller can be a tactile device that can be used to interact with the AR system (e.g., via a virtual user interface). In other embodiments, the electromagnetic field transmitter can be simply incorporated as part of a belt pack, such as Figure 2D The handheld controller 606 may include a battery 610 or other power source for supplying power to the electromagnetic field transmitter 602 .

[0084] It should be understood that the electromagnetic field transmitter 602 may also include or be coupled to an IMU component 650 that is configured to assist in determining the position and / or orientation of the electromagnetic field transmitter 602 relative to other components. This is useful in situations where both the electromagnetic field transmitter 602 and the sensor 604 are mobile (discussed in further detail below). In some embodiments, the electromagnetic field transmitter 602 is placed in a handheld controller rather than in a belt pack, such as Figure 6 , ensuring that the electromagnetic field transmitter does not compete for resources at the belt pack, but instead uses its own battery power at the handheld controller 606.

[0085] In one or more embodiments, the electromagnetic sensor 604 may be placed at one or more locations on the user's headset 58 along with other sensing devices such as one or more IMUs or additional magnetic flux capture coils 608. For example, Figure 6 As shown in FIG, sensors 604, 608 may be placed on either side of the headset 58. Since these sensors 604, 608 are designed to be relatively small (and therefore may not be very sensitive in some cases), it is important to include multiple sensors in order to increase the efficiency and accuracy of the measurements.

[0086] In one or more embodiments, one or more sensors 604, 608 may also be placed on the belt pack 620 or any other part of the user's body. The sensors 604, 608 may be wirelessly or via a computing device 607 (e.g., a controller). In communication, the computing device 607 determines the attitude and orientation of the sensors 604, 608 (and the AR headset 58 to which they are attached) relative to the known magnetic field emitted by the electromagnetic field transmitter 602. In one or more embodiments, as Figure 6 As shown, the computing device 607 can be located at the belt pack 620. In other embodiments, the computing device 607 can be located at the headset 58 itself or even at the handheld controller 604. In one or more embodiments, the computing device 607 can receive the measurements of the sensors 604, 608 and determine the position and orientation of the sensors 604, 608 relative to the known electromagnetic field emitted by the electromagnetic field transmitter 602.

[0087] In one or more embodiments, the mapping database 632 may be referenced to determine the location coordinates of the sensors 604, 608. In some embodiments, the mapping database 632 may be located in the belt pack 620. In the illustrated embodiment, the mapping database 632 is located on the cloud resource 630. Figure 6As shown, computing device 607 is in wireless communication with cloud resource 630. The determined pose information is then transmitted to cloud resource 630 in conjunction with the points and images collected by the AR system and added to transferable world model 634.

[0088] As mentioned above, conventional electromagnetic transmitters may be too bulky for use in AR devices. Therefore, the electromagnetic field transmitter can be designed to be compact, using smaller coils, compared to conventional systems. However, assuming that the strength of the electromagnetic field decreases as a cubic function of the distance from the field transmitter, the distance from the field transmitter is smaller than that of conventional systems (such as in Figure 4 A shorter radius (e.g., approximately 3-3.5 feet) between electromagnetic sensor 604 and electromagnetic field transmitter 602 can reduce power consumption while maintaining acceptable field strengths, compared to the systems detailed in .

[0089] In one or more embodiments, this feature can be used to extend the life of the battery 610 that powers the controller 606 and the electromagnetic field transmitter 602. Alternatively, this feature can be used to reduce the size of the coils that generate the magnetic field at the electromagnetic field transmitter 602. However, in order to obtain the same magnetic field strength, the power of the electromagnetic field transmitter 602 may need to be increased. This allows the electromagnetic field transmitter unit 602 to be compactly mounted at the handheld controller 606.

[0090] When using an electromagnetic tracking system for an AR device, several other changes may be made. In one or more embodiments, IMU-based posture tracking may be used. In such embodiments, keeping the IMU as stable as possible improves the efficiency of the posture detection process. The IMUs may be designed so that they remain stable for 50-100 milliseconds, which results in a stable signal with a posture update / reporting rate of 10-20Hz. It should be understood that some embodiments may utilize an external posture evaluation module (because the IMU may drift over time), which may enable posture updates to be reported at a rate of 10-20Hz. By keeping the IMU stable for a reasonable amount of time, the rate of posture updates can be significantly reduced to 10-20Hz (as compared to the higher frequencies in traditional systems).

[0091] Another way to save power for an AR system can be to run the electromagnetic tracking system at a 10% duty cycle (e.g., only pinging the ground every 100 milliseconds). In other words, the electromagnetic tracking system works for 10 milliseconds out of every 100 milliseconds to generate a pose estimate. This translates directly into power savings, which in turn can impact the size, battery life, and cost of the AR device.

[0092] In one or more embodiments, this reduction in duty cycle can be strategically exploited by providing two handheld controllers (not shown) instead of just one. For example, a user may be playing a game that requires two totem devices, etc. Or, in a multi-user game, two users may have their own totem devices / handheld controllers to play the game. When two controllers (e.g., symmetrical controllers for each hand) are used instead of one controller, the controllers can operate at an offset duty cycle. The same concept can also be applied, for example, to controllers used by two different users playing a multi-player game.

[0093] Reference now Figure 7 , illustrating an example flow chart describing an electromagnetic tracking system in the context of an AR device. At 702, a handheld controller 606 emits a magnetic field. At 704, an electromagnetic sensor 604 (placed on the headset 58, belt pack 620, etc.) detects the magnetic field. At 706, the position and orientation of the headset / belt are determined based on the behavior of the coil / IMU 608 at the sensor 604. In some embodiments, the detected behavior of the sensor 604 is transmitted to a computing device 607, which determines the position and orientation of the sensor 604 relative to the electromagnetic field (e.g., relative to the coordinates of the handheld component). Of course, it should be understood that these coordinates can then be converted into world coordinates, because the head pose relative to the world can be known through SLAM processing, as described above.

[0094] At 708, the pose information is transmitted to the computing device 607 (e.g., at the belt pack 620 or the headset 58). At 710, the transferable world model 634 can optionally be referenced to determine virtual content to be displayed to the user based on the determined head pose and hand pose. At 712, based on the correlation, the virtual content can be delivered to the user at the AR headset 58. It should be understood that the above flowchart is for illustration purposes only and should not be considered limiting.

[0095] Advantageously, using something like Figure 6 The electromagnetic tracking system outlined in [1] is capable of performing pose tracking (e.g., head position and orientation, position and orientation of totem devices, and other controllers) at higher refresh rates and lower latency. This allows AR systems to project virtual content with higher accuracy and lower latency than optical tracking techniques used to calculate pose information.

[0096] refer to Figure 8 , shows a system configuration featuring many sensing components similar to the sensors described above. It should be understood that Figures 2A-2D and Figure 6 The reference mark is Figure 8The head mounted wearable assembly 58 is shown operatively coupled 68 to a local processing and data module 70, such as a belt pack (similar to Figure 2D ), where a physical multi-core lead is used, which is also referenced as follows Figures 9A-9F The control and quick release module 86 is characterized. The local processing and data module 70 can be operatively coupled 100 to the handheld assembly 606 (similar to Figure 6 In one or more embodiments, the local processing and data module 70 can be connected via a wireless connection (such as a low power ) is coupled to the handheld component 606. In one or more embodiments, the handheld component 606 may also be operably coupled directly to the head-mounted wearable component 58, such as via a wireless connection (such as a low power ).

[0097] Typically, in the case of transmitting IMU data in order to detect attitude information of various components, a high frequency connection is desired, such as in the range of hundreds or thousands of cycles / second or higher. On the other hand, tens of cycles per second may be sufficient for electromagnetic positioning sensing, such as through the pairing of sensor 604 and transmitter 602. A global coordinate system 10 is also shown, representing fixed objects in the real world around the user, such as wall 8. Cloud resources 46 can also be operably coupled 42, 40, 88, 90 to local processing and data module 70, head-mounted wearable component 58, resources that can be coupled to wall 8 or other items fixed relative to global coordinate system 10. Resources coupled to the wall 8 or having a known position and / or orientation relative to the global coordinate system 10 may include a Wi-Fi transceiver 114, an electromagnetic transmitter 602 and / or receiver 604, a beacon or reflector 112 configured to emit or reflect a given type of radiation (such as an infrared LED beacon), a cellular network transceiver 110, a radar transmitter or detector 108, a lidar transmitter or detector 106, a GPS transceiver 118, a poster or marker with a known detectable pattern 122, and a camera 124.

[0098] The head mounted wearable assembly 58 features similar components as shown, with the addition of a light emitter 130 configured as an auxiliary camera 124 detector, such as an infrared emitter 130 for an infrared camera 124. In one or more embodiments, the head mounted wearable assembly 58 may further include one or more strain gauges 116, which may be fixedly coupled to a frame or mechanical platform of the head mounted wearable assembly 58 and configured to determine deflection of such platform between components such as electromagnetic receiver sensors 604 or display elements 62, wherein understanding whether the bending of the platform is at a thinned portion of the platform (such as Figure 8The portion above the nose of the glasses-like platform shown) has occurred to be valuable.

[0099] The head mounted wearable assembly 58 may also include a processor 128 and one or more IMUs 102. Preferably, each component is operably coupled to the processor 128. A handheld assembly 606 is shown featuring similar components and a local processing and data module 70. Figure 8 As shown, with so many sensing and connection devices, such a system is likely to be heavy, large, relatively expensive, and likely to consume a lot of power. However, for illustrative purposes, such a system can be used to provide a very high level of connectivity, system component integration, and position / orientation tracking. For example, with this configuration, various major mobile components (58, 70, 606) can be located in terms of their position relative to a global coordinate system using Wi-Fi, GPS, or cellular signal triangulation algorithms; beacons, electromagnetic tracking (as described above), radar, and lidar systems can provide further position and / or orientation information and feedback. Markers and cameras can also be used to provide further information about relative and absolute position and orientation. For example, various camera components 124 coupled to a head-mounted wearable component 58, such as shown, can be used to capture data that can be used in a simultaneous positioning and mapping (SLAM) protocol to determine the position of the component 58 relative to other components and how it is oriented relative to other components.

[0100] refer to Figures 9A-9F , showing various aspects of the control and quick release module 86. Fig. 9A The two housing 134 components are coupled together using a magnetic coupling arrangement, which may be enhanced with a mechanical latch. A button 136 may be included for operation of the associated system. Fig. 9B A partial cutaway view is shown showing the button 136 and the top printed circuit board 138 underneath. Fig. 9C , removing the button 136 and the top printed circuit board 138 underneath, the female header pin array 140 is visible. Fig.9D , removing the corresponding portion of the housing 134, the bottom printed circuit board 142 is visible. Fig.9E As shown, the bottom printed circuit board 142 is removed and the male header pin array 144 is visible.

[0101] refer to Fig.9FIn a cross-sectional view, at least one male pin or female pin is configured as a spring support so that they can be compressed along the longitudinal axis of each pin. In one or more embodiments, the pins can be referred to as "spring pins" and can generally include a highly conductive material such as copper or gold. When assembled, the configuration shown can mate 46 male pins with female pins, and the entire assembly can be quickly decoupled in half by manually pulling it apart and overcoming the magnetic interface 146 load generated using north and south magnets oriented around the perimeter of the pin array 140, 144.

[0102] In one embodiment, a load of approximately 2 kg compressing the 46 spring pins is counterbalanced by a closure maintenance force of approximately 4 kg. In one embodiment, the pins in the arrays 140, 144 may be separated by approximately 1.3 mm, and the pins may be operably coupled to various types of wires (such as twisted pairs or other combinations supporting USB 3.0, HDMI 2.0, I2S signals, GPIO, and MIPI configurations) as well as high current analog lines and ground configured up to approximately 4 amps / 5 volts.

[0103] refer to Fig.10 , enabling a minimized component / feature set to minimize the weight and volume of various components and result in a relatively slim head-mounted assembly (e.g., in Fig.10 Those components of the head-mounted assembly 58 characterized in (i) are beneficial. Therefore, it is possible to utilize Figure 8 Various arrangements and combinations of the various components shown in .

[0104] refer to Fig.11A , shows an electromagnetic sensing coil assembly (604, e.g., 3 separate coils coupled to a housing) coupled to a head mounted assembly 58. This configuration adds additional geometry (i.e., protrusions) to the overall assembly, which may not be desirable. Fig. 11B , not like Fig.11A The individual coils may be integrated into various configurations of the head-mounted assembly 58, such as Fig. 11B For example, the x-axis coil 148 can be placed in one portion of the head-mounted assembly 58 (e.g., the center of the frame). Similarly, the y-axis coil 150 can be placed in another portion of the head-mounted assembly 58 (e.g., either bottom side of the frame). Similarly, the z-axis coil 152 can be placed in yet another portion of the head-mounted assembly 58 (e.g., either top side of the frame).

[0105] Figures 12A-12E Various configurations are shown for featuring a ferrite core coupled to an electromagnetic sensor to increase magnetic field sensitivity. Fig. 12A, the ferrite core may be a solid cube 1202. Although a solid cube may be most effective in increasing magnetic field sensitivity, when compared with Figures 12B-12E It is also likely to be the heaviest when compared to the rest of the configurations shown. Fig. 12B , a plurality of ferrite disks 1204 can be coupled to the electromagnetic sensor. Similarly, referring to Fig. 12C , a solid cube 1206 with a uniaxial air core can be coupled to an electromagnetic sensor. Fig. 12C As shown, an open space (i.e., an air core) can be formed in a solid cube along one axis. This can reduce the weight of the cube while still providing the necessary magnetic field sensitivity. In another embodiment, reference Fig.12D , a solid cube 1208 with a triaxial air core can be coupled to an electromagnetic sensor. In this configuration, the solid cube is hollowed out along all three axes, thereby significantly reducing the weight of the cube. Fig.12E , a ferrite rod 1210 having a plastic housing can also be coupled to the electromagnetic sensor. As described above, it should be understood that Figures 12B-12E The embodiment is heavier than Fig. 12A The solid core configuration is lighter and can be used to save mass as described above.

[0106] refer to Figures 13A-13C , time division multiplexing (TDM) can also be used to save quality. For example, refer to Fig.13A , shows a conventional local data processing configuration for a 3-coil electromagnetic receiver sensor, wherein analog current enters from each of the X, Y, and Z coils (1302, 1304, and 1306), then enters a separate preamplifier 1308, enters a separate bandpass filter 1310, a separate preamplifier 1312, passes through an analog-to-digital converter 1314, and finally enters a digital signal processor 1316.

[0107] refer to Fig. 13B The transmitter configuration and Fig. 13C For receiver configurations, time division multiplexing can be used to share hardware so that each coil sensor chain does not require its own amplifier, etc. This can be achieved by TDM switch 1320, such as Fig. 13BAs shown, the TDM switch 1320 facilitates the use of the same set of hardware components (amplifiers, etc.) to process signals to multiple transmitters and from multiple receivers. In addition to removing the sensor housing and multiplexing to save hardware overhead, the signal-to-noise ratio can be increased by having more than one set of electromagnetic sensors, each set of electromagnetic sensors being relatively small relative to a single large coil group. In addition, the low-frequency limit that is usually required to have multiple sensing coils that are very close can be improved to facilitate improved bandwidth requirements. It should be noted that there may be a trade-off in multiplexing, because multiplexing usually extends the reception of radio frequency signals in time, which results in a generally wider signal. Therefore, a multiplexed system may require a larger coil diameter. For example, if a multiplexed system may require a cubic coil sensor box with a side length of 9mm, a non-multiplexed system may only require a cubic coil box with a side length of 7mm for similar performance. Therefore, it should be noted that there is a trade-off in minimizing geometry and quality.

[0108] In another embodiment, where a particular system component, such as head mounted assembly 58, features two or more electromagnetic coil sensor groups, the system may be configured to selectively utilize sensor and electromagnetic transmitter pairings that are in closest proximity to each other to optimize system performance.

[0109] refer to Fig.14 In one embodiment, after a user turns on his or her wearable computing system at 160, a head mounted assembly may capture a combination of IMU data and camera data (camera data is used, for example, for SLAM analysis, such as at a belt pack processor where more RAW processing power is present) to determine and update head pose (i.e., position and orientation) relative to a real world global coordinate system at 162. At 164, the user may also activate a hand-held assembly to, for example, play an augmented reality game, and at 166, the hand-held assembly may include an electromagnetic transmitter operably coupled to one or both of the belt pack and the head mounted assembly. At 168, one or more electromagnetic field coil receiver groups (e.g., a group of 3 separate coils in different orientations) coupled to the head mounted assembly may be used to capture magnetic flux from the electromagnetic transmitter. This captured magnetic flux may be used to determine a position or orientation difference (or delta) between the head mounted assembly and the hand-held assembly.

[0110] In one or more embodiments, the combination of a head-mounted component that assists in determining a pose relative to a global coordinate system and a hand-held component that assists in determining a relative position and orientation of a hand-held component relative to the head-mounted component allows the system to generally determine the position of each component relative to the global coordinate system, so that at 170, the user's head pose and hand-held pose can preferably be tracked with relatively low latency for presenting augmented reality image features and interacting using movement and rotation of the hand-held component.

[0111] refer to Fig.15, showing the Fig.14 The embodiment is somewhat similar to the embodiment of, except that at 172, 176, and 178, the system has more sensing devices and configurations that can be used to assist in determining the posture of the head-mounted component and the hand-held component so that at 180, the user's head posture and hand-held posture can preferably be tracked with relatively low latency for presenting augmented reality image features and interacting using movement and rotation of the hand-held component.

[0112] Specifically, after a user turns on his or her wearable computing system at 160, the head-mounted assembly captures a combination of IMU data and camera data for SLAM analysis to determine and update the head pose relative to the real-world global coordinate system at 172. The system can be further configured to detect the presence of other positioning resources in the environment, such as Wi-Fi, cellular, beacons, radar, lidar, GPS, landmarks, and / or other cameras that can be connected to various aspects of the global coordinate system or one or more movable components.

[0113] At 174, the user may also activate the handheld assembly to, for example, play an augmented reality game, and at 176, the handheld assembly may include an electromagnetic transmitter operably coupled to one or both of the belt pack and the head-mounted assembly. Other positioning resources may also be similarly utilized. At 178, one or more electromagnetic field coil receiver groups (e.g., a group of 3 independent coils in different orientations) coupled to the head-mounted assembly may be used to capture magnetic flux from the electromagnetic transmitter. The captured magnetic flux may be used to determine a position or orientation difference (or delta) between the head-mounted assembly and the handheld assembly.

[0114] Thus, at 180 , the user's head pose and hand gesture may be tracked with relatively low latency for use in presenting AR content and / or for use in interacting with the AR system using movement or rotation of the handheld component.

[0115] Various exemplary embodiments of the present invention are described herein. Reference is made to these examples in a non-limiting sense. These examples are provided to illustrate the wider application aspects of the present invention. Various changes and replacement equivalents may be made to the described invention without departing from the spirit and scope of the present invention. In addition, many modifications may be made to adapt specific situations, materials, compositions of matter, processes, process actions or steps to the goals, spirit or scope of the present invention. In addition, as will be appreciated by those skilled in the art, each of the various variations described and illustrated herein has separate components and features that may be easily separated or combined from any of the features in other several embodiments without departing from the scope or spirit of the present invention. All of these modifications are intended to be within the scope of the claims associated with the present disclosure.

[0116] The present invention includes methods that can be performed using the subject apparatus. The method can include an act of providing such suitable apparatus. Such providing can be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, access, locate, set up, activate, turn on, or otherwise provide the necessary apparatus in the method. The methods described herein can be performed in any order of the events described that is logically possible and in the order of events described.

[0117] Exemplary aspects of the present invention and details about material selection and manufacturing have been described above. For other details of the present invention, these can be understood in conjunction with the patents and publications referenced above and what is generally known or understood by those skilled in the art. The same can be established in terms of additional actions that are generally or logically utilized with respect to aspects of the basic method according to the present invention.

[0118] In addition, although the present invention has been described with reference to several examples that optionally incorporate various features, the present invention is not limited to the invention described or indicated for each variant of the present invention. Various changes may be made to the described invention without departing from the actual spirit and scope of the present invention, and equivalents (whether or not included herein for the sake of brevity) may be substituted. In addition, where a range of values ​​is provided, it is understood that each intermediate value between the upper and lower limits of the range and any other described or intermediate values ​​within the described range are included in the present invention.

[0119] In addition, it is contemplated that any optional feature of the described variations of the invention may be stated and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that the same item is present in the plurality. More specifically, as used herein and in the associated claims, the singular forms "one," "an," "said," and "the" include plural objects unless otherwise expressly stated. In other words, in the above description and in the claims associated with the present disclosure, the use of the article "at least one" target item is permitted. It should be further noted that such claims may be drafted to exclude any optional elements. Therefore, in conjunction with claim elements or using a "negative" limitation, this statement is intended to serve as an antecedent basis for the use of exclusive terms such as "exclusively," "only," and the like.

[0120] Without using such exclusive terms, the term "comprising" in the claims associated with the present disclosure shall allow for the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claim or the addition of features may be considered to alter the nature of the elements described in the claim. Except as specifically defined herein, all technical and scientific terms used herein shall be given as broad a commonly understood meaning as possible while maintaining claim validity.

[0121] The breadth of the present invention is not limited to the examples provided and / or subject specification, but is limited only by the scope of the claim language associated with this disclosure.

Claims

1. An augmented reality display system, comprising: an electromagnetic field transmitter configured to emit a known magnetic field; an electromagnetic sensor configured to measure a parameter related to a magnetic flux at the electromagnetic sensor due to the emitted known magnetic field, wherein the electromagnetic sensor comprises at least three coils housed at different locations of a head mounted assembly of the augmented reality display system and configured to measure magnetic flux in three directions; a processor configured to determine attitude information relative to the electromagnetic field transmitter based at least in part on the measured parameter related to the magnetic flux at the electromagnetic sensor; and A display is configured to display virtual content to a user based at least in part on the determined posture information relative to the electromagnetic field transmitter.

2. The system of claim 1, further comprising a moving assembly, wherein: The electromagnetic field transmitter is located in the mobile assembly.

3. The system according to claim 2, wherein: The mobile assembly is a handheld assembly.

4. The system according to claim 2, wherein: The moving component is a totem device.

5. The system according to claim 2, wherein: The mobile component is a head-mounted component of the augmented reality display system.

6. The system of claim 1, further comprising a head-mounted assembly, in, The display is housed in the head mounted assembly, and Wherein, the electromagnetic sensor is operatively coupled to the head-mounted assembly.

7. The system according to claim 1, wherein: The electromagnetic sensors include one or more inertial measurement units (IMUs).

8. The system according to claim 1, wherein: The posture information corresponds to at least a position and an orientation of the electromagnetic field transmitter relative to the world.

9. The system according to claim 1, wherein: The posture information is analyzed to determine world coordinates corresponding to the electromagnetic field transmitter.

10. The system according to claim 1, wherein: The processor is further configured to detect interaction with one or more virtual content based at least in part on the gesture information corresponding to the electromagnetic field transmitter.

11. The system according to claim 10, wherein: The processor is further configured to display virtual content to the user based at least in part on the detected interaction.

12. The system of claim 1, further comprising: A control and quick release module is configured to decouple the magnetic field emitted by the electromagnetic field emitter.

13. The system of claim 1, further comprising: One or more additional positioning resources configured to determine world coordinates of the electromagnetic field transmitter.

14. The system according to claim 13, wherein: The additional positioning resource includes a GPS receiver.

15. The system of claim 13, wherein: The additional positioning resources include beacons.

16. The system of claim 1, wherein: The electromagnetic sensor includes a non-solid ferrite cube.

17. The system of claim 1, wherein: The electromagnetic sensor comprises a stack of ferrite disks.

18. The system of claim 1, wherein: The electromagnetic sensor includes a plurality of ferrite rods, each ferrite rod having a polymer coating.

19. The system of claim 1, wherein: The electromagnetic sensor includes a time division multiplexing switch.

20. A method for displaying augmented reality, comprising: emitting a known magnetic field by means of an electromagnetic field transmitter; measuring, by means of an electromagnetic sensor, a parameter related to the magnetic flux at the electromagnetic sensor due to the emitted known magnetic field; determining attitude information relative to the electromagnetic field transmitter based at least in part on the measured parameter related to the magnetic flux at the electromagnetic sensor; as well as displaying virtual content to a user based at least in part on the determined posture information relative to the electromagnetic field transmitter, The electromagnetic sensor includes at least three coils to measure magnetic flux in three directions, and the at least three coils are accommodated at different positions of a head-mounted component of the augmented reality display system.

Images