Systems and methods for augmented reality

Abstract

A method for positioning a user operating multiple sensing components in an augmented or mixed reality environment is disclosed, the method comprising sending gesture data from a fixed control and processing module and receiving the gesture data at a first sensing component, and then transforming the gesture data into a first component relative gesture in a coordinate system based on the control and processing module. A display unit in communication with the first sensing component is updated with the transformed first component relative gesture to render virtual content with improved environmental awareness.

Description

[0001] Related Applications

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 62 / 627,155, filed on February 6, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to systems and methods for locating the position or orientation of one or more objects in the context of an augmented reality system. Background Art

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears to be real or can be perceived as real. Virtual reality (or "VR") scenes generally involve the presentation of digital or virtual image information that is opaque to other actual real-world visual input; augmented reality (or "AR") scenes generally involve the presentation of digital or virtual image information as an enhancement to the visualization of the actual world around the user.

[0005] For example, refer to Figure 1 , depicting an augmented reality scene (4) in which the user of the AR technology sees a realistic park-like scene (6) with people, trees, buildings, and a concrete platform (1120) in the background. In addition to these items, the user of the AR technology also perceives that he "sees" a robotic statue (1110) standing on the real-world platform (1120) and a cartoon avatar character (2) flying by, which appears to be a personification of a bee, even though these elements (2, 1110) do not exist in the real world. It turns out that the human visual perception system is very complex, and it is challenging for VR or AR technology to produce a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements.

[0006] For example, a head-mounted AR display (or helmet display, or smart glasses) is typically at least loosely coupled to the user's head so that it moves when the user's head moves. If the display system detects movement of the user's head, the displayed data can be updated to account for changes in head posture. For example, if a user wearing a head-mounted display views a virtual representation of a three-dimensional (3D) object on the display and walks around an area where the 3D object appears, the 3D object can be re-rendered for each viewpoint, giving the user the feeling that he or she is walking around the object occupying real space. If the head-mounted display is used to present multiple objects in a virtual space (e.g., a rich virtual world), measurements of head posture (i.e., the position and orientation of the user's head) can be used to re-render the scene to match the user's dynamically changing head position and orientation and provide a higher sense of immersion in the virtual space.

[0007] In an AR system, detection or calculation of head pose can assist the display system in rendering virtual objects so that they appear to occupy space in the real world in a way that makes sense to the user. In addition, detecting the position and / or orientation of real objects (such as handheld devices (also referred to as "totems"), haptic devices, or other real physical objects) relative to the user's head or the AR system can also help the display system present display information to the user so that the user can effectively interact with certain aspects of the AR system. When the user's head moves in the real world, the virtual objects can be re-rendered as a function of the head pose so that the virtual objects appear to remain stable relative to the real world. At least for AR applications, setting virtual objects in spatial relationship to physical objects (e.g., being presented as being spatially close to physical objects in two or three dimensions) can be a non-trivial problem. For example, head movement can make the placement of virtual objects in the view of the environment very complicated. This is true whether the view is captured as an image of the surrounding environment and then projected or displayed to the end user, or whether the end user directly perceives a view of the surrounding environment. For example, head movement may cause the end user's field of view to change, which may require updating the positions of various virtual objects displayed in the end user's field of view.

[0008] Furthermore, head movements may occur over a wide range of ranges and speeds. Head movement speed may vary not only between different head movements, but also within or within the range of a single head movement. For example, head movement speed may initially increase from a starting point (e.g., a straight line or non-straight line), and when reaching an end point, the head movement speed may decrease, achieving a maximum speed somewhere between the starting point and the end point of the head movement. Rapid head movements may even exceed the ability of a particular display or projection technology to present an even and / or smoothly moving image to an end user.

[0009] Accuracy and latency of head tracking (i.e., the time that elapses between the time a user moves his or her head and the time the image is updated and displayed to the user) has long been a challenge for VR and AR systems. Particularly for display systems that fill a large portion of the user's field of view with virtual elements, it is critical that the accuracy of head tracking is high and that the overall system latency from first detecting head movement to updating the light delivered by the display to the user's visual system is very low. If latency is high, the system can create a mismatch between the user's vestibular and visual sensory systems and produce a scene that the user perceives that causes nausea or simulation sickness. If system latency is high, the apparent position of virtual objects will appear unstable during rapid head movements.

[0010] In addition to head-mounted display systems, other display systems can benefit from accurate and low-latency head pose detection. These include head-tracked display systems, where the display is not worn on the user's body, but is mounted, for example, on a wall or other surface. The head-tracked display acts like a window on the scene, and as the user moves their head relative to the "window", the scene is re-rendered to match the user's changing viewpoint. Other systems include head-mounted projection systems, where the head-mounted display projects light onto the real world.

[0011] In addition, in order to provide a realistic augmented reality experience, the AR system can be designed to interact with the user. For example, multiple users can play a ball game using a virtual ball and / or other virtual objects. One user can "catch" the virtual ball and then throw it back to another user. In another embodiment, a totem (e.g., a real bat communicatively coupled to the AR system) may be provided to the first user to hit the virtual ball. In other embodiments, a virtual user interface may be presented to the AR user to allow the user to select one of multiple options. The user may interact with the system using a totem, a haptic device, a wearable component, or simply by touching the virtual screen.

[0012] Detecting the user's head pose and orientation, as well as detecting the physical position of real objects in space, enables AR systems to display virtual content in an effective and enjoyable manner. However, while these capabilities are key to AR systems, they are difficult to implement. In other words, the AR system must identify the physical position of real objects (e.g., the user's head, totems, haptic devices, wearable components, the user's hands, etc.) and associate the physical coordinates of the real objects with the virtual coordinates corresponding to one or more virtual objects displayed to the user. This requires highly accurate sensors and sensor recognition systems that track the position and orientation of one or more objects at a fast rate. Current methods cannot perform positioning at a satisfactory speed or accuracy standard.

[0013] Therefore, in the context of AR and VR devices, better positioning systems are needed. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings illustrate the design and utility of various embodiments of the invention. It should be noted that the drawings are not drawn to scale and that elements of similar structure or function are represented by similar reference numerals throughout the drawings. In order to better understand how the above and other advantages and objects of the various embodiments of the invention are obtained, the invention briefly described above will be described in more detail with reference to specific embodiments of the invention, which are shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are not to be considered limiting of the scope of the invention, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings,

[0015] Figure 1 A plan view of an AR scene displayed to a user of an AR system is shown according to one embodiment.

[0016] Figures 2A-2D Various embodiments of wearable AR devices are shown.

[0017] Figure 3 An example embodiment of a wearable AR device interacting with one or more cloud servers of a system according to an embodiment of the present invention is shown.

[0018] Figure 4 An example embodiment of a wearable AR device interacting with one or more handheld controller devices according to an embodiment of the present invention is shown.

[0019] Figure 5 An example of a user operating a positioning protocol in a physical environment according to an embodiment of the present invention is shown.

[0020] Figures 6A-6C An example of a user operating a positioning protocol in a physical environment using one or more supporting modules according to an embodiment of the present invention is shown.

[0021] Figures 7A-7C An example of a support module according to an embodiment of the present invention is shown.

[0022] Figures 8A-8B An example of a configuration utilizing an embodiment of the present invention is shown.

[0023] Figures 9A-9I An example of a coupling device for a wearable AR system and a supporting module according to an embodiment of the present invention is shown.

[0024] Figure 10-11 A configuration of an ecosystem for utilizing positioning tools according to an embodiment of the present invention is shown.

[0025] Figures 12A-12C An example of an environment-specific support module according to an embodiment of the present invention is shown.

[0026] Fig.13 An example configuration for utilizing an ecosystem of positioning tools including environment-specific support modules according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] refer to Figures 2A-2D , which illustrates some common component options. Figures 2A-2D In the detailed description section following the discussion of the present invention, various systems, subsystems, and components are presented to address the goal of providing a high-quality, comfortable-to-perceive display system for human VR and / or AR.

[0028] like Figure 2A As shown, an AR system user (60) is depicted as wearing a head-mounted assembly (58) characterized by having a frame (64) structure coupled to a display system (62) located in front of the user's eyes. A speaker (66) is coupled to the frame (64) in the described configuration and is positioned near the user's ear canal (in one embodiment, another speaker not shown is positioned near the user's other ear canal to provide stereo / shapeable sound control). The display (62) is operably coupled (68) to a local processing and data module (70), such as by a wired lead or a wireless connection, and the local processing and data module (70) can be mounted in a variety of configurations, such as fixedly attached to the frame (64), fixedly attached to a helmet or cap (80), such as Figure 2B As shown in the embodiment, embedded in the earphone, Figure 2C The backpack-like configuration shown in the embodiment of the present invention can be removably attached to the torso (82) of the user (60), or in a Figure 2D The embodiment shown in FIG. 1 is removably attached to the hip (84) of the user (60) in a belt-coupled configuration.

[0029] The local processing and data module (70) may include an energy-efficient processor or controller, and digital memory, such as flash memory, both of which may be used to assist in processing, caching and storing: data a) captured from sensors (e.g., image capture devices (e.g., cameras), infrared IR transmitters and receivers, microphones, inertial measurement units, accelerometers, compasses, GPA units, radios and / or gyroscopes) operably coupled to the frame (64); and / or data b) acquired and / or processed using the remote processing module (72) and / or remote data repository (74), which may be used for transmission to the display (62) after such processing or retrieval. The local processing and data module (70) may be operably coupled (76, 78) (e.g., via a wired or wireless communication link) to the remote processing module (72) and the remote data repository (74), such that these remote modules (72, 74) are operably coupled to each other and available as resources to the local processing and data module (70).

[0030] In one embodiment, the remote processing module (72) may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository (74) may include a relatively large-scale digital data storage facility that may be available via the Internet or other network configuration in the form of a "cloud" resource configuration. In one embodiment, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from any remote module.

[0031] Reference now Figure 3 , schematic diagram illustrating coordination between cloud computing assets (46) and local processing assets, which may reside, for example, in a head-mounted assembly (58) coupled to a user's head (120) and a local processing and data module (70) coupled to a user's belt (308; thus assembly 70 may also be referred to as a "belt pack" 70), such as Figure 3 In one embodiment, cloud (46) assets such as one or more server systems (110) are operably coupled (115) directly to one or both of the local computing assets such as processors and memory configurations coupled to the user's head (120) and belt (308) as described above, such as via a wired or wireless network (wireless is preferred for mobility, wired is preferred for certain high bandwidth or high data capacity transmissions that may be required) (40, 42). These computing assets local to the user may also be operably coupled (68) to each other via a wired and / or wireless connection configuration (44), such as described above with reference to Figure 2AIn one embodiment, in order to maintain a low inertia and small size subsystem mounted to the user's head (120), the primary transmission between the user and the cloud (46) can be via a link between a subsystem mounted on a waistband (308) and the cloud, where the head-mounted (120) subsystem is primarily data-tethered to the waistband-based (308) subsystem using a wireless connection (such as an ultra-wideband ("UWB") connection) as currently employed, for example, in personal computing peripheral connection applications.

[0032] Through efficient local and remote processing coordination and for users such as Figure 2A With the user interface or user display system (62) shown, or a suitable display device of a variation thereof, aspects of a world related to the user's current real or virtual location can be transmitted or "delivered" to the user and updated in an efficient manner. In other words, a map of the world can be continuously updated at a storage location that can reside partially on the user's AR system and partially in a cloud resource.

[0033] A map (also called a "transferable world model") can be a large database of raster images, 3-D and 2-D points, parametric information, and other information about the real world. As more AR users continuously capture information about their real environment (e.g., through cameras, sensors, IMUs, etc.), the map becomes more and more accurate and complete.

[0034] With the configuration described above, where there is a model of the world that can be located on a cloud computing resource and distributed from there, such a world can be "deliverable" to one or more users in a relatively low bandwidth form, preferably attempting to deliver real-time video data, etc. Figure 1 The augmented experience of people nearby (as shown) can be informed by a cloud-based world model, a subset of which can be passed down to the cloud-based world model and their local display device to complete the view. A person sitting at a remote display device (which can be as simple as a personal computer sitting on a desk) can effectively download the same portion of information from the cloud and present it on their display. In fact, a person physically present in a park near the statue might take a distantly located friend for a walk in that park, where the friend joins through virtual and augmented reality. The system will need to know where the street is, where the tree is, where the statue is - but with that information on the cloud, the joined friend can download it from the cloud aspect of the scene and then begin walking together as an augmented reality local to the person actually in the park.

[0035] Features or otherwise 3-D points can be detected and captured or otherwise extracted from the environment, and the pose of the camera that captured those images or points (i.e., vector and / or origin position information relative to the world) can be determined so that the points or images can be "tagged" or associated with the pose information. The points captured by a second camera (either a physically second camera, or the same camera but captured at a second moment) can then be used to determine the pose of the second camera. In other words, the second camera can be oriented and / or positioned based on a comparison with the tagged image from the first camera. This knowledge can then be used to extract textures, make maps, and create a virtual copy of the real world (e.g., around which two cameras are registered, or two moments in time that can be used to compare data from a common registration).

[0036] Thus, at a basic level, in one embodiment, a person-worn system may be utilized to both capture 3-D points and generate 2-D images of these points, and these points and images may be transmitted to cloud storage and processing resources. They may also be cached locally with embedded pose information (i.e., cache-tagged images); thus, the cloud may have ready (i.e., in available cache) tagged 2-D images (i.e., tagged with 3-D poses) as well as 3-D points. If the user is viewing something dynamic, he may also transmit additional information to the cloud related to motion (e.g., if viewing another person's face, the user may take a texture map of the face and push it up at an optimized frequency, even if the surrounding world is essentially static).

[0037] In order to capture points that can be used to create a "transferable world model", it is helpful to accurately know the user's position, posture, and orientation relative to the world. More specifically, the user's position must be located to a granular level, as it may be important to know the user's head posture as well as hand posture (if the user is grasping a handheld component, gesturing, etc.). In one or more embodiments, GPS and other positioning information can be used as input to this processing. Highly accurate positioning of the user's head, totems, gestures, haptic devices, etc. is critical in displaying appropriate virtual content to the user.

[0038] refer to Figure 4, showing an AR system featuring an operably coupled handheld controller (606). In one or more embodiments, the handheld controller may be referred to as a "totem" for use in various user scenarios such as gaming. In other embodiments, the handheld controller may be a haptic device. The handheld controller (606) may include a battery or other power source to power various interconnected components, which may include cameras, IMUs, and other devices, as described below. As described herein, the local processing and data module (70) may be operably coupled to cloud resources (630) and operably coupled to a head-mounted wearable component (58).

[0039] In various embodiments, it may be desirable to aggregate the computational, perceptual, and sensory capabilities of various components to, for example, have an enhanced understanding of various issues related to a user's operations in a local space, such as where components are positioned and / or oriented in space relative to each other, the type of local environment the user is dealing with (e.g., the geometry of the room, the location and / or orientation of objects within the room, etc.), and / or which objects and features can be mapped (e.g., using SLAM analysis).

[0040] refer to Figure 5 , showing a top view of a room (200) with a user (60) standing looking generally toward a desk (206), a coffee table (204), and a door (202) that is configured to swing open into the room when utilized. As described above, in various embodiments, a wearable computing system includes operably coupled components, such as a head mounted component (58), a local processing and data module (70), and a hand held component (606). Also as described above, each of such components may include various sensors and processors and their own associated functionality - e.g., camera, IMU, and processor components, which may be configured to, among other things, provide data to constitute posture determination algorithms, which are used by one or more processors to determine the position and / or orientation of such components relative to each other and / or relative to various aspects of their surrounding environment (as described above, so-called "pose" determination).

[0041] Reference again Figure 5 In one embodiment, the relative proximity of the handheld assembly (606) and local processing and data module (70) to the head mounted assembly (58) may be advantageous with respect to determining the position and / or orientation of such assemblies relative to each other, but may not be optimized to collect information about the person and environment surrounding such assemblies. For example, in some positions and orientations, the user's own body may obscure or otherwise interfere with the handheld assembly (606) and prevent effective communication with the head mounted assembly (58). Other non-user environmental factors may also affect performance, such as, Figure 5The door (202), due to its movable nature as a functional door, can change position and / or orientation relative to the assembly (58, 70 and / or 606), and in the described configuration, the door (202) is almost completely across the room from the operator (60). The distance and angular movement of such a structure may not be easily detected by the handheld assembly (606) or the head-mounted assembly (58).

[0042] refer to Fig. 6A In various embodiments, it may be desirable to place one or more instrument components at a location and orientation deemed of interest - e.g., a location near a door (202) with the instrument oriented toward such door, the one or more instrument components being, for example, an additional handheld component module (606) that may be wirelessly interconnected to the processing and storage resources of other components (58, 70, and / or 606) coupled to the operator (60), e.g., via Bluetooth or other wireless connection as described above. In other words, the additional handheld component module (606) may be placed in a location where it is not actually held in the hand of the user, but rather is positioned and oriented relative to a room or other object (i.e., by being placed on a table, floor, or other object, supported by a shelf coupled to another object, movably and / or fixedly attached to another object, etc.); in such a configuration, the additional handheld component module (606) may be referred to as a "support" or "capture and processing" module (606). In one embodiment, as Fig. 6A As shown in , the capture and processing module (606) can be placed in a temporary position / orientation configuration on a floor or some other structure, or can be fixedly attached to a structure, such as Figure 6B As shown in an embodiment, an additional capture and processing module (606) is coupled to the wall, again located at a selected position and orientation to assist in collecting data related to a nearby door (202), and again, the additional capture and processing module (606) is operably coupled to other components (58, 70, 606) of the subject system, such as via a Bluetooth or other wireless connection configuration, for example as described above.

[0043] In such an embodiment, each component physically coupled (58, 70, 606) to the operator (60) may be configured to benefit from the additional data capture and processing capabilities residing within the additional capture and processing module (606), which is shown here as an additional version of the component being used as a handheld component. Figure 6CMultiple additional capture and processing modules (606) are shown that can be positioned and oriented around the operator's environment to assist in processing and analyzing the environment around them. It should be understood that multiple capture and processing modules can be optimized for different measurements. For example, a capture and processing module (606) near a door can be configured to update positioning data for the system used by the user (60) only when the door (202) is opened and closed. While a raycast or cone projection from a sensor on a remote head-mounted component (58) or handheld component (606) may not be able to tell the difference between a half-open door and a closed door, a sensor close to the door with a time-of-flight sensor oriented in the angular direction of the door's opening and closing can more quickly and accurately determine the updated detectable feature location of the door (202) (and by extension the room (200)).

[0044] Similarly, when a capture and processing module (606) at a corner of a room (200) detects that a user (60) is approaching and has a head posture that is at least partially directed toward the capture and processing module (606), it can be activated and provide positioning data. The benefit of such an embodiment is that when the user approaches the physical limit of the space, such as a wall or a corner, the sensors on the head-mounted component (50) or the handheld component (606) detect and extract fewer features for positioning, and the capture and processing module (606) can provide a "lighthouse" effect for positioning.

[0045] Fig. 7A and 7B An embodiment of a portable additional capture and processing module (606) is shown which may include a handle (210), a table stand (212), and components such as those described above including, but not limited to, a camera (124), an IR transmitter and / or receiver, an IMU (102), a WiFi communication device (114), a Bluetooth communication device (115), and an embedded and / or mobile processor (128) appropriately coupled to other sensors. Figure 7C A close up view of one embodiment of a camera module (124) is shown that includes a plurality of monochrome camera devices (125) (e.g., four as shown), a color camera (127), and a depth camera (129) configured to utilize time-of-flight data to assist in placing image elements in space relative to a capture interface of such depth camera (129).

[0046] refer to Fig.9A , such as Figure 5 and 7A-7C can be used to determine the relative poses of various components. A user can wear a head-mounted wearable component (58) that is operably coupled to a handheld controller (606), which is also coupled to an operator (214), for example via the operator's hand. The head-mounted wearable component (58) can be configured to determine certain aspects of the pose (e.g., position and / or orientation) of the head-mounted wearable component relative to an environment (e.g., a room and its elements) surrounding the head-mounted wearable component (58) using sensors coupled thereto, and / or certain features of such an environment, such as the shape of the room or objects therein (216).

[0047] The handheld controller (606) may be configured to utilize sensors coupled thereto to determine certain aspects of the posture (e.g., position and / or orientation) of the handheld controller (606) relative to an environment surrounding the handheld controller (606), and / or certain features about such environment (e.g., the shape of a room) (218).

[0048] The handheld controller and the head-mounted wearable component can be configured to share determined posture information via their operable coupling (e.g., via a Bluetooth connection as described above) so that the relative position and / or orientation between the handheld controller and the head-mounted wearable component can be determined (220). Such operation can be frequently updated to maintain timely refreshed information about the relative position and / or orientation between the handheld controller and the head-mounted wearable component (222). The positioning of any component can be provided as a function of any coordinate system. In other words, the handheld component can operate its sensors to determine its position and orientation in a "handheld component frame" or "coordinate system", and then couple with the head-mounted component to determine the relative position of the head-mounted component so as to provide it with a position and orientation in the same frame. The opposite fact is that the head-mounted component can determine its position and orientation, and then determine the position and orientation of the controller relative to the head-mounted component, thereby providing the handheld component with a positioning in the head-mounted component frame or head-mounted coordinate system. A matrix transformation between the two, such as a Euclidean distance matrix, a quaternion orientation, or a 3D rotation matrix, can be used to facilitate this coordinate system transformation between components.

[0049] In some embodiments, any one component can position itself relative to multiple other components. For example, a handheld controller (606) can detect its position relative to a head mounted component (58) and a capture and processing module (606). The means of relative sensing can further be subjective. The hand assembly (606) can utilize IR array tracking to detect IR signals on the head mounted component (58) and use visible light tracking (e.g., feature detection and extraction or fiducial detection) to detect the capture and processing module (606). In these aspects, translation or rotation measurements between a pair of components can replace or supplement image capture feature detection between other pairs of components to provide an optimized positioning data set for the user at any given time.

[0050] Referring to the earlier example, if the user were to approach an empty wall gap that had many detectable features, using image-based tracking would reduce accuracy or increase latency because the system would have fewer sensing features to make calculations. However, if the control and processing module (606) provided additional detectable data, such as IR signals, time-of-flight pulses, or electromagnetic transmissions, the rotation and translation of the user relative to the control and processing module could at least supplement any positioning logic to compensate for the lack of detectable features of the image-based tracking sensor.

[0051] refer to Figure 8B , such as the above reference Figures 6A-6C Those configurations described in 7A-7C can be used to determine the relative or absolute posture of various components. Figure 8B As shown, a user can wear a head-mounted wearable assembly (58) that is operably coupled to multiple handheld controllers (606) and / or fixedly mounted (8) controllers, one of which can be physically coupled to the operator's body (224), for example via a handheld coupling. The head-mounted wearable assembly (58) can be configured to determine certain aspects of the posture (e.g., position and orientation) of the head-mounted wearable assembly relative to the environment (e.g., a room) surrounding the head-mounted wearable assembly (58) and / or certain features of such an environment (e.g., the shape of the room) (226) using sensors coupled thereto. Multiple handheld controllers (606) can be configured to determine certain aspects of the posture (e.g., position and orientation) of such controllers relative to the environment (e.g., a room) surrounding the controllers, and / or certain features of such an environment (e.g., the shape of the room) (228) using sensors coupled thereto, the handheld controllers can be placed on other objects or fixedly mounted thereon, one of which can be physically coupled to the operator, for example on Figures 6A-6CThe plurality of handheld controllers (606) and the head-mounted wearable assembly (58) can be configured to share determined posture information via their operational coupling (e.g., via a Bluetooth connection) so that a relative position and / or orientation (230) between one or more of the plurality of handheld controllers (606) and the head-mounted wearable assembly (58) can be determined, the plurality of handheld controllers (606) being fixedly mounted to other objects (on which the handheld controllers are placed) or coupled to an operator.

[0052] The matrix transformations among each of the components may be optimized to provide absolute positions, if not simply relative positions among the components. For example, if the positions and orientations of the fixed mounted components are known in the physical environment, the handheld component (606) or head mounted component (58) may determine their respective poses relative to those fixed mounted components, and by extension determine a "real world coordinate system" or "environment coordinate system" position and orientation, which may be referred to as an absolute pose.

[0053] It will be appreciated that other intermediate coordinate systems may exist or be useful, as described above. For example, a handheld controller that receives only position data (e.g., from an inertial measurement unit, accelerometer, or camera coupled to a handheld assembly) may determine its pose in a "handheld coordinate system," and similarly for a "head-mounted coordinate system" and pose information detected and extracted by its sensors. In some embodiments, positioning systems and protocols for determining position using the disclosed sensors and sensing assemblies may only need to operate in an intermediate coordinate system, without requiring absolute or complete environmental coordinate system information.

[0054] Such operations may be frequently updated to maintain timely information regarding the relative positioning and / or orientation between the plurality of handheld controllers and / or fixed mounted controllers and the head mounted wearable assembly (232).

[0055] refer to Fig.9A Small fiducials (246) may be used to assist in quickly determining the position and / or orientation between modules of the system using known or predetermined dimensions between one or more of the fiducials (246). The small fiducials may represent transmitting elements such as infrared emitters or reflectors such as a small number of mirrors, glass balls, or other reflective elements. For example, with reference to Fig. 9B In one embodiment of a head-mounted assembly (58) having multiple fiducials (246) coupled thereto, relatively long known lengths between the fiducials (246) can be used to determine and track the relative position and / or orientation of such a head-mounted assembly (58) in space relative to a global coordinate system or relative to another component configured to track the fiducials utilizing a camera or other device. Fig. 9CAnother head mounted assembly (58) configuration is shown having an additional fiducial (246) for providing additional information regarding the position and / or orientation of such a configuration.

[0056] Fig.9D A local processing and data module (70) configuration is shown that also has a plurality of fiducials (246) coupled thereto to assist in relative pose determination. Fig.9E Another embodiment of a local processing and data module (70) configuration is shown having a larger number of fiducials (246) coupled thereto to assist in relative pose determination.

[0057] Fig.9F A handheld controller module (606) configuration is shown that also has a plurality of fiducials (246) coupled thereto to assist in relative pose determination. Figure 9G Another embodiment of a handheld controller (606) configuration is shown having a larger number of fiducials (246) coupled thereto to assist in relative pose determination.

[0058] Figure 9H A handheld controller module (606) configuration is shown having a multiple reference array (248) coupled thereto to assist in relative posture determination; such a reference array having known positions of sub-portions may be useful in posture determination when the geometry for such a reference array can be provided, such as where an attached handheld controller module (606) is to be placed on a coffee table or the like without as many geometric constraints as a wearable component of a mobile computing system. Fig.9I A configuration is shown in which, in addition to a handheld controller (606) already coupled to an operator (60), multiple additional controllers, such as handheld controller modules (606), may also be equipped with a reference (246) to assist in rapid pose determination of components relative to each other.

[0059] refer to Fig.10 , in references such as Fig.9AIn the described embodiment, a user (60) may wear a head-mounted wearable assembly (58) (214) that is operably coupled to a handheld controller (606). The head-mounted wearable assembly (58) may be configured to utilize sensors coupled thereto to determine certain aspects of the posture (e.g., position and orientation) of the head-mounted wearable assembly relative to an environment (e.g., a room) surrounding the head-mounted wearable assembly, and / or certain features of such an environment (e.g., the shape of the room) (216). The handheld controller (606) and the head-mounted wearable assembly, such as being physically coupled to an operator's hand, may be configured to utilize references coupled thereto (e.g., via a transmitter / receiver pair (e.g., an infrared spectrum beacon and sensor in a single lap configuration) or a transceiver / reflector pair (e.g., an infrared spectrum transmitter / receiver device and reflector in a dual lap configuration)) to determine certain aspects of the posture (e.g., position and orientation) of the handheld controller relative to the posture of the head-mounted wearable assembly (234). The handheld controller (606) and the head-mounted wearable assembly (58) can be configured to share the determined posture information via their operable coupling (e.g., via a Bluetooth connection) so that the relative position and / or orientation (236) between the handheld controller and the head-mounted wearable assembly can be determined. Such operation can be frequently updated to maintain refreshed information (238) regarding the relative position and / or orientation between the handheld controller and the head-mounted wearable assembly.

[0060] refer to Fig.11 , in such Fig.9IIn the illustrated embodiment, a user (60) may wear a head-mounted wearable assembly (58) (224) operably coupled to a plurality of hand-held controllers (606), which may be physically coupled to the user or to other objects in a surrounding room, such as a wall (8). The head-mounted wearable assembly (58) may be configured to utilize sensors coupled thereto to determine certain aspects of the posture (e.g., position and orientation) of the head-mounted wearable assembly relative to an environment (e.g., a room) surrounding the head-mounted wearable assembly (58), and / or certain features (e.g., the shape of the room) (226) regarding such an environment. Multiple handheld controllers (606) and head-mounted wearable components (58) can be configured to utilize a reference configuration such as a transmitter / receiver pair (such as an infrared spectrum beacon and sensor in a single overlap configuration) or a transceiver / reflector pair (such as an infrared spectrum transmitter / receiver device and reflector in a dual overlap configuration) coupled thereto to determine certain aspects (240) of the posture (e.g., position and orientation) of the handheld controller (606) relative to the posture of the head-mounted wearable component (58). Multiple handheld controllers (606) and head-mounted wearable components (58) can be configured to share the determined posture information via their operable coupling (e.g., via a Bluetooth connection) so that the relative positioning and / or orientation (242) between the multiple handheld controllers (606) and the head-mounted wearable components (58) can be determined. Such operation can be frequently updated to maintain refreshed information (244) regarding the relative positioning and / or orientation between the multiple handheld controllers and / or fixedly mounted controllers and the head-mounted wearable components.

[0061] refer to Fig. 12A , another embodiment may be configured such that the cyclical motion (248) of the handheld controller (606) may be utilized to further utilize the instrumentation contained thereon for additional posture determination data from different positions / orientations of the handheld controller (606). In other words, if the operator (60) is particularly interested in the movement of the area surrounding the door, the operator (60) may forgo the surrounding handheld controller (60) to assist the controller in utilizing the sensors included with the controller to capture additional information about such local environment obtained from different angles, vectors, and / or orientations.

[0062] This handheld cyclic motion requires the operator to be close to the target structure, which may be undesirable or impractical; Fig. 12B An embodiment of the invention is characterized in that the user (60) is relatively far away from the subject door (202), but there are two handheld controllers (606) mounted to the area of ​​interest - one mounted to a nearby wall and one mounted to the door (202) itself. Fig. 12C Shown Fig. 12AAn embodiment characterized by periodic motion (248) and additional sensing means (606).

[0063] Fig.13 Shown with Fig.11 A similar configuration, except that the user (60) moves one or more handheld controllers around an object or volume of interest (e.g., around a door or other object), such as via a periodic motion or movement, to further capture information about the object or volume of interest and its position and / or orientation relative to the handheld controller (606) and other devices (e.g., one or more fixed-mounted controllers (606) and a head-mounted wearable assembly (58). Such operations may be frequently updated to maintain refreshed information (270) about the relative position and / or orientation between multiple handheld controllers and / or fixed-mounted controllers and head-mounted wearable assemblies.

[0064] Various exemplary embodiments of the present invention are described herein. Reference is made to these examples in a non-limiting sense. They are provided to illustrate the more widely applicable aspects of the present invention.

[0065] The present invention includes methods that can be performed using subject devices. The method may include the action of providing such suitable devices. Such provision may be performed by an end user. In other words, the "providing" action only requires the end user to obtain, access, approach, locate, set, activate, power on or otherwise act to provide the necessary devices in the subject method. The method described herein may be performed in any order of logically possible events being described and the order in which the events are described.

[0066] The present invention includes methods that can be performed using the subject apparatus. The method can include an act of providing such suitable apparatus. Such providing can be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, access, locate, set up, activate, turn on, or otherwise provide the necessary apparatus in the method. The methods described herein can be performed in any order of the events described that is logically possible and in the order of events described.

[0067] Likewise, it is contemplated that any optional features of the described inventive variations may be set forth and claimed independently or in combination with any one or more of the features described herein. Reference to a single item includes the possibility of multiple identical items being present. More specifically, as used herein and in the claims associated therewith, the singular forms "one," "an," "said," and "the" include plural referents unless expressly stated otherwise. In other words, the use of articles allows for "at least one" of the subject items in the above description and in the claims associated with the present disclosure. It should also be noted that such claims may be drafted to exclude any optional elements. Thus, this statement is intended to serve as a prior basis for the use of exclusive terms such as "only," "only" or the use of "negative" limitations in conjunction with the narration of the claim elements.

[0068] Without using such exclusive terms, the term "comprising" in the claims associated with the present disclosure shall allow for the inclusion of any additional elements, regardless of whether a given number of elements are recited in these claims or the addition of a feature may be considered to alter the nature of the elements presented in these claims. Except as expressly defined herein, all technical and scientific terms used herein shall be given the broadest possible meaning as commonly understood while maintaining claim validity.

Claims

1. A method for locating a user operating a plurality of sensing components, the method comprising: sending first gesture data by the capture and processing module; Receiving the first posture data at a head mounted assembly; converting the first pose data into a head mounted assembly relative pose in a coordinate system based on the capture and processing module; updating a display unit in communication with the head mounted assembly with the relative posture of the head mounted assembly; receiving the first gesture data at a handheld component, wherein the handheld component is separate from the head-mounted component; converting the first pose data into a handheld component relative pose in a coordinate system based on the capture and processing module; sharing the relative posture of the handheld component with the head-mounted component; as well as The head mounted component relative posture is updated based on the hand held component relative posture.

2. The method according to claim 1, wherein: Sending the first posture data includes sending an electromagnetic field signal.

3. The method according to claim 2, wherein: The electromagnetic field signal is within the visible spectrum.

4. The method according to claim 2, wherein: The electromagnetic field signal is within the infrared spectrum.

5. The method according to claim 1, wherein: Receiving the first gesture data at the head mounted assembly includes visual capture of an infrared marker.

6. The method according to claim 1, wherein: Receiving the first gesture data at the handheld assembly includes visual capture of a fiducial marker.

7. A method for locating a user operating a plurality of sensing components, the method comprising: receiving an environmental characteristic indicating position information at a head mounted assembly; determining a head mounted assembly pose in an environmental coordinate system of the head mounted assembly based on the received environmental features; receiving at least one hand-held component at the head-mounted component in a relative position relative to the head-mounted component, wherein the at least one hand-held component is detached from the head-mounted component; determining a relative hand-held component pose of the at least one hand-held component in a head-mounted component coordinate system; receiving, at the at least one hand-held component, a relative positional aspect with respect to the head-mounted component; converting the relative positional aspects into relative handheld assembly poses in the head mounted assembly coordinate system; sharing the head mounted component pose in the environment coordinate system with the at least one hand mounted component and sharing the relative hand mounted component pose in the head mounted component coordinate system with the head mounted component; updating the head mounted component pose in the environment coordinate system based on the relative hand held component pose; as well as Based on the head mounted component posture in the environment coordinate system, the relative hand-held component posture in the head mounted component coordinate system is updated.

8. The method according to claim 7, further comprising: Prior to updating the relative hand-held assembly pose, the relative positional aspect is sent from a capture and processing module in the environmental coordinate system to the head-mounted assembly.

9. The method according to claim 8, wherein: Prior to updating the relative handheld assembly pose in the head mounted assembly coordinate system, updating of the head mounted assembly pose in the environment coordinate system is performed.

10. The method according to claim 7, further comprising: The relative handheld component pose in the head mounted component coordinate system is transformed into a handheld component pose in the environment coordinate system.

11. The method according to claim 7, further comprising: The relative positional aspects are transmitted from a control and processing module in the environmental coordinate system to the at least one handheld component.

12. The method according to claim 11, further comprising: A pose of the handheld component in the environment coordinate system is determined.

13. The method according to claim 12, further comprising: The relative hand-held component pose shared with the head-mounted component is updated with the hand-held component pose in the environment coordinate system.

14. The method according to claim 7, wherein: Receiving environmental features includes image-based feature extraction and positioning.

Images