System and method for virtual and augmented reality
Patent Information
- Application Number
- JP2024098003
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2018-12-27
- Filing Date
- 2024-06-18
- Publication Date
- 2024-11-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing VR and AR systems are limited by size, power requirements, and lack of robustness for critical operational scenarios, and they often require users to remove the headset to see their surroundings, disrupting the flow of information.
A distributed computing and networking system for mixed reality systems that includes a head-mounted device with a camera and inertial measurement unit, transmitting images to a remote server for neural network training and returning the trained network for on-device processing, with optional intermediate repeaters like drones or cellular towers.
Enables persistent, compact, and robust wearable computing for scenarios like emergency response and military operations, allowing users to maintain a connected AR experience while navigating their environment.
Abstract
Description
[Technical field]
[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 62 / 785,370, filed December 27, 2018, the contents of which are incorporated herein by reference in their entirety.
[0002] The present invention relates to connected mobile computing systems, methods, and configurations, and more particularly to mobile computing systems, methods, and configurations featuring at least one wearable component that may be utilized for virtual and / or augmented reality operations. [Background technology]
[0003] Mixed reality ("MR"), or augmented reality ("AR"), eyepiece displays are desirably lightweight, low cost, have a small form factor, have a wide virtual image field of view, and are as transparent as possible. In addition, it is desirable to have a configuration that presents virtual image information in multiple focal planes (e.g., two or more) to be practical for a wide variety of use cases without exceeding acceptable tolerances for vergence-accommodation misalignment. Referring to FIG. 1, an augmented reality system is illustrated that features a head-mounted viewing component (2), a handheld controller component (4), and an interconnected auxiliary computing or controller component (6) that may be configured to be worn on a user as a beltpack or equivalent. Each of these components may be operatively coupled (10, 12, 14, 16, 17, 18) to each other and to other connected resources (8), such as cloud computing or cloud storage resources, via wired or wireless communication configurations, such as those defined by IEEE 802.11, Bluetooth® (RTM), and other connectivity standards and configurations. For example, as described in U.S. patent application Ser. Nos. 14 / 555,585, 14 / 690,401, 14 / 331,218, 15 / 481,255, 62 / 518,539, 62 / 693,891, and 62 / 743,492 (each of which is incorporated by reference in its entirety herein), various aspects of such components are described, such as various embodiments of two depicted optical elements (20) through which a user may see the world around them, along with visual components that may be produced by an associated system component for an augmented reality experience.
[0004] A variety of virtual reality ("VR") systems and some effective AR systems exist on the market. Many of these systems are configured to utilize local data (such as gaming software stored on a local flash memory) or connect with some remote data (such as high scores stored on cloud storage resources at a remote location), but few are configured to not only interconnect with other user nodes of various types, but also to effectively utilize and balance local and cloud resources for both processing / operation and storage.
[0005] Most VR systems are essentially surround-type monitors, similar to computer or television monitors placed near the eyes, providing a stereoscopic viewpoint for convergent cues, and connected to computing and storage resources via a connected computing device, such as a high-performance gaming laptop or desktop computer. VR systems are limited in their capabilities in many human motion scenarios, since users are generally unable to see the world around them, and would therefore be challenged to remove the head-mounted display component to view ongoing content, then place the head-mounted display component back on the eye, and continue to view the computer-developed or displayed information. Some variations of VR systems may accommodate so-called "pass-through" video, whereby a forward-oriented camera captures video that can be broadcast to the user within the VR wearable component, but due to latency, viewpoint shift, image fidelity, and negative physiological cues, such systems may not be desirable in many important human motion scenarios. A further limitation is the size of the computing resources, which generally need to be tethered for most contemporary wearable VR display systems. Even if computational capabilities were present, the power requirements to meet the physiological demands of a VR system would require a backpack-sized battery. Furthermore, there is a lack of such technology that is secure and robust enough to be utilized in critical operational scenarios such as emergency medical response, fire response, police operations, and / or military operations. The systems, configurations, and methods described herein are designed to address the various challenges of portable, robust, highly connected, and highly capable wearable computing deployment in a variety of human scenarios. There is a need for compact and persistently connected systems and assemblies that are optimized for use in wearable computing systems. Summary of the Invention [Means for solving the problem]
[0006] An embodiment of the present disclosure describes a system and method for distributed computing and / or networking for a mixed reality system. According to an embodiment of the present disclosure, the method may include capturing an image via a camera of a head-mounted device. Inertial data may be captured via an inertial measurement unit of the head-mounted device. A position of the head-mounted device may be estimated based on the image and the inertial data via one or more processors of the head-mounted device. The image may be transmitted to a remote server. A neural network may be trained based on the image via the remote server. The trained neural network may be transmitted to the head-mounted device. The present specification also provides, for example, the following items: (Item 1) 1. A method comprising: Capturing an image via a camera of a head-worn device; Capturing inertial data via an inertial measurement unit of the head-mounted device; estimating, via one or more processors of the head-mounted device, a position of the head-mounted device based on the images and the inertial data; and transmitting said image to a remote server; training a neural network based on the images via the remote server; and Transmitting the trained neural network to the head-worn device. A method comprising: (Item 2) 2. The method of claim 1, wherein transmitting the image to a remote server includes transmitting the image to an intermediate repeater. (Item 3) 3. The method of claim 2, wherein the intermediate repeater comprises a drone network. (Item 4) 3. The method of claim 2, wherein the intermediate repeater comprises a communication array, the communication array being mountable on a person. (Item 5) 3. The method of claim 2, wherein the head-mounted device is a first head-mounted device and the intermediate repeater comprises a second head-mounted device. (Item 6) 3. The method of claim 2, wherein the intermediate repeater comprises a cellular tower. (Item 7) 1. A system comprising: A camera in a head-mounted device; an inertial measurement unit of the head-worn device; A remote server; one or more processors, the one or more processors comprising: Capturing an image via a camera of the head-worn device; Capturing inertial data via an inertial measurement unit of the head-mounted device; estimating, via one or more processors of the head-mounted device, a position of the head-mounted device based on the images and the inertial data; and transmitting said image to said remote server; training a neural network based on the images via the remote server; and transmitting the trained neural network to the head-worn device; one or more processors configured to execute a method comprising: A system comprising: (Item 8) 8. The system of claim 7, wherein transmitting the image to a remote server includes transmitting the image to an intermediate repeater. (Item 9) 9. The system of claim 8, wherein the intermediate repeater comprises a drone network. (Item 10) 9. The system of claim 8, wherein the intermediate repeater comprises a communications array, the communications array being mountable on a person. (Item 11) 9. The system of claim 8, wherein the head-mounted device is a first head-mounted device and the intermediate repeater comprises a second head-mounted device. (Item 12) Item 9. The system of item 8, wherein the intermediate repeater comprises a cellular tower. (Item 13) A non-transitory computer readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: Capturing an image via a camera of a head-worn device; Capturing inertial data via an inertial measurement unit of the head-mounted device; estimating, via one or more processors of the head-mounted device, a position of the head-mounted device based on the images and the inertial data; and transmitting said image to a remote server; training a neural network based on the images via the remote server; and transmitting the trained neural network to the head-worn device; A non-transitory computer readable medium for carrying out a method comprising: (Item 14) Item 14. The non-transitory computer-readable medium of item 13, wherein transmitting the image to a remote server includes transmitting the image to an intermediate repeater. (Item 15) Item 15. The non-transitory computer-readable medium of item 14, wherein the intermediate repeater comprises a drone network. (Item 16) Item 15. The non-transitory computer-readable medium of item 14, wherein the intermediate repeater comprises a communications array, the communications array being mountable on a person. (Item 17) Item 15. The non-transitory computer-readable medium of item 14, wherein the head-mounted device is a first head-mounted device and the intermediate repeater comprises a second head-mounted device. (Item 18) Item 15. The non-transitory computer-readable medium of item 14, wherein the intermediate repeater comprises a cellular tower. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 illustrates an example mixed reality system, according to some embodiments.
[0008] [Diagram 2] FIG. 2 illustrates an example mixed reality system, according to some embodiments.
[0009] [Diagram 3] FIG. 3 illustrates an exemplary mixed reality environment, according to some embodiments.
[0010] [Figure 4] FIG. 4 illustrates an example mixed reality system, according to some embodiments.
[0011] [Diagram 5] FIG. 5 illustrates an example process for building a virtual model, according to some embodiments.
[0012] [Figure 6] FIG. 6 illustrates an example head-mounted component of a mixed reality system, according to some embodiments.
[0013] [Figure 7] FIG. 7 illustrates an example wearable pack component of a mixed reality system, according to some embodiments.
[0014] [Figure 8] FIG. 8 illustrates an exemplary mixed reality computing architecture, according to some embodiments.
[0015] [Figure 9] FIG. 9 illustrates an exemplary mixed reality computing architecture, according to some embodiments.
[0016] [Figure 10] FIG. 10 illustrates an exemplary mixed reality computing architecture, according to some embodiments.
[0017] [Figure 11] FIG. 11 illustrates an exemplary mixed reality computing architecture, according to some embodiments.
[0018] [Figure 12] FIG. 12 illustrates an example use of a connected mixed reality system, according to some embodiments.
[0019] [Figure 13] FIG. 13 illustrates an example use of a connected mixed reality system, according to some embodiments.
[0020] [Figure 14] FIG. 14 illustrates an example use of a connected mixed reality system, according to some embodiments.
[0021] [Figure 15] FIG. 15 illustrates an example use of a connected mixed reality system, according to some embodiments.
[0022] [Figure 16] FIG. 16 illustrates an example use of a connected mixed reality system, according to some embodiments.
[0023] [Figure 17] FIG. 17 illustrates an example use of a connected mixed reality system, according to some embodiments.
[0024] [Figure 18] FIG. 18 illustrates an example use of a connected mixed reality system, according to some embodiments.
[0025] [Figure 19] FIG. 19 illustrates an example use of a connected mixed reality system, according to some embodiments.
[0026] [Figure 20] FIG. 20 illustrates an example use of a connected mixed reality system, according to some embodiments.
[0027] [Figure 21] FIG. 21 illustrates an example use of a connected mixed reality system, according to some embodiments.
[0028] [Figure 22] FIG. 22 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0029] [Figure 23] FIG. 23 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0030] [Figure 24] FIG. 24 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0031] [Diagram 25] FIG. 25 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0032] [Figure 26] FIG. 26 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0033] [Figure 27] FIG. 27 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0034] [Figure 28] FIG. 28 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0035] [Figure 29] FIG. 29 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0036] [Diagram 30] FIG. 30 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0037] [Diagram 31] FIG. 31 illustrates an example mixed reality network architecture, according to some embodiments.
[0038] [Diagram 32] FIG. 32 illustrates an example mixed reality computing architecture, according to some embodiments.
[0039] [Diagram 33] FIG. 33 illustrates an example use of a mixed reality system, according to some embodiments.
[0040] [Diagram 34] FIG. 34 illustrates an example mixed reality computing architecture, according to some embodiments.
[0041] [Diagram 35] FIG. 35 illustrates an example sensor component of a mixed reality system, according to some embodiments.
[0042] [Diagram 36] FIG. 36 illustrates an example sensor component of a mixed reality system, according to some embodiments.
[0043] [Figure 37] FIG. 37 illustrates an example use of a mixed reality system, according to some embodiments.
[0044] [Figure 38] FIG. 38 illustrates an example use of a mixed reality system, according to some embodiments.
[0045] [Figure 39] FIG. 39 illustrates an example use of a mixed reality system, according to some embodiments.
[0046] [Diagram 40] FIG. 40 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0047] [Diagram 41] FIG. 41 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0048] [Diagram 42] FIG. 42 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0049] [Diagram 43] FIG. 43 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0050] [Diagram 44] FIG. 44 illustrates an example use of a mixed reality system in an emergency situation, according to some embodiments.
[0051] [Diagram 45] FIG. 45 illustrates an example mixed reality computing architecture, according to some embodiments.
[0052] [Figure 46]FIG. 46 illustrates an example mixed reality computing architecture, according to some embodiments.
[0053] [Figure 47] FIG. 47 illustrates an example mixed reality computing architecture, according to some embodiments.
[0054] [Figure 48] FIG. 48 illustrates an example mixed reality computing architecture, according to some embodiments.
[0055] [Figure 49] FIG. 49 illustrates an example mixed reality computing architecture, according to some embodiments.
[0056] [Figure 50] FIG. 50 illustrates an example mixed reality computing architecture, according to some embodiments.
[0057] [Figure 51] FIG. 51 illustrates an example mixed reality computing architecture, according to some embodiments.
[0058] [Figure 52] FIG. 52 illustrates an example mixed reality computing architecture, according to some embodiments.
[0059] [Diagram 53] FIG. 53 illustrates an example mixed reality computing architecture, according to some embodiments.
[0060] [Figure 54] FIG. 54 illustrates an example mixed reality computing architecture, according to some embodiments.
[0061] [Figure 55] FIG. 55 illustrates an example mixed reality computing architecture, according to some embodiments.
[0062] [Figure 56] FIG. 56 illustrates an example mixed reality computing architecture, according to some embodiments.
[0063] [Figure 57] FIG. 57 illustrates an example mixed reality computing architecture, according to some embodiments.
[0064] [Figure 58] FIG. 58 illustrates an example mixed reality computing architecture, according to some embodiments.
[0065] [Figure 59] FIG. 59 illustrates an example mixed reality computing architecture, according to some embodiments.
[0066] [Figure 60] FIG. 60 illustrates an example mixed reality computing architecture, according to some embodiments.
[0067] [Figure 61] FIG. 61 illustrates an example mixed reality computing architecture, according to some embodiments.
[0068] [Figure 62] FIG. 62 illustrates an example mixed reality computing architecture, according to some embodiments.
[0069] [Figure 63] FIG. 63 illustrates an example mixed reality computing architecture, according to some embodiments.
[0070] [Figure 64] FIG. 64 illustrates training use of an exemplary mixed reality system, according to some embodiments.
[0071] [Figure 65] FIG. 65 illustrates training use of an exemplary mixed reality system, according to some embodiments.
[0072] [Figure 66] FIG. 66 illustrates training use of an exemplary mixed reality system, according to some embodiments.
[0073] [Figure 67] FIG. 67 illustrates training use of an exemplary mixed reality system, according to some embodiments.
[0074] [Figure 68] FIG. 68 illustrates training use of an exemplary mixed reality system, according to some embodiments.
[0075] [Figure 69] FIG. 69 illustrates training use of an exemplary mixed reality system, according to some embodiments.
[0076] [Figure 70] FIG. 70 illustrates an example use of a mixed reality system, according to some embodiments.
[0077] [Figure 71] FIG. 71 illustrates an example authorization architecture for a mixed reality system, according to some embodiments.
[0078] [Figure 72] FIG. 72 illustrates an example authorization architecture for a mixed reality system, according to some embodiments.
[0079] [Figure 73]FIG. 73 illustrates an example authorization architecture for a mixed reality system, according to some embodiments.
[0080] [Figure 74] 74A-74D illustrate example dimming functionality of a mixed reality system, according to some embodiments.
[0081] [Figure 75] FIG. 75 illustrates an example targeting feature of a mixed reality system, according to some embodiments.
[0082] [Figure 76] FIG. 76 illustrates an example targeting feature of a mixed reality system, according to some embodiments.
[0083] [Figure 77] FIG. 77 illustrates an example mixed reality computing architecture, according to some embodiments.
[0084] [Figure 78] FIG. 78 illustrates an example sensor component of a mixed reality system, according to some embodiments.
[0085] [Figure 79] FIG. 79 illustrates an example sensor component of a mixed reality system, according to some embodiments.
[0086] [Figure 80] FIG. 80 illustrates an example sensor component of a mixed reality system, according to some embodiments.
[0087] [Figure 81] FIG. 81 illustrates an example mixed reality computing architecture, according to some embodiments.
[0088] [Figure 82]FIG. 82 illustrates an example mixed reality computing architecture, according to some embodiments.
[0089] [Figure 83] FIG. 83 illustrates an example mixed reality computing architecture, according to some embodiments.
[0090] [Fig. 84A] 84A-84B illustrate an example mixed reality computing architecture, according to some embodiments. [Fig. 84B] 84A-84B illustrate an example mixed reality computing architecture, according to some embodiments.
[0091] [Figure 85] FIG. 85 illustrates an example wearable component of a mixed reality system, according to some embodiments.
[0092] [Figure 86] 86A-86D illustrate example sensor components of a mixed reality system, according to some embodiments.
[0093] [Figure 87] FIG. 87 illustrates an example mixed reality computing architecture, according to some embodiments.
[0094] [Figure 88] FIG. 88 illustrates an example mixed reality computing architecture, according to some embodiments.
[0095] [Figure 89] FIG. 89 illustrates an example mixed reality computing architecture, according to some embodiments.
[0096] [Figure 90]FIG. 90 illustrates an example virtual assistant, according to some embodiments.
[0097] [Figure 91] FIG. 91 illustrates an example mapping component of a mixed reality system, according to some embodiments.
[0098] [Figure 92] FIG. 92 illustrates an example virtual model, according to some embodiments.
[0099] [Figure 93] FIG. 93 illustrates an example cuff component of a mixed reality system, according to some embodiments.
[0100] [Figure 94] FIG. 94 illustrates an example mixed reality computing architecture, according to some embodiments.
[0101] [Figure 95] FIG. 95 illustrates an example mixed reality computing architecture, according to some embodiments.
[0102] [Figure 96] FIG. 96 illustrates an example mixed reality computing architecture, according to some embodiments.
[0103] [Figure 97] FIG. 97 illustrates an example mixed reality computing architecture, according to some embodiments.
[0104] [Figure 98] FIG. 98 illustrates an example mixed reality computing architecture, according to some embodiments.
[0105] [Figure 99]FIG. 99 illustrates an example mixed reality computing architecture, according to some embodiments.
[0106] [Figure 100] FIG. 100 illustrates an exemplary mixed reality computing architecture, according to some embodiments.
[0107] [Figure 101] FIG. 101 illustrates an example mixed reality computing architecture, according to some embodiments.
[0108] [Figure 102] FIG. 102 illustrates an example mixed reality computing architecture, according to some embodiments.
[0109] [Figure 103] FIG. 103 illustrates an example head-mounted component of a mixed reality system, according to some embodiments.
[0110] [Figure 104] FIG. 104 illustrates an example wearable pack component of a mixed reality system.
[0111] [Figure 105A] 105A-105C illustrate an example mixed reality environment according to some embodiments. [Figure 105B] 105A-105C illustrate an example mixed reality environment according to some embodiments. [Figure 105C] 105A-105C illustrate an example mixed reality environment according to some embodiments.
[0112] [Figure 106A] 106A-106D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment, according to some embodiments. [Figure 106B] 106A-106D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment, according to some embodiments. [Figure 106C] 106A-106D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment, according to some embodiments. [Figure 106D] 106A-106D illustrate components of an example mixed reality system that can be used to generate and interact with a mixed reality environment, according to some embodiments.
[0113] [Figure 107A] FIG. 107A illustrates an example mixed reality handheld controller that can be used to provide input to a mixed reality environment, according to some embodiments.
[0114] [Figure 107B] FIG. 107B illustrates an example auxiliary unit that may be used with an example mixed reality system, according to some embodiments.
[0115] [Figure 108] FIG. 108 illustrates an example functional block diagram for an example mixed reality system, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0116] Referring to Figure 2, a user is shown wearing a head mounted display component (2) and a belt or "computing capability" puck (6) and holding a handheld controller component (4). With the system operatively connected to other resources, for example through an 802.11 WiFi connection, the user or operator may navigate around the environment and utilize the computing power of the system while also seeing things around them. Figure 3 illustrates some basic concepts of how a user may see and / or perceive things in an augmented reality experience as the user walks through a park setting (22) and views (24) the world through a head mounted wearable component (2) such as those described herein. In this illustration, the user may see aspects of the real world such as trees, a raised concrete platform (1120), and the rest of the park (22) setting, while also seeing insect characters (26) and statues (1110) in three dimensions that do not exist in the real physical world, but are presented to the user as if they did by using the augmented reality capabilities of the subject system. With reference to FIG. 4, as discussed above with reference to FIG. 1, the subject AR system may be operatively coupled to many different resources, such as storage and computing resources, as well as other important operational resources, such as other portable computing systems, smartphones, smartwatches, storage and / or processing hubs, wireless connectivity hubs or nodes, various types of personal radios, aircraft or air support, emergency response vehicles, network control centers or command and control centers, unmanned vehicles such as drones, mobile or wearable camera devices, and / or sensors or many types, including LIDAR and other sensors. In various embodiments, the user is preferably connected to many resources.
[0117] With reference to FIG. 5, in one embodiment, a connected constellation involving many connected resources may be utilized as a kind of 3-D sensor to reconstruct a mesh or model of one or more objects that may be located between or surrounded by one or more transmitting / receiving devices, such as a cell phone with a 5G antenna array or equivalent. In other words, each phone, tablet, wearable, or other connected device may be utilized as a sensor. For example, the direction and intensity, scattering, transmission, and absorption of each signal-source pair may be utilized to create an interference map, which can then be visualized by a user, for example, within the field of view of a wearable computing system. In the case of a 5G connectivity environment, the source / signal pairing may provide a millimeter wave reconstruction of the surrounding world. This may be very computationally intensive, but is one means of creating a general model of the world that can be updated using modern frequency / spatial domain holographic reconstructions.
[0118] With reference to Figures 6 and 7, as described in the above incorporated references, in various embodiments, it may be desirable to have many types of sensors and technologies integrated into the wearable component (2) and / or the belt pack / computation pack component (6). For example, in the depicted embodiment of Figure 6, the deflection or bending sensor may comprise an internal interferometer with two different directions. Based on phase change detection, bending deflection may be correlated and determined. Also shown is a hybrid dynamic visual visible light sensor, where a differential sensor and a visible light sensor may be utilized on the same optical path, using a beam splitter or the like. The signals may be analyzed together and processed for differences / similarity. As depicted in Figure 7, in one embodiment, the belt pack component (6) may be removably coupleable from the head mounted component and removably coupleable / replaceable from the battery component to ensure sustained operation. The beltpack component (6) may have an interface such as an LCD touch panel / display, may be utilized as a mobile phone if required, and may have one or more cameras or sensing devices integrated therein to provide additional insight for the operator, such as a rear or other orientation relative to the operator.
[0119] With reference to FIG. 8, various aspects of system connectivity are illustrated along with other elements operatively coupled to the operator headset (2), beltpack (6), communication radio, and various other resources. For example, in various embodiments, a "cuff assistant" element may comprise a cuff or band type device that may be mounted around the user's wrist in certain scenarios, such as critical actions such as escaping danger. Such a device may be configured to sense temperature, blood pressure, location (such as by GPS and / or mobile connectivity triangulation and / or IP address mapping), partial pressure of blood oxygen, and other variables related to such operator and his / her location and condition. These may be reported to emergency medical services or other systems or personnel, such as to dispatch resources to the operator, assist in managing the scenario, etc. FIG. 8 also illustrates the concept of connectivity (which may be referred to as the "MagicVerse") with a database and universal or semi-universal collection of information about the operator's world. This may include connectivity to emergency medical systems, for example through emergency and prevention networks or other medical networks, which may contain full patient records, for example with information regarding allergies or medical conditions. Also connected may be various types of home, business, or city-wide networks, such as groups of web cameras, servers with related information and data, and resources from what may be referred to as the "Guardian Net," a network of resources accessible to law enforcement and certain military personnel that gain access to web cameras, microphones, satellites, and large-scale storage devices for certain investigative, emergency, and other purposes.
[0120] With reference to Figures 9, 10, and 11, a firefighter, law enforcement officer, or military personnel, respectively, may utilize a wearable computing system (2, 6) as a type of mobile operating hub to integrate the utility of numerous other connected resources, such as smartphones, EMS radios, connectivity and / or storage hubs at home, business, or any location, various cameras and associated sensors, GPS, medical and emergency databases, law enforcement, Guardian Net connectivity, weapon system connectivity, connectivity to other manned or unmanned vehicles, and the like.
[0121] With reference to Figures 12-21, the subject technology may be utilized and adapted for office productivity scenarios as well. For example, with reference to Figures 12 and 13, a meeting may take place with six people convened to discuss something, such as something visually complex, such as a three-dimensional automobile part, that may be displayed to each participant through their head-mounted computing component (2). One or more of the attendees may be physically present in the room ("local"), while one or more may be physically present somewhere but represented in the meeting by a presented image, such as an avatar, of themselves ("remote"). Attendees may utilize not only mobile computing technology (2) to be part of the meeting, but also various other "Internet of Things" connected devices, such as displays, lights, microphones, speakers, cameras, and the like, in the room, as shown in Figure 14. In various embodiments, devices available from a particular room or environment may be mapped into such room or environment such that they are available to the user when actually or virtually present in such room. With reference to FIG. 15, in various embodiments, translation technology such as those available for translating language / text and text / different languages may be utilized to facilitate real-time or near real-time engagement of members who speak a different language than that of other participants in the conference. With reference to FIGS. 16 and 17, the system may be configured to allow participants to select aspects of their own avatars, such as customized views of their own face or parts thereof, characters such as selected cartoon characters, and / or similar treatments for other participants. For example, user A may decide for a particular conference to appear to others as an avatar of Abraham Lincoln, while automatically assigning a "pig" avatar to another member of the conference (user B), and also automatically placing a presentation of a funny hat and fake mustache on the otherwise standard avatar of a third member of the conference (user C).With reference to FIG. 18, various user interface configurations may be configured to display a menu above each participant that is visible to other participants and may be utilized, for example, to share notes, share a video perspective, and / or share certain meeting or information controls. With reference to FIG. 19, the system may be configured to provide a private chat messaging user interface between subgroups of meeting attendees, or a more public chat visible to all attendees, for example, to share notes from the meeting. As noted above, languages may be translated, such as by automated settings, to provide access and usability in a multilingual meeting environment. With reference to FIG. 20, in various embodiments, the system may be configured to present a user interface at each participant's actual or virtually represented desk, such that the participant may utilize their hands to move things, actuate or activate various things, control various aspects of the room, camera views, microphones, speakers, documents the participant is viewing, sharing of notes, and the like. Such a user interface is preferably configurable by and to provide each user with a customized and efficient view of information and controls within the meeting environment. With reference to FIG. 21, as described above with reference to FIG. 12, the unique three-dimensional viewing capability possible with some version of the head-mounted computing component (2) presents a unique opportunity for participants in a meeting to view some complex material or information in three dimensions together "around the table," regardless of whether all participants are actually physically present in the room.
[0122] With reference to Figures 22-30, in some firefighting environments, wearable computing systems may be configured for high usability. For example, with reference to Figure 22, firefighters may wear a head-mounted component (2) so that they can visualize routes and mapping to an incident scene. With reference to Figure 23, Internet of Things connected resources such as door locks, smoke alarms, cameras, electrical and gas resources, and lighting may be visualized (i.e., by head-mounted component 2, etc.) and made accessible by firefighter operators in the field or back in a control center for emergency control. With reference to FIG. 24, in various embodiments, the system may be configured to allow each member of the team connected with a mobile computing system, such as a head mounted component (2), enabling hands-free usability, to visualize location (i.e., based on GPS, connectivity triangulation, IP address, mapping / SLAM / computer vision, etc.) and status (health status, etc., based on biometric sensors that may be presented on the user's head mounted component (2) upon encountering stress, danger, smoke, etc., which may be detected and / or visualized using various sensors and cameras present on such head mounted component and shared with other collaborators via wireless connectivity)). For example, with reference to FIG. 26, the user's head mounted component (2) may be configured to monitor eye-based and facial indicators of stress, as well as heart rate, heart rate variability, etc., and report this information to others. Additionally, the user's head mounted component (2) may be configured to utilize computer vision resources, such as cameras and local or remote / connected computing resources, to not only scan rooms around them for mapping and machine learning functionality, but also to perform facial recognition of personnel, such as injured patients, within a building. Additionally, with reference to FIG. 27, visible light camera technology, such as may be operatively coupled to the user's head mounted component (2), may be configured to capture and share photographs of various scenes, personnel, etc. with other operators at various locations, such as a remote control center.With reference to FIG. 28, thermal / infrared (i.e., long or normal wavelength infrared, etc.), visible light, and / or short wavelength infrared imaging techniques ("SWIR" has the advantage of being able to image very successfully through smoke that may be visually obscured by other imaging modalities) may be combined to facilitate visualization of various things within the emergency operator environment, such as fire ignition location, people, pets, etc. With reference previously to FIGS. 86A-86D, SWIR imaging may be utilized to significantly improve user visibility over normal visible light depending on the environmental scenario. With reference to FIG. 29, an operator may utilize the user's head mounted component (2) to tag various people or things within the emergency environment, such as patients with various illnesses or injuries, for sharing with other operators. With reference to FIG. 30, connected systems, such as wearable components, may be configured to provide outgoing information regarding operator conditions, not only operator biometrics and environmental concerns, but also other important connected devices, such as oxygen tank fill level sensors.
[0123] With reference to Figs. 31-39, in a police / law enforcement environment, the wearable computing system may be configured for high availability. With reference to Fig. 31, the wearable computing system (2) may be configured for various members of a police operational team to utilize hands-free messaging to / from various disparate locations. With reference to Fig. 32, in various embodiments, each member is connected, each member being another sensing "node" to the overall system, providing not only data regarding its operator biometrics, but also information regarding the environment surrounding such operator, such as evidence collection, personnel or structure identification, video / audio / photo capture, thermal / IR / SWIR imaging, simultaneous localization and mapping ("SLAM"), wireless connectivity triangulation, GPS, and / or location via IP address, traffic or congestion detection / reporting, access to remote databases such as crime or health care databases, sensing or receiving mapping data related to the world around each operator. For example, with reference to FIG. 33, as a police operator walks around with its connected head-mounted computing component (2), the system may be configured to analyze each face that comes within close visual contact with the officer and perform biometric analysis, such as facial or iris recognition. Such information may be utilized in conjunction with a connected database search to inform an officer, for example, that the person he just passed on the sidewalk is a felon with an active arrest warrant, who is armed and likely dangerous. With reference to FIG. 34, utilizing connected resources, such as crime / location databases, in conjunction with location determined, for example, by GPS, wireless connectivity location, IP address, and / or SLAM / computer vision techniques using the connected head-mounted computing component (2), the system may be configured to provide an operator in the field, at a command center, or anywhere, with an indication regarding crime rates within a particular area.For example, in one embodiment, if the operator is in a high crime dark alley area in South Chicago, especially at 2:30 a.m. local time, everything around the operator may be colored red or otherwise indicated as relatively high crime. Elsewhere, one side of the street may be indicated as having a relatively lower crime rate than the other side of the street based on recent data. Thus, if an officer is walking or driving through a known crime zone, at least this will not be surprising. Additionally, recent events or information may be marked for the operator, such as pointers to known high concentrations of street drug transactions in the last week. Referring to FIGS. 35-37, sensors such as inward facing cameras on the head mounted computing component (2) may be utilized to track variables related to the operator's transient stress level, such as pupil diameter variability over time. Significant variability over time, when normalized for incident light variability, has been shown to be a relatively good indicator of an individual's stress state (e.g., a small constant pupil diameter may be correlated with a relatively still state, while high variability and a relatively large pupil diameter may be correlated with a relatively high stress state). Pupil diameter may be measured over time and reported to connected systems and personnel. For example, in one embodiment, a Eulerian video magnification technique that involves scanning a line across a captured scene (such as the eye) and looking for changes / deltas, along with segmentation to identify the pupil vs. sclera of the eye, etc., can provide an efficient window into some biometric variability. Referring to FIG. 37, such techniques may be utilized not only for the biometrics of operators wearing the appropriate components (2), such as police officers, but also for analyzing the biometrics of others, if in sufficient close proximity, using outwardly facing sensors.With reference to FIG. 38, it is shown that people under extreme stress conditions may have difficulty or inefficiency in processing certain things that would be considered relatively easy under non-stress conditions, and therefore various embodiments may be configured to assist with all kinds of information-related tasks, such as labeling a person with a wound as such or a dead body as a corpse, without the need for each operator to walk over, check a pulse, confirm, etc. With reference to FIG. 39, criminal activity may be mapped in real-time or near real-time and visualized, such as in an AR mapping user interface, by an operator wearing the appropriate component (2). In one embodiment, the AR mapping interface may also be configured to provide time-domain control to the operator, such that they may rewind in hours or any time frame and visualize the scenario as it was at that time.
[0124] An operator or responder, such as a firefighter, police, or emergency medical responder, may also utilize connected medical resources through their connected wearable computing component (2) in various situations. For example, referring to Figs. 40-44, an operator may utilize their connected wearable computing component (2) to perform spectrum analysis on environmental issues around them (smoke, gas, chemicals, etc., using technology such as a SWIR device that may be coupled to the operator's headset 2), have direct access to patient history, and / or have a specialist physician who may be able to connect to their headset (2) share their view and audio feed of the patient in high resolution (i.e., via an outward facing microphone on the headset 2), perform biometric analysis (such as patient recognition by face, iris, and / or transient connectivity to specialized resources on the patient's smartphone, etc.), understand allergies or the patient's important case history, etc. For example, an operator utilizing a connected wearable computing component (2) with direct connectivity to a remote specialist may encounter an unconscious patient who appears to be in cardiac arrest. The operator may request expert emergency medical triage assistance, and the expert may participate in a video conference and / or avatar presentation that appears with audio within a portion of the operator's computing component (2) field of view. Facial recognition, other biometrics, specialized emergency responder patient smartphone access, and / or simple patient wallet identification card information may be utilized to identify the patient, and a securely connected resource may be utilized to establish that the patient is a known heroin addict who, from what the attending emergency medical professional can discern from the operator's shared field of view, appears to have overdosed and is dying, and should be urgently administered a non-opiate naloxone hydrochloride-infused drug product, such as that sold under the trademark NarCan.
[0125] With reference to Figs. 45-63, many configurations may be utilized to assist various types of military personnel in a myriad of scenarios. For example, with reference to Figs. 45-46, a military personnel with a connected wearable computing system (2, 6) may be wirelessly connected to various other resources, such as a secure communications radio (such as an ultra-wideband radio available to the military personnel), one or more remote weapon designators, one or more battery packs configured to be removably coupled to the wearable computing system, databases, maps, images, video, audio, and many types of communications (all interconnected and operable by the military personnel). Such configurations may be communicated from and / or interconnected with military systems such as those known as ATAC and NetWarrior. The wearable computing system may also be configured to provide certain protection to the operator, such as laser eye protection through the head mounted component (2) and hearing protection through earplug type headphones configured to pass only audio up to a certain intensity, above which transmission is retracted and the earplug blocks the sound (in other words, the earplug may be operatively coupled to a microphone, such as a directional microphone or array thereof, which may be capable of capturing and processing sounds and selectively reproducing them or not in the powered earplug). With reference to Figures 48 and 55, the weapon or a portion thereof may be tracked in a manner similar to that described with reference to the six degrees of freedom ("DOF") tracking of the handheld component (4), described in detail in the incorporated references above, and the control system may be configured to assist the operator in pointing at a desired target and / or to facilitate activation or firing of the weapon only when the weapon is in the correct position / orientation relative to the desired target. Referring back to FIG. 52, the system may also be configured to provide relevant information about each visualized object (enemy combatant status, rules of engagement, suspected weaponry status, any detectable or known stress or medical information, etc.) after identification using previous information from the outward facing cameras and / or sensors and / or other sources about specific objects within the view.
[0126] In another embodiment, one or more directional microphones may be operatively coupled to highly sophisticated signal processing capabilities to help direct and identify the sound capture. For example, at night, around a military combat vehicle, a unit may place its headset (2) perimeter to act as a sound capture perimeter, which may be monitored locally or remotely (i.e., as a group of individual sound capture devices or as an array) for specific sound patterns, such as "sound pattern from the north at 1:39 AM local time - diesel transport truck, one of the friendly vehicles" or "sound pattern from due south at 2:58 AM local time - AK-47 small arms fire, along with small motorcycle activity; 83% confidence; to wake up military commanders and provide updates on potential insurgent activity."
[0127] 49-53, preferably the system utilized by operator (2) is connected with those of other operators and remote systems and resources, such that, for example, the local ground operator can take advantage of video, audio, and other analysis and activities related to the use of the unmanned aerial vehicle, other vehicles, other operators, and remote command centers, all of which may be connected by a variety of technologies, including, but not limited to, ultra-wideband transceivers, 802.11 / WiFi configurations, 3G, 4G, 5G, LTE, and other mobile connectivity modalities, satellite, etc. For example, in FIG. 49, the ground operator is seen visualizing three armed enemies using imagery captured by the unmanned aerial vehicle from a position where the operator would otherwise be unable to see such enemies (e.g., because they are on the other side of a ridge, etc.). With reference to FIG. 51, short-wave infrared ("SWIR"), or NIR-based (near infrared) time-of-flight sensors, or structured light configurations, LIDAR, RADAR, and the like, may be utilized to understand the surroundings and look for specific things, such as people. With reference to FIG. 53, in various embodiments, the connected system may be configured to provide each operator on the ground and others at other locations with a real-time or near real-time feed of activities and issues related to a given situation. With reference to FIG. 54, this may include information regarding medical evacuation, such as by using air resources (so-called "medical evacuation" air resources). For example, in various embodiments, the system may be configured such that operators wearing their connected computing systems (2), both in the air and on the ground (and connected operators at base camps, headquarters, etc.) may be able to share useful information that would otherwise be difficult to communicate effectively in a hands-free manner, such as a roughly cone-shaped (or other geometric shape) air traffic avoidance to avoid contact with small arms fire from enemy combatants, while still being able to reach and contain the location of the wounded.The operator on the ground may be able to adjust or change the geometry and / or location of this avoidance zone by using gestures made by his headset and pointer, his connected weapon or weapon sight, etc. Referring to Fig. 56, the operator wearing his computing system (2) may be able to orient himself and other resources, such as weapons, by using computer vision, object recognition, SLAM, and other image-based techniques when GPS, mobile telecommunications triangulation, IP addresses, and other location techniques are not readily available or operational.
[0128] With reference to Figures 57-58, 81, and 87-89, the level of connectivity of operators and available resources with the configuration of the present subject matter provides detailed quantitative analysis of various aspects of military operations like never before, through facts and performance, not recommendations or blanket standardized testing. The system may be configured so that the lowest ranking operators have access to a minimal subset of information for them and their peers, and operators up the chain of command have access to more and more information, classified and otherwise, all the way to the top. With reference to Figures 59-63, everyone in an operation can benefit from sharing and capturing information in an efficient manner, primarily hands-free. Operators can collaborate with each other, visualize things like never before, and communicate with and utilize other resources such as aircraft, other imaging systems, database resources, experts, team members, and / or health-related information, all with their wearable computing systems, preferably head-mounted AR visualization components, at the heart of the functionality and enablement requirements.
[0129] With reference to FIGS. 64-69, the wearable computing system of the present subject matter may be utilized to support advanced training exercises in nearly any location, providing repeatability, improved access, and also diagnostic / assessment visualization tools. For example, with reference to FIG. 64, in one embodiment, a user may be able to select one from a variety of prepared training scenarios. With reference to FIG. 65, in nearly any environment, the wearable computing asset (2) may be utilized to present a user with a predefined training / operation scenario in which they encounter and visualize one or more members of an adversary, for example, as shown in FIGS. 66 and 67, with content / imagery relating to the adversary being shown to the operator in the exercise as augmented reality content through its head-mounted component (2). With reference to FIGS. 68 and 69, the exercise may be reviewed in three-dimensions versus time for improved diagnostics and learning.
[0130] In various embodiments, at the core of the simulation / training architecture is the Training Simulation Software Architecture ("TSS"), which represents a rigorous simulation engine. In other words, the TSS is essentially a set of rules that govern how a synthetic version of the world functions, i.e., how fast bullets fall, what warriors, police officers, firefighters, and others can see while standing in a location, how fast enemies, fires, suspects move, etc. In various embodiments, the server-side component of the TSS, i.e., "TSS / S," may be configured to allow users to connect to and interact with an ongoing simulation for training purposes. The TSS / S may be configured to provide a synthetic environment to connected users, receive information from users about their actions, take these actions into account in the simulation, and then update the synthetic environment based on those actions and their simulated results. The client-side component of the TSS, i.e., "TSS / C," may be configured to allow users to interact with the TSS. The simulation running on the server (TSS / S) may be configured to provide information about the state of the simulated world to each connected client (TSS / C). For example, the TSS / C running on the MAGIC LEAP Body Computer may utilize information from the TSS / S to determine how to present the world behind the head-mounted component (2) to the eyes and ears. The TSS / C may also be configured to provide input mechanisms and capture sensor data from the user and communicate these to the TSS / S, where the simulation processes these and determines how they modify the state of the simulated world. A training management tool ("TMT") may be configured to provide a suite of tools with access to applications designed to support training / simulation operations.For example, in various embodiments, the TMT may be configured to provide server tools for configuring and deploying instances of the TSS / S to enable warfighters, law enforcement officers, firefighters, and others to rapidly plan and organize training events; a scenario editor used to create or modify training scenarios; an after-action review ("AAR") tool configured to provide users with various mechanisms for replaying training sessions and evaluating their performance in engaging them; a data client that manages access to data captured during training, generates general performance reports, allows arbitrary queries, and creates specialized reports as needed; a simulation editor that provides the ability to modify the underlying rules of the simulation (e.g., to provide ballistic data for new munitions, or to adjust the behavior of synthetic enemies, or fires, chemical spills, etc.); and administration tools for managing user accounts.
[0131] In various embodiments, the training system services may be remotely hosted resources, including, for example, a relatively comprehensive database, which may be referred to as a "data lake" for storage of user accounts and training performance data, a file store for collecting and sharing training scenarios, available server resources dedicated to cloud hosting of the TSS / S training servers, if necessary, and access to what may be referred to as an "Authoritative Lifestream World Map" (or "LWM"), which contains data for use in creating training scenarios and for processing raw data streams captured from the wearable components (2) into a preferred LWM format. The system may also be configured to provide certain "training aids," which may be any items used in support of training with the training system (e.g., training versions of weapons, tasers, fire suppression equipment, and CDC equipment). In various embodiments, the training system itself does not involve custom training tools, but instead provides a mechanism for integrating various equipment into training and representing an extensible ecosystem. In various embodiments, the TSS / S may be a simulation engine of the training environment, configured to generate accurate synthetic representations of the operational environment required to support collective training.This may include realistic explosion and ballistic simulation for weapons and ammunition; hit / injury decisions on synthetic entities and trainees (e.g., the TSS / S understands when a shot hits a synthetic enemy force, a synthetic civilian, or a real-world agent involved in training); general pathfinding functionality that allows synthetic entities to understand traversable and obstructed areas of the simulated space; unit-level behavior (a synthetic entity has knowledge of its surroundings based on simulated sensing, has the ability to move, has reaction times, is affected by conditions (e.g., can be inhibited when fired upon), can make basic decisions (e.g., fight or flee), and has a general proficiency level (e.g., accuracy or training in putting out a fire)); and group-level behavior (using scripts, multiple synthetic entities can decide on a course of action based on if / then conditions and coordinate their behavior). In various embodiments, the TSS / S may be specifically configured to support group behaviors relating to civilians, wildfire vs. home fires, drug dealers, and enemy sniper teams, as well as simulation of audio data of environments, entities, and actions and playback or other output on the head-mounted component (2).
[0132] In various embodiments, a TSS / S may be deployed to conduct a training exercise. A user participating in a training exercise then connects to the TSS / S intended for that exercise. Multiple TSS / S instances may be deployed simultaneously, allowing different groups to conduct different training exercises simultaneously. In various embodiments, the TSS / S may be designed to allow flexibility in deployment. Using a server browser in the TMT, a TSS / S instance may be hosted on a dedicated platform (a server in the cloud reserved for TSS / S use) or on commonly available local hardware (a typical laptop computer).
[0133] Self-hosted, with the TSS / S and TSS / C running on the same platform, is an alternative configuration that allows for standalone training by using only the head-mounted component (2) and computing pack (6) pair.
[0134] These configuration options allow the training system to provide training when needed and in a variety of Internet-enabled environments. With stable Internet access, the TSS / S can be launched in the cloud at the time of perceived need. In the field with zero connectivity, the TSS / S can be on the operator's laptop, and without connectivity (i.e., Internet connectivity via 802.11, or mobile wireless connectivity via 5G, 4G, LTE, and the like), the operator may train alone using only the operator's mobile computing system (2, 6).
[0135] In various embodiments, the goal of the TSS / S is to enable an end-user client-independent system. With properly formatted data, the TSS / S can receive actions from and send simulation state information to any client (the subject matter's wearable computing systems 2, 6, laptops, smartphones, etc.). One intent of this is to accommodate other existing or planned training simulation systems and, after minimal integration, enable maximum compatibility with the present Simulation / Training Environment (which may also be referred to as "STE" for short). In variants where training is performed without network access, the TSS / S may be configured to temporarily store training-related data and will then upload this stored data to the data lake when it is possible to establish a connection to the STE service. At one level in various embodiments, any interaction with the training system occurs on the client and starts with a "launcher". The launcher may be configured to check the user's credentials, using a biometric when accessed via the wearable component (2), or a standard name and password when accessed from a laptop. Once authorized, when a connection to the training service is enabled, the launcher will check for updates to the training-related software and apply these if necessary. The TSS / C may be configured to include a rendering engine that is used to display visual content. In the primary use case, the TSS / C is connected to the TSS / S and the server may be configured to describe the synthetic environment to a client, which projects the world to the user through a wearable headset (2). Alternatively, a version of the rendering engine may run on a laptop or PC or other device, providing a "flat screen", i.e., non-mixed reality rendering of the world data. In various embodiments, the goal of the TSS / C is to enable the user to experience the world data in different modes, supporting different purposes.The mixed reality client display may be configured to be operational only when using a specific head-wearable headset (2). This display mode may employ proprietary spatial computing techniques to insert realistic synthetic content into the user's field of view. In this mode, the user may physically move through real-world space. Such a mode may be used primarily for tactical purposes. The virtual client display may be used with a wearable component (2) or a laptop / PC. On a PC, this mode may be configured to show a flat, non-mixed reality display of world content, similar to that used in existing PC-based training simulations. On a wearable headset (2) such as those described herein, this mode may be configured to present content in a virtual reality mode. Regardless of the display being used, the user may be physically stationary when in this mode and navigate an avatar through a synthetic environment via controls connected to the client (i.e., keyboard and mouse, d-pad, or handheld controller 4). This mode can be used to virtually explore a space (e.g., chunks of LWM data) or control synthetic entities in a training exercise.
[0136] Hybrid client display may be operational only when using a specific headset (2) such as the one described herein. Like mixed reality displays, this display mode utilizes proprietary spatial computing technology. In hybrid display mode, the headset does not attempt to realistically insert content into the user's field of view, but instead presents the physical space without consideration of where it is projected in the real world. In this mode, the user moves through the projected space by physically walking, as in mixed reality display mode. This mode is primarily intended for rehearsals. For example, this mode would allow the user to project part of a village taken from the LWM inside an empty warehouse and then walk around in a scale-accurate projection of the real location.
[0137] In various embodiments, when connecting to a TSS / S for training, a client connects with a specific role assigned to them as a trainee, a trainer, or an observer. The client's role may dictate the information presented to the client and the controls made available to the client. For example, a trainer has the control to view performance data and modify training scenarios, whereas a trainee does not.
[0138] The TSS / C may be configured to manage receipt of user input (e.g., when a trainee fires his or her weapon) and capture and communicate relevant data (user location, rifle orientation, number of shots fired, instructor trigger to end the exercise, etc.) to connected TSS / S.
[0139] When a client is running on the wearable computing headset (2) of the present subject matter, the headset may be configured to capture biometric data, including heart rate, eye gaze, and perceived pupil dilation. This data is used in the AAR to present the instructor with information to resolve performance issues and coach performance improvement. A server browser may be used to find or deploy training servers.
[0140] In various embodiments, when deploying, the user may specify the type of deployment, the authorized participants, and the training scenario to be used. Once this is completed, the user may confirm that the training server is added to the list of available servers.
[0141] Upon connecting to a TSS / S, the user may be presented with a list of known TSS / S instances, along with basic information about each. The user selects a server to connect to for training or observation.
[0142] Options are provided to adjust the advertising and access of deployed servers, allowing everything in between an open "anyone can join" server and a restricted server visible only to certain users.
[0143] The TMT may be configured to provide a training scenario editor that allows users to custom create training exercises.
[0144] Scenario creation may begin with the specification of a geographic location for training. A portion of the LWM can be used for this, or a user can create a mesh using a depth sensor on a headset, such as the subject wearable computing headset (2). Through services provided by the OS, this mesh may be uploaded to the LWM for use by others.
[0145] Once a training area is defined, the scenario editor may be configured to allow a user to select appearances for objects within the training area and place composite entities for training.
[0146] Once completed, a scenario can be saved and loaded at a later time. Scenarios are associated with a training location, allowing others training in the same location to access a library of existing training scenarios. Templates for common exercises may be utilized to guide the creation of standard training scenarios. In addition, so-called "enablers" may be used to modify any training scenario. Enablers may comprise modular "sub-scenarios" that are intended to enable variable training. For example, if a squad destroyed a bunker earlier using a grenade, during the next maneuver exercise, the instructor wants them to execute the objective using CAS. In various embodiments, it is anticipated that nearly anyone can use this editor with minimal training. Underlying simulation data resources may be made accessible to users with sufficient credentials. This data is extensible and can be parameterized, allowing modification of the core simulation "rules". Such configuration allows for rapid modification of existing simulation data as more precise details are reported, and the creation of new simulation entities is introduced as new systems. A "principle editor" may be configured to allow users with sufficient credentials to modify and create behaviors that govern how synthetic entities behave within a simulation.
[0147] In various embodiments, this may be what a trainer would use, for example, to create an "opponent combatant" entity and assign it characteristics that cause it to perform within the simulation in the same manner as would be observed on the battlefield.
[0148] Baseline elements of behavior may be parameterized and easily adjustable (e.g., "These enemies are typically armed with these weapons and demonstrate this level of accuracy at these ranges").
[0149] More complex behavior (eg, defining specific ambush techniques) may require scripting, but may also be handled from within the principle editor.
[0150] Once completed, the behaviors defined here may be saved with the concrete entity. This means that when a user places a concrete entity in the editor, the entity will bring these behaviors with it. Thus, more technical users may define behaviors and less technical users can easily utilize them.
[0151] Training management tools may be configured to provide core administration functionality for the STE. Users with appropriate credentials may use the administration tools to manage user accounts, modify permissions, review logs, foster new application versions, and perform other administrative functions.
[0152] In various embodiments, all users who access STE may have an STE account. Account tools may be used to set up or modify this account, provide reminders about upcoming scheduled workouts, and show performance data.
[0153] "AAR" tools may be accessible by trainers during a training session (to review performance as the training occurs) or from historical data (to review performance of past training or training conducted by different groups). Such tools may be configured to provide trainers with the option to view a replay of the training exercise. This can be done "to scale" (a life-size reproduction of the training) or "on a map" (a top-down view of a scaled-down image of the training).
[0154] The playback controls may be configured to allow modification of the replay so that the trainee can present sections of the training at a slower speed, jump to a different point in the timeline, rewind from a point, or bookmark sections for review.
[0155] Filters may be configured to allow a trainee to visualize detailed performance information within a replay. For example, the system may be utilized to visualize a particular operator's heart rate at a given time, or whether a particular operator passed a given corner when entering a room.
[0156] A data tool may be configured to provide access to training performance data stored within the data lake. Various general reports may be automatically generated from the tool. These may be organized for different levels of access (individual, squad leader, platoon leader, etc.).
[0157] In addition, the tool may be configured to manage access to the data pool and allow for more involved custom-built queries. As there are already many commercially available data visualization tools, this may not be intended to replicate this functionality, but rather to manage the receipt and transmission of the required data to the visualizer.
[0158] An authentication server may be a service that processes authentication requests when a client is launched. If approved, the user is allowed access to other services.
[0159] In denied environments where a user cannot connect to an authentication server, authentication may be configured to occur at a local level, allowing use of only specific devices or local networks.
[0160] Server resources may be reserved for TSS / S use to enable cloud deployment of servers for training events.
[0161] LWM information may be required for several different aspects of the training operation. Access to this information may be managed by the service. Mesh sensor data, in the form of a versioned raw data stream, is sent from the sensor-equipped wearable computing headset (2) to the service and may be pre-processed before sending to the LWM.
[0162] As mentioned above, the cloud-based data storage for the TSE may comprise a "data lake" that may contain, for example, all account information, logs, training performance data, etc. Referring back to Figure 77, the data lake information may be made available from nearly any connected location using an access point such as a secure laptop PC.
[0163] Drive space may be maintained to provide cloud storage of scenarios, application installs, patches, archives, and training backups.
[0164] The various synthetic training environment implementations contemplated herein may combine information about any equipment used by the military. Various examples are listed below.
[0165] Various embodiments relate to an infantry fighting vehicle ("IFV") personnel trainer. This may be a complete personnel trainer, which can be implemented in a vehicle yard without any additional resources in advance of personnel, IFV, wearable computing technology (2, 6), training scenarios, and time. The IFV driver may utilize the wearable system (2, 6) to locate its position in the driver's seat based on pattern recognition of control gauges or visible / scannable markers. This may then provide a world frame for the device to perform digital overlays based on the control layout of the area, which is the same for all positions in the IFV (gunner, commander (TC), loader, and infantry). Here, the wearable computing system (2, 6) may be configured to overlay rendered replicas of control panel readings (gunner's sight, driver's periscope, etc.) and track user movements for user and IFV interaction. The device may then be configured to render the correct stimulus information at the correct depth (as seen through the gunner's sights) so that, for example, the gunner may practice target acquisition and interception while interacting with the rest of the soldier. The device may be configured to track it based on the device's localized location and subsequent head pose. The weapon system may be tracked by the vehicle command, which is recorded through a directional microphone and ear-protecting active noise canceling headphones. This may be configured to locate the enemy that has been found and intercepted, and from which shots are fired. The principle procedure for enemy interception may be to verbally shout to each other, such as: Contact on the infantry's left (gunner (g))-Confirm (TC)-Target acquisition (g)-Fire (TC)-Target destruction (g). The gunner may fire a three-round burst or a multi-second burst depending on the target and ammunition selection (e.g., High Explosive (HE), Armor-Piercing, and Machine Gun). This data may then be collected and transmitted to the TMT, where soldier and user performance is aggregated with previous training, combat, and orthogonal data to update its overall mortality score and training roadmap for future training scenarios.In various embodiments, this may involve taking sensor data from a wearable computing system (2, 6) on the user, creating a dense mesh model of the environment, and also tracking the position / orientation of the user's head within this environment (the user's "head pose"). Such information may be passed through an application programming interface ("API") to the application layer of the system, where a scanning mode may exist in the system that is configured to allow the option to view this 3D mesh locally without any external information. Non-locally, the system may be configured to pull data from other nodes in the intelligence framework, to include position and heading information from other dismounted and mounted Intelligence, Surveillance, and Reconnaissance ("ISR"), and from external sources, to include in the 3D mesh of the world visible by the warfighter. This may include all Army, Air Force, Navy, and Space forces present in a given situation. This data may be time-stamped and geo-tagged so that the translation of where spatial data resides to the user's eye may be constrained by the user's location and line of sight of the wearable computing system (2, 6).
[0166] To correctly overlay information from external sources onto the user's viewpoint, features must be recognized and overlapped in a repeatable manner, otherwise temporal and spatial aliasing artifacts will provide confusing data. For correct overlay, factual data may be used, and a passable world type architecture may be used to segment and orient different objects, both virtual and real, relative to the same location.
[0167] This map of the world, which may originate from external sources to internal sources, may also be used to see what other people are seeing from their device perspective and that of the particular user once the transformation has been calculated. This facilitates functionality such as augmented reality vision through walls, or the viewing of remotely captured information, such as video feeds from unmanned aerial vehicles. Using a red / green / blue ("RGB") or monochrome photo camera of appropriate resolution, the system may be configured to overlay image information onto the dense mesh information to give the user a more contextual understanding of the content contained within the environment shared through Net Warrior or some other information transfer protocol and system.
[0168] Again, referring to law enforcement, fire fighting, and / or hazardous materials environments, locally, similar to the military style implementation described above, the user may collect data from the world through sensors on the head and on the user, create a mesh of the world and overlay it within the real world, and also display a miniature map of the displayed data along with the included interaction elements, all routed through an API and initiated through the application layer of the device for the user. Non-locally, the system may be configured to utilize many of the same types of off-user peripheral devices and information to relay data to the user of the device, such as in a remote forest fire fighting scenario.
[0169] Learning networks, neural networks, and / or so-called "artificial intelligence" (or "AI") computing configurations may be utilized to live stream the adaptive soldier architecture to learn operational information likely to increase mortality, survivability, and mobility. This may be accomplished via machine learning, where the soldier is given a training mission and the model runs a series of parameters and test cases. Based on the output data from the training event, the system may be configured to optimize the heads-up display aspects of the wearable computing system (2, 6) based on the level of data shown to the individual. This is a way to personalize the level of data fidelity displayed to a particular user. Another implementation is the use of machine learning models to dynamically change the data received and displayed in stressful situations to reduce cognitive load on the user. Virtual assistants or artificially synthesized characters such as those depicted in FIG. 90 and described in the previously incorporated references may be utilized to aid in effective communication using the wearable computing configurations (2, 6) of the present subject matter in roles such as general assistants, supervisors, colleagues, and the like.
[0170] Planning capabilities may be improved as well, utilizing the subject system configuration. For example, information about the world may be captured through sensors and then transmitted to an authority server and, once authenticated, to a commander at a headquarters or base. Police, firefighters, military, and / or other services may be able to use such a system configuration to plan en route and situationally rehearse events prior to arriving at the scene or operating environment. With reference to Figures 71-73, the system may be configured to allow a commander to use his connected wearable computing system (2, 6) to keep an eye on everything deployed on his floor or coffee table, for example, to use hand gestures to help control engagements / teams / resources, to use gestures to redirect a team within a building, etc. In other words, the system may be configured such that a commander monitors a multi-perspective information feed through his wearable computing system (2, 6) and then, using the big picture in his head, provides hand gestures to his local device, where a gesture recognition machine learning technology configuration captures the movements, interprets them based on an application definition of the gestures, and performs the desired task based on the previously determined interaction method.
[0171] As mentioned above, the system configuration may involve integration with other communication systems utilized by the operator and may show visualization of various marker types such as obstacles or hazards to avoid, enemy combatant locations and status, all of which may be geo-registered to the operator's operating area using visualized way points, markers, audio, and the like. The wearable system may be configured to create a mesh of the geometry and location of objects around the operator as shown in FIG.
[0172] As mentioned above and explained in the above-incorporated references, global and local dynamic dimming may be utilized to enhance the visibility of various presented images using the head-mounted component (2) in the local environment. With reference to Figs. 74A-74D, global dynamic dimming is illustrated, where the entire display is dimmed or darkened (i.e., darkened in Figs. 74A-74B, and similar results are shown with darkening in Figs. 74C-74D) to help enhance the visibility of certain presented images or content. Localized dynamic dimming specifically relates to contrast adjustment around a particular portion or portions of a presented image or content, such as the area where the operator's eye gaze is directed. Such a configuration may help the operator avoid visual washout, flash blindness, and optical vulnerability to certain weapons, which may be known as "directed energy weapons." Dynamic dimming may also be utilized to selectively prevent light leakage when the operator is in a generally dark environment, since leakage may be detected by the other party, and transiently leaking bright light may cause the user's pupil constriction changes that may be suboptimal for the remainder of viewing through the headset (2) (also known as "maintaining light security" or simply "light security"). Thus, in many scenarios there is a reason to block light leakage in and out of the user's viewing area.
[0173] With reference to Figures 75 and 76, as discussed above, the system may be integrated with various local target visualization and designator tools, such as thermal, infrared, and / or RBG camera configurations, which may be operatively coupled to weapons or other implementations, such as those shown in Figures 81-82, and may also be coupled to a data lake and many other resources through hardware and software configurations, such as those described and illustrated herein, such as Figures 83 and 84.
[0174] With reference to Fig. 78, the head wearable component (2) may be physically integrated with a thermal and / or infrared camera, as discussed above, to otherwise assist in identifying certain nearby objects, such as humans or animals in dark environments, as shown in Fig. 79. Fig. 80 illustrates certain aspects of a low light camera sensor configuration featuring CMOS sensor technology along with firmware, system software, and hardware optimized for low light configurations.
[0175] Thermal imaging devices may be utilized in many of the configurations discussed herein to assist, for example, in identifying the boundaries of a human hand versus other spaces or objects in gesture tracking, observing the attenuation of heat after hand contact such as a handprint on a wall, and removing certain features (such as humans or animals) or "stray objects" for purposes of creating a room mesh (i.e., a human-shaped object at 98° F. may be assumed not to be a wall or other fixed object). Other known objects such as heater vents, windows with sun exposure, water heaters, furnaces, fireplaces, and the like may be identified, and such identification may be utilized to improve the quality, accuracy, and speed with which wearable computing resources (2, 6) may understand the world around them and their associated users. For example, the wearable component (2) may be configured to engage in facial recognition analysis rather than searching the entire room for facial features matching a given database. In one embodiment, the system may be configured to first identify easily accessible faces that are presented uncovered and can be easily thermally identified at approximately 98° F., and then, once one or more faces are identified, to focus more computing resources on those pixels rather than the room in general. In other words, when performing facial recognition, focus on the face rather than everything in a superset of things local to the user (a system capable of using thermal configurations to identify faces to do so would be beneficial). Items may be labeled "Do not touch due to high temperature" for emergency responders or firefighters.An inward facing sensor may be utilized to detect the operator's body temperature, or an outward facing sensor may be utilized to sense other conditions (e.g., to not only indicate the presence of another person across the room, but also that they may have a fever, or that they may be carrying a long gun that may have been recently shot, or perhaps they may have some type of heat-generating object in their bag similar to a recently operated laptop computer, or perhaps the person viewed may have died 8 hours ago based on body temperature). Thermal sensor configurations may also be utilized to tag certain objects, such as dogs, cats, or people, and further aid in visually associating them with certain content or images (e.g., the system may be configured to place a bright red hat on each identified enemy combatant, or a pink mustache that moves with the face of a particular dog or cat). Thermal conditions and gradients may be visualized and color-coded, for example, for firefighters rushing into unknown situations.
[0176] In other embodiments, thermal imaging may be utilized to aid in mapping lighting conditions in a given room (i.e., the location and directionality of one or more lighting sources) to aid in shadow simulation for virtual content presented in AR. Additionally, some objects change shape with temperature changes, providing another data point if that information is known. In various embodiments, if a user with a wearable computing system (2) is in a room and aligned to the room based on visible light camera head pose detection and has certain thermal elements (heater vents, fish tank, one or more lights, electrical appliances, etc.) mapped into this spatial understanding of the room, then if the light suddenly goes out, the user can remain aligned to the room by using the thermal elements. Thermal imaging may also be utilized to aid in horizon determination and matching solar-exposed parts of the world where temperatures are elevated.
[0177] In various embodiments, such as the one illustrated in Fig. 78, it may be beneficial to correct the image for parallax based on the user's actual line of sight, or to perform a real-time or near real-time Jacobian transformation to account for the fact that one or more imaging devices may not be positioned at the same capture location as the user's eye line of sight. This may be done per eye, based on one imaging device, so long as such imaging device captures the field of view for both eyes' lines of sight.
[0178] Various embodiments may also be configured to rapidly identify windows as planar elements that are at least partially transparent to conventional visible light cameras, but that may have time-of-flight signals, scattering, and thermal properties that distinguish them from open space. Traditionally, identifying windows is a particular challenge for computer vision systems.
[0179] Referring to Fig. 85, another embodiment of the wearable component (2) is illustrated with a variety of advanced sensors integrated therein. Various configurations may be coupleable directly to an operator's head, a helmet, the inside of a vehicle, an aircraft, or the like.
[0180] Referring to Fig. 91, a mapping home appliance embodiment is shown, removably coupled to the charger base, where the mapping / imaging unit, when in operation, is configured to create a point cloud of a room and overlay RGB images onto a captured point cloud mesh of such room, updating this in real-time or near real-time. Fig. 92 illustrates a point cloud version of a person's geometry, which may also be viewed in the right light, for example, by an RGB camera.
[0181] 93, an embodiment of the cuff assistant module is illustrated, where a wrist-wearable cuff, similar to a somewhat large watch band, may be coupled to a user when some type of biometric monitoring is desired, such as in an emergency or danger situation, so that others can utilize the data collected and made available (i.e., by cloud connectivity, etc.), to monitor factors such as location (e.g., by GPS, mobile transceiver triangulation, and / or IP address), cardiac condition / output, temperature, etc. The cuff may also contain certain medical components, such as painkillers, that are introduced into the user's body through the cuff, either manually or automatically (i.e., remotely, etc., from a command center or connected colleague). In various embodiments, it may be wirelessly coupled to the wearable computing assets (2, 6) described herein.
[0182] Mixed Reality Environment
[0183] Like all people, users of a mixed reality system exist in a real environment, i.e., the three-dimensional portion of the "real world" and all of its content are perceivable by the user. For example, the user perceives the real environment using normal human senses, i.e., sights, sound, touch, taste, and smell, and interacts with the real environment by moving his or her body within the real environment. Locations within the real environment can be described as coordinates in a coordinate space. For example, coordinates can include latitude, longitude, and altitude relative to sea level, distance in three orthogonal dimensions from a reference point, or other suitable values. Similarly, a vector can describe a quantity, having a direction and magnitude in a coordinate space.
[0184] The computing device may maintain a representation of the virtual environment, for example, in a memory associated with the device. As used herein, a virtual environment is a computed representation of a three-dimensional space. The virtual environment may include representations of any objects, actions, signals, parameters, coordinates, vectors, or other properties associated with that space. In some examples, the circuitry (e.g., a processor) of the computing device may maintain and update the state of the virtual environment. That is, the processor may determine the state of the virtual environment at a second time t1 based on data associated with the virtual environment and / or input provided by a user at a first time t0. For example, if an object in the virtual environment is located at a first coordinate at time t0 and has certain programmed physical parameters (e.g., mass, coefficient of friction), and input received from a user indicates that a force should be applied to the object in a certain directional vector, the processor may apply the laws of kinematics and use basic mechanics to determine the location of the object at time t1. The processor may use any suitable information known about the virtual environment and / or any suitable input to determine the state of the virtual environment at time t1. In maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software related to creating and deleting virtual objects in the virtual environment, software (e.g., scripts) for defining behavior of virtual objects or characters in the virtual environment, software for defining behavior of signals (e.g., audio signals) in the virtual environment, software for creating and updating parameters associated with the virtual environment, software for generating audio signals in the virtual environment, software for handling inputs and outputs, software for implementing network operations, software for applying asset data (e.g., animation data for moving a virtual object over time), or many other possibilities.
[0185] An output device, such as a display or speaker, can present any or all aspects of the virtual environment to the user. For example, the virtual environment may include virtual objects (which may include representations of inanimate objects, people, animals, lights, etc.) that may be presented to the user. The processor can determine a view of the virtual environment (e.g., corresponding to a "camera," with its origin coordinates, viewing axis, and frustum) and render on the display a viewable scene of the virtual environment corresponding to that view. Any suitable rendering technique may be used for this purpose. In some examples, the viewable scene may include only some virtual objects in the virtual environment and exclude certain other virtual objects. Similarly, the virtual environment may include audio aspects that may be presented to the user as one or more audio signals. For example, a virtual object in the virtual environment may generate sounds originating from the object's location coordinates (e.g., a virtual character may speak or create a sound effect), or the virtual environment may be associated with musical cues or ambient sounds that may or may not be associated with a particular location. The processor can determine audio signals corresponding to "listener" coordinates, e.g., audio signals corresponding to the synthesis of sounds in the virtual environment and that are mixed and processed to simulate the audio signals that would be heard by a listener at the listener coordinates, and present the audio signals to the user via one or more speakers.
[0186] Because the virtual environment exists only as a computational structure, the user cannot directly perceive the virtual environment using normal senses. Instead, the user can only indirectly perceive the virtual environment, as presented to the user, for example, by a display, a speaker, a tactile output device, etc. Similarly, the user cannot directly touch, manipulate, or otherwise interact with the virtual environment, but input data can be provided via an input device or sensor to a processor, which can use the device or sensor data to update the virtual environment. For example, a camera sensor can provide optical data indicating that a user is attempting to move an object in the virtual environment, and the processor can use that data to cause the object to respond appropriately in the virtual environment. As another example, an accelerometer, gyroscope sensor, or other type of sensor in a peripheral device (e.g., controller 300 described below) can provide motion data and / or device orientation data indicating that a user is attempting to manipulate an object in the virtual environment using the peripheral device, and the processor can use that data to cause the object to respond appropriately in the virtual environment. Other suitable types of sensor data may also be employed.
[0187] A mixed reality system can present a user with a mixed reality environment ("MRE") that combines aspects of real and virtual environments, for example, using a see-through display and / or one or more speakers (which may be incorporated, for example, into a wearable head device). In some embodiments, the one or more speakers may be external to the head-mounted wearable unit. As used herein, an MRE is a simultaneous representation of a real environment and a corresponding virtual environment. In some examples, the corresponding real and virtual environments share a single coordinate space. In some examples, the real coordinate space and the corresponding virtual coordinate space are related to each other by a transformation matrix (or other suitable representation). Thus, a single coordinate (in some examples, together with the transformation matrix) may define a first location in the real environment and also a second corresponding location in the virtual environment, and vice versa.
[0188] In an MRE, a virtual object (e.g., in a virtual environment associated with the MRE) may correspond to a real object (e.g., in a real environment associated with the MRE). For example, if the real environment of the MRE includes a real lamppost (real object) at some location coordinates, the virtual environment of the MRE may include a virtual lamppost (virtual object) at the corresponding location coordinates. As used herein, a real object combines with its corresponding virtual object to comprise a "mixed reality object." It is not necessary for a virtual object to perfectly match or match a corresponding real object. In some examples, a virtual object can be a simplified version of a corresponding real object. For example, if the real environment includes a real lamppost, the corresponding virtual object may include a cylinder of approximately the same height and radius as the real lamppost (reflecting that a lamppost may be approximately cylindrical in shape). Simplifying virtual objects in this manner can enable computational efficiencies and simplify calculations to be performed on such virtual objects. Furthermore, in some examples of an MRE, not all real objects in the real environment may be associated with a corresponding virtual object. Similarly, in some embodiments of the MRE, not all virtual objects in the virtual environment may be associated with corresponding real objects, i.e., some virtual objects may exist solely within the virtual environment of the MRE without any real-world counterpart.
[0189] In some embodiments, virtual objects may have characteristics that are sometimes significantly different from those of the corresponding real object. For example, a real environment in an MRE may contain a green, two-pronged cactus, a thorny, inanimate object, while the corresponding virtual object in the MRE may have the characteristics of a green, two-armed virtual character with human facial features and a surly attitude. In this embodiment, the virtual object resembles its corresponding real object in some characteristics (color, number of arms) but differs from the real object in other characteristics (facial features, personality). In this way, virtual objects have the potential to represent real objects in creative, abstract, exaggerated, or fictitious ways, or to impart behaviors (e.g., human personality) to otherwise inanimate real objects. In some embodiments, a virtual object may be a purely fictitious creation with no real-world counterpart (e.g., a virtual monster in a virtual environment, perhaps in a location that corresponds to a void in the real environment).
[0190] Compared to a VR system that presents a virtual environment to a user while obscuring the real environment, a mixed reality system that presents an MRE offers the advantage that the real environment remains perceptible while the virtual environment is presented. Thus, a user of a mixed reality system can experience and interact with a corresponding virtual environment using visual and audio cues associated with the real environment. As an example, a user of a VR system may struggle to perceive or interact with a virtual object displayed in the virtual environment because, as mentioned above, the user cannot directly perceive or interact with the virtual environment, whereas a user of an MR system may find it intuitive and natural to interact with a virtual object by seeing, hearing, and touching the corresponding real object in his or her real environment. This level of interaction may enhance the user's sense of immersion, connection, and engagement with the virtual environment. Similarly, by presenting a real environment and a virtual environment simultaneously, a mixed reality system may reduce the negative psychological sensations (e.g., cognitive dissonance) and negative physical sensations (e.g., motion sickness) associated with a VR system. Mixed reality systems also offer many possibilities for applications that can augment or modify our experience of the real world.
[0191] FIG. 105A illustrates an exemplary real environment 100 in which a user 110 uses a mixed reality system 112. The mixed reality system 112 may include a display (e.g., a see-through display) and one or more speakers, and one or more sensors (e.g., cameras), for example, as described below. The illustrated real environment 100 includes a rectangular room 104A in which the user 110 is standing, and real objects 122A (lamp), 124A (table), 126A (sofa), and 128A (painting). The room 104A further includes a location coordinate 106, which may be considered as the origin of the real environment 100. As shown in FIG. 105A, an environment / world coordinate system 108 (with an x-axis 108X, a y-axis 108Y, and a z-axis 108Z), with its origin at point 106 (world coordinates), may define a coordinate space for the real environment 100. In some embodiments, the origin 106 of the environment / world coordinate system 108 may correspond to where the mixed reality system 112 was powered on. In some embodiments, the origin 106 of the environment / world coordinate system 108 may be reset during operation. In some examples, the user 110 may be considered a real object in the real environment 100. Similarly, the body parts (e.g., hands, feet) of the user 110 may be considered real objects in the real environment 100. In some examples, the user / listener / head coordinate system 114 (comprising an x-axis 114X, a y-axis 114Y, and a z-axis 114Z) with its origin at point 115 (e.g., user / listener / head coordinate) may define a coordinate space for the user / listener / head on which the mixed reality system 112 is located. The origin 115 of the user / listener / head coordinate system 114 may be defined with respect to one or more components of the mixed reality system 112. For example, the origin 115 of the user / listener / head coordinate system 114 may be defined relative to the display of the mixed reality system 112, such as during an initial calibration of the mixed reality system 112. A matrix (which may include a transformation matrix and a quaternion or other rotation matrix) or other suitable representation may characterize the transformation between the user / listener / head coordinate system 114 space and the environment / world coordinate system 108 space.In some embodiments, the left ear coordinates 116 and right ear coordinates 117 may be defined relative to the origin 115 of the user / listener / head coordinate system 114. Matrices (which may include transformation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the left ear coordinates 116 and right ear coordinates 117 and the user / listener / head coordinate system 114 space. The user / listener / head coordinate system 114 can simplify the representation of location relative to the user's head or head-mounted device, for example, relative to the environment / world coordinate system 108. Using simultaneous localization and mapping (SLAM), visual odometry, or other techniques, the transformation between the user coordinate system 114 and the environment coordinate system 108 can be determined and updated in real time.
[0192] FIG. 105B illustrates an example virtual environment 130 that corresponds to the real environment 100. The shown virtual environment 130 includes a virtual rectangular room 104B that corresponds to the real rectangular room 104A, a virtual object 122B that corresponds to the real object 122A, a virtual object 124B that corresponds to the real object 124A, and a virtual object 126B that corresponds to the real object 126A. Metadata associated with the virtual objects 122B, 124B, 126B may include information derived from the corresponding real objects 122A, 124A, 126A. The virtual environment 130 additionally includes a virtual monster 132, which does not correspond to any real object in the real environment 100. A real object 128A in the real environment 100 does not correspond to any virtual object in the virtual environment 130. A persistent coordinate system 133 (with x-axis 133X, y-axis 133Y, and z-axis 133Z) with its origin at point 134 (persistent coordinate) may define a coordinate space for the virtual content. The origin 134 of the persistent coordinate system 133 may be defined relative to / with respect to one or more real objects, such as real object 126A. A matrix (which may include a transformation matrix and a quaternion matrix or other rotation matrix) or other suitable representation may characterize the transformation between the persistent coordinate system 133 space and the environment / world coordinate system 108 space. In some embodiments, virtual objects 122B, 124B, 126B, and 132 may each have their own persistent coordinate point relative to the origin 134 of the persistent coordinate system 133. In some embodiments, there may be multiple persistent coordinate systems, and virtual objects 122B, 124B, 126B, and 132 may each have their own persistent coordinate point relative to one or more persistent coordinate systems.
[0193] With reference to Figures 105A and 105B, the environment / world coordinate system 108 defines a shared coordinate space for both the real environment 100 and the virtual environment 130. In the illustrated embodiment, the coordinate space has its origin at point 106. Furthermore, the coordinate space is defined by the same three orthogonal axes (108X, 108Y, 108Z). Thus, a first location in the real environment 100 and a second corresponding location in the virtual environment 130 can be described with respect to the same coordinate space. This simplifies the steps of identifying and displaying corresponding locations in the real and virtual environments, since the same coordinates can be used to identify both locations. However, in some embodiments, the corresponding real and virtual environments need not use a shared coordinate space. For example, in some embodiments (not shown), matrices (which may include transformation matrices and quaternion matrices or other rotation matrices) or other suitable representations can characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.
[0194] 105C illustrates an exemplary MRE 150 that simultaneously presents aspects of real environment 100 and virtual environment 130 to user 110 via mixed reality system 112. In the example shown, MRE 150 simultaneously presents to user 110 real objects 122A, 124A, 126A, and 128A from real environment 100 (e.g., via a transparent portion of a display of mixed reality system 112) and virtual objects 122B, 124B, 126B, and 132 from virtual environment 130 (e.g., via an active display portion of a display of mixed reality system 112). As described above, origin 106 acts as the origin for a coordinate space corresponding to MRE 150, and coordinate system 108 defines the x-, y-, and z-axes for the coordinate space.
[0195] In the illustrated example, the mixed reality objects include corresponding pairs of real and virtual objects (i.e., 122A / 122B, 124A / 124B, 126A / 126B) that occupy corresponding locations in coordinate space 108. In some examples, both real and virtual objects may be visible to user 110 at the same time. This may be desirable in instances where, for example, a virtual object presents information designed to augment the view of the corresponding real object (such as in a museum application where a virtual object presents a missing portion of an ancient damaged statue). In some examples, the virtual objects (122B, 124B, and / or 126B) may be displayed so as to occlude the corresponding real objects (122A, 124A, and / or 126A) (e.g., via active pixelated occlusion using a pixelated occlusion shutter). This may be desirable, for example, in instances where a virtual object acts as a visual replacement for a corresponding real object (such as in interactive storytelling applications where inanimate real objects become "living" characters).
[0196] In some examples, real objects (e.g., 122A, 124A, 126A) may be associated with virtual content or helper data that does not necessarily constitute a virtual object. The virtual content or helper data may facilitate processing or handling of the virtual object within a mixed reality environment. For example, such virtual content may include a two-dimensional representation of the corresponding real object, a custom asset type associated with the corresponding real object, or statistical data associated with the corresponding real object. This information may enable or facilitate calculations involving the real object without incurring unnecessary computational overhead.
[0197] In some embodiments, the presentation described above may also incorporate audio aspects. For example, in MRE 150, virtual monster 132 may be associated with one or more audio signals, such as footstep effects, that are generated as the monster walks around MRE 150. As described further below, a processor in mixed reality system 112 may calculate an audio signal that corresponds to a mixed and processed combination of all such sounds within MRE 150 and present the audio signal to user 110 via one or more speakers included within mixed reality system 112 and / or one or more external speakers.
[0198] Exemplary Mixed Reality System
[0199] An exemplary mixed reality system 112 may include a wearable head device (e.g., a wearable augmented reality or mixed reality head device) with a display (which may include left and right transmissive displays, which may be eyepiece displays, and associated components for coupling light from the displays to the user's eyes), left and right speakers (e.g., positioned adjacent the user's left and right ears, respectively), an inertial measurement unit (IMU) (e.g., mounted on a temple arm of the head device), a quadrature coil electromagnetic receiver (e.g., mounted on the left temple piece), left and right cameras (e.g., depth (time of flight) cameras) oriented away from the user, and left and right eye cameras (e.g., for detecting the user's eye movements) oriented toward the user. However, the mixed reality system 112 may incorporate any suitable display technology and any suitable sensors (e.g., optical, infrared, acoustic, LIDAR, EOG, GPS, magnetic). In addition, the mixed reality system 112 may incorporate networking features (e.g., Wi-Fi capabilities) to communicate with other devices and systems, including other mixed reality systems. The mixed reality system 112 may further include a battery (which may be mounted in an auxiliary unit, such as a belt pack designed to be worn around the waist of the user), a processor, and a memory. The wearable head device of the mixed reality system 112 may include a tracking component, such as an IMU or other suitable sensor, configured to output a set of coordinates of the wearable head device relative to the user's environment. In some examples, the tracking component may provide input to the processor and implement simultaneous localization and mapping (SLAM) and / or visual odometry algorithms. In some examples, the mixed reality system 112 may also include a handheld controller 300 and / or an auxiliary unit 320, which may be a wearable belt pack, as described further below.
[0200] 2A-2D illustrate components of an example mixed reality system 200 (which may correspond to the mixed reality system 112) that may be used to present an MRE (which may correspond to the MRE 150) or other virtual environment to a user. FIG. 106A illustrates a perspective view of a wearable head device 2102 included within the example mixed reality system 200. FIG. 106B illustrates a top view of the wearable head device 2102 worn on a user's head 2202. FIG. 106C illustrates a front view of the wearable head device 2102. FIG. 106D illustrates an edge view of an example eyepiece 2110 of the wearable head device 2102. As shown in FIGS. 2A-2C , the exemplary wearable head device 2102 includes an exemplary left eyepiece (e.g., left transparent waveguide set eyepiece) 2108 and an exemplary right eyepiece (e.g., right transparent waveguide set eyepiece) 2110. Each eyepiece 2108 and 2110 can include a transmissive element through which the real environment is visible, and a display element for presenting a display (e.g., via image-wise modulated light) that is superimposed on the real environment. In some examples, such a display element can include a surface diffractive optical element for controlling the flow of image-wise modulated light. For example, the left eyepiece 2108 can include a left internal coupling grating set 2112, a left orthogonal pupil expansion (OPE) grating set 2120, and a left exit (output) pupil expansion (EPE) grating set 2122. Similarly, the right eyepiece 2110 can include a right internal coupling grating set 2118, a right OPE grating set 2114, and a right EPE grating set 2116. The light modulated for each image can be transferred to the user's eye via the internal coupling gratings 2112 and 2118, the OPEs 2114 and 2120, and the EPEs 2116 and 2122. Each internal coupling grating set 2112, 2118 can be configured to deflect light toward its corresponding OPE grating set 2120, 2114. Each OPE grating set 2120, 2114 can be designed to progressively deflect light downward toward its associated EPE 2122, 2116, thereby extending the exit pupil formed horizontally.Each EPE 2122, 2116 can be configured to progressively redirect at least a portion of the light received from its corresponding OPE grating set 2120, 2114 outwardly to a user eyebox location (not shown), defined behind the eyepieces 2108, 2110, to vertically extend an exit pupil formed in the eyebox. Alternatively, instead of the internal coupling grating sets 2112 and 2118, the OPE grating sets 2114 and 2120, and the EPE grating sets 2116 and 2122, the eyepieces 2108 and 2110 can include gratings and / or other arrangements of refractive and reflective features to control the coupling of the image-wise modulated light into the user's eye.
[0201] In some embodiments, the wearable head device 2102 can include a left temple arm 2130 and a right temple arm 2132, with the left temple arm 2130 including a left speaker 2134 and the right temple arm 2132 including a right speaker 2136. A quadrature coil electromagnetic receiver 2138 can be located in the left temple piece or another suitable location in the wearable head unit 2102. An inertial measurement unit (IMU) 2140 can be located in the right temple arm 2132 or another suitable location in the wearable head device 2102. The wearable head device 2102 can also include a left depth (e.g., time-of-flight) camera 2142 and a right depth camera 2144. The depth cameras 2142, 2144 can be preferably oriented in different directions so that both cover a wider field of view.
[0202] 2A-2D, a left source of image-wise modulated light 2124 can be optically coupled into the left eyepiece 2108 through a left internal coupling grating set 2112, and a right source of image-wise modulated light 2126 can be optically coupled into the right eyepiece 2110 through a right internal coupling grating set 2118. The source of image-wise modulated light 2124, 2126 can include, for example, a fiber optic scanner, a projector including an electronic light modulator such as a digital light processing (DLP) chip or a liquid crystal on silicon (LCoS) modulator, or an emissive display such as a micro light emitting diode (μLED) or micro organic light emitting diode (μOLED) panel that is coupled into the internal coupling grating sets 2112, 2118 using one or more lenses per side. The input coupling grating sets 2112, 2118 can deflect light from the source of image-wise modulated light 2124, 2126 to an angle above the critical angle for total internal reflection (TIR) for the eyepieces 2108, 2110. The OPE grating sets 2114, 2120 progressively deflect the propagating light downwards by TIR towards the EPE grating sets 2116, 2122. The EPE grating sets 2116, 2122 progressively couple the light towards the user's face, including the pupils of the user's eyes.
[0203] In some examples, as shown in FIG. 2D , the left eyepiece 2108 and the right eyepiece 2110 each include multiple waveguides 2402. For example, each eyepiece 2108, 2110 can include multiple individual waveguides, each dedicated to a separate color channel (e.g., red, blue, and green). In some examples, each eyepiece 2108, 2110 can include multiple sets of such waveguides, each set configured to impart a different wavefront curvature to the emitted light. The wavefront curvature may be convex with respect to the user's eye, for example, to present a virtual object located at a distance in front of the user (e.g., a distance corresponding to the inverse of the wavefront curvature). In some examples, the EPE grating sets 2116, 2122 can include curved grating grooves to provide a convex wavefront curvature by modifying the Poynting vector of the light exiting across each EPE.
[0204] In some examples, stereoscopically accommodated left and right eye images can be presented to the user through per-image light modulators 2124, 2126 and eyepieces 2108, 2110 to create the perception that the displayed content is three-dimensional. The perceived realism of the presentation of three-dimensional virtual objects can be enhanced by selecting the waveguides (and thus the corresponding wavefront curvatures) such that the virtual objects are displayed at distances that approximate the distances shown by the stereoscopic left and right images. The technique can also reduce motion sickness suffered by some users, which can be caused by differences between the depth perception cues provided by the stereoscopic left and right eye images and the automatic accommodation (e.g., object distance-dependent focus) of the human eye.
[0205] FIG. 106D illustrates an edge view from the top of the right eyepiece 2110 of the exemplary wearable head device 2102. As shown in FIG. 2D, the plurality of waveguides 2402 can include a first subset of three waveguides 2404 and a second subset of three waveguides 2406. The two subsets of waveguides 2404, 2406 can be differentiated by different EPE gratings that feature different grating line curvatures to impart different wavefront curvatures to the exiting light. Within each of the subsets of waveguides 2404, 2406, each waveguide can be used to couple a different spectral channel (e.g., one of the red, green, and blue spectral channels) to the user's right eye 2206. Although not shown in FIG. 2D, the structure of the left eyepiece 2108 is similar to that of the right eyepiece 2110.
[0206] FIG. 107A illustrates an example handheld controller component 300 of a mixed reality system 200. In some examples, the handheld controller 300 includes a grip portion 346 and one or more buttons 350 disposed along a top surface 348. In some examples, the buttons 350 may be configured for use as an optical tracking target to track six degrees of freedom (6DOF) movement of the handheld controller 300, for example, in conjunction with a camera or other optical sensor (which may be mounted in a head unit (e.g., wearable head device 2102) of the mixed reality system 200). In some examples, the handheld controller 300 includes a tracking component (e.g., an IMU or other suitable sensor) for detecting a position or orientation, such as a position or orientation relative to the wearable head device 2102. In some examples, such a tracking component may be positioned in a handle of the handheld controller 300 and / or may be mechanically coupled to the handheld controller. The handheld controller 300 can be configured to provide one or more output signals corresponding to one or more of a button press state, or a position, orientation, and / or movement (e.g., via an IMU) of the handheld controller 300. Such output signals may be used as inputs to a processor of the mixed reality system 200. Such inputs may correspond to the position, orientation, and / or movement of the handheld controller (or, for that matter, the position, orientation, and / or movement of a user's hand holding the controller). Such inputs may also correspond to a user pressing a button 350.
[0207] FIG. 107B illustrates an example auxiliary unit 320 of the mixed reality system 200. The auxiliary unit 320 can include a battery for providing energy and operating the system 200, and can include a processor for executing programs and operating the system 200. As shown, the example auxiliary unit 320 includes a clip 2128 for attaching the auxiliary unit 320 to a user's belt, etc. It will be apparent that other form factors are also suitable for the auxiliary unit 320, including form factors that do not involve mounting the unit on a user's belt. In some embodiments, the auxiliary unit 320 is coupled to the wearable head device 2102 through a multi-tube cable, which may include, for example, electrical wires and optical fibers. A wireless connection between the auxiliary unit 320 and the wearable head device 2102 can also be used.
[0208] In some examples, the mixed reality system 200 can include one or more microphones to detect sound and provide a corresponding signal to the mixed reality system. In some examples, the microphones may be attached to or integrated with the wearable head device 2102 and configured to detect the user's voice. In some examples, the microphones may be attached to or integrated with the handheld controller 300 and / or the auxiliary unit 320. Such microphones may be configured to detect environmental sounds, ambient noise, the user's or a third party's voice, or other sounds.
[0209] FIG. 108 illustrates an example functional block diagram that may correspond to an example mixed reality system, such as the mixed reality system 200 described above (which may correspond to the mixed reality system 112 with respect to FIG. 1). As shown in FIG. 108, the example handheld controller 400B (which may correspond to the handheld controller 300 ("totem")) includes a totem / wearable head device six degree of freedom (6DOF) totem subsystem 404A, and the example wearable head device 400A (which may correspond to the wearable head device 2102) includes a totem / wearable head device 6DOF subsystem 404B. In an example, the 6DOF totem subsystem 404A and the 6DOF subsystem 404B cooperate to determine six coordinates (e.g., offsets in three translation directions and rotations along three axes) of the handheld controller 400B relative to the wearable head device 400A. The six degrees of freedom may be expressed relative to the coordinate system of the wearable head device 400A. The three transformation offsets may be represented as X, Y, and Z offsets in such coordinate system, a transformation matrix, or some other representation. The rotational degrees of freedom may be represented as a sequence of yaw, pitch, and roll rotations, a rotation matrix, a quaternion, or some other representation. In some examples, the wearable head device 400A, one or more depth cameras 444 (and / or one or more non-depth cameras) included within the wearable head device 400A, and / or one or more optical targets (e.g., buttons 350 of handheld controller 400B as described above or dedicated optical targets included within handheld controller 400B) may be used for 6DOF tracking. In some examples, the handheld controller 400B may include a camera as described above, and the wearable head device 400A may include an optical target for optical tracking in conjunction with the camera. In some embodiments, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids that are used to wirelessly transmit and receive three distinguishable signals.By measuring the relative magnitudes of the three distinguishable signals received in each of the coils used to receive, the 6DOF of the wearable head device 400A relative to the handheld controller 400B can be determined. Additionally, the 6DOF totem subsystem 404A can include an inertial measurement unit (IMU), which is useful for providing improved accuracy and / or more timely information regarding high speed movements of the handheld controller 400B.
[0210] In some examples, it may be necessary to transform coordinates from a local coordinate space (e.g., a coordinate space fixed relative to the wearable head device 400A) to an inertial coordinate space (e.g., a coordinate space fixed relative to the real environment), e.g., to compensate for movement of the wearable head device 400A relative to the coordinate system 108. For example, such a transformation may be necessary so that the display of the wearable head device 400A presents virtual objects in an expected position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair facing forward, regardless of the position and orientation of the wearable head device), rather than in a fixed position and orientation on the display (e.g., the same position in the bottom right corner of the display), preserving the illusion that the virtual objects are present in the real environment (and do not appear unnaturally positioned in the real environment, e.g., as the wearable head device 400A shifts and rotates). In some examples, a compensation transformation between coordinate spaces can be determined by processing images from the depth camera 444 using SLAM and / or visual odometry procedures to determine a transformation of the wearable head device 400A relative to the coordinate system 108. In the example shown in FIG. 108, the depth camera 444 can be coupled to the SLAM / visual odometry block 406 and provide images to the block 406. The SLAM / visual odometry block 406 implementation can include a processor configured to process the images and then determine a position and orientation of the user's head, which can be used to identify a transformation between the head coordinate space and another coordinate space (e.g., an inertial coordinate space). Similarly, in some examples, an additional source of information regarding the user's head pose and location is obtained from the IMU 409. Information from the IMU 409 can be integrated with information from the SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information regarding fast adjustments of the user's head pose and position.
[0211] In some examples, the depth camera 444 can provide 3D images to a hand gesture tracker 411, which can be implemented within a processor of the wearable head device 400A. The hand gesture tracker 411 can identify the user's hand gestures, for example, by matching the 3D images received from the depth camera 444 to stored patterns representing hand gestures. Other suitable techniques for identifying the user's hand gestures will also be apparent.
[0212] In some embodiments, the one or more processors 416 may be configured to receive data from the 6DOF headgear subsystem 404B, the IMU 409, the SLAM / visual odometry block 406, the depth camera 444, and / or the hand gesture tracker 411 of the wearable head device. The processor 416 may also send and receive control signals to and from the 6DOF totem system 404A. The processor 416 may be wirelessly coupled to the 6DOF totem system 404A, such as in embodiments where the handheld controller 400B is not tethered. The processor 416 may further communicate with additional components, such as an audiovisual content memory 418, a graphical processing unit (GPU) 420, and / or a digital signal processor (DSP) audio spatializer 422. The DSP audio spatializer 422 may be coupled to a head-related transfer function (HRTF) memory 425. The GPU 420 may include a left channel output coupled to a left source of imagewise modulated light 424 and a right channel output coupled to a right source of imagewise modulated light 426. The GPU 420 may output stereoscopic image data to the sources of imagewise modulated light 424, 426, for example, as described above with respect to FIGS. 2A-2D. The DSP audio spatializer 422 may output audio to the left speaker 412 and / or the right speaker 414. The DSP audio spatializer 422 may receive an input from the processor 419 indicating a direction vector from the user to a virtual sound source (e.g., which may be moved by the user via the handheld controller 320). Based on the direction vector, the DSP audio spatializer 422 may determine a corresponding HRTF (e.g., by accessing an HRTF or by interpolating multiple HRTFs). The DSP audio spatializer 422 may then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object.This can improve the believability and realism of virtual sounds by incorporating the user's relative position and orientation to the virtual sounds in the mixed reality environment, i.e., by presenting virtual sounds that match the user's expectations of what they would hear if the virtual sounds were real sounds in a real environment.
[0213] In some implementations, such as that shown in FIG. 108, one or more of the processor 416, GPU 420, DSP audio spatializer 422, HRTF memory 425, and audio / visual content memory 418 may be included within an auxiliary unit 400C (which may correspond to auxiliary unit 320 described above). The auxiliary unit 400C may include a battery 427 to power its components and / or provide power to the wearable head device 400A or handheld controller 400B. Including such components within an auxiliary unit, which may be mounted on the user's waist, can limit the size and weight of the wearable head device 400A, which in turn can reduce fatigue in the user's head and neck.
[0214] FIG. 108 presents elements corresponding to various components of an exemplary mixed reality system, although various other suitable arrangements of these components will be apparent to those skilled in the art. For example, elements presented in FIG. 108 as being associated with the auxiliary unit 400C may instead be associated with the wearable head device 400A or the handheld controller 400B. Further, some mixed reality systems may dispense with the handheld controller 400B or the auxiliary unit 400C entirely. Such variations and modifications should be understood as falling within the scope of the disclosed embodiments.
[0215] System Architecture
[0216] The goal of a mixed reality system may be to synthesize real-world information and provide a user with information that may not otherwise be available and / or easily accessible to the user. The benefits of a mixed reality system may be particularly evident when a user needs to process a large amount of information within a short time period to make a decision. In these situations, a mixed reality system is well suited to aid the user in decision making, particularly by making relevant information easily accessible to the user. For example, in a combat situation, a soldier may benefit from information such as the remaining ammunition in his weapon's magazine, the location of squad members, and / or the location of enemy combatants. This information, which may otherwise be unavailable to the soldier, can be presented to the soldier, for example, via a see-through display of the mixed reality system.
[0217] Because a mixed reality system may receive and analyze large amounts of real-world information, an efficient computing architecture may result in an improved user experience and increased capabilities for the mixed reality system. An efficient computing architecture may enable the mixed reality system to rapidly process inputs from various real-world information sources. For example, referring back to FIG. 8, a mixed reality system comprising a headset 2 and a beltpack 6 (which may correspond to an MR system 112, 200) may receive real-world inputs from one or more weapons, which may be communicatively coupled to the mixed reality system. The one or more weapons may each include one or more sensors coupled to a microprocessor. For example, the sensor may detect the amount of ammunition remaining in a magazine, and that information may be transmitted to the microprocessor. The weapon's microprocessor may communicate this information to the mixed reality system via wired and / or wireless communication. For example, the weapon's microprocessor may communicate with the mixed reality system using a universal serial bus ("USB") connection and / or radio frequency communication (e.g., WiFi, Bluetooth, cellular networks, 5G).
[0218] In some embodiments, the mixed reality system may be more efficient to communicate with devices that include a microprocessor. For example, a firefighter kit monitor may include a fire extinguisher, which may include one or more sensors configured to detect the amount of suppressant remaining in the extinguisher. In some embodiments, the fire extinguisher may include a microprocessor configured to communicate with the one or more sensors. The microprocessor may be configured to receive raw inputs (e.g., voltage levels) from the sensors and thus calculate the amount of suppressant remaining. The extinguisher's microprocessor may then communicate this information to the mixed reality system in a data structure that the mixed reality system can read. This computing architecture may reduce the computational burden placed on the mixed reality system by offloading one or more calculations to the microprocessor of the connected device.
[0219] In some embodiments, it may be more efficient for the mixed reality system to communicate directly with sensors on the connected devices. For example, a cuff assistant device may be worn on the user's wrist, and the cuff assistant may include a sensor configured to measure the user's heart rate. In some embodiments, the sensor may communicate directly with the mixed reality system (e.g., without an intermediate device microprocessor) using wired and / or wireless means. The mixed reality system may be configured to receive raw inputs (e.g., voltage levels) from the sensor and thus calculate the heart rate. It may be more economically efficient for the connected devices to include sensors that communicate directly with the mixed reality system, omitting individual microprocessors. In some embodiments, omitting a device-specific microprocessor may reduce manufacturing costs for the connected devices, but may increase the computational burden on the mixed reality system that serves as a hub for the connected devices. For example, the mixed reality system may be required to calculate the heart rate from voltage readings instead of offloading the calculation to a device-specific microprocessor.
[0220] FIG. 57 illustrates an example computing architecture for large-scale mixed reality applications. In some embodiments, multiple mixed reality systems (e.g., MR systems 112, 200) can be interconnected and / or connected to external computing systems (e.g., cloud-based servers). The interconnected computing architecture can enable large-scale mixed reality applications that may rely on large amounts of data and computing resources. In some embodiments, an efficient computing architecture can balance the need for fast, low-latency computations with complex, processing-intensive computations. In some embodiments, the efficient computing architecture may implement edge computing principles. Edge computing principles can include dividing computing tasks across multiple systems and / or strategically locating computing systems to minimize communication latency between systems.
[0221] For example, the exemplary computing architecture illustrated in FIG. 57 may be used in military training exercises. In large-scale mixed reality war gaming applications, many soldiers (e.g., 100, 1,000, or 10,000) may be deployed to test, develop, and / or implement military strategies. Each soldier may be equipped with its own mixed reality system, which may include a head-mounted device (e.g., MR system 112, 200). Each mixed reality system may capture a variety of real-world data. For example, each mixed reality system may capture soldier vitals, remaining ammunition, soldier position, communications, weapon status, etc. The mixed reality system may also capture information about the location of real objects and / or the soldier's location / orientation relative to real objects (e.g., via SLAM). In some embodiments, information captured by the mixed reality systems (e.g., the mixed reality systems used by the individual soldiers) may communicate with each other and / or with external computing systems (e.g., one or more servers). Mixed reality systems may communicate with each other and / or with external computing systems via any suitable means. For example, mixed reality systems may communicate via wired and / or wireless connections, including, but not limited to, USB connections, cellular networks, WiFi, Bluetooth, 5G (mmWave and / or sub-6 GHz), and the like.
[0222] In some embodiments, individual mixed reality systems can be organized into one or more groupings. For example, as shown in FIG. 57, a group of 10 soldiers may be assigned to a particular squad / platoon / company. The mixed reality systems used by those 10 soldiers may communicate with a remote operations server 5702. In some embodiments, the remote operations server 5702 may aggregate data collected by the squad's mixed reality systems. In some embodiments, computationally complex tasks that may not require low-latency operation may be performed on the remote operations server 5702. For example, each individual mixed reality system may generate its own understanding of real object locations and positions (e.g., via SLAM). In some embodiments, this information can be transmitted to the remote operations server 5702, which may optimize object locations and positions determined across multiple mixed reality systems. Multiple soldiers may result in repeated observations of the same real object from different perspectives, and combining multiple streams of SLAM data may produce a more accurate estimate of real-world object locations and positions.
[0223] In some embodiments, each mixed reality system may display the virtual object according to a local preliminary SLAM calculation. Because the mixed reality system may rely on SLAM to visually display the virtual object in relation to the real object (e.g., a virtual target may appear in the same real location as the soldier moves around), it may be beneficial for the individual mixed reality system to perform a "first-order" SLAM estimation. Computing SLAM locally may enable the mixed reality system to visually display the virtual object in a low-latency manner, which may improve the user experience (e.g., by allowing the mixed reality system to display the virtual object in real time). In some embodiments, the SLAM optimization may be performed remotely (e.g., at the remote operations server 5702). In some embodiments, the optimization data may be sent back to the mixed reality system, which may use the data to update the visual display of the virtual object. Further details regarding optimizing the SLAM calculation using additional observations and / or using optimized data to update the SLAM calculation may be found in U.S. Provisional Patent Application No. 62 / 923,317, the contents of which are incorporated herein by reference in their entirety.
[0224] Other computationally intensive tasks that may not require low latency communication may also be offloaded to a remote server, which may send results back to the individual mixed reality systems. For example, machine learning algorithms may be offloaded to a remote server (e.g., remote operational server 5702, remote tactical server 5704, remote strategic server 5706, and / or data lake 5708). In some embodiments, the machine learning algorithms may identify objects as targets based on data collected from the mixed reality system. For example, one or more cameras on the mixed reality system that approximate the soldier's field of view may capture video. The mixed reality system may be connected to a weapon, which may have one or more sensors that may determine the direction in which the weapon is aimed. For example, a laser sight from the weapon may be detected by one or more cameras of the mixed reality system to determine the weapon aim direction. In some embodiments, a sensor such as an accelerometer on the weapon may be used to determine the weapon aim direction. In some embodiments, data about the soldier's field of view and weapon aim direction can be fed into a machine learning algorithm (e.g., an image with the area where the weapon is aimed can be fed through a deep convolutional neural network) to determine objects that are likely to be targets.
[0225] In some embodiments, machine learning computations (e.g., convolution, forward propagation, back propagation, etc.) can be performed at the remote server to train the neural network. In some embodiments, the mixed reality system can utilize machine learning to visually identify objects that are likely to be targets (e.g., via colored contours). The mixed reality system can feed images to a remote server, which can send the images to a trained neural network to determine whether an area is likely to be a target. In some embodiments, the trained neural network (e.g., a neural network trained at a remote server) can be stored locally on the mixed reality system for low latency identification of likely targets. In some embodiments, the locally stored trained neural network can be periodically updated through a remote server, which can continually train the neural network with additional training data.
[0226] Similar methods of remote computation and / or local storage of completed computations can be employed for other machine learning / computationally complex applications, including automatic speech recognition. For example, the mixed reality system may record audio of a user's speech and transmit the audio to a remote server for speech processing (e.g., speech recognition and / or natural language processing). In some embodiments, the mixed reality system may receive instructions from the remote server to perform one or more functions as a result of processing the audio of the user's speech. In some embodiments, it may be desirable to perform the speech processing locally on the mixed reality system. For example, a soldier may wish to execute a voice command to the mixed reality system (e.g., to turn off night vision functionality) and it may be desirable to execute the command as quickly as possible. In some embodiments, a speech recognition system can be trained at a remote server and stored locally on the mixed reality system.
[0227] In some embodiments, some mixed reality functionality can be implemented locally on the mixed reality system. For example, the mixed reality system may employ eye tracking technology (e.g., via one or more cameras pointed at the user's eyes) to identify the direction in which the user is looking. In some embodiments, eye tracking calculations may be implemented locally on the mixed reality system. It may be desirable to calculate eye tracking in a low latency manner because video may be displayed to the user as a result of the eye tracking and high latency (e.g., on the order of 50 ms or more) may result in significant delays for the user. In some embodiments, spatialized audio rendering may be performed locally on the mixed reality system. For example, a virtual sound to be presented as coming from behind the user may be rendered and / or spatialized locally on the mixed reality device. It may be desirable to render and / or spatialize the virtual audio in a low latency manner so that the virtual audio appears "in sync" (e.g., presented at approximately the same time) with real or virtual visual events. In some embodiments, graphic rendering can be implemented locally on the mixed reality system. It may be desirable to quickly render and display virtual objects to a user, and offloading the computation (e.g., to a remote server) may produce unacceptable latency.
[0228] Mixed reality edge computing architectures may also rely on relative physical locations for efficient computing. For example, communication between nearby mixed reality systems may have lower latency than communication between a mixed reality system and a remote server. In some embodiments, computations that may be offloaded to a remote server may be distributed across connected mixed reality systems (e.g., if the remote server is not reachable). For example, mixed reality systems may communicate SLAM data to each other, which may increase the pool of available observations, which may lead to more accurate SLAM computations. In some embodiments, SLAM optimization may be performed in a single mixed reality system, which may include a more powerful processor than other standard mixed reality systems. In some embodiments, SLAM optimization may be distributed across connected mixed reality systems, and computation power may be pooled together.
[0229] In some embodiments, a remote server (e.g., remote operational server 5702) may be located physically close to a connected mixed reality system, which may reduce communication latency. For example, a mixed reality system deployed with a soldier may communicate with a remote server located at a base of operations that may be physically close to the mixed reality system (e.g., less than 50 miles away). In some embodiments, communicating with a remote server located in the vicinity of the mixed reality system may result in lower latency than communicating with a remote server located more distant (e.g., 2,000 miles away and / or on a different continent).
[0230] In some embodiments, mixed reality systems deployed in large-scale mixed reality applications can have personalized permissions / network connections. For example, each soldier's mixed reality system may transmit data (e.g., SLAM, machine learning, and / or vital data) to the remote operations server 5702, but only the squad leader's mixed reality system may have read access to each squad member's vital data. The limited access structure can be implemented via any suitable means. For example, each mixed reality system can have a unique identifier, and the remote server may transmit only sensitive data to the appropriate mixed reality system. In some embodiments, the squad leader's mixed reality system can store an encryption / decryption key, which can be used to encrypt / decrypt communications between the mixed reality system and the remote server.
[0231] In some embodiments, a group of 1,000 soldiers may be assigned to a battalion, and each soldier may be equipped with an individual mixed reality system. In some embodiments, data collected from the entire battalion may be aggregated in a remote server (e.g., a remote tactical server 5704, which may be the same or a separate server as the remote operational server 5702). In some embodiments, the aggregated data for the battalion (e.g., squad status, overall unit movements, casualties, etc.) may be accessed by a limited subset of deployed mixed reality systems (e.g., by the mixed reality system used by the unit commander). In some embodiments, the aggregated data for the battalion may be accessed by a general computing system (e.g., by a laptop used by the unit commander).
[0232] In some embodiments, data aggregated by one or more remote servers (e.g., remote operational server 5702, remote tactical server 5704, and / or remote strategic server 5706) can be further aggregated in a separate remote server (e.g., data lake 5708). In some embodiments, the data lake 5708 can further access external resources (e.g., the Internet) and internal resources (e.g., military databases) in addition to data collected from deployed mixed reality systems. In some embodiments, the data lake 5708 can be physically located in an area considered safe from enemy combatants (e.g., offshore) and / or may be decentralized (e.g., data lake 5708 may comprise one or more server farms located in different locations). It may be beneficial to utilize the data lake 5708 to perform computationally complex tasks that may not require low latency communications (e.g., in a similar manner where computation may be split between the mixed reality systems and the remote operational server 5702). For example, the remote operations server 5702 may initiate relatively simple machine learning algorithms (e.g., whether a piece of footage is likely to be a target), while more complex machine learning algorithms (e.g., what types of troop movements are more likely to win an engagement) and / or neural network training may be performed using data aggregated in the data lake 5708.
[0233] Although some embodiments of the present disclosure are described with respect to military applications, it will be understood by those skilled in the art that other applications of the disclosed embodiments (including applications not directly related to military technology or emergency services) are suitable and within the scope of the present disclosure.
[0234] FIG. 84 illustrates an exemplary mixed reality computing architecture, according to some embodiments. In some embodiments, the mixed reality computing architecture 8400 may include one or more modules, which may be components of the mixed reality computing architecture 8400. In some embodiments, the modules may include one or more computer systems (e.g., MR systems 112, 200 or remote servers) configured to execute instructions and / or store one or more data structures. For example, the instructions executed by the modules may be processes and / or sub-processes launched within the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the modules may be threads launched within the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the modules may be launched within the same process address space and / or memory space as other components of the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the modules may be launched within a different process address space and / or memory space than other components of the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the modules may be launched on different hardware than other components of the mixed reality computing architecture 8400. For example, instructions performed by one or more modules of mixed reality computing architecture 8400 may run on mixed reality systems 112 and / or 200, while other components of mixed reality computing architecture 8400 may run on a remote server. In some embodiments, instructions performed by one or more modules of mixed reality computing architecture 8400 may be instantiated within mixed reality computing architecture 8400.In some embodiments, instructions executed by and / or data structures stored within modules in the mixed reality computing architecture 8400 may communicate with other components of the mixed reality computing architecture 8400 (e.g., instructions executed by and / or data structures stored therein other modules).
[0235] In some embodiments, the mixed reality computing architecture 8400 may include one or more modules, which may include one or more sub-modules. In some embodiments, the sub-modules may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the instructions executed by the sub-modules may be processes and / or sub-processes launched within the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the sub-modules may be threads launched within the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the sub-modules may be launched within the same process address space and / or memory space as other components of the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the sub-modules may be launched within a different process address space and / or memory space than other components of the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the sub-modules may be launched on different hardware than other components of the mixed reality computing architecture 8400. For example, instructions performed by one or more sub-modules of mixed reality computing architecture 8400 may run on an audio-specific processor (e.g., a DSP), while other components of mixed reality computing architecture 8400 may run on a general-purpose processor. In some embodiments, instructions performed by one or more sub-modules of mixed reality computing architecture 8400 may be instantiated within mixed reality computing architecture 8400.In some embodiments, instructions executed by and / or data structures stored within submodules in the mixed reality computing architecture 8400 may communicate with other components of the mixed reality computing architecture 8400 (e.g., instructions executed by and / or data structures stored within other modules).
[0236] In some embodiments, the instructions executed by the wearable software module 8402 may be launched on a mixed reality system (e.g., MR system 112, 200). In some embodiments, the instructions executed by the wearable software module 8402 may be launched on a component of the mixed reality system (e.g., a head-mounted device of the mixed reality system). The instructions executed by the wearable software module 8402 may include functionality where low latency is beneficial. For example, the instructions executed by the wearable software module 8402 may include a tracking sub-module 8406. In some embodiments, the tracking sub-module 8406 may track real and / or virtual objects used to produce the mixed reality experience. For example, the tracking sub-module 8406 may track head pose, which may include the position and / or orientation of the user's head. The head pose may be used to determine which virtual objects should be presented to the user (e.g., because only virtual objects located within a real location in the user's field of view should be presented). In some embodiments, estimating head pose exclusively remotely (e.g., at a remote server) may result in latency between when the user turns and when the mixed reality system determines that the field of view has shifted. When turning, having a new virtual object appear 100 ms after it should already have been visible may disorient the user. In some embodiments, head pose can be preliminarily determined locally in the mixed reality headset and optimized using remote computation. For example, head pose data (e.g., images captured by a camera mounted on a head-mounted device and / or inertial measurement unit data) can be transmitted to the cloud module 8410. In some embodiments, instructions executed by the cloud module 8410 can be launched on a remote server (e.g., remote operations server 5702). For example, the cloud module 8410 can include a data management module 8410.In some embodiments, the data management module 8410 can manage complex calculations (e.g., neural network training and / or SLAM optimization). In some embodiments, the cloud module 8410 can include an integration sub-module 8414. The integration sub-module 8414 may manage connections between servers and / or databases (e.g., other computing systems within an intranet and / or the Internet).
[0237] Other functions, such as those performed by the fast target acquisition submodule 8408, can also be launched on the mixed reality system. In some embodiments, it may be beneficial for the mixed reality system to quickly display the target acquisition image with low latency (e.g., outline the target in red), especially when the user is rapidly changing the field of view. A rapidly changing field of view may result in the target moving quickly in and out of the sights, and it may be highly desirable for the mixed reality system to recognize and / or visually display a target indication to the user. In some embodiments, the instructions executed by the fast target acquisition submodule 8408 can be optimized by computations performed remotely. For example, a deep convolutional neural network may be trained on a remote server, and the trained model can be deployed to and launched on the fast target acquisition submodule 8408.
[0238] In some embodiments, it may be beneficial to launch instructions executed by the OS sub-module 8404 on the mixed reality system. In some embodiments, the OS sub-module 8404 may execute instructions related to base operating system functionality (e.g., drivers, services, etc.). In some embodiments, the OS sub-module 8404 may enable the mixed reality system 112, 200 to function as a computing system and may enable the mixed reality system to launch applications. In some embodiments,
[0239] FIG. 94 illustrates an example network and / or computing architecture, according to some embodiments. In some embodiments, data can be transmitted by one or more mixed reality systems 9404 worn by soldiers 9402. In some embodiments, the mixed reality systems 9404 may not be viable to communicate with a remote server (e.g., because there is no cellular network connection available). In some embodiments, large-scale mixed reality applications can utilize non-centralized networks (e.g., ad-hoc and / or peer-to-peer networks) to facilitate communication between components of a large-scale system. For example, individual mixed reality systems may transmit data (e.g., wirelessly using Bluetooth, WiFi, 5G, and / or other radio frequency communications) to nearby vehicles 9406. In some embodiments, the vehicles 9406 can include a communications array that may be more powerful than the communications array in the mixed reality systems 9404. In some embodiments, the vehicles 9406 can relay and / or transmit data to aircraft 9408 and / or aircraft 9414, which may be passing within communication range. In some embodiments, aircraft 9408 and / or aircraft 9414 may relay and / or transmit data to satellite 9410. In some embodiments, satellite 9410 may relay and / or transmit data to a remote server and / or spacecraft 9412. In some embodiments, the data may be transmitted and / or relayed to a communications array 9416, which may be portable enough to be on-board a soldier (e.g., a radio operator). It is contemplated that any combination of components may communicate bidirectionally (e.g., two-way communication) with other components.
[0240] In some embodiments, ad-hoc and / or mesh networks may effectively transmit data from mixed reality systems, and the network infrastructure may be sufficient to meet the needs of large-scale mixed reality applications. In some embodiments, ad-hoc and / or mesh communications may be suitable for latency-independent transmissions, such as SLAM optimization, neural network training, mapping / navigation, non-live communications (e.g., messaging), etc. In some embodiments, ad-hoc and / or mesh communications may produce latencies that impede latency-sensitive transmissions, such as graphical and / or audio rendering, preliminary SLAM computation, voice command processing, eye tracking, etc.
[0241] FIG. 95 illustrates an example network and / or computing architecture, according to some embodiments. In some embodiments, the communications array 9502 (which may be coupled to one or more mixed reality systems) may communicate with one or more drones 9504. In some embodiments, the one or more drones 9504 may form a network (e.g., a mesh network and / or an ad-hoc network) that may facilitate communication between one or more mixed reality systems and / or one or more remote computing systems (e.g., a remote server). For example, the communications array 9502 may not be able to communicate directly with the communications array 9506, but may be able to communicate with the communications array 9506 via one or more drones 9504 (e.g., via relaying data across multiple drones and / or via a mesh network).
[0242] FIG. 96 illustrates an example network and / or computing architecture, according to some embodiments. In some embodiments, the communications array 9602 can be mounted on a soldier, which may be in motion. One or more drones 9604 can form a network (e.g., a mesh network and / or an ad-hoc network), which can facilitate communication between one or more mixed reality systems and / or one or more remote computing systems (e.g., a remote server). In some embodiments, the network formed by the one or more drones 9604 can be mobile, with the one or more drones 9604 moving with the communications array 9602. In some embodiments, the one or more drones 9604 can fly in a predetermined pattern according to expected troop movements. In some embodiments, the one or more drones 9604 can be configured to follow a soldier with an on-board communications array 9602. In some embodiments, the one or more drones 9604 can be configured to maintain a static and / or dynamic spacing between other drones (e.g., the spacing can vary according to troop density, combat risk, network reach, etc.).
[0243] 97-102 illustrate an exemplary network and / or computing architecture, according to some embodiments. In some embodiments, the wearable component 9702 may correspond to the wearable component 2 depicted in FIG. 6. In some embodiments, the belt pack 9704 may correspond to the belt pack 6 depicted in FIG. 7. In some embodiments, the wearable component 9702 and / or the belt pack 9704 may correspond to the MR system 112, 200. In some embodiments, the wearable component 9702 and / or the belt pack 9704 may be connected to the communication unit 9706 (e.g., through a wired and / or wireless connection). In some embodiments, the communication unit 9706 may include a mobile device (e.g., a smartphone). In some embodiments, the mobile device may be configured to provide additional processing and / or communication capabilities. For example, the mobile device may include its own processor, which may be configured to perform calculations. In some embodiments, the mobile device may be configured to act as an interface between the mixed reality system and other computing systems (e.g., accessories, sensors, other mixed reality systems, and / or remote servers). In some embodiments, the mobile device may include a radio frequency antenna and / or a network modem configured to communicate with one or more cellular towers. The mobile device may act as a repeater and transmit information from the mixed reality system to other computing devices (e.g., a remote server).
[0244] In some embodiments, the communication unit 9706 may include an interconnect pack. In some embodiments, the interconnect pack may include a powerful antenna (e.g., a radio frequency antenna) and / or a transceiver. In some embodiments, the interconnect pack may be capable of high bandwidth transmission. For example, the interconnect pack may be configured to communicate with the wearable component 9702 and / or the belt pack 9704. In some embodiments, the interconnect pack may be configured to receive all or a portion of all data captured by the wearable component 9702 and / or the belt pack 9704 (e.g., video feed, depth information, SLAM data, audio feed, vital information, etc.). In some embodiments, the interconnect pack may be less constrained than other electronic devices. For example, a smartphone may be subject to power and / or transmission limitations because the smartphone may be configured to be used in the vicinity of the user's head. In some embodiments, the interconnect pack may be configured to be used away from the user and / or sensitive parts of the user. In some embodiments, the interconnect pack may be coupled to the mobile device (e.g., through wired and / or wireless means). In some embodiments, one or more mixed reality systems can be coupled to an interconnect pack, which can be configured to communicate with other computing systems.
[0245] In some embodiments, the communication unit 9706 can communicate with other computing systems. For example, the communication unit 9706 can communicate with an intermediate transceiver 9708. In some embodiments, the intermediate transceiver 9708 can be a cellular tower. In some embodiments, the intermediate transceiver 9708 can be a communication array mounted on the soldier. In some embodiments, the intermediate transceiver 9708 can transmit information to one or more cloud servers 9712. In some embodiments, the intermediate transceiver 9708 can transmit information directly to one or more cloud servers. In some embodiments, the intermediate transceiver 9708 can transmit information via one or more edge nodes 9710. The edge nodes 9710 can be network devices that are non-unitary and / or located physically near the intermediate transceiver 9708. For example, a mixed reality system can be an edge node, a mobile device can be an edge node, a wireless access point can be an edge node, a communication array mounted on the soldier can be an edge node, etc. Physical proximity can reduce communication latency, which can be important for various mixed reality features, including SLAM computation, object recognition, voice recognition, etc.
[0246] FIG. 98 depicts an exemplary embodiment of mixed reality network communication. In some embodiments, the communication unit 9706 can communicate directly with one or more edge nodes 9710. FIG. 99 depicts an exemplary embodiment of mixed reality network communication. In some embodiments, the communication 9706 can communicate directly with one or more cloud servers 9712. FIG. 100 depicts an exemplary embodiment of mixed reality network communication. In some embodiments, the wearable component 9702 and / or the belt pack 9704 can communicate with the communication unit 9706 via an intermediate transceiver 9708. FIG. 101 depicts an exemplary embodiment of mixed reality network communication. In some embodiments, the wearable component 9702 and / or the belt pack 9704 can communicate with the communication unit 9706 wirelessly (e.g., via Bluetooth, WiFi, 5G, and / or other radio frequency communication). FIG. 102 depicts an exemplary embodiment of mixed reality network communication. In some embodiments, the communication unit 9706 can communicate with one or more connectivity appliances 9714. In some embodiments, the connectivity appliances 9714 can be connected resources with very good local connectivity. For example, an enterprise may have a connectivity appliance 9714 in each room. In some embodiments, the connectivity appliances 9714 may have storage capabilities managed by local IT or the like. For example, in one embodiment, each connectivity appliance may not only have very good connectivity to communicate outside the building, but also store one or more high-resolution meshes of a particular room (e.g., one or more meshes captured / refreshed by a mixed reality system in the area during the previous day).
[0247] FIG. 103 depicts an example wearable component, which may correspond to the example wearable component depicted in FIG. 6. In some embodiments, the wearable component includes an electromagnetic sensor 1, a light field camera 2, a Bluetooth antenna 3, a WiMAX antenna 4, a WiFi antenna 5, a 4G / LTE antenna 6, an active cooling fan 7, a variable focus element 8, a thermocouple 9, an inertial measurement unit 10, a visible RGB / monochrome camera 11, a near infrared camera 12, a long wave infrared camera 13, a short wave infrared camera 14, an infrared eye tracking camera 15, a neuromorphic differential sensor 16, a thermistor 17, a 5G antenna 18, a polarized sensor 19, a hemispherical camera 20, a time-of-flight depth The imaging systems may include a depth sensor 21, a LIDAR sensor 22, a structured light depth sensor 23, a global dynamic dimmer 24, a localized dynamic dimmer 25, a refractive freeform optics 26, a diffractive lens optics 27, a MEMS scanner single source 28, a MEMS scanner dual source 29, a digital holographic display 30, a photodiode 31, a CCD sensor 32, a CMOS sensor 33, an electron-impacted active pixel sensor 34, a hybrid dynamic visual visible light sensor 35, an internal interferometer 36, an active noise cancelling device 37, and / or a microphone array 38.
[0248] FIG. 104 depicts an example beltpack component, which may correspond to the example beltpack component depicted in FIG. 7. In some embodiments, the beltpack component may include a near-range fingerprint reader 1, a multi-purpose external connector 2, an HDMI port 3, a display port 4, a shortwave infrared camera 5, a 5G antenna 6, a WiFi antenna 7, an internal Qi charger 8, a USB-C connector 9, a USB-A (version 2 and / or 3) connector 10, a light field camera 11, a 4G antenna 12, an ultra-wideband antenna 13, a visible RGB and / or monochrome camera 14, a Bluetooth camera 15, a WiMAX antenna 16, an internal battery 17, an internal memory 18, an internal computation 19, a longwave infrared camera 20, an LCD touch panel 21, an internal thermocouple 22, an internal GPU 23, an internal electromagnetic sensor 24, and / or a near-infrared camera 25.
[0249] Various exemplary embodiments of the present invention are described herein. Reference is made to these examples in a non-limiting sense. They are provided to illustrate the broader and more applicable aspects of the present invention. Various modifications may be made to the described invention, and equivalents may be substituted without departing from the true spirit and scope of the present invention. For example, some embodiments are described with respect to military or emergency service applications, but other suitable applications will be understood by those skilled in the art to be within the scope of the present disclosure. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process acts, or steps to the objective, spirit, or scope of the present invention. Furthermore, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein has discrete components and features that can be easily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present invention. All such modifications are intended to be within the scope of the claims associated with this disclosure.
[0250] The present invention includes methods that may be implemented using the subject devices. The methods may include the act of providing such a suitable device. Such provision may be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, access, position, configure, activate, power on, or otherwise act to provide the requisite device in the subject method. The methods described herein may be carried out in any order of the recited events that is logically possible, and in the recited order of events.
[0251] Exemplary aspects of the invention have been described above, along with details regarding material selection and manufacturing. As for other details of the invention, these may be understood in conjunction with the above-referenced patents and publications, and may generally be known or understood by those skilled in the art. The same may be true with respect to method-based embodiments of the invention in terms of additional acts as typically or logically adopted.
[0252] In addition, although the present invention has been described with reference to several embodiments that optionally incorporate various features, the present invention is not limited to those described and indicated as being considered with respect to each variation of the present invention. Various modifications may be made to the invention described, and equivalents (whether described herein or not included for some simplicity) may be substituted without departing from the true spirit and scope of the present invention. In addition, when a range of values is provided, it is to be understood that all intervening values between the upper and lower limits of that range, and any other stated or intervening values within that stated range, are encompassed within the present invention.
[0253] It is also contemplated that any optional features of the described inventive variations may be described and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that the same items are present in plural. More specifically, as used herein and in the claims associated herewith, the singular forms "a," "an," "said," and "the" include plural referents unless otherwise specified. In other words, the use of articles allows for "at least one" of the subject items in the above description and in the claims associated herewith. Furthermore, it is noted that such claims may be drafted to exclude any optional element. Thus, this statement is intended to serve as a predicate for the use of exclusive terminology such as "solely," "only," and the like, or the use of "negative" limitations in connection with the recitation of claim elements.
[0254] Without using such exclusive language, the term "comprising" in the claims associated with this disclosure shall be construed as allowing for the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claims or whether the addition of features can be considered as a transformation of the nature of the elements recited in such claims. Except as specifically defined herein, all technical and scientific terms used herein shall be given the broadest possible commonly understood meaning while maintaining the validity of the claims.
[0255] The scope of the present invention is not intended to be limited to the examples provided and / or specifica- tion of the present subject matter, but rather is limited only by the scope of the language of the claims associated with this disclosure.
Claims
1. A method for a mixed reality system, comprising: Capturing data about a user's field of view via a camera of the mixed reality system; receiving data from a handheld device in communication with the mixed reality system about an orientation of the handheld device detected by a sensor of the handheld device; transmitting the data on the user's field of view and the data on the orientation to a remote server, such that the remote server trains a neural network based on the data on the user's field of view and the data on the orientation; receiving, in the mixed reality system, a trained neural network from the remote server; identifying an object within the user's field of view as a target via the trained neural network; and A method comprising:
2. The method of claim 1, wherein transmitting the data about the user's field of view and the data about the orientation to the remote server includes transmitting the data about the user's field of view and the data about the orientation to an intermediate repeater.
3. The method of claim 2, wherein the intermediate repeater comprises a drone network.
4. The method of claim 2, wherein the intermediate repeater comprises a mobile network device.
5. The method of claim 2, wherein the mixed reality system comprises a first head-mounted device and the intermediate repeater comprises a second head-mounted device.
6. The method of claim 2, wherein the intermediate repeater comprises a cellular tower.
7. A system comprising: A camera of a mixed reality system; a sensor in a handheld device, the handheld device configured to communicate with the mixed reality system; and A remote server; one or more processors configured to perform the method; Equipped with The method comprises: Capturing data about a user's field of view via the camera; receiving data from the handheld device regarding an orientation of the handheld device detected by the sensor; transmitting the data on the user's field of view and the data on the orientation to the remote server, the remote server being configured to train a neural network based on the data on the user's field of view and the data on the orientation; receiving, in the mixed reality system, a trained neural network from the remote server; identifying an object within the user's field of view as a target via the trained neural network; and Including, the system.
8. The system described in claim 7, wherein transmitting the data about the user's field of view and the data about the orientation to the remote server includes transmitting the data about the user's field of view and the data about the orientation to an intermediate repeater.
9. The system described in claim 8, wherein the intermediate repeater comprises a drone network.
10. The system of claim 8, wherein the intermediate repeater comprises a mobile network device.
11. The system described in claim 8, wherein the mixed reality system comprises a first head-mounted device and the intermediate repeater comprises a second head-mounted device.
12. The system of claim 8, wherein the intermediate repeater comprises a cellular tower.
13. A non-transitory computer-readable medium, the non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: Capturing data about a user's field of view via a camera of the mixed reality system; receiving data from a handheld device in communication with the mixed reality system about an orientation of the handheld device detected by a sensor of the handheld device; transmitting the data on the user's field of view and the data on the orientation to a remote server, such that the remote server trains a neural network based on the data on the user's field of view and the data on the orientation; receiving, in the mixed reality system, a trained neural network from the remote server; identifying an object within the user's field of view as a target via the trained neural network; and 16. A non-transitory computer readable medium comprising:
14. The non-transitory computer-readable medium of claim 13, wherein transmitting the data about the user's field of view and the data about the orientation to the remote server includes transmitting the data about the user's field of view and the data about the orientation to an intermediate repeater.
15. The non-transitory computer-readable medium of claim 14, wherein the intermediate repeater comprises a drone network.
16. The non-transitory computer-readable medium of claim 14, wherein the intermediate repeater comprises a mobile network device.
17. The non-transitory computer-readable medium of claim 14, wherein the mixed reality system comprises a first head-mounted device and the intermediate repeater comprises a second head-mounted device.
18. The non-transitory computer-readable medium of claim 14, wherein the intermediate repeater comprises a cellular tower.
