Systems and methods for virtual reality and augmented reality
Through distributed computing and networking systems, the head-mounted devices capture images and inertial data and transmit them to remote servers for processing, solving the delay and computing resource problems of existing VR and AR systems when presenting virtual images in multifocal planes, and realizing the A portable, robust high-performance virtual reality experience.
Patent Information
- Application Number
- CN201980093010.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-12-27
- Filing Date
- 2019-12-27
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2039-12-27
AI Technical Summary
When existing virtual reality (VR) and augmented reality (AR) systems present virtual image information on multiple focal planes, there is delay, viewing angle offset, low image fidelity, and high demand for computing resources and power, which is difficult to be in an emergency. or is effectively used in critical operation scenarios and lacks portable, robust, and high-performance wearable computing deployment.
The distributed computing and networking system is adopted to capture images and inertial data through head-mounted devices, transmit them to remote servers for processing, and return the trained neural network to the device to realize high-performance computing and display of mixed reality systems.
It provides stable, transparent, and low-cost virtual image presentation on multiple focal planes, suitable for various human scenes, improving the portability and robustness of the system, and meeting the needs of emergency operation scenarios.
Smart Images

Figure CN113632030B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Provisional Application No. 62 / 785,370, filed on December 27, 2018, the contents of which are incorporated herein by reference in their entirety. Technical Field
[0003] The present invention relates to connected mobile computing systems, methods and configurations, and more particularly to mobile computing systems, methods and configurations featuring at least one wearable component that can be used for virtual and / or augmented reality operations. Background Art
[0004] It is desirable for a mixed reality ("MR") or augmented reality ("AR"), near-eye display to be lightweight, low cost, have a small form factor, have a wide virtual image field of view, and be as transparent as possible. Furthermore, it is desirable to have a configuration that presents virtual image information in multiple focal planes (e.g., two or more) so as to be suitable for a variety of use cases without exceeding acceptable tolerances for vergence accommodation mismatch. Reference Figure 1 , an augmented reality system is illustrated as featuring a head mounted viewing component (2), a handheld controller component (4), and an interconnected auxiliary computing or controller component (6) that may be configured to be worn on a user as a waist pack or the like. Each of these components may be operably coupled to each other (10, 12, 14, 16, 17, 18) and to other connected resources (8), such as cloud computing or cloud storage resources, via wired or wireless communication configurations, such as those specified by IEEE 802.11, Bluetooth (RTM), and other connection standards and configurations. For example, as described in U.S. patent application serial numbers 14 / 555,585, 14 / 690,401, 14 / 331,218, 15 / 481,255, 62 / 518,539, 62 / 693,891, and 62 / 743,49, each of which is incorporated herein by reference in its entirety, various aspects of such components are described, such as various embodiments of the two depicted optical elements (20), through which a user can see the world around them and the visual components that can be generated by associated system components for an augmented reality experience.
[0005] There are various virtual reality ("VR") systems and some effective AR systems on the market. Many of these systems are configured to utilize local data (such as game software stored on local flash memory), or connect to some remote data (such as high scores stored on cloud storage resources at remote locations), but few are configured to not only interconnect with various types of other user nodes, but also effectively utilize and balance local and cloud resources for processing / operation and storage.
[0006] Most VR systems are essentially close-range monitors, similar to computer or television monitors placed close to the eyes, providing stereoscopic view for convergence cues, and connected to computing and storage resources, such as high-performance gaming laptops or desktop computers, via connected computing devices. VR systems are limited in their capabilities in many human operation scenarios because users are generally unable to see the world around them, so the user will face the challenge of removing the head-mounted display assembly to see what he or she is doing, and then wearing the head-mounted display assembly back over the eyes to continue viewing the information developed or displayed by the computer. Some variants of VR systems can accommodate so-called "pass-through" video, whereby a forward-facing camera captures video that can be broadcast to the user in a VR wearable assembly, but such systems may be less than ideal in many critical human operation scenarios due to latency, perspective offset, image fidelity, and negative physiological cues. Another limitation is the size of the computing resource, which is typically required to be connected for most modern wearable VR display systems. Even if there is computing, the power requirements to meet the physiological needs of the VR system require a backpack-sized battery. In addition, there is a lack of such technologies that are sufficiently safe and robust for use in critical operation scenarios (such as emergency medical response, fire response, police operations, and / or military operations). The systems, configurations, and methods described herein are intended to address the various challenges of portable, robust, highly connected, and high-performance wearable computing deployed in a variety of human scenarios. Systems and components optimized for compact and persistent connectivity for wearable computing systems are needed. Summary of the invention
[0007] Examples of the present disclosure describe systems and methods for distributed computing and / or networking for mixed reality systems. According to examples of the present disclosure, a method may include capturing images via a camera of a head-mounted device. Inertial data may be captured via an inertial measurement unit of the head-mounted device. The position of the head-mounted device may be estimated based on the images and the inertial data via one or more processors of the head-mounted device. The images may be transmitted to a remote server. A neural network may be trained based on the images via the remote server. The trained neural network may be transmitted to the head-mounted device. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 An example mixed reality system is shown in accordance with some embodiments.
[0009] Figure 2 An example mixed reality system is shown in accordance with some embodiments.
[0010] Figure 3 An example mixed reality environment is shown in accordance with some embodiments.
[0011] Figure 4 An example mixed reality system is shown in accordance with some embodiments.
[0012] Figure 5 An example process for building a virtual model according to some embodiments is shown.
[0013] Figure 6 An example head mounted assembly of a mixed reality system is shown in accordance with some embodiments.
[0014] Figure 7 An example wearable package assembly of a mixed reality system is shown in accordance with some embodiments.
[0015] Figure 8 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0016] Fig. 9 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0017] Fig.10 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0018] Fig.11 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0019] Fig.12 An example use of a connected mixed reality system is shown in accordance with some embodiments.
[0020] Fig.13 An example use of a connected mixed reality system is shown in accordance with some embodiments.
[0021] Fig.14 An example use of a connected mixed reality system is shown in accordance with some embodiments.
[0022] Fig.15 An example use of a connected mixed reality system is shown in accordance with some embodiments.
[0023] Fig.16 An example use of a connected mixed reality system is shown in accordance with some embodiments.
[0024] Fig.17 An example use of a connected mixed reality system is shown in accordance with some embodiments.
[0025] Fig.18 An example use of a connected mixed reality system is shown in accordance with some embodiments.
[0026] Fig.19 An example use of a connected mixed reality system is shown in accordance with some embodiments.
[0027] Fig. 20 An example use of a connected mixed reality system is shown in accordance with some embodiments.
[0028] Fig.21 An example use of a connected mixed reality system is shown in accordance with some embodiments.
[0029] Fig. 22 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0030] Fig.23 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0031] Fig.24 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0032] Fig.25 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0033] Fig.26 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0034] Fig. 27 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0035] Fig.28 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0036] Fig.29 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0037] Fig.30 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0038] Fig.31 An example mixed reality network architecture is shown in accordance with some embodiments.
[0039] Fig.32 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0040] Fig.33 An example use of a mixed reality system in accordance with some embodiments is shown.
[0041] Fig.34 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0042] Fig.35 An example sensor assembly of a mixed reality system is shown in accordance with some embodiments.
[0043] Fig.36An example sensor assembly of a mixed reality system is shown in accordance with some embodiments.
[0044] Fig.37 An example use of a mixed reality system in accordance with some embodiments is shown.
[0045] Fig.38 An example use of a mixed reality system in accordance with some embodiments is shown.
[0046] Fig.39 An example use of a mixed reality system in accordance with some embodiments is shown.
[0047] Fig.40 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0048] Fig.41 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0049] Fig.42 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0050] Fig.43 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0051] Fig.44 An example use of a mixed reality system in an emergency situation is shown in accordance with some embodiments.
[0052] Fig.45 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0053] Fig.46 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0054] Fig.47 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0055] Fig.48 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0056] Fig.49 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0057] Fig.50 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0058] Fig.51 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0059] Fig.52An example mixed reality computing architecture is shown in accordance with some embodiments.
[0060] Fig.53 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0061] Fig.54 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0062] Fig.55 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0063] Fig.56 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0064] Fig.57 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0065] Fig.58 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0066] Fig.59 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0067] Fig.60 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0068] Fig.61 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0069] Fig.62 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0070] Fig.63 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0071] Fig.64 An example training use of a mixed reality system is shown in accordance with some embodiments.
[0072] Fig.65 An example training use of a mixed reality system is shown in accordance with some embodiments.
[0073] Fig.66 An example training use of a mixed reality system is shown in accordance with some embodiments.
[0074] Fig.67 An example training use of a mixed reality system is shown in accordance with some embodiments.
[0075] Fig.68 An example training use of a mixed reality system is shown in accordance with some embodiments.
[0076] Fig.69 An example training use of a mixed reality system is shown in accordance with some embodiments.
[0077] Fig.70 An example use of a mixed reality system in accordance with some embodiments is shown.
[0078] Fig.71 An example permissions architecture for a mixed reality system is shown in accordance with some embodiments.
[0079] Fig.72 An example permissions architecture for a mixed reality system is shown in accordance with some embodiments.
[0080] Fig.73 An example permissions architecture for a mixed reality system is shown in accordance with some embodiments.
[0081] Figures 74A-74D An exemplary dimming function of a mixed reality system according to some embodiments is shown.
[0082] Fig.75 An exemplary target functionality of a mixed reality system according to some embodiments is shown.
[0083] Fig.76 An exemplary target functionality of a mixed reality system according to some embodiments is shown.
[0084] Fig.77 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0085] Fig.78 An example sensor assembly of a mixed reality system is shown in accordance with some embodiments.
[0086] Fig.79 An example sensor assembly of a mixed reality system is shown in accordance with some embodiments.
[0087] Fig.80 An example sensor assembly of a mixed reality system is shown in accordance with some embodiments.
[0088] Fig.81 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0089] Fig.82 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0090] Fig.83 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0091] Figures 84A-84B An example mixed reality computing architecture is shown in accordance with some embodiments.
[0092] Fig.85 Example wearable components of a mixed reality system are shown in accordance with some embodiments.
[0093] Figures 86A-86D An example sensor assembly of a mixed reality system is shown in accordance with some embodiments.
[0094] Fig.87 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0095] Fig.88 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0096] Fig.89 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0097] Fig.90 An example virtual assistant is shown in accordance with some embodiments.
[0098] Fig.91 An example mapping component of a mixed reality system is shown in accordance with some embodiments.
[0099] Fig.92 An example virtual model is shown in accordance with some embodiments.
[0100] Fig.93 An example cuff assembly of a mixed reality system is shown in accordance with some embodiments.
[0101] Fig.94 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0102] Fig.95 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0103] Fig.96 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0104] Fig.97 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0105] Fig.98 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0106] Fig.99 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0107] Fig.100 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0108] Fig.101 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0109] Fig.102 An example mixed reality computing architecture is shown in accordance with some embodiments.
[0110] Fig.103 An example head mounted assembly of a mixed reality system is shown in accordance with some embodiments.
[0111] Fig.104 An example wearable package assembly of a mixed reality system is shown.
[0112] Figures 105A-105C An example mixed reality environment is shown in accordance with some embodiments.
[0113] Figures 106A-106D Components of an example mixed reality system that may be used to generate and interact with a mixed reality environment are shown in accordance with some embodiments.
[0114] Fig.107A An example mixed reality handheld controller that can be used to provide input to a mixed reality environment is shown in accordance with some embodiments.
[0115] Fig.107B An example auxiliary unit is shown that may be used with an example mixed reality system in accordance with some embodiments.
[0116] Fig.108 An example functional block diagram of an example mixed reality system is shown in accordance with some embodiments. DETAILED DESCRIPTION
[0117] refer to Figure 2 , showing a user wearing a head mounted display assembly (2), a waist belt or "computing" bag (6) and holding a handheld controller assembly (4). With the system operatively connected to other resources, such as via an 802.11 WiFi connection, the user or operator can navigate around his environment and utilize the computing capabilities of the system while also being able to see what is around him. Figure 3 Some basic concepts of how a user may see and / or perceive things in an augmented reality experience are shown as the user walks through a park environment (22) and views (24) the world through a head mounted wearable assembly (2) such as those described herein; in this illustration, the user can see aspects of the actual world such as trees, concrete elevated pads (1120), and the rest of the park environment (22), while also seeing in three dimensions insect characters (26) and statues (1110) that do not exist in the actual physical world but are presented to the user as if they exist in the actual physical world using the augmented reality capabilities of the present system. Reference Figure 4 , as referenced above Figure 1 As described, the AR system can be operatively coupled to many different resources, such as storage and computing resources, and other important operational resources, such as other portable computing systems, smart phones, smart watches, storage and / or processing centers, wireless connection centers or nodes, various types of personal radios, aircraft or air support, emergency response vehicles, network operation centers or operation control centers, unmanned vehicles such as drones, mobile or wearable camera devices and / or sensors or multiple types, including lidar and other sensors. In various embodiments, preferably, the user is connected to many resources.
[0118] refer to Figure 5 In one embodiment, with many connected resources, the connected collection can be used as a 3-D sensor of sorts to reconstruct a mesh or model of one or more objects that may be positioned between or surrounded by one of multiple transmit / receive devices (such as a cell phone with a 5G antenna array, etc.). In other words, each cell phone, tablet, wearable, or other connected device can be used as a sensor; for example, the direction and intensity, scattering, penetration, and absorbance of each signal source pair can be used to create an interference pattern that can then be visualized, for example, by a user in the field of view of a wearable computing system. In the case of a 5G connected environment, the source / signal pairings can provide a millimeter wave reconstruction of the surrounding world; this can be computationally intensive, but it is one way to create an updateable, universal world model using modern frequency-to-spatial domain holographic reconstruction.
[0119] refer to Figure 6 and Figure 7 As noted in the above-incorporated references, in various embodiments, it may be desirable to integrate many types of sensors and technologies into the wearable component (2) and / or the waist pack / computing pack component (6). For example, in Figure 6 In the illustrated embodiment, the deflection or bending sensor may include an internal interferometer with two different orientations; based on phase change detection, the bending deflection can be correlated and determined; a hybrid dynamic vision-visible light sensor is also shown, where the differential sensor and the visible light sensor can be used in the same optical path, such as using a beam splitter; the signals can be analyzed together to process differences / similarity. Figure 7 As shown in , in one embodiment, the waist pack assembly (6) can be removably coupled from the head mounted assembly and can be removably coupled / swapped from the battery assembly to ensure continuous operation. The waist pack assembly (6) can have an interface, such as an LCD touch screen / display that can be used as a cell phone if desired, and can have one or more cameras or sensing devices integrated therein, for example to provide additional sensing to the operator, such as rearward or other directions relative to the operator.
[0120] refer to Figure 8 , showing various aspects of system connectivity, with an operator headset (2), a waist pack (6), a communications radio, and other elements operably coupled to various other resources. For example, in various embodiments, a "cuff assistant" element may include a cuff or belt-type device that may be mounted around a user's wrist in certain scenarios (such as critical operations such as escaping danger); such a device may be configured to sense temperature, blood pressure, location (such as via GPS and / or mobile connection triangulation and / or IP address mapping), blood oxygen partial pressure, and other variables related to such operator and his or her location and condition; these may be reported to emergency medical services or other systems or personnel so that they can allocate resources to the operator, assist in managing the scenario, etc. Figure 8 Also shown is the concept of connection to a universal or semi-universal collection of databases and information related to the operator's world (which may be referred to as the "MagicVerse"); this may include, for example, connection to emergency medical systems via emergency and prevention networks or other medical networks that may contain, for example, complete patient records with information about allergies or medical conditions; various types of home, business, or city-sized networks may also be connected, such as groups of webcams, servers containing relevant information and data, and resources from what may be referred to as the "Guardian Net", which may be a network of resources accessible to law enforcement and certain military personnel who may access webcams, microphones, satellites, and large storage devices for certain investigations, emergencies, and other purposes.
[0121] refer to Fig. 9 , 10 and 11, firefighters, law enforcement officers, or military operators can respectively utilize wearable computing systems (2, 6) as various mobile operation centers for integrating the utility of a large number of other connected resources, such as smart phones; EMS radios; connection and / or storage centers in homes, businesses, or other places; various cameras and related sensors; GPS; medical and emergency databases; law enforcement; protection network connections; weapon system connections; connections to other manned or unmanned vehicles, etc.
[0122] refer to Figure 12-21 , this technology can also be utilized and adapted to office productivity scenarios. For example, refer to Fig.12 and 13, a meeting of six people can be held to discuss something, such as something visually complex, such as a 3D car part, which can be displayed to each participant through his or her head-mounted computing component (2); one or more participants can be physically present in the room ("local"), while one or more participants can be physically present elsewhere, but represented in the meeting ("remote") by a presented image (such as their own avatar). Participants can participate in the meeting not only using mobile computing technology (2), but also using various other "Internet of Things" connected devices, such as Fig.14 , such as displays, lighting, microphones, speakers, cameras, etc. within a room; in various embodiments, available devices from a particular room or environment can be mapped to such room or environment so that they are available when a user is physically or virtually present in such room. Fig.15 In various embodiments, translation technologies such as those that can be used to translate language into text and text into a different language can be used to facilitate real-time or near real-time participation by members who speak a language different from the language of the other participants in the conference. Fig.16 and 17 , the system may be configured so that participants may select aspects of their own avatars, such as customized views of their own faces or portions thereof, personas such as selected cartoon characters, and / or similar treatments for other participants. For example, user A may decide to display an avatar of a well-known person to others in a particular meeting, while automatically assigning an animal avatar to another member of the meeting (user B), and simultaneously automatically placing a funny hat and fake beard presentation over an otherwise standard avatar of a third member of the meeting (user C). Fig.18 Various user interface configurations may be configured to display a menu above each participant that other participants may see and utilize, for example, to share notes, share a video perspective, and / or share certain conference or information controls. Fig.19 The system can be configured to provide a private chat messaging user interface between subgroups of conference attendees, or to provide a more public chat visible to all attendees, such as for sharing notes from the conference. As described above, languages can be translated, such as by automatic settings, to provide access and utilization in a multi-lingual conference environment. Reference Fig. 20 In various embodiments, the system may be configured to present a user interface on each participant's actual or virtually presented desk so that they can use their hands to move things around, actuate or activate various things, control various aspects of the room, camera views, microphones, speakers, documents the participant is viewing, sharing of notes, etc. Such a user interface is preferably configurable by each user to provide each user with a customized and efficient view of the information and controls in the conference environment. Fig.21 , as referenced above Fig.12 As described, the unique 3D viewing capability of possible certain versions of the head mounted computing assembly (2) presents participants in a meeting with a unique opportunity to view certain complex material or information together “around the table” in three dimensions, regardless of whether all participants are actually in the room.
[0123] refer to Figure 22-30 ,In the fire environment, wearable computing systems can be configured for high ,utility.,For example, refer to Fig. 22 , firefighters can wear a head mounted assembly (2) that allows them to visualize routes and incident maps. Fig.23 , IoT-connected resources, such as door locks, smoke alarms, cameras, power and gas resources, and lighting can be visualized (i.e., such as through the head mounted assembly 2) and made available to firefighter operators on-site or back at a control center for emergency control. Fig.24 In various embodiments, the system can be configured to allow each member of a team connected to a mobile computing system (such as a head mounted component (2) that allows hands-free use) to visualize location (i.e., based on GPS, connection triangulation, IP address, mapping / SLAM / computer vision, etc.) and status (such as health status based on biometric sensors, when the user is experiencing stress, danger, smoke, etc., which can be present on the user's head mounted component 2, and which can be detected and / or visualized by various sensors and cameras on such head mounted component and shared with other collaborators via a wireless connection). For example, referring to Fig.26 , the user's head-mounted component (2) can be configured to monitor eye- and face-based stress indicators as well as heart rate, heart rate variability, etc., and report this information to others; further, the user's head-mounted component (2) can be configured to utilize computer vision resources, such as cameras and local or remote / connected computing resources to not only scan the room around them for mapping and machine learning functions, but also to perform facial recognition of persons (such as injured patients within the building); further, reference Fig. 27 , visible light camera technologies, such as those operably coupled to a user's head mounted assembly (2), can be configured to capture pictures of various scenes, people, etc., for sharing with other operators in different locations (such as at a remote control center). Fig.28 , thermal / infrared (i.e., infrared such as long or normal wavelength), visible light, and / or short wavelength infrared imaging techniques ("SWIR"; having the advantage of being able to image fairly successfully through smoke that is visually invisible to other imaging modalities) can be combined to facilitate visualization of various things in the emergency operator's environment (such as the location of fire hot spots, the location of people, pets, etc.). Figures 86A-86D, depending on the environmental scenario, SWIR imaging can be used to significantly enhance user visibility relative to normal visible light. Fig.29 , the operator can utilize the user's head mounted assembly (2) to mark the conditions of various people within the emergency environment, such as patients suffering from various illnesses or injuries, for sharing with other operators. Fig.30 , connected systems (such as wearable components) can be configured to provide outgoing information about operator conditions that are related not only to operator biometrics and environmental issues, but also to other critical connected devices (such as oxygen tank fill level sensors).
[0124] refer to Figure 31-39 ,In the police / law enforcement environment, wearable computing systems can be configured for high utility. Fig.31 , the wearable computing system (2) can be configured for various members of a police action team to utilize hands-free messaging to / from various different locations. Fig.32 , in various embodiments, each member is connected, and each member becomes another sensing "node" to the overall system, providing not only data related to operator biometrics, but also information related to the environment surrounding such operators, such as for evidence collection, person or structure identification, video / audio / photo capture, thermal / IR / SWIR imaging, simultaneous localization and mapping ("SLAM"), positioning via wireless connection triangulation, GPS and / or IP address, traffic or congestion sensing / reporting, access to remote databases (such as criminal or healthcare databases), sensing or receiving map data related to the world around each operator, etc. For example, with reference to Fig.33 As a police operator walks around with his or her connected head-mounted computing assembly (2), the system can be configured to analyze each face that comes into visual contact with the officer in the vicinity for biometric analysis, such as facial or iris recognition. Such information can be used in conjunction with a connected database search to, for example, notify an officer that the person they just walked past on the sidewalk is a felon with an active warrant and may be armed and dangerous. Fig.34, utilizing connected resources such as crime / location databases, and locations determined, for example, by GPS, wireless connection positioning, IP address, and / or SLAM / computer vision technology using a connected head-mounted computing component (2), the system can be configured to provide an operator in the field, at an operations headquarters, or anywhere else, with indications of crime rates within a particular area; for example, in one embodiment, when an operator is in a dark alley area of high crime rate in the South Side of Chicago at 2:30 a.m. local time, everything around the operator may be colored red or otherwise indicated as having a relatively high crime rate; in other locations, based on recent data, one side of the street may be indicated as having a relatively lower crime rate than the other side of the street. Thus, even if an officer is walking or driving through a known crime area, at least this will not be a surprise; in addition, recent events or information can be tagged for the operator, such as pointing to a known location of a high density of street crack transactions last week. Reference Figure 35-37 , sensors (such as an inward-facing camera on a head-mounted computing assembly (2)) can be used to track variables related to the operator's transient stress level, such as variability in pupil diameter over time; significant variability over time when normalized for incident light variability has been shown to be a relatively good indicator of an individual's stress state (e.g., a small constant pupil diameter may be associated with a relatively static state, while high variability and a relatively large pupil diameter may be associated with a relatively high stress state). Pupil diameter can be measured over time and reported to connected systems and personnel; for example, in one embodiment, Euler video magnification techniques involving scanning a line across a captured scene (such as an eye) and looking for changes / deltas and segmentation of the pupil and sclera of the eye, etc. can provide an effective window for certain biometric variability. Reference Fig.37 , such technology can be used to detect the biometrics of an operator (such as a police officer) wearing the appropriate components (2), but can also be used in situations where one is close enough to analyze the biometrics of others using outward-facing sensors. Fig.38 , it has been shown that people in extremely stressful situations may have difficulty or inefficiency in handling certain things that are considered relatively simple in non-stressful situations, and thus various embodiments may be configured to assist with various information-related tasks, such as marking a person with a particular wound as such, or marking a corpse as a corpse so that every operator does not have to walk up to check for a pulse to confirm, etc. Reference Fig.39 , criminal activity can be mapped in real time or near real time and visualized in, for example, an AR mapping user interface by an operator wearing appropriate components (2). In one embodiment, the AR mapping interface can be configured to also provide the operator with time domain controls so that they can go back an hour or any time frame to visualize what the scene was like at that time.
[0125] Certain operators or responders (such as firefighters, police officers, or emergency medical responders) can also utilize connected medical resources through their connected wearable computing components (2) in various situations. For example, refer to Figure 40-44 , an operator can utilize their connected wearable computing component (2) to perform spectral analysis of environmental issues around them (such as smoke, gases, chemicals; using technology such as a SWIR device that can be coupled to the operator's headset 2); directly access expert physicians and / or patient medical histories who may be able to connect to their headset (2) to share their patient view and audio feed in high resolution (i.e., via an outward-facing microphone on the headset 2); perform biometric analysis (such as patient identification through face, iris, and / or instantaneous connection to special resources on the patient's smartphone); learn about the patient's allergies or important medical history, etc. For example, an operator utilizing a connected wearable computing component (2) to connect directly with a remote specialist may encounter an unconscious patient who appears to be in cardiac arrest; the operator can request specialist emergency medical triage assistance and the specialist can enter the scene, such as via video teleconference and / or avatar presentation appearing in a portion of the operator's computing component (2) field of view, as well as audio; facial recognition, other biometrics, dedicated first responder patient smartphone access, and / or simple patient wallet identification card information can be used to identify the patient, and securely connected resources can be utilized to determine that the patient is a known heroin addict, and from this situation, the emergency medical specialist who appears to be overdosing and near death can see from the operator's shared field of view that it is time to urgently administer an anti-opioid naloxone hydrochloride injection medication, such as that sold under the trademark NarCan.
[0126] refer to Figure 45-63 , can be utilized in many configurations to assist various types of military operators in countless scenarios. For example, Figures 45-46, a military operator with a connected wearable computing system (2, 6) can wirelessly connect to a variety of other resources, such as a secure communications radio (such as an ultra-wideband radio available to the military operator), one or more long-range weapon sights, one or more battery packs configured to be removably coupled to the wearable computing system, databases, maps, images, video, audio, and a variety of communications, all of which can be interconnected and operated by the military operator; such configuration can be informed and / or interconnected with military systems (such as those systems known as ATAC and NetWarrior). The wearable computing system can also be configured to provide certain protections to the operator, such as laser eye protection through the head-mounted assembly (2), and hearing protection through earbud headphones configured to only pass audio of a certain intensity, after which the transmission is diverted and the earbuds block the sound (in other words, the earbuds can be operably coupled to microphones, such as directional microphones or arrays thereof, which are capable of capturing and processing sounds and selectively reproducing those sounds in the active earbuds, or not reproducing them). Reference Fig.48 and 55 In a manner similar to that described with reference to the 6 degrees of freedom ("DOF") tracking of the handheld assembly (4) described in detail in the above-incorporated references, the weapon or a portion thereof may be tracked and the control system may be configured to assist in directing the operator to the desired target and / or cause the weapon to activate or fire only when the weapon is in the correct position / orientation relative to the desired target. Fig.52 The system can also be configured to use outward-facing cameras and / or sensors and / or prior information from other sources regarding specific objects in the field of view to provide relevant information about each visualized object after identification (such as enemy combatant status, rules of engagement, suspected weapons conditions, any detectable or known stress or medical information, etc.).
[0127] In another embodiment, one or more directional microphones are operably coupled to highly sophisticated signal processing capabilities to assist in directional and identification sound capture; for example, around a troop combat vehicle at night, troops may place their headsets (2) around the perimeter as sound capture perimeters that may monitor locally or remotely (i.e., as a set of individual sound capture devices, or as an array) for specific sound patterns such as, "sound pattern due north at 1:39 a.m. local time - one of our diesel transport trucks"; or, "86% confidence; sound pattern due south at 2:58 a.m. local time - AK-47 small arms fire and small motorcycle activity; 83% confidence; wake up troop leaders to provide updates on potential insurgent activity."
[0128] refer to Figure 49-53Preferably, the system utilized by operator (2) is connected to the systems of other operators and remote systems and resources so that, for example, a local ground operator can gain the benefit of video, audio and other analysis and activities related to the use of the drone, other vehicles, other operators and remote command centers, all of which can be connected through various technologies including but not limited to ultra-wideband transceivers, 802.11 / WiFi configurations, 3G, 4G, 5G, LTE and other mobile connection modes, satellite, etc. For example, in Fig.49 , a ground operator can be seen using imagery captured by a drone to visualize three armed enemies from a location where the operator would otherwise not be able to see such enemies (e.g., because they are on the other side of a ridge, etc.). Fig.51 , can use short wavelength infrared ("SWIR") or NIR (near infrared) based time-of-flight sensors or structured light configurations, LIDAR, RADAR, etc. to understand the surrounding environment and find specific things, such as people. Fig.53 In various embodiments, the connected system can be configured to provide each operator on the ground, as well as other operators at other locations, with a real-time or near real-time feed of activities and events related to a given situation. Fig.54 , which may include information related to medical evacuation, such as through the use of air resources (so-called "Med-Evac" air resources). For example, in various embodiments, the system may be configured so that operators in the air and on the ground wearing their connected computing systems (2) (as well as connected operators at base camps, headquarters, etc.) may be able to share valuable information that would otherwise be difficult to effectively communicate in a hands-free manner, such as an approximate cone (or other geometry) of air traffic avoidance to avoid contact with small arms fire from enemy combatants while still being able to reach and pick up the wounded; the operator on the ground may be able to adjust or change the geometry and / or location of this avoidance zone by using their headset and a pointer to create gestures, their connected weapon or weapon sight, etc. Reference Fig.56 , when GPS, mobile telecommunications triangulation, IP addresses and other positioning technologies are not readily available or operational, an operator wearing his computing system (2) may be able to locate himself and other resources, such as weapons, by using computer vision, object recognition, SLAM and other image-based technologies.
[0129] refer to Figure 57-58, 81 and 87-89, the level of connectivity between operators and resources available through this configuration provides unprecedented detailed quantitative analysis of all aspects of military operations through facts and performance rather than recommendations or comprehensive standardized testing. The system can be configured so that the lowest-level operator has access to a minimal subset of information relevant to them and their peers, with operators moving up the chain of command all the way to the top having access to increasing amounts of information, classified and otherwise. References Figure 59-63 , everyone in the operation can benefit from sharing and capturing information in an efficient manner, such as a largely hands-free manner. Operators can collaborate with each other, visualize things in unprecedented ways, communicate with and utilize other resources, such as aircraft, other imaging systems, database resources, experts, team members, and / or health-related information, all through their wearable computing system, preferably through a head-mounted AR visualization component, which is at the center of functionality and enablement.
[0130] refer to Figure 64-69 , the present wearable computing system can be used to support complex training exercises in almost any location - providing repeatability, enhanced access, and diagnostic / assessment visualization tools. For example, reference Fig.64 In one embodiment, the user may be able to select a scenario from a variety of prepared training scenarios. Fig.65 , in virtually any environment, wearable computing assets (2) can be used to present a user with a predetermined training / operation scenario in which they encounter and visualize one or more counterattack members, e.g. Fig.66 and 67 , where content / images regarding the counterattack are displayed to the operators in the exercise as augmented reality content via their head mounted components (2). Fig.68 and 69 , exercises can be reviewed in 3 dimensions versus time to enhance diagnosis and learning.
[0131] In various embodiments, the core of the simulation / training configuration is the training simulation software configuration ("TSS"), which represents the exact simulation engine; in other words, the TSS is essentially a set of rules that govern how the synthetic version of the world works; how fast bullets fall; what soldiers, police, firefighters, and others can see when standing in a certain position; how fast enemies, fire, suspects move, etc. In various embodiments, the server-side component of the TSS, "TSS / S", can be configured to allow users to connect to and interact with the simulation being conducted for training purposes. The TSS / S can be configured to provide the synthetic environment to connected users, receive information from the users about their actions, incorporate those actions into the simulation, and then update the synthetic environment based on those actions and their simulated results. The client-side component of the TSS, "TSS / C", can be configured to allow users to interact with the TSS. The simulation running on the server (TSS / S) can be configured to provide information about the state of the simulated world to each connected client (TSS / C). For example, the TSS / C running on the MAGIC LEAP human computer can use information from the TSS / S to determine how to present the world to the eyes and ears behind the head-mounted assembly (2). TSS / C can also be configured to provide input mechanisms, capture sensor data from users and transmit these data to TSS / S, where the simulation processes these data and determines how they change the state of the simulated world. Training Management Tool ("TMT") can be configured to provide a tool suite with access to applications designed to support training / simulation operations. For example, in various embodiments, TMT can be configured to provide: server tools for configuring and deploying TSS / S instances, allowing warfighters, law enforcement officers, firefighters, and others to quickly plan and organize training activities; scenario editors for creating or modifying training scenarios; after-action review ("AAR") tools, which are configured to provide users with various mechanisms to replay training courses and evaluate the performance of those involved; data clients that manage access to data captured during training, generate general performance reports and allow arbitrary queries to create specialized reports as needed; simulation editors that provide the ability to modify the basic rules of the simulation (e.g., provide ballistic data for new ammunition or adjust the behavior of synthetic enemies, or fires, chemical leaks, etc.); administrative tools for managing user accounts.
[0132] In various embodiments, the training system service can be a remotely hosted resource and can include, for example: a relatively comprehensive database, which can be referred to as a "data lake", for storing user accounts and training performance data; file storage for collecting and sharing training scenarios; available server resources designated for cloud hosting of TSS / S training servers as needed; access to what can be referred to as an "authoritative lifestream world map" (or "LWM"), which contains data for creating training scenarios and processing raw data streams captured from wearable components (2) into a preferred LWM format. The system can also be configured to provide certain "training aids", which can be any items used to support training with the training system (e.g., weapons, tasers, fire equipment, and training versions of CDC devices, for example). In various embodiments, the training system itself does not involve custom training tools, but rather provides mechanisms for integrating various devices into training and represents an extensible ecosystem. In various embodiments, TSS / S is a simulation engine for the training environment and can be configured to generate accurate synthetic representations of the operating environment necessary to support collective training. This may include: realistic explosion and trajectory simulation of weapons and ammunition; hit / damage determination for synthetic entities and trainees (e.g., TSS / S understands when a shot hits a synthetic enemy, synthetic civilian, or real-world agent participating in the training); general pathfinding capabilities that allow synthetic entities to understand navigable and obstructed areas of simulated space; unit-level behavior - synthetic entities understand their surroundings based on simulated senses, have mobility, have reaction times, are subject to conditions (e.g., if they are hit, they may be suppressed), can make basic decisions (e.g., fight or flee), and have general proficiency (e.g., their accuracy or discipline in a firefight); group-level behavior - using scripts, multiple synthetic entities can determine a course of action and coordinate their behavior based on if / then conditions. In various embodiments, TSS / S can be specifically configured to support group behaviors against civilians, forest and house fires, drug dealers, and enemy sniper teams; simulation of audio data for the environment, entities, and actions and playback on the head-mounted assembly (2) or other outputs.
[0133] In various embodiments, TSS / S can be deployed for training. Users participating in training then connect to the TSS / S designed for their practice. Multiple TSS / S instances can be deployed simultaneously, allowing different groups to conduct different training at the same time. In various embodiments, TSS / S can be designed to allow flexibility in deployment. Using the server browser in TMT, TSS / S instances can be hosted on dedicated platforms (servers in the cloud reserved for TSS / S use) or on common local hardware (typical laptops).
[0134] Self-hosting, where TSS / S and TSS / C operate on the same platform, is an alternative configuration that allows separate training of the pairs using only the head-mounted component (2) and the computing package (6).
[0135] These configuration options allow the training system to provide training when needed and in a variety of Internet-available environments. At a perceived point of need with stable Internet access, the TSS / S can run in the cloud. In the realm of zero connectivity, the TSS / S can be on an operator's laptop, and even without connectivity (i.e., an Internet connection such as via 802.11, or a mobile wireless connection via 5G, 4G, LTE, etc.), the operator can train individually using only the operator's mobile computing system (2, 6).
[0136] In various embodiments, the goal of TSS / S is to allow end-user client-independent systems. With properly formatted data, TSS / S can receive actions from any client (the present wearable computing system 2, 6; laptop; smart phone, etc.) and send simulation state information to any client. One purpose of this is to accommodate other training simulation systems - existing or planned - to allow maximum compatibility with the simulation / training environment (also referred to as "STE") after minimal integration. In a variant where training is performed without network access, TSS / S can be configured to temporarily store training-related data and upload the stored data to the data lake when a connection to the STE service can be established next time. At one level in various embodiments, any interaction with the training system occurs on the client and is initiated through a "launcher". The launcher can be configured to check the user's credentials using biometrics when accessed via the wearable component (2) or a standard username and password if accessed from a laptop. Once authorized, if a connection to the training service can be made, the launcher will check for updates to the training-related software and apply these updates if necessary. TSS / C can be configured to include a rendering engine for displaying visual content. In the primary use case, the TSS / C is connected to the TSS / S, the server can be configured to describe the synthetic environment to the client, and the client projects the world to the user through a wearable headset (2). Alternatively, a version of the rendering engine can be executed on a laptop or PC or other device, providing a "flat screen", non-mixed reality rendering of the world data. In various embodiments, the goal of the TSS / C is to allow users to experience the world data in different modes to support different goals: The mixed reality client display can be configured to be operational only when using a specific headset (2). This display mode can employ proprietary spatial computing technology to insert realistic synthetic content into the user's field of view. In this mode, the user can physically move through the real world space; such a mode may be primarily used for tactical purposes. The virtual client display can be used with the wearable component (2) or with a laptop / PC. On a PC, this mode can be configured to show a flat non-mixed reality display of the world content, similar to what is used in existing PC-based training simulations. On a wearable headset, such as those described herein (2), this mode can be configured to present the content in a virtual reality mode. Regardless of the display used, while in this mode, the user may be physically stationary and may navigate the avatar through the synthetic environment via controls connected to their client (i.e. keyboard and mouse, directional pad, or handheld controller 4). This mode may be used to virtually explore a space (e.g., a large amount of LWM data) or to control a synthetic entity in a training exercise.
[0137] The mixed client display may only be operable when using specific headsets, such as headset (2) described herein. Like the mixed reality display, this display mode utilizes proprietary spatial computing technology. In mixed display mode, the headset does not attempt to realistically insert content into the user's field of view, but rather presents the physical space without regard to where it is projected in the real world. In this mode, the user moves through the projected space by physically walking, just as in the mixed reality display mode. This mode is primarily used for rehearsals. For example, this mode would allow a user to project a portion of a village taken from LWM inside an empty warehouse, and then walk around inside a to-scale projection of that location.
[0138] In various embodiments, when connecting to a TSS / S for training, a client connects with a specific role assigned: as a trainee, as a trainer, or as an observer. The client's role may dictate the information presented to the client and the controls available to the client. For example, a trainer sees performance data and has controls to change training scenarios, while a trainee does not.
[0139] The TSS / C can be configured to manage the receipt of user input (e.g., when a trainee fires their weapon) and capture relevant data (user's position, rifle's orientation, shots taken, instructor triggering ENDEX, etc.) for communication with a connected TSS / S.
[0140] When the client operates on the wearable computing headset (2), the headset can be configured to capture biometric data including heart rate, eye gaze, and perceived pupil dilation. This data is used in the AAR to present information to the instructor to address performance issues and guide performance improvements. The server browser can be used to find or deploy a training server.
[0141] In various embodiments, when deploying, the user may specify the type of deployment, authorized participants, and training scenarios to use. Upon completion of this operation, the user may see the training server added to the list of available servers.
[0142] When connecting to a TSS / S, the user may be presented with a list of known TSS / S instances along with basic information about each instance. The user selects a server to connect to for training or observation.
[0143] Provides options to regulate advertising and access to deployed servers, allowing everything between open "anyone who wants to join" servers and restricted servers visible only to certain users.
[0144] The TMT may be configured to provide a training scenario editor, thereby allowing a user to custom create training exercises.
[0145] Scene creation can start with the specification of the geographic location for training. Parts of LWM can be used for this, or the user can create a mesh using the depth sensor on a headset (such as the present wearable computing headset (2)). Via services provided by the operating system, this mesh can be uploaded into LWM for use by others.
[0146] Once a training area is specified, the scene editor may be configured to allow the user to select an appearance for objects in the training area and to place composite entities for training.
[0147] Once completed, the scenario can be saved and loaded later. The scenario is associated with the training location, allowing others training in the same location to access a library of existing training scenarios. Templates for common exercises can be used to guide the creation of standard training scenarios. In addition, so-called "enables" can be used to modify any training scenario. Enablers can include modular "sub-scenarios" designed to allow for changed training. For example, if a class earlier used a grenade to blow up a bunker, during the next evolution, the instructor wants them to use CAS to complete the objective. In various embodiments, it is expected that almost anyone can use the editor with minimal training. Users with sufficient credentials can access the underlying simulation data resources. The data can be extended and parameterized to allow modification of the core simulation "rules". Such configurations allow existing simulation data to be quickly modified when reporting more accurate details, and new simulation entities to be created when new systems are introduced. The "doctrine editor" can be configured to allow users with sufficient credentials to modify and create behaviors that control how synthetic entities act in simulations.
[0148] In various embodiments, this may be what a trainer would use to, for example, create an "opposing fighter" entity and assign characteristics to it so that it performs in the simulation in the same manner as would be observed on the battlefield.
[0149] Baseline elements of behavior may be parameterized and easily adjustable (eg, "these enemies are typically equipped with these weapons and demonstrate this level of accuracy at these ranges").
[0150] More complex behavior (e.g. defining specific ambush techniques) may require scripting, but can also be addressed from within the Doctrine Editor.
[0151] Once completed, the behaviors defined here can be saved with a specific entity. This means that when a user places a specific entity in the editor, that entity carries these behaviors with it. Thus, more technical users can define behaviors, and less technical users can easily take advantage of them.
[0152] The training management tool can be configured to provide core management functions for STE. Users with appropriate credentials can use the management tool to manage user accounts, change permissions, review logs, promote new application versions, and perform other management functions.
[0153] In various embodiments, each user accessing STE may have an STE account. Account tools may be used to set up or modify the account, to provide reminders about scheduled training, and to show performance data.
[0154] The "AAR" tool may be accessed by a trainer during a training session (to review performance as the training occurs) or from historical data (to review performance of past training or training performed by a different group). Such a tool may be configured to provide the trainer with the option of displaying a playback of the training exercise. This may be done "to scale" (a life-size rendition of the training) or "on a map" (a top view on a scaled-down image of the training).
[0155] The playback controls can be configured to allow modification of the playback so that the trainer can show portions of the training at a slower speed, jump to different points in the timeline, rewind from a certain point, or bookmark sections for review.
[0156] Filters can be configured to allow trainers to visualize detailed performance information in the replay. For example, the system can be used to visualize the heart rate of a specific operator at a given point, or whether a specific operator cleared a given corner when entering a room.
[0157] The data tool can be configured to provide access to training performance data stored in the data lake. Various common reports can be automatically generated from the tool. These can be organized for different access levels (individual, squad leader, platoon leader, etc.).
[0158] Furthermore, the tool can be configured to manage access to the data pool to allow for more involved, customized queries. Since many commercial data visualization tools already exist, this may aim to manage the receipt and transmission of the required data to the visualizer rather than re-create that functionality.
[0159] An authentication server can be a service that handles the authentication request when the client is started. When approved, the user is allowed to access other services.
[0160] In a denial environment where users cannot connect to the authentication server, authentication can be configured to occur at the local level and only allow use of specific devices or local networks.
[0161] Server resources may be reserved for TSS / S use, allowing cloud deployment of servers for training events.
[0162] Several different aspects of the training operation may require LWM information. Access to this information may be managed by the service. Grid sensor data from a sensor-equipped wearable computing headset (2) may be sent to the service in the form of a versioned raw data stream for pre-processing before being sent to the LWM.
[0163] As described above, cloud-based data storage for TSE may include a "data lake" that may, for example, contain all account information, logs, and training performance data. Fig.77 , data lake information can be made available from virtually any connected location, leveraging access points such as a secure laptop PC.
[0164] Drive space can be maintained to provide cloud storage for scenarios, application installations, patches, archives, and training backups.
[0165] The various synthetic training environment implementations contemplated herein can incorporate information related to any and all equipment used by the military. Various examples are listed below.
[0166] Various embodiments relate to an Infantry Fighting Vehicle ("IFV") crew trainer. This can be a complete crew trainer that can be implemented in a motor pool without any additional resources before the crew, IFV, wearable computing technology (2, 6), training scenario and time. The driver of the IFV can use the wearable system (2, 6) and locate his position in the driver's seat based on pattern recognition of control instruments or visible / scannable markers. This can then provide a world frame for the device to digitally overlay a region-based control layout, which is the same for all positions within the IFV (gunner, tracking commander (TC), loader and vehicle-mounted infantry (dismount)). Now, the wearable computing system (2, 6) can be configured to overlay a rendered copy of the control panel readout (gunner's sight, driver's periscope, etc.) and track user movement for interaction between the user and the IFV. For example, the device can then be configured to render the correct stimulus information at the correct depth (as seen through the gunner's sight) so that the gunner can practice target acquisition and engagement while interacting with other members of the crew. The device may be configured to track this based on the device's location and subsequent head pose. The weapon system may be tracked by vehicle commands, which are recorded via directional microphones and ear protection active noise canceling headphones. This may be configured to record shots fired by enemies that are found and engaged. The theoretical procedure for enemy engagement may be verbally shouted to each other, such as: contact left vehicle-mounted infantry (gunner (g)) - identification (TC) - acquisition target (g) - fire (TC) - destroy target (g). Depending on the target and ammunition selection (e.g. high explosive (HE), armor-piercing and machine gun), the gunner may fire a three-round burst or a rare two-round burst. This data may then be collected and sent to the TMT, where crew and user performance is aggregated with previous training, combat and orthogonal data to update their overall lethality score and training roadmap for future training scenarios. In various embodiments, this may involve taking sensor data from a wearable computing system (2, 6) on the user and creating a dense mesh model of the environment, and also tracking the position / orientation of the user's head within the environment (the user's "head pose"). Such information may be passed to the application layer of the system through an application programming interface ("API"), where there may be a scan mode in the system that is configured to allow the option to view this 3D grid locally without any external information. Off-locally, the system may be configured to extract data from other nodes within the information framework, including position and heading information from other offloaded, onloaded, intelligence surveillance reconnaissance ("ISR"), and external sources included into the 3D grid of the world visible to the warfighter. This may include all land, air, sea, and space forces present in a given situation.The data may be time stamped and geotagged so that the transformation of the location of the spatial data to the user's eyes can be constrained by the location and gaze of the user of the wearable computing system (2, 6).
[0167] In order to correctly overlay information from external sources to the user's perspective, features can be identified and overlaid in a repeatable manner, otherwise artifacts of temporal and spatial aliasing would provide confusing data. For correct overlay, factual data can be used, and a traversable world type architecture can also be used to segment and orient different virtual and real objects to the same location.
[0168] This world map from external sources to internal sources can also be used to see what others are looking at from their device perspective as well as from the perspective of a particular user after the transformation is calculated. This facilitates capabilities such as through-wall augmented reality vision or viewing of remotely captured information (such as a video feed from a drone). Employing a red / green / blue ("RGB") picture camera or a monochrome picture camera of sufficient resolution, the system can be configured to overlay image information on top of the dense grid information. Providing the user with more contextual understanding of what is contained within an environment shared via Net Warrior or some other information transfer protocol and system.
[0169] Referring again to a law enforcement, firefighting, and / or hazardous materials environment, locally, much like the military-style implementation described above, a user can collect data from the world through sensors on the head and body of the user, create a grid of the world, and display it overlaid on a miniature map of the real world and the data displayed and with interactive elements included. All of this is brought together through an API and run for the user through the device's application layer. Non-locally, the system can be configured to utilize many of the same kinds of non-user peripherals and information to relay data to the user of the device, such as in a remote forest firefighting scenario.
[0170] Learning networks, neural networks, and / or so-called "artificial intelligence" (or "AI") computing configurations can be used to live stream adaptive soldier architectures to understand which action information may increase lethality, survivability, and mobility. This can be accomplished via machine learning, where soldiers are given training tasks and the models run through a series of parameters and test cases; based on the output data from the training events, the system can be configured to optimize aspects of the heads-up display of the wearable computing system (2, 6) based on the level of data shown to the individual. This is a way to personalize the level of data fidelity displayed to a specific user. Another implementation is to use machine learning models to dynamically change the data received and displayed in stressful situations, reducing the cognitive burden on the user. Virtual assistants or synthetic characters, such as Fig.90The roles depicted in and described in the above-incorporated references can be used to assist in efficient communication using the present wearable computing configuration (2, 6) in roles such as general assistant, supervisor, colleague, etc.
[0171] Planning capabilities may also be enhanced using configurations of the present system. For example, information about the world may be captured by sensors and then sent to an authoritative server and, once authenticated, to a commander at an operations headquarters or location. Police, firefighters, military, and / or other services may be able to use such system configurations to perform on-the-go planning and scenario rehearsals of incidents prior to arriving at a scene or operational environment. Reference Figures 71-73 , the system can be configured so that the commander can use their connected wearable computing system (2, 6) to watch everything unfold on their floor or coffee table, and can, for example, use gestures to assist in controlling the battle / team / resources; use gestures to redirect teams within a building, etc. In other words, the system can be configured so that the commander views the multi-view information feed through his wearable computing system (2, 6), and then provides gestures to his local device with the overall picture in his mind, where the gesture recognition machine learning technology configuration captures the action and interprets it based on the application definition of the gesture to perform the required task based on the predetermined interaction method.
[0172] As discussed above, system configuration may involve integration with other communication systems utilized by the operator, and may display visualizations of various marker types for obstacles or hazards to be avoided, enemy combatant locations and status, etc., all of which may be geo-registered with the operator's area of action using visualized waypoints, markers, audio, etc. Fig.70 As shown in , the wearable system can be configured to create a mesh related to the geometry and positions of objects around the operator.
[0173] As described above and in the above-incorporated references, global and local dynamic dimming can be used to enhance the visualization of various rendered images in a local environment using a head mounted assembly (2). Figures 74A-74D , showing global dynamic dimming, where the entire display is dimmed or dimmed (i.e., from FIG. 74A to FIG. 74B dimmed, and similarly, the results are shown as FIG. 74C to FIG. 74DLocal dynamic dimming involves contrast adjustment specifically around one or more specific portions of a presented image or content, such as the area toward which the operator's eye gaze is directed. Such configurations can assist the operator in avoiding visual washout, flash blindness, and optical vulnerability to certain weapons that may be referred to as "directed energy weapons." Dynamic dimming can also be used to selectively prevent light leakage (also known as "light safety maintenance" or just "quasi-safety") when the operator is generally in a dark environment, as leakage may be detectable by an adversary and brief leakage of bright light may cause changes in the user's pupil constriction that become suboptimal for the rest of the viewing through the headset (2); therefore, in many scenarios, there is reason to prevent light from leaking into or out of the user's viewing area.
[0174] refer to Fig.75 and 76 , as described above, the system can be integrated with various local target visualization and targeting tools, such as thermal, infrared and / or RBG camera configurations, which can be operably coupled to weapons or other implementations, such as Figures 81-82 and also by hardware and software configurations (such as those described and shown herein, such as in Fig.83 , 84A and 84B) coupled to the data lake and many other resources.
[0175] refer to Fig.78 , the head mounted assembly (2) may be physically integrated with a thermal and / or infrared camera, as discussed above, to assist in identifying certain nearby objects, such as people or animals in an otherwise dark environment, such as Fig.79 as shown in . Fig.80 Certain aspects of a low light camera sensor configuration featuring CMOS sensor technology are shown, where the firmware, system software, and hardware are optimized for the low light configuration.
[0176] Thermal imaging devices can be used in many of the configurations discussed herein, such as in gesture tracking to assist in identifying the boundaries of a human hand relative to other spaces or objects, observing the heat decaying after a hand contact (such as a handprint on a wall), removing certain features (such as humans or animals) or "floating objects" to create a mesh of a room (i.e., it can be assumed that a human-shaped object at 98 degrees Fahrenheit is not a wall or other fixed object). Other known objects can be identified, such as heater vents, windows exposed to sunlight, water heaters, furnaces, fireplaces, etc., and such identification can be used to improve the quality, accuracy, and speed with which the wearable computing resources (2, 6) can be used to understand the world around them and the associated users. For example, in embodiments where the wearable component (2) can be configured to engage in facial recognition analysis, rather than searching the entire room for facial features that match a given database, the system can be configured to first identify easily accessible faces that are likely uncovered and easily thermally identifiable as about 98 degrees Fahrenheit; then once one or more faces are located, more computational resources are spent on those pixels rather than the entire room; in other words, when performing facial recognition, the focus should be on the face, rather than everything in the user's local superset of things, and to this end, a system that can use thermal configuration to identify faces is valuable. For first responders or firefighters, items may be marked as too hot to touch. Inward-facing sensors may be used to detect the operator's body temperature, or outward-facing sensors may be used to sense the condition of other people (e.g., not only indicating the presence of another person across the room, but also indicating that they appear to have a fever, or that they appear to be holding a long gun that appears to have been recently fired; or perhaps their bag appears to have some type of exothermic object similar to a laptop that has been recently operated; or perhaps based on body temperature, the person being viewed appears to have died 8 hours ago). Thermal sensor configurations can also be used to assist in tagging certain objects, such as dogs, cats, or people, and even visually associating certain content or images with them (e.g., the system can be configured to put a bright red hat on each identified enemy combatant; or a pink mustache that moves around with the face of a particular dog or cat). Thermal conditions and gradients can be visualized and color-coded, such as for firefighters encountering an unknown situation.
[0177] In other embodiments, thermal imaging can be used to assist in mapping lighting conditions (i.e., the location and directionality of one or more lighting sources) in a given room to assist in shadow simulation of virtual content presented in AR. Additionally, some objects change shape as temperature changes, which, if known, provides another data point. In various embodiments, if a user with a wearable computing system (2) is in a room and is registered to the room based on visible light camera pose detection - and certain thermal elements (such as heater vents, fish tanks, one or more lights, appliances) are also mapped into this spatial understanding of the room, and then the lights suddenly go out, the user can remain registered with the room by using the thermal elements. Thermal imaging can also be used to assist in horizon determination and matching - where the sun-exposed parts of the world rise in temperature.
[0178] In such Fig.78 In various embodiments shown in , images are corrected for parallax based on the user's actual gaze, or a real-time or near real-time Jacobian transform is performed to account for the fact that one or more imaging devices may not be positioned at the same capture location as the user's eye gaze; this can be done for each eye based on one imaging device, as long as such imaging devices capture two gaze fields of view for both eyes.
[0179] Various embodiments may be configured to also quickly identify windows as planar elements that are at least partially transparent to conventional visible light cameras, but may have time-of-flight signals, scattering, and thermal properties that distinguish them from open space. Traditionally, identification of windows has been a particular challenge for computer vision systems.
[0180] refer to Fig.85 , shows another embodiment of a wearable component (2) that integrates various complex sensors. Various configurations can be directly coupled to the operator's head, helmet, vehicle, aircraft interior, etc.
[0181] refer to Fig.91 , showing a mapping device embodiment in which a mapping / imaging unit removably coupled to a charger base is configured to create a point cloud of a room and overlay an RBG image on a captured point cloud grid of such room, and update it in real time or near real time while operating. Fig.92 A point cloud version of the geometry of a person is shown which can also be viewed, for example, by an RGB camera in sufficient light.
[0182] refer to Fig.93, shows an embodiment of a cuff auxiliary module, where a wrist-wearable cuff similar to a slightly larger watchband can be coupled to a user when certain types of biometric monitoring are needed (such as in an emergency or dangerous situation) so that others can utilize the collected and available data (i.e., such as through a cloud connection) to monitor factors such as location (such as through GPS, mobile transceiver triangulation and / or IP address), heart condition / output, temperature, etc. The cuff can also contain certain medical components, such as analgesics, which can be introduced into the user's body through the cuff manually or automatically (i.e., such as remotely from the operations center or a connected colleague). In various embodiments, it can be wirelessly coupled to the wearable computing assets (2, 6) described herein.
[0183] Mixed reality environment
[0184] Like all people, users of mixed reality systems exist in a real environment, that is, the three-dimensional portion of the "real world" and all of its contents that are perceptible to the user. For example, users perceive the real environment using ordinary human senses—sight, hearing, touch, taste, smell—and interact with the real environment by moving their bodies through the real environment. Positions in the real environment can be described as coordinates in a coordinate space; for example, coordinates can include latitude, longitude, and altitude relative to sea level; distances in three orthogonal dimensions from a reference point; or other suitable values. Similarly, vectors can describe quantities that have direction and magnitude in a coordinate space.
[0185] A computing device may maintain a representation of a virtual environment, for example, in a memory associated with the device. As used herein, a virtual environment is a computational representation of a three-dimensional space. A virtual environment may include representations of any objects, actions, signals, parameters, coordinates, vectors, or other features associated with the space. In some examples, a circuit (e.g., a processor) of a computing device may maintain and update the state of a virtual environment; that is, the processor may determine the state of the virtual environment at a second time t1 based on data associated with the virtual environment and / or input provided by a user at a first time t0. For example, if an object in a virtual environment is located at a first coordinate at time t0 and has certain programmed physical parameters (e.g., mass, coefficient of friction); and an input received from a user indicates that a force should be applied to the object in a direction vector; the processor may apply the laws of kinematics using basic mechanics to determine the position of the object at time t1. The processor may use any suitable information known about the virtual environment and / or any suitable input to determine the state of the virtual environment at time t1. In maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software related to the creation and deletion of virtual objects in the virtual environment; software (e.g., scripts) for defining the behavior of virtual objects or characters in the virtual environment; software for defining the behavior of signals (e.g., audio signals) in the virtual environment; software for creating and updating parameters associated with the virtual environment; software for generating audio signals in the virtual environment; software for processing input and output; software for implementing network operations; software for applying asset data (e.g., animation data that moves virtual objects over time); or many other possibilities.
[0186] An output device such as a display or a speaker can present any or all aspects of the virtual environment to the user. For example, the virtual environment can include virtual objects (which can include inanimate objects; people; animals; representations of lights, etc.) that can be presented to the user. The processor can determine a view of the virtual environment (e.g., corresponding to a "camera" with origin coordinates, view axes, and a view cone); and render the visual scene of the virtual environment corresponding to the view to the display. Any suitable rendering technology can be used for this purpose. In some examples, the visual scene can include only some virtual objects in the virtual environment, while excluding certain other virtual objects. Similarly, the virtual environment can include audio aspects that can be presented to the user as one or more audio signals. For example, a virtual object in the virtual environment can generate sounds originating from the position coordinates of the object (e.g., a virtual character can speak or cause a sound effect); or the virtual environment can be associated with a music cue or ambient sound, which may or may not be associated with a specific location. The processor may determine audio signals corresponding to the “listener” coordinates—e.g., audio signals corresponding to a sound synthesis in the virtual environment and mixed and processed to simulate the audio signals heard by the listener at the listener coordinates—and present the audio signals to the user via one or more speakers.
[0187] Since the virtual environment exists only as a computing structure, the user cannot directly perceive the virtual environment using ordinary senses. Instead, the user can only indirectly perceive the virtual environment presented to the user, for example, through a display, a speaker, a tactile output device, etc. Similarly, the user cannot directly touch, manipulate, or otherwise interact with the virtual environment; but can provide input data to the processor via an input device or sensor, and the processor can use the device or sensor data to update the virtual environment. For example, a camera sensor can provide optical data indicating that the user is trying to move an object in the virtual environment, and the processor can use the data to make the object respond accordingly in the virtual environment. As another example, an accelerometer, a gyroscope sensor, or other types of sensors in a peripheral device (e.g., the controller 300 described below) can provide motion data and / or device orientation data indicating that the user is trying to use a peripheral device to manipulate an object in the virtual environment, and the processor can use the data to make the object respond accordingly in the virtual environment. Other suitable types of sensor data can also be used.
[0188] A mixed reality system may present a mixed reality environment ("MRE") that combines aspects of a real environment and a virtual environment to a user, for example using a transmissive display and / or one or more speakers (which may be incorporated, for example, into a wearable head device). In some embodiments, the one or more speakers may be external to the head-mounted wearable unit. As used herein, an MRE is a simultaneous representation of a real environment and a corresponding virtual environment. In some examples, the corresponding real and virtual environments share a single coordinate space; in some examples, the real coordinate space and the corresponding virtual coordinate space are related to each other by a transformation matrix (or other suitable representation). Thus, a single coordinate (in some examples, together with a transformation matrix) may define a first location in the real environment, and a second corresponding location in the virtual environment; or vice versa.
[0189] In an MRE, a virtual object (e.g., in a virtual environment associated with the MRE) may correspond to a real object (e.g., in a real environment associated with the MRE). For example, if the real environment of the MRE includes a real lamp post (real object) at a position coordinate, the virtual environment of the MRE may include a virtual lamp post (virtual object) at the corresponding position coordinate. As used herein, a real object and its corresponding virtual object are combined to form a "mixed reality object". It is not necessary for a virtual object to perfectly match or align with the corresponding real object. In some examples, a virtual object may be a simplified version of the corresponding real object. For example, if the real environment includes a real lamp post, the corresponding virtual object may include a cylinder with approximately the same height and radius as the real lamp post (reflecting that the lamp post may be approximately cylindrical in shape). Simplifying virtual objects in this manner may allow for computational efficiency and may simplify calculations performed on such virtual objects. In addition, in some examples of the MRE, not all real objects in the real environment may be associated with corresponding virtual objects. Similarly, in some examples of the MRE, not all virtual objects in the virtual environment may be associated with corresponding real objects. That is, some virtual objects may only be in the virtual environment of the MRE without any real-world counterparts.
[0190] In some examples, virtual objects may have characteristics that are different, and sometimes even completely different, from those of corresponding real objects. For example, while the real environment in the MRE may include a green, two-armed cactus—an inanimate object with thorns—the corresponding virtual object in the MRE may have the characteristics of a green, two-armed virtual character with human facial features and a menacing look. In this example, the virtual object is similar to its corresponding real object in some characteristics (color, number of arms); but is different from the real object in other characteristics (facial features, personality). In this way, it is possible for virtual objects to represent real objects in creative, abstract, exaggerated, or fantasy ways; or to assign behaviors (e.g., human personalities) to otherwise inanimate real objects. In some examples, virtual objects may be purely fantasy creations that have no real-world counterparts (e.g., virtual monsters in a virtual environment may be located in locations that correspond to empty spaces in the real environment).
[0191] Compared to VR systems that present a virtual environment to the user while obscuring the real environment, a mixed reality system that presents an MRE provides the advantage that the real environment remains perceptible while the virtual environment is presented. Therefore, users of the mixed reality system are able to experience and interact with the corresponding virtual environment using visual and audio cues associated with the real environment. For example, while users of VR systems may find it difficult to perceive or interact with virtual objects displayed in a virtual environment—because, as described above, users cannot directly perceive or interact with the virtual environment—users of MR systems may find it intuitive and natural to interact with virtual objects by seeing, hearing, and touching the corresponding real objects in his or her own real environment. This level of interactivity can enhance the user's sense of immersion, connection, and participation in the virtual environment. Similarly, by presenting the real environment and the virtual environment simultaneously, the mixed reality system can reduce the negative psychological feelings (such as cognitive dissonance) and negative physical feelings (such as motion sickness) associated with VR systems. Mixed reality systems further provide many possibilities for applications that may enhance or change our experience of the real world.
[0192] Fig.105A An example real environment 100 is shown in which a user 110 uses a mixed reality system 112. For example, as described below, the mixed reality system 112 may include a display (e.g., a transmissive display) and one or more speakers, as well as one or more sensors (e.g., a camera). The real environment 100 shown includes a rectangular room 104A in which the user 110 is standing; and real objects 122A (lamp), 124A (table), 126A (couch), and 128A (painting). The room 104A further includes a position coordinate 106, which can be considered to be the origin of the real environment 100. As shown in Fig.105AAs shown in , an environment / world coordinate system 108 (including an x-axis 108X, a y-axis 108Y, and a z-axis 108Z) with its origin at point 106 (world coordinates) can define a coordinate space for the real environment 100. In some embodiments, the origin 106 of the environment / world coordinate system 108 can correspond to the location where the mixed reality system 112 is powered on. In some embodiments, the origin 106 of the environment / world coordinate system 108 can be reset during operation. In some examples, the user 110 can be considered a real object in the real environment 100; similarly, the body parts of the user 110 (e.g., hands, feet) can be considered real objects in the real environment 100. In some examples, a user / listener / head coordinate system 114 (including an x-axis 114X, a y-axis 114Y, and a z-axis 114Z) with its origin at point 115 (e.g., user / listener / head coordinates) can define a coordinate space for the user / listener / head in which the mixed reality system 112 is located. The origin 115 of the user / listener / head coordinate system 114 may be defined relative to one or more components of the mixed reality system 112. For example, such as during initial calibration of the mixed reality system 112, the origin 115 of the user / listener / head coordinate system 114 may be defined relative to a display of the mixed reality system 112. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation may characterize the transformation between the user / listener / head coordinate system 114 space and the environment / world coordinate system 108 space. In some embodiments, left ear coordinates 116 and right ear coordinates 117 may be defined relative to the origin 115 of the user / listener / head coordinate system 114. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation may characterize the transformation between the left ear coordinates 116 and right ear coordinates 117 and the user / listener / head coordinate system 114 space. The user / listener / head coordinate system 114 can simplify the representation of a position relative to the user's head or head mounted device (e.g., relative to the environment / world coordinate system 108). The transformation between the user coordinate system 114 and the environment coordinate system 108 can be determined and updated in real time using simultaneous localization and mapping (SLAM), visual odometry, or other techniques.
[0193] Fig.105BAn example virtual environment 130 corresponding to the real environment 100 is shown. The virtual environment 130 shown includes a virtual rectangular room 104B corresponding to the real rectangular room 104A; a virtual object 122B corresponding to the real object 122A; a virtual object 124B corresponding to the real object 124A; and a virtual object 126B corresponding to the real object 126A. Metadata associated with the virtual objects 122B, 124B, 126B may include information derived from the corresponding real objects 122A, 124A, 126A. The virtual environment 130 additionally includes a virtual monster 132 that does not correspond to any real object in the real environment 100. The real object 128A in the real environment 100 does not correspond to any virtual object in the virtual environment 130. A persistent coordinate system 133 (including an x-axis 133X, a y-axis 133Y, and a z-axis 133Z) having its origin at point 134 (persistent coordinates) may define a coordinate space for virtual content. The origin 134 of the persistent coordinate system 133 may be defined relative to / with respect to one or more real objects, such as the real object 126A. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation may characterize the transformation between the persistent coordinate system 133 space and the environment / world coordinate system 108 space. In some embodiments, each of the virtual objects 122B, 124B, 126B, and 132 may have their own persistent coordinate point relative to the origin 134 of the persistent coordinate system 133. In some embodiments, there may be multiple persistent coordinate systems and each of the virtual objects 122B, 124B, 126B, and 132 may have their own persistent coordinate point relative to one or more persistent coordinate systems.
[0194] about Fig.105A and 105B , the environment / world coordinate system 108 defines a shared coordinate space for both the real environment 100 and the virtual environment 130. In the example shown, the coordinate space has an origin at point 106. In addition, the coordinate space is defined by the same three orthogonal axes (108X, 108Y, 108Z). Therefore, a first position in the real environment 100 and a second corresponding position in the virtual environment 130 can be described with respect to the same coordinate space. This simplifies the identification and display of corresponding positions in the real and virtual environments because the same coordinates can be used to identify the two positions. However, in some examples, the corresponding real and virtual environments do not need to use a shared coordinate space. For example, in some examples (not shown), a matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.
[0195] Fig.105CAn example MRE 150 is shown that simultaneously presents aspects of a real environment 100 and a virtual environment 130 to a user 110 via a mixed reality system 112. In the example shown, the MRE 150 simultaneously presents real objects 122A, 124A, 126A, and 128A from the real environment 100 (e.g., via a transmissive portion of a display of the mixed reality system 112); and virtual objects 122B, 124B, 126B, and 132 from the virtual environment 130 (e.g., via an actively displayed portion of the display of the mixed reality system 112) to the user 110. As described above, the origin 106 serves as the origin of a coordinate space corresponding to the MRE 150, and the coordinate system 108 defines an x-axis, a y-axis, and a z-axis for the coordinate space.
[0196] In the example shown, the mixed reality objects include corresponding real objects and virtual object pairs (i.e., 122A / 122B, 124A / 124B, 126A / 126B) occupying corresponding positions in the coordinate space 108. In some examples, both the real object and the virtual object can be visible to the user 110 at the same time. For example, this may be desirable in cases where the virtual object presents information designed to enhance the view of the corresponding real object (such as in a museum application, where the virtual object presents the missing part of an ancient damaged sculpture). In some examples, the virtual object (122B, 124B, and / or 126B) can be displayed (e.g., via active pixelated occlusion using a pixelated occlusion shutter) so as to occlude the corresponding real object (122A, 124A, and / or 126A). For example, this may be desirable in cases where the virtual object acts as a visual substitute for the corresponding real object (such as in an interactive storytelling application, where an inanimate real object becomes a "living" character).
[0197] In some examples, real objects (e.g., 122A, 124A, 126A) can be associated with virtual content or assistant data that does not necessarily constitute a virtual object. The virtual content or assistant data can facilitate the processing or handling of virtual objects in a mixed reality environment. For example, such virtual content may include a two-dimensional representation of the corresponding real object; a custom asset type associated with the corresponding real object; or statistics associated with the corresponding real object. This information can enable or facilitate calculations involving real objects without incurring unnecessary computational overhead.
[0198] In some examples, the above presentations may also incorporate audio aspects. For example, in the MRE 150, the virtual monster 132 may be associated with one or more audio signals, such as footstep effects generated when the monster walks around the MRE 150. As further described herein, the processor of the mixed reality system 112 may calculate an audio signal corresponding to a composite of the mixing and processing of all such sounds in the MRE 150, and present the audio signal to the user 110 via one or more speakers included in the mixed reality system 112 and / or one or more external speakers.
[0199] Example Mixed Reality System
[0200] An example mixed reality system 112 may include: a wearable head device (e.g., a wearable augmented reality or mixed reality head device) including a display (which may include a left transmissive display and a right transmissive display that may be a near-eye display, and associated components for coupling light from the display to the user's eyes); a left speaker and a right speaker (e.g., positioned adjacent to the user's left and right ears, respectively); an inertial measurement unit (IMU) (e.g., mounted to the temple of the head device); an orthogonal coil electromagnetic receiver (e.g., mounted to the left temple piece); a left camera and a right camera oriented away from the user (e.g., a depth (time of flight) camera); and a left eye camera and a right eye camera oriented toward the user (e.g., for detecting eye movements of the user). However, the mixed reality system 112 may incorporate any suitable display technology and any suitable sensor (e.g., optical, infrared, acoustic, LIDAR, EOG, GPS, magnetic). In addition, the mixed reality system 112 may incorporate networking features (e.g., Wi-Fi capabilities) to communicate with other devices and systems including other mixed reality systems. The mixed reality system 112 may further include a battery (which may be installed in an auxiliary unit, such as a waist pack designed to be worn around the user's waist), a processor, and a memory. The wearable head device of the mixed reality system 112 may include a tracking component, such as an IMU or other suitable sensor, which is configured to output a set of coordinates of the wearable head device relative to the user's environment. In some examples, the tracking component may provide input to a processor that performs simultaneous localization and mapping (SLAM) and / or visual odometry. In some examples, as further described below, the mixed reality system 112 may also include a handheld controller 300 and / or an auxiliary unit 320 that may be a wearable waist pack.
[0201] Figures 106A-106D Components of an example mixed reality system 200 (which may correspond to mixed reality system 112 ) that may be used to present an MRE (which may correspond to MRE 150 ) or other virtual environment to a user are shown. Fig.106AA perspective view of a wearable head device 2102 included in an example mixed reality system 200 is shown. Fig.106B A top view of a wearable head device 2102 worn on a user's head 2202 is shown. Fig.106C A front view of the wearable head device 2102 is shown. Fig.106D An edge view of an example eyepiece 2110 of a wearable head device 2102 is shown. Figures 106A-106C As shown in , the example wearable head device 2102 includes an example left eyepiece (e.g., a left transparent waveguide group eyepiece) 2108 and an example right eyepiece (e.g., a right transparent waveguide group eyepiece) 2110. Each eyepiece 2108 and 2110 may include a transmissive element through which a real environment can be seen, and a display element for presenting a display (e.g., via image modulated light) that overlaps with the real environment. In some examples, such display elements may include surface diffractive optical elements for controlling the flow of image modulated light. For example, the left eyepiece 2108 may include a left incoupling grating group 2112, a left orthogonal pupil expansion (OPE) grating group 2120, and a left exit (output) pupil expansion (EPE) grating group 2122. Similarly, the right eyepiece 2110 may include a right incoupling grating group 2118, a right OPE grating group 2114, and a right EPE grating group 2116. The image modulated light may be transmitted to the user's eyes via the in-coupling gratings 2112 and 2118, the OPEs 2114 and 2120, and the EPEs 2116 and 2122. Each in-coupling grating group 2112, 2118 may be configured to deflect light toward its corresponding OPE grating group 2120, 2114. Each OPE grating group 2120, 2114 may be designed to deflect light gradually downward toward its associated EPE 2122, 2116, thereby horizontally extending the formed exit pupil. Each EPE 2122, 2116 may be configured to gradually redirect at least a portion of the light received from its corresponding OPE grating group 2120, 2114 outward to a user's eye-tracking position (not shown) defined behind the eyepieces 2108, 2110, thereby vertically extending the exit pupil formed at the eye-tracking position. Alternatively, in place of the in-coupling grating groups 2112 and 2118, the OPE grating groups 2114 and 2120, and the EPE grating groups 2116 and 2122, the eyepieces 2108 and 2110 may include other arrangements of gratings and / or refractive and reflective features for controlling the coupling of image modulated light to the user's eyes.
[0202] In some examples, the wearable head device 2102 may include a left temple 2130 and a right temple 2132, wherein the left temple 2130 includes a left speaker 2134 and the right temple 2132 includes a right speaker 2136. The orthogonal coil electromagnetic receiver 2138 may be located in the left temple piece, or another suitable location in the wearable head unit 2102. An inertial measurement unit (IMU) 2140 may be located in the right temple 2132, or another suitable location in the wearable head unit 2102. The wearable head device 2102 may also include a left depth (e.g., time of flight) camera 2142 and a right depth camera 2144. The depth cameras 2142, 2144 may be appropriately oriented in different directions so as to collectively cover a wider field of view.
[0203] exist Figures 106A-106D In the example shown in , a left image modulated light source 2124 can be optically coupled into the left eyepiece 2108 via a left incoupling grating set 2112, and a right image modulated light source 2126 can be optically coupled into the right eyepiece 2110 via a right incoupling grating set 2118. The image modulated light sources 2124, 2126 can include, for example, a fiber scanner; a projector including an electronic light modulator such as a digital light processing (DLP) chip or a liquid crystal on silicon (LCoS) modulator; or an emissive display such as a micro light emitting diode (μLED) or micro organic light emitting diode (μOLED) panel coupled into the incoupling grating set 2112, 2118 using one or more lenses on each side. The incoupling grating set 2112, 2118 can deflect light from the image modulated light sources 2124, 2126 to an angle above the critical angle for total internal reflection (TIR) of the eyepieces 2108, 2110. The OPE grating groups 2114, 2120 gradually deflect light that propagates via TIR downward toward the EPE grating groups 2116, 2122. The EPE grating groups 2116, 2122 gradually couple the light toward the user's face, including the pupils of the user's eyes.
[0204] In some examples, such as Fig.106D As shown in , each of the left eyepiece 2108 and the right eyepiece 2110 includes a plurality of waveguides 2402. For example, each eyepiece 2108, 2110 may include a plurality of separate waveguides, each dedicated to a corresponding color channel (e.g., red, blue, and green). In some examples, each eyepiece 2108, 2110 may include multiple groups of such waveguides, each group being configured to impart a different wavefront curvature to the emitted light. The wavefront curvature may be convex with respect to the user's eyes, for example to present a virtual object located at a certain distance in front of the user (e.g., a distance corresponding to the inverse of the wavefront curvature). In some examples, the EPE grating groups 2116, 2122 may include curved grating grooves to achieve a convex wavefront curvature by changing the Poynting vector of the outgoing light across each EPE.
[0205] In some examples, to create the perception that the displayed content is three-dimensional, stereoscopically accommodative left-eye and right-eye images can be presented to the user through the image light modulators 2124, 2126 and the eyepieces 2108, 2110. The perceived realism of the presentation of three-dimensional virtual objects can be enhanced by selecting the waveguides (and therefore the corresponding wavefront curvature) so that the virtual objects are displayed at a distance that is approximately the distance indicated by the stereoscopic left and right images. This technique can also reduce motion sickness experienced by some users, which can be caused by the difference between the depth perception cues provided by the stereoscopic left and right eye images and the autonomous accommodation of the human eye (e.g., focus related to object distance).
[0206] Fig.106D An edge-facing view from the top of the right eyepiece 2110 of the example wearable head device 2102 is shown. Fig.106D As shown in , the plurality of waveguides 2402 may include a first subset 2404 of three waveguides and a second subset 2406 of three waveguides. The two subsets 2404, 2406 of waveguides may be distinguished by different EPE gratings characterized by different grating line curvatures to impart different wavefront curvatures to the outgoing light. Within each of the waveguide subsets 2404, 2406, each waveguide may be used to couple a different spectral channel (e.g., one of the red, green, and blue spectral channels) to the user's right eye 2206. Although in Fig.106D Not shown in the figure, the structure of the left eyepiece 2108 is similar to that of the right eyepiece 2110.
[0207] Fig.107AAn example handheld controller assembly 300 of a mixed reality system 200 is shown. In some examples, the handheld controller 300 includes a grip portion 346 and one or more buttons 350 disposed along a top surface 348. In some examples, the button 350 may be configured to be used as an optical tracking target, for example, for tracking six degrees of freedom (6DOF) motion of the handheld controller 300 in conjunction with a camera or other optical sensor (which may be mounted in a head unit (e.g., wearable head device 2102) of the mixed reality system 200). In some examples, the handheld controller 300 includes a tracking component (e.g., an IMU or other suitable sensor) for detecting a position or orientation (such as a position or orientation relative to the wearable head device 2102). In some examples, such a tracking component may be positioned in a handle of the handheld controller 300 and / or may be mechanically coupled to the handheld controller. The handheld controller 300 may be configured to provide one or more output signals (e.g., via an IMU) corresponding to one or more pressed states of a button or a position, orientation, and / or motion of the handheld controller 300. Such output signals may be used as inputs to a processor of the mixed reality system 200. Such input may correspond to the position, orientation and / or movement of the handheld controller (and, by extension, to the position, orientation and / or movement of the user's hand holding the controller). Such input may also correspond to the user pressing button 350.
[0208] Fig.107B An example auxiliary unit 320 of the mixed reality system 200 is shown. The auxiliary unit 320 may include a battery to provide energy to operate the system 200, and may include a processor for executing a program to operate the system 200. As shown, the example auxiliary unit 320 includes a clip 2128, such as for attaching the auxiliary unit 320 to a user's belt. Other form factors are suitable for the auxiliary unit 320 and will be apparent, including form factors that do not involve mounting the unit to a user's belt. In some examples, the auxiliary unit 320 is coupled to the wearable head device 2102 via a multi-conductor cable that may include, for example, wires and optical fibers. A wireless connection between the auxiliary unit 320 and the wearable head device 2102 may also be used.
[0209] In some examples, the mixed reality system 200 may include one or more microphones to detect sounds and provide corresponding signals to the mixed reality system. In some examples, the microphone may be attached to or integrated with the wearable head device 2102 and may be configured to detect the user's voice. In some examples, the microphone may be attached to or integrated with the handheld controller 300 and / or the auxiliary unit 320. Such microphones may be configured to detect ambient sounds, ambient noise, the voice of the user or a third party, or other sounds.
[0210] Fig.108 An example functional block diagram is shown, which may correspond to an example mixed reality system, such as the mixed reality system 200 described above (which may correspond to the example mixed reality system 200 described above). Figure 1 Mixed reality system 112). Fig.108 As shown in , an example handheld controller 400B (which may correspond to the handheld controller 300 ("totem")) includes a totem to wearable head device six degrees of freedom (6DOF) totem subsystem 404A, and an example wearable head device 400A (which may correspond to the wearable head device 2102) includes a totem to wearable head device 6DOF subsystem 404B. In the example, the 6DOF totem subsystem 404A and the 6DOF subsystem 404B cooperate to determine the six coordinates (e.g., offsets in three translation directions and rotations along three axes) of the handheld controller 400B relative to the wearable head device 400A. The six degrees of freedom can be expressed relative to the coordinate system of the wearable head device 400A. The three translation offsets can be expressed as X, Y, and Z offsets in such a coordinate system, a translation matrix, or some other representation. The rotational degrees of freedom can be expressed as a sequence of yaw, pitch, and roll rotations, a rotation matrix, a quaternion, or some other representation. In some examples, the wearable head device 400A; one or more depth cameras 444 (and / or one or more non-depth cameras) included in the wearable head device 400A; and / or one or more optical targets (e.g., the button 350 of the handheld controller 400B as described above, or a dedicated optical target included in the handheld controller 400B) can be used for 6DOF tracking. In some examples, the handheld controller 400B may include a camera as described above; and the wearable head device 400A may include an optical target for optical tracking in conjunction with the camera. In some examples, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids that are used to wirelessly send and receive three distinguishable signals. By measuring the relative size of the three distinguishable signals received in each of the coils for receiving, the 6DOF of the wearable head device 400A relative to the handheld controller 400B can be determined. Additionally, the 6DOF totem subsystem 404A may include an inertial measurement unit (IMU) that may be used to provide improved accuracy and / or more timely information regarding rapid movements of the handheld controller 400B.
[0211] In some examples, it may be necessary to transform coordinates from a local coordinate space (e.g., a coordinate space that is fixed relative to the wearable head device 400A) to an inertial coordinate space (e.g., a coordinate space that is fixed relative to the real environment), for example, in order to compensate for movement of the wearable head device 400A relative to the coordinate system 108. For example, such a transformation may be necessary for the display of the wearable head device 400A to present a virtual object (e.g., a virtual person sitting in a real chair, facing forward, regardless of the position and orientation of the wearable head device) at an expected position and orientation relative to the real environment rather than at a fixed position and orientation on the display (e.g., in the same position in the lower right corner of the display) to maintain the illusion that the virtual object exists in the real environment (and, for example, does not appear unnaturally positioned in the real environment as the wearable head device 400A moves and rotates). In some examples, the compensating transformation between coordinate spaces can be determined by processing images from the depth camera 444 using a SLAM and / or visual odometry program to determine the transformation of the wearable head device 400A relative to the coordinate system 108. In such ... Fig.108 In the example shown in , the depth camera 444 is coupled to the SLAM / visual odometry block 406 and can provide images to the block 406. The SLAM / visual odometry block 406 implementation may include a processor configured to process the image and determine the position and orientation of the user's head, which can then be used to identify the transformation between the head coordinate space and another coordinate space (e.g., inertial coordinate space). Similarly, in some examples, an additional source of information about the user's head pose and position is obtained from the IMU 409. Information from the IMU 409 can be integrated with information from the SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information for rapid adjustment of the user's head pose and position.
[0212] In some examples, depth camera 444 can provide 3D images to gesture tracker 411, which can be implemented in a processor of wearable head device 400A. Gesture tracker 411 can recognize the user's gesture, for example, by matching the 3D image received from depth camera 444 with a stored pattern representing a gesture. Other suitable techniques for recognizing user gestures will be apparent.
[0213] In some examples, one or more processors 416 can be configured to receive data from the 6DOF helmet subsystem 404B, IMU 409, SLAM / visual odometry block 406, depth camera 444, and / or gesture tracker 411 of the wearable head device. The processor 416 can also send and receive control signals from the 6DOF totem system 404A. For example, in an example where the handheld controller 400B is untethered, the processor 416 can be wirelessly coupled to the 6DOF totem system 404A. The processor 416 can further communicate with additional components such as an audio-visual content memory 418, a graphics processing unit (GPU) 420, and / or a digital signal processor (DSP) audio spacer 422. The DSP audio spacer 422 can be coupled to a head-related transfer function (HRTF) memory 425. The GPU 420 may include a left channel output coupled to a left image modulated light source 424 and a right channel output coupled to a right image modulated light source 426. For example, as described above with respect to Figures 106A-106D As described, the GPU 420 may output stereoscopic image data to the image modulated light sources 424, 426. The DSP audio spatializer 422 may output audio to the left speaker 412 and / or the right speaker 414. The DSP audio spatializer 422 may receive an input from the processor 419 indicating a direction vector from the user to the virtual sound source (which may be moved by the user, for example, via the handheld controller 320). Based on the direction vector, the DSP audio spatializer 422 may determine a corresponding HRTF (e.g., by accessing the HRTF, or by interpolating multiple HRTFs). The DSP audio spatializer 422 may then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object. This may enhance the credibility and realism of the virtual sound by incorporating the user's relative position and orientation with respect to the virtual sound in the mixed reality environment—that is, by presenting a virtual sound that matches the user's expectation that the virtual sound sounds like a real sound in a real environment.
[0214] In some examples, such as Fig.108 As shown in , one or more of the processor 416, GPU 420, DSP audio spatializer 422, HRTF memory 425, and audio / visual content memory 418 may be included in an auxiliary unit 400C (which may correspond to the auxiliary unit 320 described above). Auxiliary unit 400C may include a battery 427 to power its components and / or to power the wearable head device 400A or handheld controller 400B. Including such components in an auxiliary unit that can be mounted to the user's waist can limit the size and weight of the wearable head device 400A, which in turn can reduce fatigue on the user's head and neck.
[0215] Although Fig.108Elements corresponding to various components of an example mixed reality system are presented, but various other suitable arrangements of these components will become apparent to those skilled in the art. For example, as associated with auxiliary unit 400C Fig.108 The elements presented in the embodiment may be associated with the wearable head device 400A or the handheld controller 400B instead. In addition, some mixed reality systems may completely abandon the handheld controller 400B or the auxiliary unit 400C. Such changes and modifications should be understood to be included within the scope of the disclosed examples.
[0216] System Architecture
[0217] The goal of a mixed reality system may be to synthesize real-world information and provide a user with information that may not otherwise be available and / or easily accessible to the user. The advantages of a mixed reality system may be particularly evident when a user needs to process a large amount of information in a short period of time to make a decision. In these cases, a mixed reality system is particularly suitable for assisting the user in making decisions by giving the user easy access to relevant information. For example, in a combat situation, a soldier may benefit from information such as the remaining ammunition in his weapon's magazine, the location of squad members, and / or the location of enemy combatants. This information may be presented to the soldier via, for example, a perspective display of the mixed reality system—information that may not otherwise be available to the soldier.
[0218] Because mixed reality systems can receive and interpret large amounts of real-world information, efficient computing architectures can produce improved user experiences and enhanced capabilities for mixed reality systems. Efficient computing architectures can allow mixed reality systems to quickly process input from a variety of real-world information sources. For example, Figure 8 , a mixed reality system including a headset 2 and a waist pack 6 (which may correspond to MR systems 112, 200) may receive real-world input from one or more weapons, which may be communicatively coupled to the mixed reality system. One or more weapons may each include one or more sensors coupled to a microprocessor. For example, a sensor may detect the amount of ammunition remaining in a magazine, and this information may be sent to the microprocessor. The weapon's microprocessor may transmit this information to the mixed reality system via wired and / or wireless communications. For example, the weapon's microprocessor may communicate with the mixed reality system using a universal serial bus ("USB") connection and / or radio frequency communications (e.g., WiFi, Bluetooth, cellular networks, 5G).
[0219] In some embodiments, it may be more efficient for a mixed reality system to communicate with a device that includes a microprocessor. For example, a firefighter kit monitor may include a fire extinguisher, and the fire extinguisher may include one or more sensors configured to detect the amount of suppressant remaining in the fire extinguisher. In some embodiments, the fire extinguisher may include a microprocessor configured to communicate with one or more sensors. The microprocessor may be configured to receive raw inputs (e.g., voltage levels) from the sensors and calculate the remaining amount of suppressant accordingly. The microprocessor of the fire extinguisher may then transmit this information to the mixed reality system in a data structure that the mixed reality system can read. The computing architecture may reduce the computing burden placed on the mixed reality system by offloading one or more calculations to the microprocessor of the connected device.
[0220] In some embodiments, it may be more efficient for the mixed reality system to communicate directly with sensors on the connected device. For example, the Cuff Assistant device may be worn on the user's wrist, and the Cuff Assistant may include a sensor configured to measure the user's heart rate. In some embodiments, the sensor may communicate directly with the mixed reality system (e.g., without an intermediate device microprocessor) using wired and / or wireless means. The mixed reality system may be configured to receive raw inputs (e.g., voltage levels) from the sensor and calculate the heart rate accordingly. It may be more cost-effective for the connected device to include a sensor that communicates directly with the mixed reality system, and for the connected device to omit a separate microprocessor. In some embodiments, omitting a device-specific microprocessor can reduce the manufacturing cost of the connected device, but may increase the computational burden on the mixed reality system that serves as a hub for the connected device. For example, the mixed reality system may be required to calculate the heart rate based on the voltage reading, rather than offloading the calculation to the device-specific microprocessor.
[0221] Fig.57 An exemplary computing architecture for large-scale mixed reality applications is shown. In some embodiments, multiple mixed reality systems (e.g., MR systems 112, 200) can be connected to each other and / or to an external computing system (e.g., a cloud-based server). The interconnected computing architecture can enable large-scale mixed reality applications that may rely on large amounts of data and computing resources. In some embodiments, an efficient computing architecture can balance the needs for fast, low-latency computing and complex, processing-intensive computing. In some embodiments, an efficient computing architecture can implement edge computing principles. Edge computing principles can include dividing computing tasks across multiple systems and / or strategically positioning computing systems to minimize communication delays between systems.
[0222] For example, Fig.57The exemplary computing architecture shown in can be used for military training exercises. In large-scale mixed reality war game applications, many soldiers (e.g., 100, 1,000 or 10,000) can be deployed to test, develop and / or implement military strategies. Individual soldiers may be equipped with their own mixed reality systems, which may include head-mounted devices (e.g., MR systems 112, 200). Individual mixed reality systems can capture a large amount of real-world data. For example, individual mixed reality systems can capture soldier vital signs, remaining ammunition, soldier positioning, communications, weapon status, etc. The mixed reality system can also capture information about the position of real objects and / or the position / orientation of soldiers about real objects (e.g., via SLAM). In some embodiments, the information captured by the mixed reality system (e.g., a mixed reality system used by a single soldier) can communicate with each other and / or with an external computing system (e.g., one or more servers). The mixed reality system can communicate with each other and / or with an external computing system via any suitable means. For example, the mixed reality system can communicate via wired and / or wireless connections, including but not limited to: USB connection, cellular network, WiFi, Bluetooth, 5G (millimeter wave and / or sub-6GHz), etc.
[0223] In some embodiments, the individual mixed reality systems can be organized into one or more groups. Fig.57 As shown in , a group of 10 soldiers may be assigned to a specific squad / platoon / company. The mixed reality system used by those 10 soldiers may communicate with a remote operation server 5702. In some embodiments, the remote operation server 5702 may aggregate data collected by the squad's mixed reality system. In some embodiments, computationally complex tasks that may not require low-latency operation may be performed on the remote operation server 5702. For example, each individual mixed reality system may generate its own understanding of the position and positioning of real objects (e.g., via SLAM). In some embodiments, this information may be transmitted to the remote operation server 5702, which may optimize the position and positioning of objects determined across multiple mixed reality systems. Multiple soldiers may repeatedly observe the same real object from different perspectives, and synthesizing multiple SLAM data streams may produce a more accurate estimate of the position and positioning of real-world objects.
[0224] In some embodiments, a separate mixed reality system can display virtual objects based on a local preliminary SLAM calculation. It may be beneficial for a separate mixed reality system to perform a "first pass" SLAM estimate because the mixed reality system can rely on SLAM to visually display virtual objects related to real objects (e.g., virtual targets can appear in the same real location when soldiers move around). Locally calculating SLAM can allow the mixed reality system to visually display virtual objects in a low-latency manner, which can improve the user experience (e.g., by allowing the mixed reality system to display virtual objects in real time). In some embodiments, SLAM optimization can be performed remotely (e.g., at a remote operation server 5702). In some embodiments, optimization data can be sent back to the mixed reality system, which can use the data to update the visual display of virtual objects. More details about optimizing SLAM calculations using additional observations and / or updating SLAM calculations using optimization data can be found in U.S. Provisional Patent Application Serial No. 62 / 923,317, the contents of which are incorporated herein by reference in their entirety.
[0225] Other computationally intensive tasks that may not require low-latency communications can also be offloaded to a remote server, which can transmit the results back to a separate mixed reality system. For example, a machine learning algorithm can be offloaded to a remote server (e.g., a remote operation server 5702, a remote tactical server 5704, a remote strategic server 5706, and / or a data lake 5708). In some embodiments, a machine learning algorithm can identify an object as a target based on data collected from a mixed reality system. For example, one or more cameras on a mixed reality system can capture a video approximating a soldier's field of view. The mixed reality system can be connected to a weapon, which can have one or more sensors that can determine the direction in which the weapon is aimed. For example, a laser sight from a weapon can be detected by one or more cameras of a mixed reality system to determine the direction in which the weapon is aimed. In some embodiments, sensors such as accelerometers on a weapon can be used to determine the direction in which the weapon is aimed. In some embodiments, data about the soldier's field of view and data about the direction in which the weapon is aimed can be fed into a machine learning algorithm (e.g., an image with a weapon aiming area can be fed through a deep convolutional neural network) to determine what may be a target.
[0226] In some embodiments, machine learning calculations (e.g., convolution, forward propagation, back propagation, etc.) can be performed at a remote server to train a neural network. In some embodiments, a mixed reality system can utilize machine learning to visually identify (e.g., via colored outlines) objects that may be targets. The mixed reality system can feed images to a remote server, which can run the images through a trained neural network to determine whether an area is likely to be a target. In some embodiments, a trained neural network (e.g., a neural network that has been trained at a remote server) can be stored locally on the mixed reality system for low-latency identification of possible targets. In some embodiments, a locally stored trained neural network can be periodically updated by a remote server, which can continuously train the neural network with additional training data.
[0227] Similar methods of remote computing and / or local storage of completed computations can be used for other machine learning / computationally complex applications, including automatic speech recognition. For example, a mixed reality system can record audio of a user speaking and transmit the audio to a remote server for speech processing (e.g., speech recognition and / or natural language processing). In some embodiments, as a result of processing the audio of the user speaking, the mixed reality system can receive instructions from a remote server to perform one or more functions. In some embodiments, it may be desirable to perform speech processing locally on the mixed reality system. For example, a soldier may wish to execute a voice command to the mixed reality system (e.g., turn off night vision) and expect the command to be executed as quickly as possible. In some embodiments, a speech recognition system can be trained at a remote server and stored locally on the mixed reality system.
[0228] In some embodiments, some mixed reality functions can be performed locally on the mixed reality system. For example, the mixed reality system can use eye tracking technology (e.g., via one or more cameras pointing to the user's eyes) to identify what direction the user is looking at. In some embodiments, eye tracking calculations can be performed locally on the mixed reality system. It may be desirable to calculate eye tracking in a low-latency manner, because visual effects may be displayed to the user as a result of eye tracking, and high latency (e.g., about 50ms or longer) may cause too much delay to the user. In some embodiments, spatialized audio rendering can be completed locally at the mixed reality device. For example, virtual sounds that should be presented as originating from behind the user can be rendered and / or spatialized locally on the mixed reality device. It may be desirable to render and / or spatialize virtual audio in a low-latency manner so that the virtual audio appears to be "synchronized" with real or virtual visual events (e.g., presented approximately at the same time as real or virtual visual events). In some embodiments, graphics rendering can be performed locally on the mixed reality system. It may be desirable to quickly present and display virtual objects to the user, and unloading calculations (e.g., to a remote server) may produce unacceptable delays.
[0229] The mixed reality edge computing architecture may also rely on relative physical location for efficient computing. For example, communication between nearby mixed reality systems may have lower latency than communication between a mixed reality system and a remote server. In some embodiments, computations that can be offloaded to a remote server can be distributed across connected mixed reality systems (e.g., if the remote server is unreachable). For example, mixed reality systems may transmit SLAM data to each other, which can increase the available observation pool, which can lead to more accurate SLAM calculations. In some embodiments, SLAM optimization can be performed at a single mixed reality system, which can include a more powerful processor than other standard mixed reality systems. In some embodiments, SLAM optimization can be distributed across connected mixed reality systems, and computing power can be pooled together.
[0230] In some embodiments, a remote server (e.g., teleoperation server 5702) can be physically located near a connected mixed reality system, which can reduce communication latency. For example, a mixed reality system deployed with a soldier can communicate with a remote server located at an operations base, which may be physically close to the mixed reality system (e.g., less than 50 miles away). In some embodiments, communicating with a remote server located near the mixed reality system can result in lower latency than communicating with a remote server located farther away (e.g., 2,000 miles away and / or on a different continent).
[0231] In some embodiments, mixed reality systems deployed in large-scale mixed reality applications may have personalized permissions / network connections. For example, although each soldier's mixed reality system can transmit data (e.g., SLAM, machine learning, and / or vital signs data) to the remote operation server 5702, only the squad leader's mixed reality system can have read rights to the vital signs data of each squad member. The restricted access structure can be implemented in any suitable manner. For example, each mixed reality system can have a unique identifier, and the remote server can only transmit sensitive data to the appropriate mixed reality system. In some embodiments, the squad leader's mixed reality system can store encryption / decryption keys that can be used to encrypt / decrypt communications between the mixed reality system and the remote server.
[0232] In some embodiments, a group of 1,000 soldiers may be assigned to a battalion, and each soldier may be equipped with a separate mixed reality system. In some embodiments, data collected from the entire battalion may be aggregated to a remote server (e.g., remote tactical server 5704, which may be the same as remote operation server 5702 or a separate server). In some embodiments, data aggregated about the battalion (e.g., squad status, overall troop movement, casualties, etc.) may be accessed by a limited subset of deployed mixed reality systems (e.g., mixed reality systems used by the commander). In some embodiments, data aggregated about the battalion may be accessed by a general purpose computing system (e.g., by a laptop used by the commander).
[0233] In some embodiments, data aggregated by one or more remote servers (e.g., remote operation server 5702, remote tactical server 5704, and / or remote strategic server 5706) may be further aggregated into a separate remote server (e.g., data lake 5708). In some embodiments, in addition to data collected from the deployed mixed reality system, data lake 5708 may further access more external resources (e.g., the Internet) and internal resources (e.g., military databases). In some embodiments, data lake 5708 may be physically located in an area that is considered to be away from enemy combatants (e.g., offshore) and / or may be dispersed (e.g., data lake 5708 may include one or more server farms located in different locations). It may be beneficial to utilize data lake 5708 to perform computationally complex tasks that may not require low-latency communications (e.g., in a similar manner that computation may be divided between the mixed reality system and remote operation server 5702). For example, teleoperation server 5702 may run relatively simple machine learning algorithms (e.g., whether a visual object is likely to be a target) and complex machine learning algorithms (e.g., what types of troop movements are more likely to win a battle) and / or neural network training may be performed using data aggregated at data lake 5708.
[0234] Although some embodiments of the disclosure are described with respect to military applications, skilled artisans will appreciate that other applications of the disclosed embodiments, including applications not directly related to military technology or emergency services, are suitable and within the scope of the disclosure.
[0235] Fig.84A and 84B An exemplary mixed reality computing architecture according to some embodiments is shown. In some embodiments, the mixed reality computing architecture 8400 may include one or more modules, which may be components of the mixed reality computing architecture 8400. In some embodiments, the module may include one or more computer systems (e.g., MR systems 112, 200, or remote servers) configured to execute instructions and / or store one or more data structures. For example, the instructions executed by the module may be processes and / or subprocesses running within the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the module may be threads running within the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the module may be run in the same process address space and / or memory space as other components of the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the module may be run in a process address space and / or memory space different from other components of the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the module may be run on hardware different from other components of the mixed reality computing architecture 8400. For example, instructions executed by one or more modules of the mixed reality computing architecture 8400 may be run on the mixed reality system 112 and / or 200, while other components of the mixed reality computing architecture 8400 may be run on a remote server. In some embodiments, instructions executed by one or more modules of the mixed reality computing architecture 8400 may be instantiated within the mixed reality computing architecture 8400. In some embodiments, instructions executed by modules within the mixed reality computing architecture 8400 and / or data structures stored in these modules may communicate with other components of the mixed reality computing architecture 8400 (e.g., communicate with instructions executed by other modules and / or data structures stored in other modules).
[0236] In some embodiments, the mixed reality computing architecture 8400 may include one or more modules and the one or more modules may include one or more sub-modules. In some embodiments, the sub-module may include one or more computer systems configured to execute instructions and / or store one or more data structures. For example, the instructions executed by the sub-module may be processes and / or sub-processes running within the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the sub-module may be threads running within the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the sub-module may be run in the same process address space and / or memory space as other components of the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the sub-module may be run in a process address space and / or memory space different from other components of the mixed reality computing architecture 8400. In some embodiments, the instructions executed by the sub-module may be run on hardware different from other components of the mixed reality computing architecture 8400. For example, the instructions executed by one or more sub-modules of the mixed reality computing architecture 8400 may be run on a dedicated audio processor (e.g., DSP), while other components of the mixed reality computing architecture 8400 may be run on a general-purpose processor. In some embodiments, instructions executed by one or more sub-modules of the mixed reality computing architecture 8400 may be instantiated within the mixed reality computing architecture 8400. In some embodiments, instructions executed by sub-modules within the mixed reality computing architecture 8400 and / or data structures stored in these sub-modules may communicate with other components of the mixed reality computing architecture 8400 (e.g., communicate with instructions executed by other modules and / or data structures stored in other modules).
[0237] In some embodiments, the instructions executed by the wearable software module 8402 may be run on a mixed reality system (e.g., MR system 112, 200). In some embodiments, the instructions executed by the wearable software module 8402 may be run on a component of the mixed reality system (e.g., a head mounted device of the mixed reality system). The instructions executed by the wearable software module 8402 may include low latency beneficial functions. For example, the instructions executed by the wearable software module 8402 may include a tracking submodule 8406. In some embodiments, the tracking submodule 8406 may track real and / or virtual objects used to generate a mixed reality experience. For example, the tracking submodule 8406 may track head postures, which may include the position and / or orientation of the user's head. The head posture may be used to determine what virtual objects should be presented to the user (e.g., because only virtual objects located in real positions within the user's field of view should be presented). In some embodiments, only estimating the head posture completely remotely (e.g., at a remote server) may cause a delay between when the user turns around and when the mixed reality system has determined that the field of view has moved. It may be disconcerting for a user to turn around and see a new virtual object appear 100ms after they should have seen it. In some embodiments, the head pose may be initially determined locally in the mixed reality headset and optimized using remote computing. For example, head pose data (e.g., images captured by a camera mounted on the head mounted device and / or inertial measurement unit data) may be sent to the cloud module 8410. In some embodiments, the instructions executed by the cloud module 8410 may be run on a remote server (e.g., the remote operation server 5702). For example, the cloud module 8410 may include a data management module 8410. In some embodiments, the data management module 8410 may manage complex calculations (e.g., training of neural networks and / or SLAM optimization). In some embodiments, the cloud module 8410 may include an integration submodule 8414. The integration submodule 8414 may manage connections between servers and / or databases (e.g., other computing systems within an intranet and / or within the Internet).
[0238] Other functions, such as those performed by the fast target acquisition submodule 8408, may also be run on the mixed reality system. In some embodiments, it may be beneficial for the mixed reality system to quickly display target acquisition visuals (e.g., outlining the target in red) with low latency, particularly in situations where the user changes field of view rapidly. A rapidly changing field of view may cause targets to move in and out of sight quickly, and it may be highly desirable for the mixed reality system to recognize and / or visually display target indications to the user. In some embodiments, the instructions executed by the fast target acquisition submodule 8408 may be optimized by computations performed remotely. For example, a deep convolutional neural network may be trained at a remote server, and the trained model may be deployed to and run on the fast target acquisition submodule 8408.
[0239] In some embodiments, it may be beneficial to run instructions executed by the OS submodule 8404 on the mixed reality system. In some embodiments, the OS submodule 8404 may execute instructions related to basic operating system functions (e.g., drivers, services, etc.). In some embodiments, the OS submodule 8404 may allow the mixed reality system 112, 200 to function as a computing system and may allow the mixed reality system to run applications. In some embodiments,
[0240] Fig.94 An exemplary network and / or computing architecture according to some embodiments is shown. In some embodiments, data may be transmitted by one or more mixed reality systems 9404 worn by a soldier 9402. In some embodiments, it may not be feasible for the mixed reality system 9404 to communicate with a remote server (e.g., because no cellular network connection is available). In some embodiments, large-scale mixed reality applications may utilize decentralized networks (e.g., self-organizing and / or peer-to-peer networks) to facilitate communication between components of large-scale systems. For example, a separate mixed reality system may transmit data to a nearby vehicle 9406 (e.g., wirelessly using Bluetooth, WiFi, 5G, and / or other radio frequency communications). In some embodiments, the vehicle 9406 may include a communication array that may be more powerful than the communication array in the mixed reality system 9404. In some embodiments, the vehicle 9406 may relay and / or transmit data to an aircraft 9408 and / or an aircraft 9414 that may be passing within the communication range. In some embodiments, the aircraft 9408 and / or the aircraft 9414 may relay and / or transmit data to a satellite 9410. In some embodiments, satellite 9410 may relay and / or transmit data to a remote server and / or spacecraft 9412. In some embodiments, data may be transmitted and / or relayed to a communication array 9416, which may be portable enough to be mounted on a soldier (e.g., a communications officer). It is contemplated that any combination of components may communicate bidirectionally (e.g., 2-way communication) with other components.
[0241] In some embodiments, self-organizing and / or mesh networks can effectively transmit data from mixed reality systems, where the network infrastructure can fully meet the needs of large-scale mixed reality applications. In some embodiments, self-organizing and / or mesh communications can be suitable for delay-agnostic transmissions, such as SLAM optimization, neural network training, mapping / navigation, non-real-time communications (e.g., messages), etc. In some embodiments, self-organizing and / or mesh communications can generate delays that hinder delay-sensitive transmissions, such as graphics and / or audio rendering, preliminary SLAM calculations, voice command processing, eye tracking, etc.
[0242] Fig.95 An exemplary network and / or computing architecture according to some embodiments is shown. In some embodiments, a communication array 9502 (which may be coupled to one or more mixed reality systems) may communicate with one or more drones 9504. In some embodiments, one or more drones 9504 may form a network (e.g., a mesh network and / or an ad hoc network), which may facilitate communication between one or more mixed reality systems and / or one or more remote computing systems (e.g., remote servers). For example, a communication array 9502 may not be able to communicate directly with a communication array 9506, but may be able to communicate with the communication array 9506 via one or more drones 9504 (e.g., via relaying data across multiple drones and / or via a mesh network).
[0243] Fig.96 An exemplary network and / or computing architecture according to some embodiments is shown. In some embodiments, the communication array 9602 can be mounted on a soldier who may be moving. One or more drones 9604 can form a network (e.g., a mesh network and / or a self-organizing network), which can facilitate communication between one or more mixed reality systems and / or one or more remote computing systems (e.g., remote servers). In some embodiments, the network formed by one or more drones 9604 can move with the aid of one or more drones 9604 that move with the communication array 9602. In some embodiments, one or more drones 9604 can fly in a predetermined pattern according to the expected movement of troops. In some embodiments, one or more drones 9604 can be configured to follow a soldier with an installed communication array 9602. In some embodiments, one or more drones 9604 can be configured to maintain static and / or dynamic spacing between other drones (e.g., spacing can vary according to troop density, combat danger, network coverage, etc.).
[0244] Figures 97-102 An exemplary network and / or computing architecture according to some embodiments is shown. In some embodiments, the wearable component 9702 may correspond to Figure 6 In some embodiments, the waist bag 9704 may correspond to the wearable component 2 depicted in FIG. Figure 7 6 depicted in FIG. In some embodiments, the wearable component 9702 and / or the waist bag 9704 may correspond to the MR system 112, 200. In some embodiments, the wearable component 9702 and / or the waist bag 9704 may be connected (e.g., by a wired and / or wireless connection) to the communication unit 9706. In some embodiments, the communication unit 9706 may include a mobile device (e.g., a smart phone). In some embodiments, the mobile device may be configured to provide additional processing and / or communication capabilities. For example, the mobile device may include its own processor, which may be configured to perform calculations. In some embodiments, the mobile device may be configured to act as an interface between a mixed reality system and other computing systems (e.g., accessories, sensors, other mixed reality systems, and / or remote servers). In some embodiments, the mobile device may include a radio frequency antenna and / or a network modem configured to communicate with one or more cellular towers. The mobile device may act as a repeater to transmit information from the mixed reality system to other computing devices (e.g., remote servers).
[0245] In some embodiments, the communication unit 9706 may include an interconnection package. In some embodiments, the interconnection package may include a high-power antenna (e.g., a radio frequency antenna) and / or a transceiver. In some embodiments, the interconnection package is capable of high-bandwidth transmission. For example, the interconnection package may be configured to communicate with the wearable component 9702 and / or the waist bag 9704. In some embodiments, the interconnection package may be configured to receive all or part of all data captured by the wearable component 9702 and / or the waist bag 9704 (e.g., video feed, depth information, SLAM data, audio feed, vital signs information, etc.). In some embodiments, the interconnection package may be less constrained than other electronic devices. For example, a smart phone may be subject to power and / or transmission limitations because the smart phone may be configured to be used near the user's head. In some embodiments, the interconnection package may be configured to be used away from the user and / or the sensitive parts of the user. In some embodiments, the interconnection package may be coupled (e.g., by wired and / or wireless means) to a mobile device. In some embodiments, one or more mixed reality systems may be coupled to the interconnection package, and the interconnection package may be configured to communicate with other computing systems.
[0246] In some embodiments, the communication unit 9706 can communicate with other computing systems. For example, the communication unit 9706 can communicate with the intermediate transceiver 9708. In some embodiments, the intermediate transceiver 9708 can be a cellular tower. In some embodiments, the intermediate transceiver 9708 can be a communication array mounted on a soldier. In some embodiments, the intermediate transceiver 9708 can transmit information to one or more cloud servers 9712. In some embodiments, the intermediate transceiver 9708 can transmit information directly to one or more cloud servers. In some embodiments, the intermediate transceiver 9708 can transmit information via one or more edge nodes 9710. The edge node 9710 can be a network device that is dispersed and / or physically located near the intermediate transceiver 9708. For example, the mixed reality system can be an edge node, the mobile device can be an edge node, the wireless access point can be an edge node, the communication array mounted on the soldier can be an edge node, etc. Physical proximity can reduce communication latency, which is important for various mixed reality functions (including SLAM calculations, object recognition, voice recognition, etc.).
[0247] Fig.98 An exemplary embodiment of mixed reality network communications is depicted. In some embodiments, the communication unit 9706 can communicate directly with one or more edge nodes 9710. Fig.99 An exemplary embodiment of mixed reality network communications is depicted. In some embodiments, communications 9706 can communicate directly with one or more cloud servers 9712. Fig.100 An exemplary embodiment of mixed reality network communications is depicted. In some embodiments, wearable component 9702 and / or waist pack 9704 can communicate with communication unit 9706 via intermediate transceiver 9708. Fig.101 An exemplary embodiment of mixed reality network communications is depicted. In some embodiments, wearable component 9702 and / or waist pack 9704 can communicate wirelessly with communication unit 9706 (e.g., via Bluetooth, WiFi, 5G, and / or other radio frequency communications). Fig.102 An exemplary embodiment of mixed reality network communications is depicted. In some embodiments, the communication unit 9706 can communicate with one or more connection devices 9714. In some embodiments, the connection device 9714 can be a connection resource with very good local connectivity. For example, an enterprise may have a connection device 9714 in each room. In some embodiments, the connection device 9714 may have storage capabilities, be managed by local IT, etc. For example, in one embodiment, each connection device may not only have very good connectivity to be piped out of the building, but also store one or more high-resolution meshes of a particular room (e.g., one or more meshes captured / refreshed by the mixed reality system in that area during the previous day).
[0248] Fig.103 An exemplary wearable component is depicted that may correspond to Figure 6 In some embodiments, the wearable components may include an electromagnetic sensor 1, a light field camera 2, a Bluetooth antenna 3, a WiMax antenna 4, a WiFi antenna 5, a 4G / LTE antenna 6, an active cooling fan 7, a variable focus element 8, a thermocouple 9, an inertial measurement unit 10, a visible light RGB / monochrome camera 11, a near infrared camera 12, a long wave infrared camera 13, a short wave infrared camera 14, an infrared eye tracking camera 15, a neuromorphic differential sensor 16, a thermistor 17, a 5G antenna 18, a deflection sensor 19, a hemispherical camera 20, a time-of-flight depth sensor device 21, LIDAR sensor 22, structured light depth sensor 23, global dynamic dimmer 24, local dynamic dimmer 25, refractive free form optics 26, diffractive lens optics 27, MEMS scanner single light source 28, MEMS scanner dual light source 29, digital holographic display 30, photodiode 31, CCD sensor 32, CMOS sensor 33, electron bombardment active pixel sensor 34, hybrid dynamic vision visible light sensor 35, internal interferometer 36, active noise cancellation device 37 and / or microphone array 38.
[0249] Fig.104 An exemplary waist pack assembly is depicted, which may correspond to Figure 7 In some embodiments, the waist-pack assembly may include a near-field fingerprint reader 1, a multi-purpose external connector 2, an HDMI port 3, a display port 4, a short-wave infrared camera 5, a 5G antenna 6, a WiFi antenna 7, a built-in Qi charger 8, a USB-C connector 9, a USB-A (version 2 and / or 3) connector 10, a light-field camera 11, a 4G antenna 12, an ultra-wideband antenna 13, a visible RGB and / or monochrome camera 14, a Bluetooth camera 15, a WiMax antenna 16, an internal battery 17, an internal memory 18, an internal compute 19, a long-wave infrared camera 20, an LCD touch screen 21, an internal thermocouple 22, an internal GPU 23, an internal electromagnetic sensor 24, and / or a near-infrared camera 25.
[0250] Various exemplary embodiments of the present invention are described herein. Reference is made to these examples in a non-limiting manner. They are provided to illustrate aspects of the present invention that are more widely applicable. Various changes may be made to the described invention and equivalents may be substituted without departing from the true spirit and scope of the present invention. For example, although some embodiments are described with respect to military or emergency service applications, those skilled in the art will appreciate that other suitable applications are within the scope of the present disclosure. In addition, many modifications may be made to adapt specific circumstances, materials, compositions of matter, processes, process behaviors or steps to the purpose, spirit or scope of the present invention. In addition, those skilled in the art will appreciate that each of the various variants described and illustrated herein has discrete components and features that can be easily separated or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present invention. All such modifications are intended to fall within the scope of the claims associated with the present disclosure.
[0251] The present invention includes methods that can be performed using the present device. The method may include an action of providing such suitable equipment. Such providing may be performed by an end user. In other words, the "providing" action only requires the end user to obtain, access, approach, locate, set up, activate, power on or otherwise act to provide the necessary equipment in the present method. The methods described herein may be performed in any order of the listed events that is logically possible and in the listed order of events.
[0252] Exemplary aspects of the invention and details on material selection and manufacturing have been described above. As for other details of the invention, these can be understood in conjunction with the patents and publications cited above and what is known or appreciated by those skilled in the art. With respect to method-based aspects of the invention, the same applies in terms of additional actions that are typically or logically employed.
[0253] In addition, although the present invention has been described with reference to several examples optionally incorporating various features, the present invention is not limited to the contents described or indicated with respect to each variant of the present invention as contemplated. Various changes may be made to the described invention and equivalents (whether referenced herein or not included for the sake of brevity) may be substituted without departing from the true spirit and scope of the present invention. In addition, where a range of values is provided, it is understood that each intermediate value between the upper and lower limits of the range and any other specified or intermediate value within the specified range is included in the present invention.
[0254] In addition, it is contemplated that any optional features of the described variants of the invention may be independently proposed and claimed, or combined with any one or more features described herein. References to singular items include the possibility of having multiple identical items. More specifically, as used herein and in the claims associated therewith, the singular forms "a", "an", "said" and "the" include plural referents unless otherwise specifically stated. In other words, the use of articles allows "at least one" of the present items in the above description and claims associated with the present disclosure. It should be further noted that such claims may be drafted to exclude any optional elements. Therefore, this statement is intended to serve as a prior basis for the use of proprietary terms such as "only", "only" and the like associated with the narration of the claim elements or the use of "negative" limitations.
[0255] Without using such specific terms, the term "comprising" in the claims associated with the present disclosure shall allow for the inclusion of any additional elements - regardless of whether a given number of elements are recited in such claims or the addition of features may be considered to transform the nature of the elements recited in such claims. Unless expressly defined herein, all technical and scientific terms used herein shall be given as broad a meaning as commonly understood while maintaining claim validity.
[0256] The scope of the present invention is not limited to the examples provided and / or this description, but rather is limited by the scope of the claim language associated with this disclosure.
Claims
1. A method for a mixed reality system, comprising: capturing data about a user's field of view via a camera of the mixed reality system; receiving, from a handheld device communicatively coupled to the mixed reality system, data regarding an orientation of the handheld device as detected by a sensor of the handheld device; Sending the data about the user's field of view and the data about the orientation to a remote server so that the remote server trains a neural network based on the data about the user's field of view and the data about the orientation; as well as The trained neural network is received at the mixed reality system from the remote server.
2. The method according to claim 1, wherein: Sending the data about the user field of view and the data about the orientation to a remote server includes sending the data about the user field of view and the data about the orientation to an intermediate relay.
3. The method according to claim 2, wherein: The intermediate relay includes a drone network.
4. The method according to claim 2, wherein: The intermediate relay comprises a communication array, wherein the communication array is mountable on a person.
5. The method according to claim 2, wherein: The mixed reality system includes a first head mounted device, and wherein the intermediate relay includes a second head mounted device.
6. The method according to claim 2, wherein: The intermediate relays include cellular towers.
7. A mixed reality system, comprising: a camera configured to capture data regarding a user's field of view; as well as One or more processors configured to perform a method comprising: capturing data regarding a user's field of view via the camera; receiving, from a handheld device communicatively coupled to the mixed reality system, data regarding an orientation of the handheld device as detected by a sensor of the handheld device; sending data about the user field of view and data about the orientation to a remote server so that the remote server trains a neural network based on the data about the user field of view and the data about the orientation; and receiving the trained neural network from the remote server at the mixed reality system.
8. The mixed reality system according to claim 7, wherein: Sending the data about the user field of view and the data about the orientation to a remote server includes sending the data about the user field of view and the data about the orientation to an intermediate relay.
9. The mixed reality system according to claim 8, wherein: The intermediate relay includes a drone network.
10. The mixed reality system according to claim 8, wherein: The intermediate relay comprises a communication array, wherein the communication array is mountable on a person.
11. The mixed reality system according to claim 8, wherein: The mixed reality system includes a first head mounted device, and wherein the intermediate relay includes a second head mounted device.
12. The mixed reality system according to claim 8, wherein: The intermediate relays include cellular towers.
13. A non-transitory computer readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a method comprising: capturing data about the user's field of view via a camera of the mixed reality system; receiving, from a handheld device communicatively coupled to the mixed reality system, data regarding an orientation of the handheld device as detected by a sensor on the handheld device; Sending the data about the user's field of view and the data about the orientation to a remote server so that the remote server trains a neural network based on the data about the user's field of view and the data about the orientation; and receiving, at the mixed reality system, the trained neural network from the remote server.
14. The non-transitory computer readable medium of claim 13, wherein: Sending the data about the user field of view and the data about the orientation to a remote server includes sending the data about the user field of view and the data about the orientation to an intermediate relay.
15. The non-transitory computer readable medium of claim 14, wherein: The intermediate relay includes a drone network.
16. The non-transitory computer readable medium of claim 14, wherein: The intermediate relay comprises a communication array, wherein the communication array is mountable on a person.
17. The non-transitory computer readable medium of claim 14, wherein: The mixed reality system includes a first head mounted device, and wherein the intermediate relay includes a second head mounted device.
18. The non-transitory computer readable medium of claim 14, wherein: The intermediate relays include cellular towers.
