System and method for tracking user movements

Abstract

According to aspects of the present invention, a method is provided for providing tracking data to a user of a platform in an environment, wherein the platform is associated with one or more input and output devices configured to track one or more variables of data associated with a user, the platform, or the environment, generate tracking data, and provide outputs representing the tracking data, the method comprising: processing data received from one or more input devices to determine tracking data relating to the user's relative position to at least one of the platform and the environment; identifying one or more input devices of received data, each associated with its respective priority value; determining a target tracking score level based on the required level of security; determining a first tracking score for a first input device of the one or more input devices having the highest priority; determining a second tracking score for a second input device of the one or more input devices having the second highest priority; combining the first tracking score with the second tracking score to generate an integrated tracking score; comparing the target tracking score with the integrated tracking score; creating integrated tracking data from the first input device and the second output device in response to determining that the integrated tracking score is greater than or equal to the target tracking score; and providing the integrated tracking data to the user.

Description

Technical Field

[0001] The present invention relates to a system and method for tracking a user's movement, and more particularly, but not limited thereto, to tracking the movement of a user located in a vehicle.

Background Art

[0002] The tracking of objects, as well as the determination and tracking of the orientation of objects, are known in many scenarios. Head or helmet-mounted display systems incorporating tracking systems for providing orientation data of a head or helmet for use in positioning display artifacts in a display system where the positioning of symbols or other artifacts in a display can be utilized. In some examples, these systems can be used by a user moving on or within a movable platform such as an aircraft or other vehicle.

[0003] In some situations, such systems can be subject to buffeting or similar rapidly induced orientation changes when moving on or within a movable platform such as an aircraft or other vehicle. This can make tracking more difficult.

[0004] Safety solutions in vehicles are often required to comply with civil hazard classification schemes and functional safety guidance. These schemes have different levels of classification, where lower levels are less reliable than higher levels. In some environments, low or medium levels may be appropriate, while in other environments, higher levels are essential.

[0005] In an aircraft environment, medium-level integrity / level solutions are generally easy to implement using currently known tracking systems and devices. Examples of medium-level solutions are those that can be used in critical hazard classification risk scenarios.

[0006] The use of head-mounted devices (HMDs) as a primary criterion typically inherits hazardous and / or catastrophic classifications, for example, due to the provision of hazardously misleading information (HMI) for critical flight data. This imposes a no-single-point failure requirement on HMD architectures. The loss of critical symbols on a single display typically remains a minor risk. The loss of the same information on all cockpit displays, including a head-down display (HDD) and any Get You Home instruments, is typically ranked as catastrophic, although the risk is shared across different systems. Thus, for any single display surface, the significant challenge tends to exist compared to the case of HMIs where the HMD tracker contributes to all isometrically displayed information.

[0007] Traditional practice involves using multiple similar helmet-based tracking solutions, such as dual optical trackers, to provide additional completeness, but this can add weight and lead to other drawbacks.

[0008] There is a need for tracking systems and related methods that provide high-fidelity tracking under any risk scenario, unhindered by problems exhibited in known systems. [Overview of the Initiative]

[0009] According to aspects of the present invention, a method is provided for providing tracking data to a user of a platform in an environment, wherein the platform is associated with one or more input and output devices configured to track one or more variables of data associated with a user, the platform, or the environment, generate tracking data, and provide outputs representing the tracking data, the method comprising: processing data received from one or more input devices to determine tracking data relating to the user's relative position to at least one of the platform and the environment; identifying one or more input devices of received data, each associated with its respective priority value; determining a target tracking score level based on the required level of security; determining a first tracking score for a first input device of the one or more input devices having the highest priority; determining a second tracking score for a second input device of the one or more input devices having the second highest priority; combining the first tracking score with the second tracking score to generate an integrated tracking score; comparing the target tracking score with the integrated tracking score; creating integrated tracking data from the first input device and the second output device in response to determining that the integrated tracking score is greater than or equal to the target tracking score; and providing the integrated tracking data to the user.

[0010] In one embodiment, in response to determining that the integrated tracking score is less than the target tracking score, further tracking scores are determined for additional input devices among one or more input devices having a lower priority than the first priority of the second priority, the further tracking scores are combined with the integrated tracking score to generate a further integrated tracking score, the target tracking score is compared with the further integrated tracking score, and in response to determining that the further integrated tracking score is equal to or greater than the target tracking score, further integrated tracking data is created from the first input device, the second output device, and the further input devices, and the further integrated tracking data is provided to the user.

[0011] In one embodiment, in response to determining that the integrated tracking score is below a target tracking score, an alarm is generated and any of one or more input devices identified as operating below a predetermined level are reconfigured.

[0012] In one embodiment, providing tracking data to the user includes displaying the tracking data via a display device.

[0013] In this configuration, data is captured from one or more input devices.

[0014] In this embodiment, data is captured that relates to at least one of the following: data imaging, illumination, eye tracking, sound, motion, gestures, velocity, acceleration, position and location, and features present in the environment.

[0015] In this embodiment, two or more input devices are selected from one or more input devices based on the priority level of the input devices.

[0016] In this embodiment, a first input device and a second input device are selected from one or more input devices based on the use case.

[0017] In some embodiments, data from different input devices is augmented to provide an integrated tracking score.

[0018] In one embodiment, to verify the integrated tracking score, the integrated tracking score is continuously monitored and compared to the target tracking score.

[0019] A further aspect of the present invention provides a system for providing tracking data to a user of a platform in an environment, the system comprising a processor, the processor configured to process data received from one or more input devices to determine tracking data relating to the user's relative position to at least one of the platform and the environment; identify one or more input devices of the received data, each associated with its respective priority value; determine a target tracking score level based on the required level of security; determine a first tracking score for a first input device of one or more input devices having the highest priority; determine a second tracking score for a second input device of one or more input devices having the second highest priority; combine the first tracking score with the second tracking score to generate an integrated tracking score; compare the target tracking score with the integrated tracking score; create integrated tracking data from the first input device and the second output device in response to determining that the integrated tracking score is greater than or equal to the target tracking score; and provide the integrated tracking data to the user.

[0020] In one embodiment, the processor is further configured to: determine a further tracking score for one or more input devices and further input devices having a priority lower than the first or second priority, in response to determining that the integrated tracking score is less than the target tracking score; combine the further tracking score with the integrated tracking score to generate a further integrated tracking score; compare the target tracking score with the further integrated tracking score; and, in response to determining that the further integrated tracking score is greater than or equal to the target tracking score, create further integrated tracking data from the first input device, the second output device, and further input devices; and provide the further integrated tracking data to the user.

[0021] In one aspect, the processor is further configured to generate an alert and reconfigure any of one or more input devices identified as operating below a predetermined level in response to determining that the integrated tracking score is less than the target tracking score.

[0022] In one aspect, the system further comprises a display that provides tracking data to a user.

[0023] In one aspect, the system further comprises a plurality of input devices.

[0024] In one aspect, the input device further comprises at least two of an optical sensor, a light sensor, an eye tracker sensor, an audio sensor, an inertial sensor, a motion sensor, a speed and acceleration sensor, and a position and location sensor.

[0025] In one aspect, the processor is further configured to select a first input device and a second input device from one or more input devices based on the priority level of the input devices.

[0026] In one aspect, the system selects a first input device and a second input device from one or more input devices based on a use case.

[0027] In one aspect, there is a reinforcement of data from different input devices to provide an integrated tracking score.

[0028] In one aspect, there is a continuous monitoring of the integrated tracking score and comparison with a target tracking score to verify the integrated tracking score.

[0029] In one aspect, the system comprises a head-mounted device.

[0030] In one aspect, the head-mounted device comprises a head-mounted display.

[0031] In one embodiment, the head-mounted device comprises a helmet on which a head-mounted display is installed.

[0032] According to a further aspect of the present invention, a head-mounted display including the system of the prior aspect is provided.

[0033] According to a further aspect of the present invention, a machine-readable medium is provided which, when executed on a computer, is configured to carry out the method of a previous aspect.

[0034] Embodiments of the present invention are described herein by reference only to the drawings. [Brief explanation of the drawing]

[0035] [Figure 1] This is an illustrative system according to an aspect of the present invention. [Figure 2] A diagram of a helmet having components suitable for implementing the functions according to an aspect of the present invention is shown. [Figure 3] This shows a schema for multi-source tracker integration according to an aspect of the present invention. [Figure 4] A flowchart of a method for integrating multi-source trackers according to an aspect of the present invention is shown. [Modes for carrying out the invention]

[0036] The present invention relates to a tracking system and method for accurately identifying the location of a user on a platform, such as an aircraft, as well as all nearby platform-related instruments and control devices, so that all necessary information can be directed to the user who can safely navigate and move the platform.

[0037] Referring here to Figure 1, a system 100 according to an aspect of the present invention is shown. The system 100 comprises a head-worn display (HWD) device 102 adapted to be supported on a helmet (not shown in Figure 1) worn by a user 104. The system also includes a camera 106 and a processor 108.

[0038] Figure 1 further illustrates surface 110, which may be referred to interchangeably with the platform and comprises any given surface on which a user 104 is positioned. Surface 110 may include, but is not limited to, a vehicle or flight simulation device. The user 104 positioned on surface 110 wears the HWD 102. An imaging device or camera 106 may be integrated into the body of the HWD 102 or mounted at any suitable location on the HWD 102 so as to be able to observe a number of views and instruments, as will be described in more detail below. The camera may collect images of markers 112a, 112b, 112c, and 112d positioned at predetermined locations relative to surface 110. In Figure 1, markers 112a, 112b, 112c, and 112d are illustrated as being positioned on the upper part 114 of the surface; however, it should be recognized that markers 112a, 112b, 112c, and 112d can be positioned at any suitable location on the surface 110.

[0039] Camera 106 may be integrated with HWD 102. Camera 106 may include multiple cameras. The cameras may include, but are not limited to, stereo cameras or depth cameras. In Figure 1, the camera is shown facing upward, but it is recognized that camera 106 can be positioned or positioned at any suitable location on HWD 102 so as to be configured to capture images of many aspects of the environment, such as images of trackers or devices, including optical inertial trackers, HMD camera trackers, head motion trackers, eyeball trackers, hand gesture trackers, marker trackers, platform and attitude trackers, and any other devices that may be required depending on the system's usage requirements.

[0040] A tracker may include, or may be, a sensor device that detects and collects variables in data, including imaging, illumination, eye tracking, sound, motion, gravity, heat, velocity, acceleration, position and location, or any other attributes or features present in the target environment. Sensors include optical sensors, light sensors, eye tracker sensors, audio sensors, inertial sensors, motion sensors, heat sensors, gravity sensors, velocity and acceleration sensors, position and location sensors, etc. In this example, the environment includes the aircraft cockpit and the environment outside the cockpit. It will be recognized that at least some of the sensors and associated detected data may be outside the cockpit.

[0041] Tracking data is collected from sensors and further used by the user to navigate the platform within the environment. Tracking data is used herein to describe data received from one or more input sources tracking the user and any moving elements within a fixed space, platform, or similar. As indicated elsewhere, a fixed location or platform could be the cockpit of a vehicle such as an aircraft. The tracking data is processed and configured to provide output tracking data for the user, which is either displayed to the user or otherwise supplied. In the latter case, this may include haptic feedback from a joystick or any other haptic feedback. The output tracking data further includes a real-world image of the user and a virtual image generated by the system to represent any other output data. This representation may be a symbol, a real or virtual image of a local controller which may or may not move, an external view of the outside of the platform which may be real or virtual, and any other appropriate form of output data as indicated by the use case.

[0042] Referring to Figure 2, a helmet tracker system 200 integrated with a helmet 202 is shown. The helmet tracker system comprises one or more sensor devices 204. The sensors may include one or more inertial sensor components, one or more cameras, and any other sensors suitable for capturing the environment or user characteristics that may be useful in tracking the user in any way. The inertial sensor components may include three gyro elements 204 fitted into the outer shell of the helmet 202. The optical helmet tracker system further comprises an array of light-emitting diode (LED) devices 206 integrated into or mounted on the outer shell of the helmet 202, and an array of one or more cameras 208, each mounted in fixed positions, for example in an aircraft cockpit, so as to have a line-of-sight view to at least some of the LEDs 206 in any possible orientation of the helmet 202. A system control device (controller) 210 is linked to each of the cameras 208 to receive image data resulting from the detection of light emitted by the LEDs 206 in the line of sight of the cameras. The controller 210 is also linked to the LED 206 in the helmet 202 by an appropriate link (which may be wired or wireless), and the controller 210 is configured to control the illumination of the LED 206. The controller 210 is configured to trigger, for example, periodic illumination of the LED 206 in a predetermined pattern and to receive the resulting image data from the camera 208. The controller 210 includes a digital processor, or a combination of hardware and software implementation functions, programmed to implement an appropriate optical helmet tracker algorithm, and is configured to illuminate the LED 206 and interpret the resulting image data from the camera 208 to determine the orientation of the helmet relative to a reference frame associated with the aircraft.

[0043] Controller 210 is further configured to implement the functions of the entire helmet tracker system, as will be described in more detail below. Controller 210 is linked to the helmet 202 by any suitable means (not shown, which may be wired or wireless, for example) and is configured to receive rate data outputs from one or more inertial sensor components and to transmit image data received from an image generation system (not shown) installed on the aircraft to a helmet-mounted display (not explicitly shown in Figure 2) integrated within the helmet 202 via an interface. Further sensor data may be received and analyzed within the control that generates outputs displayed to the user. The display comprises a helmet-mounted display system, which may be a visor projection display system, any type of HWD, or a waveguide-based display. The display is configured to present images to the user such that the images appear superimposed on the user's view of an external scene or environment through the visor 212.

[0044] The controller is configured to output helmet orientation measurements to an image generation system (not shown) installed on the aircraft for use in calculating the position of certain types of symbols and data intended to be displayed on the helmet-mounted display, as appropriate. The output is a representation for display to the user at any given time and changes dynamically as the vehicle, user, and environment change. The output representation is real-time or near real-time. In a typical aircraft installation of the helmet-mounted display system incorporating the hybrid helmet tracker system of the present invention, one or more further modules (not shown in Figure 2) are likely to be installed on the aircraft for the purpose of generating images for display by any display system installed on or used on the aircraft. Such display systems may include a head-up display (HUD) and / or a head-down display (HDD) in addition to the helmet-mounted display relevant to the application of the present invention. Such further modules are likely to be linked to a standard aircraft data bus and thus have access to aircraft orientation data output by the aircraft navigation system and data from other systems that need to display information to the pilot. Image data, including symbols and data for display on the helmet-mounted display, may be passed to the controller 210 as needed, along with updates to the aircraft's orientation received via the aircraft data bus by such further modules.

[0045] As mentioned above, safety is a critical aspect of vehicle control. This is especially true when piloting an aircraft, where the vehicle is moving at high speeds and is subject to numerous external and internal factors that could interfere with the user's control of the aircraft. Different scenarios require different levels of safety. In some scenarios, minor issues with sensors and image presentation can be addressed by using dual devices. However, this is not always practical, and if a certain type of tracking completely fails, it will not work even with the overlap.

[0046] This invention identifies a solution in attempting to overcome this and other problems related to safely tracking variables used when operating a vehicle. This is the use of dissimilar tracking technologies (DTT) to support high-integrity HMD host functionality. The idea goes beyond a random selection of "some other tracking technology." Instead, the invention focuses on ensuring safety in so-called hazardous and catastrophic risk scenarios. The system and method rely on augmenting data from different source devices and continuously monitoring integrity to identify problems in real time, so that action can be taken to immediately restore tracking to an effective state. Herein lies a more detailed description of how this is achieved. In addition, different combinations of tracking data or inputs may be preferred over others in different use cases. This is also considered in the generation of a high-integrity integrated tracker output.

[0047] Referring to Figure 3, a multiple source tracker integration system 300 and its scheme are shown. System 300 includes a controller 302 that generates high-level, high-completeness integrated tracker data 304a, 304b. Controller 302 communicates with multiple devices, including an optical inertial tracker 306 and an HMD camera tracker 308. The controller further communicates with several additional devices, including an eyeball tracker 310 and a hand gesture tracker 312, via an integration module 314. The controller further generates outputs 304a, 304b that are used to control the aircraft and provide data to the user for further control of the aircraft.

[0048] The optical inertial tracker 306 is multifunctional and provides data to the controller, including metrics such as validity, coasting status, tracker LOS, position, and performance index. The optical inertial tracker device 306 includes optical tracking, for example, using an IR sensor in the cockpit to track an LED on a helmet or HMD to determine the angle and position of the head. The optical inertial tracker 306 further provides inertial tracking that uses an inertial measurement unit (IMU) in the helmet or HMD to determine the angle and rate of the head in free space and resolves the local angle using the aircraft's angle and coordinates. In addition, the optical inertial tracker 306 includes a tracker prediction unit that calculates the predicted tracker angle using a performance index to indicate accuracy and reliability.

[0049] The HMD camera tracker device 308 is also multifunctional and provides the controller with data including metrics such as validity, coasting status, tracker LOS, and position. The HMD camera tracker 308 uses reference points and / or markers in the cockpit to calculate the angle and position of the HMD. Daytime or nighttime cameras are used depending on the lighting conditions. The HMD camera tracker 308 also uses the IMU(s) in the HMD (ideally in the camera assembly) to determine the angle and rate of the head.

[0050] The eye tracker device 310 communicates with the controller via the integrated module 314 to provide left and right eye line-of-sight (LOS) to the HMD. The eye tracker 310 uses reference markers in the cockpit (and on the HMD display) for periodic cross-checking, as described above.

[0051] The hand gesture tracker device 312 communicates with the controller via the integrated module 314 to provide the position of the hand and / or fingers relative to a fixed cockpit reference, as well as data from the HMD optical sensors. Pressing a button triggers a cross-check of the finger position relative to a physical button, touchscreen, or any other input device operated by the hand or fingers.

[0052] The integration module 314 communicates with a controller that provides and receives data. Integrated hand position and eye tracker gaze relative to head angle is provided to the controller to validate the accuracy of the eye and hand trackers and assist the controller in integrating multiple source tracker data for the aircraft and pilot to generate highly complete integrated tracker data 304b. This is described in more detail below.

[0053] The controller 302 receives data from various tracker devices and calculates and outputs the tracker LOS and data indicating that the position error check is within an acceptable range (validity and tolerance thresholds are met). The controller provides a cross-check to validate and / or compare the tracker angles from each source device. If the data matches the accuracy for the current application, a valid integrated set of tracker data is output, a confidence number is calculated, which can be used to allow the tracker data to be used for purposes of some lower integrity, even when 100% compliance is not achieved.

[0054] If the tracker data from the optical inertial tracker 306 and the HMD camera tracker 308 do not match, the eye tracker and hand tracker may be used to identify which source is most accurate to continue with lower completeness. When the optical inertial tracker 306 and the HMD camera tracker 308 have insufficient completeness, the integration module 314 seeks additional tracking data from other sources to extend and add to the high-completeness integrated tracker data 304a. For example, the integration module 314 then integrates the hand position relative to the head angle and the gaze of the eye tracker to validate the accuracy of the eye tracker and hand tracker against the high-completeness integrated tracker data 304b. The controller 302 may further include data 316 indicating the required accuracy for the system, which can be linked to other metrics, such as HMI level or any other considerations.

[0055] The integrated multi-source tracker data 304b is output to target systems 318, including but not limited to aircraft systems, displays, tactical, navigation, communications, mission systems, pilot health systems, and similar systems. Note that when the high-completeness integrated tracker data 304a is sufficiently accurate, the high-completeness integrated tracker data 304b is the same; and when the high-completeness integrated tracker data 304a is insufficiently accurate, the high-completeness integrated tracker data 304b is then extended with additional tracker data from other sources, including but not limited to eye and hand trackers.

[0056] The present invention relates to a concept using heterogeneous tracking technology (DTT) and is used in a tracker integration algorithm to provide a high-completeness tracker LOS, hand position, and eye tracker gaze.

[0057] This invention provides a "no single-point failure" requirement in the case of at least some misleading information (HMI), and to achieve this, provides a combination of heterogeneous and independent sensor solutions and a high-integrity tracker integrated algorithm. It should be noted that failures can be complete or partial. In the latter case, the degree of failure may be sufficient to constitute a failure if the device performance falls below a certain level. Falling below a performance threshold level can be just as dangerous as a complete failure if the device is sufficiently damaged.

[0058] The use of DTT includes, for example, the use of a combination of optical inertial tracking, HMD camera tracking, eye tracking, and hand gesture tracking. In this invention, at least two of these sources are used to provide a single high-integrity tracker function, which operates either as "continuously provided" or "on-demand," depending on the requirements of the high-integrity tracker function.

[0059] DTT provides high-integrity solutions from multiple heterogeneous sources using a high-integrity integrated tracking algorithm. Source independence is a key feature in providing this high integrity without the specific need to incorporate a high-integrity (independent and redundant) solution into the sensor itself. Independent cross-checking and solution validation are found in the integration and internal independence of the tracking algorithm or each type of tracking.

[0060] When two compatible sources of tracker information are available, the system provides fail-safe behavior, namely, detection of the first fault, detection and removal of misleading information, and / or provision of a warning. The system then returns to the normal use of other cockpit instruments.

[0061] When three compatible sources of tracker information are available, the system can provide fail-operational behavior, i.e., continued operation based on two compatible tracker sources, which include detection of a first failure, provision of warnings, and subsequent fail-safe behavior in the event of a second failure.

[0062] Referring to Figure 4, a method for generating integrated multiple source tracker data is explained. Tracker data 400a, 400b, 400c, and 400d are collected from multiple tracker data sources. In this example, these correspond to tracker 1 data, tracker 2 data, tracker 3 data, and tracker 4 data. Each tracker data has its own priority relative to the other tracker data. In this example, the data from tracker 1 has priority P1, the data from tracker 2 has priority P2, the data from tracker 3 has priority P3, and the data from tracker 4 has priority P4. For simplicity, in this example, P1 is the highest and P4 is the lowest. It will be recognized that there can be a different number of tracker data sources, and different priorities can exist for each source. Furthermore, it is possible that more than two sources can have the same priority.

[0063] Tracker data is collected from one or more sensors in or associated with the area of ​​interest, for example, the cockpit of an aircraft with a pilot inside. Tracker data is received by the system 402. Many other inputs may be provided to the system, but for simplicity, these are not shown. An example of additional information shown is a risk status assessment 404. As described above, this may include classifications associated with Misleading Information (HMI) metrics. In other words, an indication of the required fidelity of the system. For the purposes of this example, a scenario of a critical, hazardous, or catastrophic event. Other metrics and classifications may be used as alternatives.

[0064] For each risk scenario and associated classification, there is a predetermined minimum tracking requirement or threshold. For example, a 100% tracking score is the highest required for the most high-risk scenario. This tracking score can be achieved by a specific combination of trackers functioning in the normal manner. Different combinations of trackers functioning in the normal manner can produce different tracking scores. In addition, if tracking data is unavailable from a source, the tracking score is reduced. This causes problems when the tracking score does not match or exceed the target score required for the current risk state. When this occurs, it may be necessary to reconfigure the system by resetting the failing trackers, obtaining tracking data from other sources, and / or reassess the risk level to correct the problem and increase the tracking score toward the target score. If the target score cannot be met, an alarm may be generated. In addition, the maximum achievable score for tracking will be sought until the completeness of tracking can be raised to the required target score. The system continuously collects data and assesses the risk level until the system is able to achieve the target score. In this case as well, this may be indicated to the user by an appropriate signal.

[0065] Returning to Figure 4, after step 402, tracker scores are determined for all available trackers and compared with the current risk status to determine a list of available trackers and their associated priorities 406. Based on the priorities of the available trackers, a priority list of tracker data is calculated 408. In step 410, the list is updated to remove any faulty sources of tracker data.

[0066] A determination 412 is made as to whether a non-faulty tracker can meet the required risk status. If yes 414, the system outputs the integrated multi-source data to the source that needs that information 416. In this example, this is via an HMD to a pilot operating a vehicle such as an aircraft.

[0067] If the determination 412 regarding whether a non-faulty tracker can meet the required risk status result is no 418, the system proceeds to step 420. In step 420, the system seeks alternative combinations of tracker data that match or are sufficiently close to the target score. If there is another combination 422, the process returns to step 406. If there is no alternative 424, the system is forced to reconfigure the tracker and / or reassess the risk status. The process then returns to step 402.

[0068] In the example in Figure 4, different tracker data can come from any known tracking system, including those mentioned herein. The process is aware of the combination of trackers and / or target scores required in different risk situations. As shown above, two types of trackers may be appropriate for a particular risk situation, but this also requires that they contribute a sufficient amount to the score. For example, the combination of tracker data 1 and tracker data 2 may be appropriate, but the combination of tracker data 3 and tracker data 4 may not. In addition, the contribution to the score when using a particular tracker data type may differ for different activities. For example, if the tracker is an eye tracker and lighting conditions are poor, the accuracy of tracking may not be very useful, and the score and / or priority may be downgraded. The system and process are highly dynamic, and the contribution of each tracker or tracker type to tracking is continuously monitored and updated as the environment changes. Tracker priorities and scores are dynamically and continuously monitored to determine the optimal integrated multiple source tracker data to provide to the task and environment at that time.

[0069] A key aspect of this invention is that security requirements are critical in many use cases. In this context, the use of tracking data could result in a user losing control of a vehicle, potentially leading to the loss of life of the user and / or others in the vicinity of the breakdown. In addition, there could be substantial loss of assets, such as aircraft, other vehicles, buildings, and similar items. Consequently, the integrated data combination for the user must be sufficiently complete to safely navigate the vehicle without incident. This is a very different consideration from, for example, the use of tracking devices in other situations where safety and loss are not realistic.

[0070] The present invention may take the form of a system, device, method, or computer medium executed on a computer. References to optics are intended to include wavelengths extending beyond the visible spectrum, and any sensor may be adapted accordingly.

[0071] In the example shown, the scenario involves an aircraft and its associated cockpit. It will be recognized that the present invention may be used in other scenarios.

Claims

1. A method for providing tracking data to a user of a platform in an environment, wherein the platform is associated with one or more input and output devices configured to track one or more variables of data associated with the user, the platform, or the environment, generate the tracking data, and provide outputs representing the tracking data, and the method is Processing data received from one or more input devices in order to determine tracking data relating to the user's relative position to at least one of the platform and the environment, or each identifies one or more input devices of the received data associated with their respective priority values, Determining the target tracking score level based on the required level of safety, To determine a first tracking score for the first input device among the one or more input devices having the highest priority, Determining a second tracking score for a second input device among the one or more input devices having the second highest priority, Combining the first tracking score with the second tracking score to generate an integrated tracking score, The aforementioned target tracking score is compared with the integrated tracking score, In response to determining that the integrated tracking score is equal to or greater than the target tracking score, integrated tracking data is created from the first input device and the second output device. To provide the aforementioned user with the integrated tracking data. A method that includes [a certain feature].

2. In response to determining that the integrated tracking score is less than the target tracking score, To determine a further tracking score for a further input device among the one or more input devices having a lower priority than the first priority of the second priority, Combining the additional tracking score with the integrated tracking score to generate an even more integrated tracking score, The aforementioned target tracking score is compared with a further integrated tracking score, In response to determining that the further integrated tracking score is equal to or greater than the target tracking score, further integrated tracking data is created from the first input device, the second output device, and the further input device. To provide the aforementioned user with the aforementioned further integrated tracking data The method according to claim 1, further comprising:

3. The method according to claim 1 or 2, further comprising generating an alarm and reconfiguring any of the one or more input devices identified as operating below a predetermined level in response to determining that the integrated tracking score is less than the target tracking score.

4. The method according to any one of claims 1 to 3, wherein providing tracking data to the user comprises displaying the tracking data via a display device.

5. The method according to any one of claims 1 to 4, further comprising capturing data from one or more input devices.

6. The method according to claim 5, further comprising capturing data relating to data imaging, illumination, eye tracking, sound, motion, gestures, velocity, acceleration, position and location, and at least one of features present in the environment.

7. The method according to any one of claims 1 to 6, further comprising selecting two or more input devices from the one or more input devices based on the priority level of the input devices.

8. The method according to any one of claims 1 to 7, further comprising selecting the first input device and the second input device from the one or more input devices based on a usage example.

9. The method according to any one of claims 1 to 8, further comprising augmenting data from different input devices to provide the integrated tracking score.

10. The method according to any one of claims 1 to 9, further comprising continuously monitoring the integrated tracking score and comparing it with the target tracking score in order to verify the integrated tracking score.

11. A system for providing tracking data to users of a platform within an environment, wherein the system is The processor comprises, Processing data received from one or more input devices in order to determine tracking data relating to the user's relative position to at least one of the platform and the environment, or each identifies one or more input devices of the received data associated with their respective priority values, Determining the target tracking score level based on the required level of safety, To determine a first tracking score for the first input device among the one or more input devices having the highest priority, Determining a second tracking score for a second input device among the one or more input devices having the second highest priority, Combining the first tracking score with the second tracking score to generate an integrated tracking score, The aforementioned target tracking score is compared with the integrated tracking score, In response to determining that the integrated tracking score is equal to or greater than the target tracking score, integrated tracking data is created from the first input device and the second output device. To provide the aforementioned user with the integrated tracking data. A system configured to perform the following actions.

12. In response to determining that the integrated tracking score is less than the target tracking score, the processor Determining further tracking scores for one or more input devices, and further input devices, that have a priority lower than the first priority or the second priority, Combining the additional tracking score with the integrated tracking score to generate an even more integrated tracking score, The aforementioned target tracking score is compared with a further integrated tracking score, In response to determining that the further integrated tracking score is equal to or greater than the target tracking score, further integrated tracking data is created from the first input device, the second output device, and the further input device. To provide the aforementioned user with the aforementioned further integrated tracking data The system according to claim 11, further configured to perform the following:

13. The system according to claim 11 or 12, wherein the processor is further configured to generate an alarm in response to determining that the integrated tracking score is less than the target tracking score, and to reconfigure any of the one or more input devices identified as operating below a predetermined level.

14. The system according to any one of claims 11 to 13, further comprising a display that provides tracking data to the user.

15. The system according to any one of claims 11 to 14, further comprising a plurality of input devices.

16. The system according to claim 15, wherein the input device further comprises at least two of the following: an optical sensor, a light sensor, an eye tracker sensor, an audio sensor, an inertial sensor, a motion sensor, a velocity and acceleration sensor, and a position and location sensor.

17. The system according to any one of claims 11 to 16, wherein the processor is further configured to select the first input device and the second input device from the one or more input devices based on the priority level of the input devices.

18. The system according to any one of claims 11 to 17, further comprising selecting the first input device and the second input device from the one or more input devices based on usage examples.

19. The system according to any one of claims 11 to 18, further comprising augmenting data from different input devices to provide the integrated tracking score.

20. The system according to any one of claims 11 to 19, further comprising continuously monitoring the integrated tracking score and comparing it with the target tracking score in order to verify the integrated tracking score.

21. The system according to any one of claims 11 to 19, comprising a head-mounted device.

22. The system according to claim 21, wherein the head-mounted device comprises a head-mounted display.

23. The system according to claim 22, wherein the head-mounted device comprises a helmet on which the head-mounted display is mounted.

24. A head-mounted display comprising the system described in any one of claims 11 to 23.

25. A machine-readable medium configured to carry out the method described in any one of claims 1 to 10 when executed on a computer.

Images