System and method for tracking user movements

Abstract

According to aspects of the present invention, a method is provided for providing tracking data to a user of a platform in an environment, wherein the platform is associated with one or more input and output devices configured to track one or more variables of data associated with a user, the platform, or the environment, generate tracking data, and provide outputs representing the tracking data, and the method comprises processing data received from one or more input devices to determine tracking data relating to the user's relative position to at least one of the platform and the environment, wherein the input data relates to the position of a hand associated with the use of a platform-based controller by a user, the hand position is used to reinforce the validity of one or more alternative inputs of one or more input devices, and processing the input data to provide output data represented to the user.

Description

Technical Field

[0001] The present invention relates to a system and method for tracking a user's movement, and more particularly, but not limited thereto, to tracking the movement of a user located in a vehicle.

Background Art

[0002] The tracking of objects, as well as the determination and tracking of the orientation of objects, are known in many scenarios. Head or helmet-mounted display systems incorporating a tracking system for providing orientation data of a head or helmet for use in positioning display artifacts in a display system where the positioning of symbols or other artifacts in the display can be utilized. In some examples, these systems can be used by a user moving on or within a movable platform such as an aircraft or other vehicle.

[0003] In some situations, such systems can be subject to buffeting or similarly rapidly induced changes in orientation when moving on or within a movable platform such as an aircraft or other vehicle. This can make tracking more difficult.

[0004] Safety solutions in vehicles are often required to comply with civil hazard classification schemes and functional safety guidance. These schemes have different levels of classification, where lower levels are less reliable than higher levels. In some environments, low or medium levels may be appropriate, while in other environments, higher levels are essential.

[0005] In an aircraft environment, medium-level integrity / solution levels are generally easy to implement using currently known tracking systems and devices. Examples of medium-level solutions are those that can be used in critical hazard classification risk scenarios.

[0006] The use of head-mounted devices (HMDs) as a primary criterion typically inherits hazardous and / or catastrophic classifications, for example, due to the provision of hazardously misleading information (HMI) for critical flight data. This imposes a no-single-point failure requirement on HMD architectures. The loss of critical symbols on a single display typically remains a minor risk. The loss of the same information on all cockpit displays, including a head-down display (HDD) and any Get You Home instruments, is typically ranked as catastrophic, although the risk is shared across different systems. Thus, for any single display surface, the significant challenge tends to exist compared to the case of HMIs where the HMD tracker contributes to all isometrically displayed information.

[0007] Traditional practice involves using multiple similar helmet-based tracking solutions, such as dual optical trackers, to provide additional completeness, but this can add weight and lead to other drawbacks.

[0008] There is a need for tracking systems and related methods that provide high-fidelity tracking under any risk scenario, unhindered by problems exhibited in known systems. [Overview of the project]

[0009] According to aspects of the present invention, a method is provided for providing tracking data to a user of a platform in an environment, wherein the platform is associated with one or more input and output devices configured to track one or more variables of data associated with a user, the platform, or the environment, generate tracking data, and provide outputs representing the tracking data, and the method comprises processing data received from one or more input devices to determine tracking data relating to the user's relative position to at least one of the platform and the environment, wherein the input data relates to the position of a hand associated with the use of a platform-based controller by a user, the hand position is used to reinforce the validity of one or more alternative inputs of one or more input devices, and processing the input data to provide output data represented to the user.

[0010] In one embodiment, one or more data sources are captured from one or more input devices in order to capture input data and alternative input data, and the output is based on a combination of input data and alternative input data.

[0011] In this embodiment, input data is captured based on user interaction with a platform-based controller on the platform at a given point in time.

[0012] In this embodiment, input data is captured from hand, finger, or thumb movements relative to a platform-based controller.

[0013] In one embodiment, the use of a platform-based controller by a user comprises the use of one of the following: a controller, a button, a touchscreen, or another haptic controller.

[0014] In this embodiment, the hand position is augmented with alternative input data to validate the alternative input data.

[0015] In one embodiment, if the hand position is not augmented with alternative input data, the system performs one of the following: system recalibration and / or reset.

[0016] In one embodiment, if the hand position is augmented with alternative input data, the system performs one of the following: cross-checking and recalibration.

[0017] In this configuration, input data is periodically captured based on instructions given to the user using a platform-based controller.

[0018] In some embodiments, this involves capturing input data and alternative data related to at least one of the following: data imaging, lighting, sound, motion, gestures, gravity, heat, velocity, acceleration, position and location, and features present in the environment.

[0019] In one embodiment, providing tracking data to a user includes at least one of displaying the tracking data via a display device and providing haptic feedback.

[0020] A further aspect of the present invention provides a system for providing tracking data to a user of a platform in an environment, wherein the platform is associated with one or more input and output devices configured to track one or more variables of data associated with a user, the platform, or the environment, generate tracking data, and provide outputs representing the tracking data, and the system provides a processor for processing data received from one or more input devices to determine tracking data relating to the user's relative position to at least one of the platform and the environment, wherein the input data relates to the position of a hand associated with the use of a platform-based controller by a user, the hand position is used to reinforce the validity of one or more alternative inputs of one or more input devices, and processing the input data to provide output data represented to the user.

[0021] In an aspect, there are one or more input devices for capturing one or more data sources from one or more input devices to process input data and alternative input data, and the output is based on a combination of the input data and the alternative input data.

[0022] In an aspect, the input data is captured based on user interaction with a platform-based controller on the platform at a certain point in time.

[0023] In an aspect, the input data is captured from movements of the hand, finger, or thumb with respect to a platform-based controller, and the platform-based controller comprises one of a controller, a button, a touch screen, and other tactile controllers.

[0024] In an aspect, the hand position is reinforced with alternative input data to validate the alternative input data.

[0025] In an aspect, if the hand position is not reinforced with alternative input data, the system performs one of system recalibration and reset, and if the hand position is reinforced with alternative input data, the system performs one of cross-check and recalibration.

[0026] In an aspect, the input data is periodically captured based on instructions to the user using a platform-based controller.

[0027] In an aspect, there is capturing input data and alternative data related to at least one of data imaging, illumination, sound, movement, gesture, gravity, heat, speed, acceleration, position, location, and sources of features present in the environment.

[0028] In an aspect, providing tracking data to the user comprises at least one of displaying the tracking data via a display device and a tactile feedback device.

[0029] In one aspect, the system comprises at least one of a head-mounted display, a head-down display, and a head-up display.

[0030] According to a further aspect of the invention, a head-mounted display including the system of the previous aspect is provided.

[0031] According to a further aspect of the invention, a machine-readable medium is provided that, when executed on a computer, is configured to implement the method of the previous aspect.

[0032] Embodiments of the invention are now described by way of example only and with reference to the drawings.

Brief Description of the Drawings

[0033] [Figure 1] An exemplary system according to an aspect of the invention. [Figure 2] A diagram of a helmet having components suitable for implementing functions according to an aspect of the invention is shown. [Figure 3] A tracking and guidance system according to an aspect of the invention is shown. [Figure 4] A scheme for operating the system of FIG. 3 is shown. [Figure 5] A simple flowchart of a method according to an aspect of the invention is shown.

Modes for Carrying Out the Invention

[0034] The present invention generally relates to a tracking system and method for accurately identifying the location of a user on a platform such as an aircraft, as well as all nearby platform-related instruments and control devices, so that all necessary information can be directed to a user who can safely navigate and move the platform.

[0035] Referring here to Figure 1, a system 100 according to an aspect of the present invention is shown. The system 100 comprises a head-worn display (HWD) device 102 adapted to be supported on a helmet (not shown in Figure 1) worn by a user 104. The system also includes a camera 106 and a processor 108.

[0036] Figure 1 further illustrates surface 110, which may be referred to interchangeably with the platform and comprises any given surface on which a user 104 is positioned. Surface 110 may include, but is not limited to, a vehicle or flight simulation device. The user 104 positioned on surface 110 wears the HWD 102. An imaging device or camera 106 may be integrated into the body of the HWD 102 or mounted at any suitable location on the HWD 102 so as to be able to observe a number of views and instruments, as will be described in more detail below. The camera may collect images of markers 112a, 112b, 112c, and 112d positioned at predetermined locations relative to surface 110. In Figure 1, markers 112a, 112b, 112c, and 112d are illustrated as being positioned on the upper part 114 of the surface; however, it should be recognized that markers 112a, 112b, 112c, and 112d can be positioned at any suitable location on the surface 110.

[0037] Camera 106 may be integrated with HWD 102. Camera 106 may include multiple cameras. The cameras may include, but are not limited to, stereo cameras or depth cameras. In Figure 1, the camera is shown facing upward, but it is recognized that camera 106 can be positioned or positioned at any suitable location on HWD 102 so as to be configured to capture images of many aspects of the environment, such as images of trackers or devices, including optical inertial trackers, HMD camera trackers, head motion trackers, eyeball trackers, hand gesture trackers, marker trackers, platform and attitude trackers, and any other devices that may be required depending on the system's usage requirements.

[0038] A tracker may include, or may be, a sensor device that detects and collects variables in data, including imaging, illumination, sound, motion, gravity, heat, velocity, acceleration, position and location, or any other attributes or features present within the target environment. Sensors include optical sensors, light sensors, audio sensors, inertial sensors, motion sensors, thermal sensors, gravity sensors, velocity and acceleration sensors, position and location sensors, etc. In this example, the environment includes the aircraft cockpit and the environment outside the cockpit. It will be recognized that at least some of the sensors and associated detected data may be outside the cockpit.

[0039] Tracking data is collected from sensors and further used by the user to navigate the platform within the environment. Tracking data is used herein to describe data received from one or more input sources tracking the user and any moving elements within a fixed space, platform, or similar. As indicated elsewhere, a fixed location or platform could be the cockpit of a vehicle such as an aircraft. The tracking data is processed and configured to provide output tracking data for the user, which is either displayed to the user or otherwise supplied. In the latter case, this may include haptic feedback from a joystick or any other haptic feedback. The output tracking data further includes a real-world image of the user and a virtual image generated by the system to represent any other output data. This representation may be a symbol, a real or virtual image of a local controller which may or may not move, an external view of the outside of the platform which may be real or virtual, and any other appropriate form of output data as indicated by the use case.

[0040] Referring to Figure 2, a helmet tracker system 200 integrated with a helmet 202 is shown. The helmet tracker system comprises one or more sensor devices 204. The sensors may include one or more inertial sensor components, one or more cameras, and any other sensors suitable for capturing the environment or user characteristics that may be useful in tracking the user in any way. The inertial sensor components may include three gyro elements 204 fitted into the outer shell of the helmet 202. The optical helmet tracker system further comprises an array of light-emitting diode (LED) devices 206 integrated into or mounted on the outer shell of the helmet 202, and an array of one or more cameras 208, each mounted in fixed positions, for example in an aircraft cockpit, so as to have a line-of-sight view to at least some of the LEDs 206 in any possible orientation of the helmet 202. A system control device (controller) 210 is linked to each of the cameras 208 to receive image data resulting from the detection of light emitted by the LEDs 206 in the line of sight of the cameras. The controller 210 is also linked to the LED 206 in the helmet 202 by an appropriate link (which may be wired or wireless), and the controller 210 is configured to control the illumination of the LED 206. The controller 210 is configured to trigger, for example, periodic illumination of the LED 206 in a predetermined pattern and to receive the resulting image data from the camera 208. The controller 210 includes a digital processor, or a combination of hardware and software implementation functions, programmed to implement an appropriate optical helmet tracker algorithm, and is configured to illuminate the LED 206 and interpret the resulting image data from the camera 208 to determine the orientation of the helmet relative to a reference frame associated with the aircraft.

[0041] Controller 210 is further configured to implement the functions of the entire helmet tracker system, as will be described in more detail below. Controller 210 is linked to the helmet 202 by any suitable means (not shown, which may be wired or wireless, for example) and is configured to receive rate data outputs from one or more inertial sensor components and to transmit image data received from an image generation system (not shown) installed on the aircraft to a helmet-mounted display (not explicitly shown in Figure 2) integrated within the helmet 202 via an interface. Further sensor data may be received and analyzed within the control that generates outputs displayed to the user. The display comprises a helmet-mounted display system, which may be a visor projection display system, any type of HWD, or a waveguide-based display. The display is configured to present images to the user such that the images appear superimposed on the user's view of an external scene or environment through the visor 212.

[0042] The controller is configured to output helmet orientation measurements to an image generation system (not shown) installed on the aircraft for use in calculating the position of certain types of symbols and data intended to be displayed on the helmet-mounted display, as appropriate. The output is a representation for display to the user at any given time and changes dynamically as the vehicle, user, and environment change. The output representation is real-time or near real-time. In a typical aircraft installation of the helmet-mounted display system incorporating the hybrid helmet tracker system of the present invention, one or more further modules (not shown in Figure 2) are likely to be installed on the aircraft for the purpose of generating images for display by any display system installed on or used on the aircraft. Such display systems may include a head-up display (HUD) and / or a head-down display (HDD) in addition to the helmet-mounted display relevant to the application of the present invention. Such further modules are likely to be linked to a standard aircraft data bus and thus have access to aircraft orientation data output by the aircraft navigation system and data from other systems that need to display information to the pilot. Image data, including symbols and data for display on the helmet-mounted display, may be passed to the controller 210 as needed, along with updates to the aircraft's orientation received via the aircraft data bus by such further modules.

[0043] As stated above, safety is a critical aspect of controlling a vehicle. This is especially true when piloting an aircraft, where the vehicle is moving at high speeds and is subject to many external and internal factors that could interfere with the user's control of the aircraft. When eye position is tracked and used to calculate and draw a reticle (or other such eye position specifier) ​​on a see-through display for purposes such as target selection and / or navigation and / or checklist completion, the reticle or other such specifier depends on a consistent flow of accurate eye position data (with current technology, it is not feasible to add auxiliary (e.g., inertial) secondary tracking to complement optical (e.g., IR) eye tracking). Eye position is currently detected optically in most systems, and therefore, in situations where the field of view of the eye is limited (e.g., extreme + / - gravity), the present invention would employ other sources of tracker information such as the head (e.g., optical inertia) or the hand (e.g., generated reference point), which would themselves contain a level of calibration data sufficient to maintain a picture of where the eye is against see-through display artifacts.

[0044] This invention identifies a solution in attempting to overcome problems associated with insufficient optical tracking and other problems associated with safely tracking variables used when maneuvering a vehicle.

[0045] The present invention relates to using more than one source of physical tracker data to enable compensation for the display field of view of a display in various situations in which the display field of view may be impaired, such as slippage of headworn equipment (e.g., a loose helmet fit). A combination of more than one source of data (e.g., eyeball, head, inertia, body) will enable the positioning of eye gaze, free space, and isometric symbols in a multi-track system display. Examples of tracker data that may be used to achieve this include eyeball (eyeball profile, iris, pupil), optical and inertial head position, hand / gesture, etc.

[0046] This invention describes how the position (e.g., angle) of a selected tracking target can be used to generate a more seamless and accurate set of position / rotation matrices, enabling the precise positioning of the target object within the display reference frame. Essentially, the use of multiple sources of tracking data is used to create a calibration standard (e.g., what the pilot is seeing or where the HWD is positioned relative to a known outer reference) that can be used to maintain accuracy of the display position in the event of loss of a real-time tracker data source.

[0047] At a higher level, the present invention relates to the use of additional sensors, such as hand and gesture tracking, to extend so-called conventional optical tracking processes and systems when navigating and operating a vehicle. In an aircraft cockpit, for example, as described above and as described in more detail below, optical tracking of the user's head is an important aspect of determining the user's position and location relative to a platform such as an aircraft.

[0048] Referring to Figure 3, an overview of the present invention is shown. System 300 includes a calibrated display and an external position controller 302 that controls data acquisition, processing, and display of the data to a user 304 located at a platform such as an aircraft cockpit. The data displayed to the user is called tracking data and is used to enable the user to control the aircraft.

[0049] User 304 is wearing a head-mounted device (HMD) that may be mounted on a helmet or positioned in any other way relative to the user. Other suitable devices may include a head-up display (HUD), a head-down display (HDD), or any other suitable display device.

[0050] Several cameras 306 and 308 are shown within the vicinity of the user. Camera 306 is one or more cockpit-mounted cameras having user and surrounding views. Camera 308 is a head-mounted camera that may be integrated with the HMD or separate from the HMD and may generally be configured to provide the user with a forward view. The cameras capture images of the user, either together or separately, including head position and location 310, eye gaze 312, and user gestures 314 such as hand movements. In addition, the cameras capture images of the surroundings, either together or separately, via platform references and / or reference point markers located in any suitable location 316, including but not limited to controllers, buttons, touchscreens, and body or hand gestures.

[0051] As described above, there are several different sensors associated with the scenario and environment. These are indicated by a single reference numeral 310 in Figure 3, and inputs from all or any of these sensors are fed into the processor 302. Further inputs to the controller are the platform's position, location, and orientation 320. The following is an exemplary list of data acquired by the camera sensors and passed to the processor: • The position of the eyes relative to the display on the HMD. • Hand position relative to the display and the HMD. • Head position relative to the display. • A combination of multiple tracker data sources to enable cross-checking and / or improved integrity. • A combination of multiple tracker sources providing HWD, eye, and hand position data relative to the external environment, and a display crucial in a virtual cockpit using an AR display. The loss of certain tracker data types, such as gestures, will result in a return to a "failsafe" display mode, such as a non-dynamic fixed display function. • A head-mounted camera for extracting gesture and platform data. • Use of reference points or cockpit and / or cabin references to provide cross-checking and redundancy for additional head trackers. • Use of eye tracking to enhance helmet aiming and / or correction.

[0052] Figure 4 shows a scheme 400 of how the addition of the various tracking data sources already used in HMDs may be used to provide more reliable head positioning and the resulting display information and / or symbols. The scheme is intended to add integrity to the entire helmet-related system.

[0053] Scheme 400 in Figure 4 revolves around a processing unit 402, which in this case is a helmet processing unit. The processing unit is configured to communicate with several different modules, including an inertial motion unit (IMU) 404, an optical helmet tracker module 406, a display 408, an aircraft or vehicle information module 410, a head-mounted camera 412, an eye-tracking module 414, and a pilot or user characteristics module 416. It will be recognized that there may be further modules related to tracking data and other information from different sensors in or associated with the environment inside or outside the platform.

[0054] Other sensors may include, or may be, sensor devices that detect and collect variables in data including imaging, illumination, eye tracking, sound, motion, gravity, heat, velocity, acceleration, position and location, or any other attributes or features present in the environment under consideration. Sensors include optical sensors, light sensors, audio sensors, inertial sensors, motion sensors, heat sensors, gravity sensors, velocity and acceleration sensors, position and location sensors, etc.

[0055] In this example, the environment includes the aircraft cockpit and the environment outside the cockpit. It will be recognized that at least some of the sensors and associated detected data may be outside the cockpit. The tracking data is used by the user to navigate the platform within the environment by presenting information and symbols on a suitable display.

[0056] The IMU404 measures head movement to enable and assist in the precise determination of head position, and provides head tracking data to the processor 402.

[0057] The optical helmet tracker 406 serves as a source of head position and / or gaze relative to the aircraft. This data and information can be captured via a tracker light-emitting diode (LED) 418 mounted on the helmet.

[0058] Display 408 is any type of display that allows the user to view data from at least the processor 402. Generally, in the use cases described, the display will comprise one or more of the following: a head-up display (HUD) and / or a head-down display (HDD), or preferably a helmet-mounted display (HMD).

[0059] The head-mounted camera module 412 includes a head-mounted camera that uses images of a reference point 420 to provide backup head positioning in a fixed space such as a cockpit. The reference point or marker 420 is located on the ceiling or one or more other points in the cockpit, and its images are recorded so as to enable a processor to use the data to determine accurate tracking data and to help provide the user with the optimal display of symbols and other images necessary to navigate the vehicle in the environment.

[0060] The eye-tracking module 414 functions as a source of eye position relative to the helmet. The display is positioned in a known plane relative to the helmet. The eye tracker is fixed to the helmet and can therefore be used as additional head position and / or gaze data points, and as a means for drawing symbolic artifacts corresponding to the eye position and / or head position.

[0061] The pilot or user features module 416 is used to capture data related to gestures 422, haptics 424, and other cameras 426. Gestures may be collected from optical images and / or other sources of data such as hand, finger, and / or thumb movement detection. The method by which gestures are collected may be linked to any interaction of the gesture with the controller or haptics 424. In addition, gestures may be captured using, for example, daytime or nighttime cameras, as indicated by lighting conditions.

[0062] Hand and / or gesture position tracking can be achieved either during the day or at night, using multiple sources of cameras mounted on the head, the user, and / or any cockpit surface. The system can be configured for one hand or both hands. Hand gestures can be determined from sensors located on a glove or hand controller worn by the user. These gestures can be determined from images or other sensing means, such as an IMU located on the glove or controller.

[0063] One place where hand gestures are particularly useful is in relation to the use of controllers, buttons, touchscreens, or other haptic controllers in a cockpit. By working with a controller in or near a known location, the hand's position at that moment can be considered known only by the fact that it is working with or cooperating with a known controller. In this way, the hand's position can be augmented with other tracking data from other sources or modules to validate the validity of other sources of input tracking data. For example, the hand's position can be used to "recalibrate" or "cross-check" with other data periodically. For example, a user might periodically press a button. This is an independent (e.g., non-optical) source used for calibration, but not necessarily the position of the captured hand and / or gesture, to augment the validity of other input data from other sources. Instead, this is an indication of where the hand is located in the cockpit at a precise moment. Knowing the hand's location allows the system to reset its positioning if there is any discrepancy or lack of effective augmentation in any tracking data from a light source. For example, if a user presses a button, but the optical data shows that the user's hand is positioned at a certain distance from the button, the system can detect the anomaly and attempt to correct it.

[0064] If an anomaly is identified, the system may recalibrate or reset all or any optical sensors that appear to be giving incorrect data. Pressing a button may be used as a cross-check to determine that the hand is actually positioned near the button or other controller. This is achieved by strictly adhering to pilot and / or user-related data.

[0065] In the case of hand gestures, the glove or hand controller may include different types of sensors. For simplicity, the following description will refer to the glove, but may apply equally to a handheld controller or other controller associated with other parts of the body. As mentioned above, the glove may include sensors, which may be reference points or markers picked up by a camera. Other sensors may capture motion, temperature, velocity, and position location. Sensors may include an IMU or other types of motion sensors. In some cases, there may be a temperature sensor located on the glove, or an infrared (IR) camera directed towards the hand that detects the hand's position based on a known temperature profile. In the latter case, the IR camera may capture head and / or hand data and determine their relative positions so that this relationship can be cross-checked by any other sensors.

[0066] In some examples of the present invention, the use of hand or gesture tracking (e.g., hand recognition and camera data, and / or a hand-mounted tracker) serves as an additional source of data for tracking the helmet, i.e., along with other sources such as camera / reference point tracking and IMU tracking on the helmet, which is referenced against general platform IMU / GPS tracking.

[0067] In addition, determining the hand alone can be used to confirm the physical location of the hand relative to the cockpit. For example, as mentioned above, when a particular dashboard button is pressed or a control stick is manipulated, the location of the hand at that moment is known and can be fed back to the system. In some situations, this mode of operation (a means of tracking the hand position by itself) can be used alone without the additional tracking methods described above.

[0068] It will be recognized that means of tracking hand position on their own can be used in combination with some or all of the other types of tracking methods and modules described above to produce multiple integrated tracking data from one or more different sources. The combination can be any combination depending on the system requirements or use case, and may also depend on the nature of the high-completeness context, in particular the context of the vehicle being driven.

[0069] The method according to the present invention may be carried out according to the simplified flowchart in Figure 5. Method 500 includes a step 502 of capturing hand position input data, as described in more detail above. In addition, the method includes capturing alternative input data 504, which is also described in more detail above. The captured hand position input data 502 and the captured alternative input data 504 are compared to determine whether they reinforce each other. If there is no invalid reinforcement 508, the method triggers a system reset or recalibration 510, the method returns to recapturing the alternative input data 504, and, if necessary, recapturing the captured hand position input data 502. If reinforcement is effective 512, the method proceeds to output the hand position based on the captured hand position input data 514. The output may be a tactile or visual representation presented to the user 516. The method may further output reinforced combined output data based on the captured hand position input data 502 and the captured alternative input data 504 518. This is then presented to the user 520 via at least one of a display device and a haptic feedback device. The display device may include one or more of a head-mounted display (HMD), a head-down display (HDD), and a head-up display (HUD).

[0070] In some situations, the user may receive a prompt or command to interact with one of the platform-based controllers 522, which may or may not be a specified controller, as the use case indicates. The commands are periodic and may act as a predetermined cross-check process for different input sources and devices. In addition, the prompts are obtained based on invalid augmentative feedback, so that the input data from the captured hand position input data 502 and the captured alternative input data 504 are simultaneous and serve as a known decision of the hand at a particular point in time. This may be achieved using time stamping or similar methods.

[0071] A key aspect of this invention is that security requirements are critical in many use cases. In this context, the use of tracking data could result in a user losing control of a vehicle, potentially leading to the loss of life of the user and / or others in the vicinity of the breakdown. In addition, there could be substantial loss of assets, such as aircraft, other vehicles, buildings, and similar items. Consequently, the integrated data combination for the user must be sufficiently complete to safely navigate the vehicle without accidents. In this context, the integrated data must contain the data necessary to meet the safety requirements of any given use case. This is a very different consideration from, for example, the use of tracking devices in other situations where safety and loss are not realistic.

[0072] The present invention may take the form of a system, device, method, or computer medium executed on a computer. References to optics are intended to include wavelengths extending beyond the visible spectrum, and any sensor may be adapted accordingly.

[0073] In the example shown, the scenario involves an aircraft and its associated cockpit. It will be recognized that the present invention may be used in other scenarios.

Claims

1. A method for providing tracking data to a user of a platform in an environment, wherein the platform is associated with one or more input and output devices configured to track one or more variables of data associated with the user, the platform, or the environment, generate the tracking data, and provide outputs representing the tracking data, and the method is Processing data received from one or more input devices in order to determine tracking data relating to the user's relative position to at least one of the platform and the environment, wherein the input data relates to the position of the hand associated with the user's use of a platform-based controller, and the hand position is used to reinforce the validity of one or more alternative inputs from the one or more input devices. To process the input data in order to provide the output data represented to the user. A method that includes [a certain feature].

2. To capture the aforementioned input data and the aforementioned alternative input data, capture one or more data sources from one or more input devices. The method according to claim 1, further comprising, wherein the output is based on a combination of the input data and the alternative input data.

3. The method according to claim 1 or 2, wherein the input data is captured based on a user interaction with the platform-based controller on the platform at a given point in time.

4. The method according to claim 3, wherein the input data is captured from the movement of the hand, finger, or thumb relative to the platform-based controller.

5. The method according to any one of claims 1 to 4, wherein the user's use of the platform-based controller comprises the use of one of a controller, a button, a touchscreen, or another haptic controller.

6. The method according to any one of claims 1 to 5, wherein the hand position is augmented with the alternative input data to validate the alternative input data.

7. The method according to claim 6, wherein if the hand position is not reinforced by the alternative input data, the system performs one of recalibration and reset of the system.

8. The method according to claim 6, wherein if the hand position is augmented with the alternative input data, the system performs one of cross-checking and recalibration.

9. The method according to any one of claims 1 to 8, wherein the input data is periodically captured based on instructions to the user using the platform-based controller.

10. The method according to any one of claims 1 to 9, further comprising capturing input data and alternative data relating to at least one of data imaging, lighting, sound, motion, gestures, gravity, heat, velocity, acceleration, position and location, and features present in the environment.

11. The method according to any one of claims 1 to 10, wherein providing tracking data to the user comprises at least one of displaying tracking data via a display device and providing haptic feedback.

12. A system for providing tracking data to users of a platform in an environment, wherein the platform is associated with one or more input and output devices configured to track one or more variables of data associated with the user, the platform, or the environment, generate the tracking data, and provide outputs representing the tracking data, and the system is Processing data received from one or more input devices in order to determine tracking data relating to the user's relative position to at least one of the platform and the environment, wherein the input data relates to the position of the hand associated with the user's use of a platform-based controller, and the hand position is used to reinforce the validity of one or more alternative inputs from the one or more input devices. To process the input data in order to provide the output data represented to the user. A system equipped with a processor for performing certain tasks.

13. One or more input devices for capturing one or more data sources from one or more input devices in order to process the input data and the alternative input data. The system according to claim 12, further comprising, wherein the output is based on a combination of the input data and the alternative input data.

14. The system according to claim 12 or 13, wherein the input data is captured based on a user interaction with the platform-based controller on the platform at a given point in time.

15. The system according to claim 14, wherein the input data is captured from the movement of the hand, finger, or thumb relative to the platform-based controller, and the platform-based controller comprises one of a controller, a button, a touchscreen, and other haptic controllers.

16. The system according to any one of claims 12 to 15, wherein the hand position is augmented with the alternative input data to validate the alternative input data.

17. The system according to claim 16, wherein if the hand position is not reinforced by the alternative input data, the system performs one of recalibration and reset of the system, and if the hand position is reinforced by the alternative input data, the system performs one of cross-checking and recalibration.

18. The system according to any one of claims 12 to 17, wherein the input data is periodically captured based on instructions to the user using the platform-based controller.

19. The system according to any one of claims 12 to 18, further comprising capturing input data and alternative data related to at least one of the sources of data imaging, lighting, sound, motion, gestures, gravity, heat, velocity, acceleration, position, location, and features present in the environment.

20. The system according to any one of claims 12 to 19, wherein providing tracking data to the user comprises at least one of displaying the tracking data via a display device and a haptic feedback device.

21. The system according to any one of claims 12 to 20, comprising at least one of a head-mounted display, a head-down display, and a head-up display.

22. A head-mounted display comprising the system described in any one of claims 12 to 21.

23. A machine-readable medium configured to carry out the method described in any one of claims 1 to 11 when executed on a computer.

Images