Mixed reality flight simulators and training systems

Abstract

A mixed reality flight simulator comprising a headset with a view screen aligned with a student's eyes and an outwardly facing passthrough camera, a flight simulator console comprising tangible and simulated aircraft controls and at least one control screen, the passthrough camera and control screen having the same refresh rate, and a computer having flight training simulation software, the computer integrating input from the passthrough camera with a simulated image of the tangible aircraft flight controls so the student can view the student's own hands through a portal on the view screen while interacting with the flight simulator console and the flight training simulation software.

Description

MIXED REALITY FLIGHT SIMULATORS AND TRAINING SYSTEMSTECHNICAL FIELD

[0001] The present disclosure relates to mixed reality flight simulators and training systems.BACKGROUND

[0002] A worldwide pilot shortage is currently affecting airlines and is expected to worsen in the coming decades. Industry experts project that over 260,000 new pilots will be needed between 2020 and 2029. Moreover, flight training institutions are faced with a multitude of challenges including finding ways to increase student throughput, decrease student attrition, streamline training processes, reduce costs, maximize flight instructor time and efficiency, and modernize outdated forms of flight training. Unfortunately, many currently available large-scale flight training programs are currently incapable of meeting the demand requested from the military and commercial air transportation industry for professionally trained pilots.

[0003] In attempts to address some of the deficiencies of existing flight training programs, virtual reality has been used. Virtual reality flight simulators have been well received by the flight training world because it allows the student to become immersed and familiar with the flight environment, while reducing the need to rely completely on human flight instructors to teach the student. In this regard, the virtual reality simulators can teach the students the basics on their own while allowing the human instructors to focus on higher level learning.

[0004] However, a downside to virtual reality is that the students generally can only see the virtual world. This means that while the students may be touching a flight control yoke in the real world, they only see a virtual yoke in the virtual world and they do not see their hands touching the yoke. This can be problematic when attempting toteach fine motor skills in the virtual world. For example, if a student needs to find a small button or switch in the virtual world, they may struggle to differentiate between multiple buttons in the physical world because they cannot actually see their hand in the virtual world.

[0005] Accordingly, mixed reality simulator systems and training systems that allow quick adoption of improved flight training with reduced effort from instructors and staff members, encourage more people to learn to fly, provide better instructor to student ratios, and provide technology that enhances the training experience are all desirable.SUMMARY

[0006] The present disclosure provides mixed reality simulator systems and training systems that allow quick adoption of improved flight training with reduced effort from instructors and staff members, encourage more people to learn to fly, provide better instructor to student ratios, and provide technology that enhances the training experience.

[0007] As used herein, “mixed reality” means a combination of a user’s virtual and real world environments. For example, in accordance with the present disclosure, a flight simulator comprises a headset having a view screen aligned with a student’s eyes and an outwardly facing passthrough camera, a flight simulator console comprising tangible simulated aircraft controls and at least one control screen, wherein the refresh rate of the passthrough camera is synchronized with a refresh rate of the control screen so that the control screen can be viewed on the view screen without visual interference, and at least one computer having flight training simulation software, the computer integrating input from the passthrough camera with a simulated image of the tangible aircraft flight controls so the student can view the student’s ownhands through a portal (as described below) on the view screen while interacting with the flight simulator console and controls and the flight training simulation software. In accordance with various aspects of the present disclosure, the flight simulator may further comprise a Learning Record Store (LRS) and a Learning Management System (LMS) for interaction between the student, one or more instructors and / or the flight simulator and software.

[0008] In accordance with various aspects of the present disclosure, the computer can provide instructions to the student, in mixed reality in a simulated environment such as on the view screen. In various embodiments, the simulator console controls may be configured to provide tactile feedback to the student correlated with the simulation.

[0009] Thus, a flight simulator in accordance with the present disclosure allows for the quick adoption of a novel approach to flight training with minimal effort from flight training staff, thereby encouraging more youth to learn to fly, providing a better instructor to student ratio, and technology that enhances the training experience, bridging a gap between written training materials and the aircraft, and reducing the time to solo flying up to and exceeding 30%. Additionally, a flight simulator in accordance with the present disclosure provides a spectrum of uses for mixed reality flight from free flight to fully integrated instruction including, among other things, manipulating the electronic characteristics of the electronic flight bag, aircraft situation, and artificial instruction.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure, and together with thedescription serve to explain the principles of the disclosure, wherein:

[0011] Figure 1 is a perspective view of a student using a mixed reality simulator in accordance with the present disclosure;

[0012] Figure 2 is a illustrates the view from the perspective of a student from inside a headset using a mixed reality simulator showing the virtual world and the student’s physical hands and controls in accordance with the present disclosure;

[0013] Figure 3 is a diagram illustrating a landscape of a virtual world, real world, a portal, placards and focus areas in accordance with the present disclosure;

[0014] Figure 4 is another diagram illustrating a landscape of a virtual world, real world, a portal, placards and focus areas in accordance with the present disclosure; and

[0015] Figures 5A and 5B are diagrams illustrating a landscape of a virtual world, real world, a portal, placards and focus areas in accordance with the present disclosure.DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

[0016] Persons skilled in the art will readily appreciate that various aspects of the present disclosure can be realized by any number of methods and systems configured to perform the intended functions. Stated differently, other methods and systems can be incorporated herein to perform the intended functions. It should also be noted that the accompanying drawing figures referred to herein are not all drawn to scale but may be exaggerated to illustrate various aspects of the present disclosure, and in that regard, the drawing figures should not be construed as limiting. Finally, although the present disclosure can be described in connection with various principles and beliefs, the present disclosure should not be bound by theory.

[0017] In general, the present disclosure provides devices, systems andmethods for mixed reality simulator systems and training systems that allow quick adoption of improved flight training with reduced effort from instructors and staff members, encourage more people to learn to fly, provide better instructor to student ratios, and provide technology that enhances the training experience.

[0018] Preliminarily, in accordance with various aspects of the present disclosure, the terms below have the following meanings (and may be further defined or expanded upon elsewhere in the present disclosure):

[0019] “Aircraft Flight Controls” refers to physical devices such as yokes, pedals, throttles, and switches that a pilot uses to control an aircraft. In accordance with the present disclosure, these are tangible and often equipped with tactile or haptic feedback.

[0020] “Control Screen” refers to a screen that displays flight simulation data and is synchronized with the passthrough camera to avoid visual interference in mixed reality.

[0021] “Haptic Feedback” refers to physical sensations (e.g., vibration or resistance) provided by simulator controls to mimic real-life aircraft feedback.

[0022] “Immersive Training Device” or “ITD” refers to a training system that fully immerses the user in a simulated environment, enhancing realism and learning effectiveness.

[0023] “Learning Management System” or “LMS” refers to a software platform (e.g., LearnDash) used to manage student progress, provide dashboards, track goals, and display performance metrics.

[0024] “Learning Record Store” or “LRS” refers to a system (e.g., GrassBlade) that collects and stores student performance data for use in analytics and integration with the LMS.

[0025] “Mixed Reality” or “MR” refers to a combination of real and virtual environments where students can interact with both simultaneously, such as seeing their own hands while wearing a headset.

[0026] “Passthrough Camera” refers to a camera on the mixed reality headset that captures real-world visuals (such as a user’s hands and controls) to overlay in the virtual display.

[0027] “Placards” are a visual way of displaying information broken into three categories.

[0028] “Instructional Placards” are placards used to provide visual representation of instruction and guidance such as check lists, configuration, objectives the student should be performing, corrective actions, and the like.

[0029] “Focus Placards” are placards with informational areas indicating an area of interaction or area that should be referenced or monitored, such as, for example, Engine Tachometer, Airspeed Indicator, Throttle control, or the like.

[0030] “Supplemental Placards” are placards with additional informational displays that can be used to provide additional information or details that do not fit into the categories above. Supplemental placards may be viewed through the simulator or using supplemental tools or devices.

[0031] “Portals” are “windows” or “cut outs” in the virtual world that is where the real world is viewed.

[0032] “Refresh Rate” refers to a frequency at which a display or camera updates its image. For example, in accordance with one aspect of the present disclosure, a passthrough camera and control screens are synchronized at a rate of 60Hz to prevent visual disruption.

[0033] “Simulated Environment” refers to a virtual setting generated by thecomputer system to replicate a flight experience for training purposes.

[0034] “Tactile Feedback” refers to physical responses from simulator components (e.g., resistance or vibrations) designed to simulate real-world sensations during use.

[0035] “Turn-Key Solution” is a ready-to-use product that includes all necessary hardware and software for immediate deployment and operation.

[0036] “Virtual Instructor” is an artificial intelligence (Al) based or preprogrammed teaching system within the simulator that provides automated guidance and feedback to the student.

[0037] “Virtual Reality” or “VR” refers to a fully immersive digital environment that replaces the user’s real-world view, in contrast to mixed reality which blends the two.

[0038] “View Screen” refers to a screen within the headset aligned with the user’s eyes, displaying the integrated virtual and real-world content.

[0039] The above being noted, a mixed reality flight simulator as disclosed herein provides safe and effective flight training using a turn-key solution by allowing students to become familiar with an aircraft cockpit environment (which may vary depending on the aircraft). The mixed reality flight simulator provides guided coaching and instruction from instructors along with active, real time feedback. In some embodiments, the mixed reality flight simulator also may provide for self-paced learning for the students. Additionally, in some embodiments, the mixed reality flight simulator may include the ability to toggle between the mixed reality world and full virtual reality. In this respect, a student can pick any location and aircraft available on the simulator to be flown and manipulated using the controls available.

[0040] In accordance with various aspects of the present disclosure, initialconditions may be preset environments that are be loaded to make setup of the mixed reality flight simulator easier. By pre-configuring the time of day, position, weather, etc., the simulation can quickly be reset to these initial settings for training or familiarization purposes.

[0041] Moreover, from an instructor perspective, a mixed reality flight simulator as disclosed herein provides for increased student capacity and throughput, automation and standardization of various training aspects, data collection and processing, all of which provide for a better prepared new pilot.

[0042] Additionally, in some embodiments, the mixed reality flight simulator may allow an instructor to present additional monitoring and instruction to the student. This may include being able to point to peripherals which are visible in the mixed reality configuration. As the instructor monitors, they can converse and provide insight and instructions to the student.

[0043] In some embodiments, the mixed reality flight simulator may provide an additional supplemental instructor station that allows the instructor to manipulate the virtual simulation while the student is flying. For example, this supplemental instructor station allows the instructor to manipulate the simulation for configuration, adjustments, and interaction. The controls allow for the loading of Initial conditions or free manipulation of the environment including position, time of day, weather, etc.

[0044] In accordance with the present disclosure, various off the shelf and / or custom hardware may be used. In this regard, hardware is generally selected and built to best represent what is being simulated such as buttons, switches, knobs, and other interfaces. Motion may be added to help stabilize the utilization of virtual and mixed reality and to increase the immersion and experience.

[0045] For example, a flight simulator comprises a headset having a viewscreen aligned with a student’s eyes and an outwardly facing passthrough camera and a view screen that are synchronized so that a flight simulator screen can be viewed on the view screen without visual interference, a flight simulator console comprising tangible simulated aircraft controls (described below), and at least one computer having flight training simulation software, the computer integrating input from the passthrough camera with a simulated image of the tangible aircraft flight controls so the student can view the student’s own hands on the view screen while interacting with the flight simulator console and aircraft flight controls and one more screens, as well as the flight training simulation software.

[0046] In accordance with the present disclosure, an interactive audio component using either augmented or real-time communications with the user to help teach proper communication may be provided. For example, in aviation, communication is important from clearances and position reports to instructions from air traffic control. Additionally, the audio component may be adapted to other subjects of training. For example, having speech to text modules implemented allow one to listen and verify the communication including allowing for corrective prompts when simulating the communications. Processing may be provided to ensure no matter how something may be stated, the point and key aspects are communicated, including the ability to confirm the student has the current weather or necessary information.

[0047] In accordance with various aspects of the present disclosure, the ability for the student to view their own hands on the view screen while interacting with the flight simulator via “mixed reality” is different from the more common “virtual reality” as students typically have to view the virtual world only in a headset and relying on touch as their only connection to the physical world. In contrast, when using the presently described mixed reality flight simulator, the student can be fully immersed in theenvironment being simulated, while simultaneously being grounded in the physical world by seeing and interacting with real-world objects around them.

[0048] The above being noted, in accordance with various aspects of the present disclosure and with reference to FIG. 1 , a mixed reality flight simulator 100 may comprise a motion platform 110, a seat 120, a computer 130 (which may be placed in any feasible location, such as for example, under the seat 120 and thus not visible in FIG. 1 ), physical flight controls 140, one or more control screens 145 and a mixed reality headset 150 with passthrough portals programmed to allow the user (or student) 160 to see and differentiate between the physical “real world” and virtual world, all within one view in the headset 150. In accordance with various aspects, components may be custom or “off the shelf” from manufactures such as, among others, D-Box Motion Actuators and Virtual Fly flight controls, as well as various off the shelf mixed reality headsets.

[0049] In various embodiments and with reference to FIG. 2, the headset 150 has a view screen 170 therein aligned with the student’s 160 eyes and an outwardly facing passthrough camera. As noted above, by selecting a particular refresh rate of the passthrough camera the passthrough camera and the view screen 170 such that they are synchronized so that a flight simulator screen 105 can be viewed on the view screen 170 without visual interference. For example, the passthrough camera and the flight simulator screen 105 may have a refresh rate of at least 60 Hz to eliminate visual interference.

[0050] In accordance with various aspects, the computer 130 integrates input from the passthrough camera with a simulated image of the tangible aircraft flight controls 140, whereby the student 160 can view the student’s own hands 175 on the flight simulator screen 105 while interacting with the flight simulator console andaircraft flight controls, the tangible aircraft flight controls 140, the control screen(s) 145, as well as the flight training simulation software.

[0051] Stated another way, the mixed reality flight simulator 100 provides real world “portals” in the virtual world and overlays passthrough from cameras, allowing a student 160 to see their hands 175, lap, feet, physical controls, and essentially anything above the “dashboard” of the plane (outside of the real world portals) remains virtual so that the simulations can be performed in any environment. Meanwhile, the real world below the dashboard remains the same and is an actual passthrough representation using cameras instead of the virtual world.

[0052] In accordance with various aspects of the present disclosure, the passthrough camera may be color or black and white. The passthrough camera may have a resolution of at least 2064 x 2208 and may have a resolution of at least 2064 x 2208 pixels and / or at least 4 total megapixels and 18 pixels per degree though these may vary based on the application.

[0053] Additionally, in accordance with various aspects, there may be additional screens such as smartphones and tablets (not shown) that may be handheld, rest on a student’s knee or other support devices that provide additional information such as checklists, maps, and the like. In various embodiments, the various screens may be touch sensitive screens with or without haptic feedback.

[0054] In accordance with various aspects of the present disclosure, the flight controls 140 are tangible simulated aircraft controls that the student 160 can touch and feel just like in a real plane and in various embodiments, the flight controls 140 may provide tactile and / or haptic feedback correlated with the simulation. For example, tactile feedback may include vibrations, centrifugal flight related forces and resistance and the like. Exemplary aircraft flight controls 140 include, but are notlimited to, a yoke or other control stick, pedal(s), throttle and mixture controls, flaps controls, as well as various buttons and switches, and the like).

[0055] In accordance with various aspects, the computer 130 may have integrated flight training simulation software. Using the flight training simulation software, the flight simulator 100 can provide instructions to the student 160 in mixed reality in a simulated environment, which in turn may be provided on the view screen 170. In various embodiments, a Learning Record Store (LRS) such as GrassBlade and a Learning Management System (LMS) such as LearnDash may provide for interaction between the student 160, instructors and / or the flight simulator 100. The LRS allows the simulator to send back student performance data, to be stored for use in analytics as well as to be sent to the LMS. The LMS provides a dashboard for the students to view their performance in a user-friendly interface. Students can view the total number of goals they had per training event and how many they successfully completed. Instructors may also use this LMS portal to track progress and learning objectives for their students.

[0056] In this respect, by matching a specific hardware solution such as described above with a fully pre-programmed learning course, LRS and / or LMS, students can learn how to fly, receive active feedback from the virtual instructor, record grades and performance, progress through lessons and curriculum including outstanding tasks in need of completion, without corresponding “task saturation” where a student is overwhelmed with too many tasks to consider at a particular moment or period of time.

[0057] With reference now to FIG. 3, in accordance with various aspects of the present disclosure, the mixed reality flight simulator 100 may use a course simulation from “off the shelf” current offerings, with modifications of the lesson to teach with thestudent’s ability to their hands through Mixed Reality (MR). This allows the use of proven courses showing the training by moving the placards and changing dialogue lines to appear above the camera portal in the virtual world.

[0058] With reference now to FIG. 4, in accordance with various aspects of the present disclosure, the mixed reality flight simulator 100 may use a course simulation from “off the shelf” current offerings, with modifications of the lesson to teach with the student’s ability to their hands through Mixed Reality (MR) and to Implement Focus Points. This allows the use of proven courses showing the training by moving the placards and changing dialogue lines to appear above the camera portal in the virtual world and from there, allowing modification of the portal, the overlay of objects, and / or the manipulation of focus points to enhance instruction. In accordance with various aspects of the present disclosure, the foregoing may be implemented using, for example, X-Plane and / or Prepar3d software and specific headsets such as the Oculus Quest 3 and HTC VIVE Focus Vision.

[0059] With reference now to FIGS. 5A and 5B, in accordance with various aspects of the present disclosure, the mixed reality flight simulator 100 may use a custom application, for example using Unity software that integrates the software development kit for a MR Headset of choice.Missions

[0060] In accordance with various aspects of the present disclosure, various missions may be provided. For example, Intro Missions may be provided that require minimum interaction and help introduce concepts, procedures, and understanding to the student, allowing the student to follow along and experience what they will be learning and doing.

[0061] In accordance with various aspects of the present disclosure, trainingmission requiring more interaction and action on the students part with additional ways of teaching the student may be provided. For example, a primary way of teaching is by providing audio and visual instructions and guidance. Audio may be provided by the simulation and the scripted voice lines triggered by the mission programming, providing instruction, guidance, and correction. In accordance with various aspects of the present disclosure, various methods may be used:1 . Visually the student will be able to see the virtual and the physical worlds. In the virtual world placards providing information including instructions, correction, area of interest, etc. are shown to the student.2. Similar to the above but including placards in both the virtual and overlaying the portal to the physical world, providing the ability to place information where it can be conveniently be seen and highlight focus points more directly.3. In addition to the above, a third capacity allows the control of where and how much is visible in the MR portal by manipulating the portal size, allowing the creation of more targeted areas of training and help by removing areas that may be points of fixation.

[0062] These training mission can be used without a physical instruction relying only on the programed instructor to tech the concepts. The difficulty and level of interaction vary depending on the concept and goal.

[0063] In accordance with various aspects of the present disclosure, modular training missions, Al and scripted / pre-programed training provide feedback on areas of deficiency providing guidance on lessons or module that should be repeated to help the student develop skills. Modular training missions may provide the ability of generating mission for the students to fly personalized to target the areas of deficiency and to create a tailored check mission to ensure that areas of deficiency have been brought up to an acceptable skill level. This capacity may also provide more accurate correction and coaching to improve student skill, leading to a point where in the mission more adaptive measures can take place such as auto-repositioning to re- attempt a maneuver.

[0064] In accordance with various aspects of the present disclosure, a training program may be structured to give students information to read, watch and to see what they are reading in action, and for interaction through the missions. For example, a training program may start with a Content and Intro Mission and then move on to the Training and Modular Training with increasing difficulties, slowly increasing the student workload as they become more familiar and proficient in operation and maneuvers.

[0065] Thus, a mixed reality flight simulator provides numerous advantages over existing flight simulators by providing an experience of full immersion while being able to see real-world physical objects to enhance realism. For example, a full pilot training courseware solution that covers all aspects of the pilot training requirements is possible. Additionally, “virtual instructors” allow students to practice and receive feedback on their performance without being tied to a human instructor’s actual presence. Because the mixed reality flight simulator is an immersive training device (ITD), it provides the student with all the hardware needed to immerse the student in a virtual environment specifically designed to familiarize the student with all aspects of the flight experience. A virtual instructor in accordance with the present disclosure allows the harvest of a significant amount of data on student performance and learning behaviors, and such data may be used with artificial intelligence to optimize the training program, identify ideal flight candidates, and provide the training institution with a library of system outputs that can be analyzed for their own research purposes.

[0066] Finally, it will be apparent to those skilled in the art that various modifications and variations can be made in the present disclosure without departing from the spirit or scope of the disclosure. Thus, it is intended that the present disclosure cover the modifications and variations of this disclosure provided they come within the scope of the appended claims and their equivalents.

[0067] Likewise, numerous characteristics and advantages have been set forth in the preceding description, including various alternatives together with details of the structure and function of the devices and / or methods. The disclosure is intended as illustrative only and as such is not intended to be exhaustive. It will be evident to those skilled in the art that various modifications may be made, especially in matters of structure, deposition materials, elements, components, shape, size, and arrangement of parts including combinations within the principles of the invention, to the full extent indicated by the broad, general meaning of the terms in which the appended claims are expressed. To the extent that these various modifications do not depart from the spirit and scope of the appended claims, they are intended to be encompassed therein.

Claims

CLAIMSWe claim:

1. A flight simulator comprising: a headset having a view screen aligned with a student’s eyes and an outwardly facing passthrough camera having a the passthrough camera refresh rate of at least 60 Hz; a flight simulator console comprising tangible simulated aircraft controls; and a computer having flight training simulation software, the computer integrating input from the passthrough camera with a simulated image of the tangible aircraft flight controls, whereby the student can view the student’s own hands through a portal on the view screen while interacting with the flight simulator console and the flight training simulation software in a mixed reality environment.

2. The flight simulator of claim 1 , wherein the passthrough camera refresh rate of the passthrough camera is synchronized with a control screen refresh rate of a control screen so that the control screen can be viewed on the view screen without visual interference.

3. The flight simulator of claim 2, wherein the passthrough camera has a resolution of at least 2064 x 2208.

4. The flight simulator of claim 2, wherein the passthrough camera has a resolution of at least 4 megapixels and 18 pixels per degree.

5. The flight simulator of claim 1 , wherein the passthrough camera is color.

6. The flight simulator of claim 1 , wherein the computer provides mixed reality instructions to the student.

7. The flight simulator of claim 6, wherein the mixed reality instructions are provided on the view screen.

8. The flight simulator of claim 1 , wherein the simulator console is configured to provide tactile feedback to the student correlated with the simulation.

9. The flight simulator of claim 1 , further comprising a Learning Record Store and / or a Learning Management System for facilitating interaction between the student, and instructor and / or the flight simulator.

10. A flight simulator comprising: a headset having a view screen aligned with a student’s eyes and an outwardly facing passthrough camera; a flight simulator console, including tangible simulated aircraft controls; at least one computer having flight training simulation software, the computer integrating input from the passthrough camera with a simulated image of aircraft flight controls, whereby the student can view the student’s own hands through a portal on the view screen while interacting with the flight simulator console; and wherein the simulator console is configured to provide tactile feedback to the student correlated with the simulation.

11. The flight simulator of claim 10, whereby a refresh rate of the passthrough camera has a passthrough camera refresh rate, and the passthrough camera refresh rate is synchronized with a control screen refresh rate of a control screen so that the control screen can be viewed on the view screen without visual interference.

12. The flight simulator of claim 11 , wherein the passthrough camera refresh rate and the control screen refresh rate are at least 60 Hz.

13. The flight simulator of claim 10, further comprising a Learning Record Store and / or a Learning Management System for facilitating interaction between the student, and instructor and / or the flight simulator.

Images