Flight skill enhancement training support platform fusing virtual imaging and motion capture

Abstract

The invention belongs to the technical field of flight training platforms, and provides a flight skill enhancement training supporting platform fusing virtual imaging and motion capture, the flight skill enhancement training supporting platform comprises a real object cabin and a virtual device, the real object cabin provides a cabin real object part for airplane training and comprises a cabin body, a flight seat and a control component, the virtual device is arranged in the cabin body, and the flight seat is arranged in the cabin body. And providing a virtual cabin based on a virtual scene for the flight trainee, wherein the virtual cabin comprises a simulation flight picture and a cabin scene of a non-cabin real object part. The system is integrally designed on the basis of a virtual-real combination technology, an XR video straight-through acquisition motion capture technology and a commercial shelf product building platform mode, the technology is safe and reliable, implementation is convenient and feasible, and use and maintenance guarantee are simple and convenient. By constructing a platform comprising an XR head-mounted display and a computer component, flight skill enhanced training fusing virtual imaging and motion capture is realized. Verification shows that the design is reliable and effective, and has great significance for realizing flight skill enhancement training with low cost and high interactivity combined with virtuality and reality.

Description

Technical Field

[0001] The present invention belongs to the technical field of flight training platforms, and relates to a flight skill enhancement training support platform, and in particular to a flight skill enhancement training support platform integrating virtual imaging and motion capture. Background Art

[0002] Based on a typical primary teaching aircraft, an operation support system based on an immersive virtual scene is customized and developed to form a hardware environment for an embedded enhanced training system for flight skills during the sensitive window period. In view of the problems of low familiarity with cockpits and difficulty in full virtual interaction among young people during the sensitive window period, as many commonly used switches, buttons and other control devices as possible are configured, and internationally advanced XR (extended reality) technology is used to achieve a low-cost, high-immersion combination of virtual and real, meeting the needs of young people in the sensitive window period who are willing to fly, dare to fly and can fly. In order to meet the requirements of embedded enhanced training and virtual-reality combination of flight skills for young people during the sensitive window period, it is necessary to build a flight skill enhancement training support platform with virtual-reality combination display capabilities and motion capture capabilities. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a flight skill enhancement training support platform integrating virtual imaging and motion capture, which can provide a hardware environment for an embedded enhanced training system for flight skills during the sensitive window period, and has virtual-real combined display capabilities and motion capture capabilities, achieving low-cost and high-immersion virtual-real combination, and solving the problems of difficult interaction, dizziness during long-term flight, and low clarity brought about by the full virtual environment of traditional VR technology.

[0004] The technical solution of the present invention is as follows:

[0005] A flight skill enhancement training support platform integrating virtual imaging and motion capture includes a physical cockpit and virtual equipment. The physical cockpit provides the physical part of the cockpit for aircraft training, including a cockpit body, a flight seat and control components. The virtual equipment is arranged in the cockpit body to provide a virtual cockpit based on virtual vision for flight trainees. The virtual cockpit includes simulated flight images and in-cockpit scenes of non-physical parts of the cockpit.

[0006] Furthermore, the virtual equipment includes an XR headset and a locator. The XR headset is worn on the flight trainee's head, and the locator is located at the left and right diagonals of the front and back sides of the physical cockpit.

[0007] Furthermore, the physical camera of the XR headset and the virtual camera of the XR headset imaging are set in the same position.

[0008] Furthermore, a locator is used to accurately position the XR headset. After positioning is completed, a regional position correction algorithm is used to perform software correction on the deviation between the camera position and the human eye position to minimize the impact of the deviation.

[0009] Furthermore, the regional position correction algorithm is specifically as follows:

[0010] The XR headset uses a marker position calculation method to place the same QR code pattern at the same position in the physical cockpit and the virtual cockpit, and uses QR code coincidence matching technology to match the physical cockpit and correct positioning errors.

[0011] Furthermore, the physical camera of the XR headset captures area images of the physical cockpit, as well as real-time movement images of the flight trainee’s hands and legs in the physical cockpit, to achieve motion capture.

[0012] Furthermore, the method of combining the physical cockpit and the virtual cockpit into a virtual cockpit is:

[0013] The high-definition image is acquired through the physical camera of the XR headset, cropped, and the mixed area with the real cockpit is selected. The objects in the mixed area are modeled in 1:1 three-dimensional modeling. After the modeling is completed, the model is placed at the corresponding position of the mixed area in the virtual cockpit; then a special material is given to the mixed area, and the real image projection is used to replace the virtual cockpit of the mixed area to retain the model.

[0014] Furthermore, when flight trainees use the physical cockpit equipment, the physical camera collects real-time images of the user's hands and legs operating the equipment in the physical area, and splices them with the virtual cockpit, presenting a complete virtual-reality cockpit in the glasses. The status of the equipment operated in the virtual-reality cockpit is consistent with the status of the external real hardware equipment.

[0015] Furthermore, the virtual-reality cockpit receives the status of instruments and indicator lights sent by the flight simulation software, and drives the virtual-reality combined cockpit instruments and indicator lights to display and indicate correctly, simulating the display effects of the cockpit during the day and at night, and simulating the equipment functions of the aircraft shell, including elevators, rudders, ailerons, landing gear, flaps, and landing gear position indicator rods.

[0016] The beneficial effects of the present invention are as follows:

[0017] The present invention is based on the overall system design based on the combination of virtual and real technology, the use of XR video direct acquisition of motion capture technology and the model of commercial shelf product construction platform. The technology is safe and reliable, the implementation is convenient and feasible, and the use and maintenance are simple and convenient. By building a platform including XR head display and computer components, flight skill enhancement training integrating virtual imaging and motion capture is realized. After verification, it is shown that the design is reliable and effective, and it is of great significance to realize low-cost and high-interactive virtual and real flight skill enhancement training. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a lateral schematic diagram of the flight skill enhancement training support platform of the present invention;

[0019] Figure 2 A three-dimensional schematic diagram of the flight skill enhancement training support platform of the present invention;

[0020] Figure 3 The XR head display intent used in the present invention;

[0021] Figure 4 A schematic diagram of a virtual-real cockpit hybrid area of ​​the present invention;

[0022] Figure 5 It is a schematic diagram of virtual device connection of the present invention. DETAILED DESCRIPTION

[0023] This section is an embodiment of the present invention, which is used to explain and illustrate the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0024] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the orientations or positional relationships in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or case referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implying the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0025] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be interpreted in a broad sense, for example, it can be a fixed connection, a detachable connection or an integrated connection; it can be a mechanical connection or a point connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0026] Embodiment 1:

[0027] A flight skill enhancement training support platform integrating virtual imaging and motion capture includes a physical cockpit and virtual equipment. The physical cockpit provides the physical part of the cockpit for aircraft training, including a cockpit body, a flight seat and control components. The virtual equipment is arranged in the cockpit body to provide a virtual cockpit based on virtual vision for flight trainees. The virtual cockpit includes simulated flight images and in-cockpit scenes of non-physical parts of the cockpit.

[0028] The virtual equipment includes an XR headset and a locator. The XR headset is worn on the flight trainee's head, and the locator is located at the left and right diagonals of the front and back sides of the physical cockpit.

[0029] The physical camera of the XR headset is located in the same position as the virtual camera used for XR headset imaging.

[0030] Use a locator to accurately position the XR headset. After positioning is completed, use the regional position correction algorithm to perform software correction on the deviation between the camera position and the human eye position to minimize the impact of the deviation.

[0031] The specific regional position correction algorithm is as follows:

[0032] The XR headset uses a marker position calculation method to place the same QR code pattern at the same position in the physical cockpit and the virtual cockpit, and uses QR code coincidence matching technology to match the physical cockpit and correct positioning errors.

[0033] The physical camera of the XR headset captures the area image of the physical cockpit, as well as the real-time movement images of the flight trainee's hands and legs in the physical cockpit to achieve motion capture.

[0034] The method of combining the physical cockpit and the virtual cockpit into a virtual cockpit is:

[0035] The high-definition image is acquired through the physical camera of the XR headset, cropped, and the mixed area with the real cockpit is selected. The objects in the mixed area are modeled in 1:1 three-dimensional modeling. After the modeling is completed, the model is placed at the corresponding position of the mixed area in the virtual cockpit; then a special material is given to the mixed area, and the real image projection is used to replace the virtual cockpit of the mixed area to retain the model.

[0036] When flight trainees use the physical cockpit equipment, the physical camera captures real-time images of the user's hands and legs operating the equipment in the physical area, and splices them with the virtual cockpit, presenting a complete virtual-reality cockpit in the glasses. The status of the equipment operated in the virtual-reality cockpit is consistent with the status of the external real hardware equipment.

[0037] The virtual-reality cockpit receives the status of instruments and indicator lights sent by the flight simulation software, and drives the virtual-reality combined cockpit instruments and indicator lights to display and indicate correctly, simulating the display effects of the cockpit during the day and at night, and simulating the equipment functions of the aircraft shell, including elevators, rudders, ailerons, landing gear, flaps, and landing gear position indicator rods.

[0038] Embodiment 2:

[0039] The present invention relates to a flight skill enhancement training support platform integrating virtual imaging and motion capture, which mainly includes a virtual imaging and motion capture system and a computer component: wherein the virtual imaging and motion capture system includes an XR head display, supporting cables, a locator and a supporting bracket; the computer component includes a high-performance computer, a display and a keyboard and mouse that support the operation of the system.

[0040] The present invention realizes flight skill simulation integrating virtual imaging and motion capture by constructing an enhanced training support platform including an XR head display, supporting cables, a locator, a supporting bracket, and computer components.

[0041] ① Combination of virtual and real

[0042] The physical area of ​​the cockpit obtains high-definition images through the external front camera of the XR headset and splices them with the virtual cockpit, presenting a complete cockpit picture on the glasses.

[0043] ②Motion capture

[0044] Rendering Offset

[0045] The XR headset is located approximately 10 cm in front of the user's eyes, which causes depth perception to shift, so real-world objects in the video pass-through image appear larger than in real life. To address the offset, it is important that the relative geometry (position, size, and parallax) matches each other when mixing the video image with the virtual content. The virtual camera can be placed in the same position as the physical camera, and when the virtual perspective and the real perspective view objects simultaneously, the virtual and real things appear at the same distance and on the same plane. Although this may cause an apparent jump in the position of virtual objects compared to purely virtual content, it will not cause any distortion in the object geometry or position.

[0046] The physical camera refers to the front-facing high-definition wide-angle camera (12 million pixels) on the outside of the XR headset; the virtual camera refers to the pilot's virtual perspective in the interactive training software.

[0047] Region Position Correction

[0048] Use a locator to accurately position the XR headset. After positioning is completed, use the regional position correction algorithm to perform software correction on the deviation between the camera position and the human eye position to minimize the impact of the deviation.

[0049] When blending video images with virtual content, it is important that their relative geometry (position, size, and parallax) matches each other. To do this, the VR camera position needs to be the same as the physical camera position. This will make virtual and real-world objects appear at the same distance and on the same plane when the virtual perspective is viewing objects at the same time as the real view.

[0050] If the cutting position is found to be offset, the position can be corrected. The XR headset uses the Marker (QR code) position calculation method to place the same QR code pattern at the same position of the physical cockpit and the virtual cockpit. The QR code overlap matching technology is used to make the virtual and real cockpits perfectly match and correct the positioning error.

[0051] Motion Capture

[0052] The camera captures images of the physical area of ​​the cockpit, as well as the real-time movements of the user's hands and legs in that area. Therefore, one can observe one's own movements and the physical area scenes in the cabin in real time to achieve motion capture.

[0053] XR headsets have depth awareness, naturally blending real and virtual elements together, providing the most immersive mixed reality experience ever, with realistic visual fidelity within the maximum field of view of any XR headset. The combination of real cockpits and virtual modeled cockpits and external visual scenes is achieved through video cropping.

[0054] This system uses Varjo's video pass-through mode to achieve the combination of virtual and real. The principle is as follows:

[0055] 1) Get external images

[0056] The two high-definition images on the left and right are obtained through the external front-facing high-definition wide-angle camera (12 million pixels) of the XR headset.

[0057] 2) Customized cropping of image blending areas

[0058] The red-framed area in the above figure is the area that needs to be blended with the physical cockpit. The objects in this area need to be 3D modeled in a 1:1 ratio based on the area corresponding to the actual physical cockpit. After the modeling is completed, the model is placed in the corresponding position of the red-framed area in the virtual scene.

[0059] 3) Mixing of virtual and real images

[0060] The customized cut area is given a special material, and a matching test is performed based on the color and transparency in the area. When the color and transparency in the area meet the requirements, the real image projection will be used instead of the virtual image. The virtual and real image mixing uses image recognition and XR depth recognition technology.

[0061] 4) Cockpit equipment interaction module

[0062] When the user manipulates the cockpit equipment simulated by hardware, the camera can collect real-time images of the user's hands and legs manipulating the equipment in the physical area, and splice them with the virtual cockpit to present a complete virtual-reality combined cockpit in the glasses. The status of the equipment in the virtual-reality combined cockpit is consistent with the status of the external real hardware equipment;

[0063] The user can see the hardware devices in the hand and leg control cabin in real time through a combination of virtual and real vision, including the position, posture, movement of the palm and each finger, and the status of the operating hardware devices, such as pressing a button and turning the power switch up or down.

[0064] 5) Virtual and real cockpit equipment status display

[0065] Receive the status of instruments and indicator lights sent by the flight simulation software, and drive the virtual and real cockpit instruments and indicator lights to display and indicate correctly;

[0066] Simulate daytime and nighttime cockpit display effects;

[0067] The functions of the equipment simulating the aircraft shell mainly include elevator, rudder, aileron, landing gear, flaps, and landing gear position indicator rod.

[0068] 6) Use XR video pass-through to obtain precise positioning technology for motion capture

[0069] Rendering Offset

[0070] The XR headset is positioned about 10 centimeters (3.9 inches) in front of the user's eyes, which causes depth perception to shift so that real-world objects in the video pass-through image appear larger than they are in real life.

[0071] This magnification effect is noticeable for objects that are close to the user. For example, their hands may appear unnaturally large in the image. The effect decreases with distance, so objects at 2 meters away already appear close to their actual size, and the sizes eventually converge to infinity. Note that while the objects may be different sizes, their geometry, relative sizes, positions, etc. remain accurate. The extent of the magnification effect ultimately depends on the application itself and the physiology of the user, as the human visual system is highly adaptable in this type of setting.

[0072] When blending video images with virtual content, it is important that their relative geometry (position, size, and parallax) matches one another. To achieve this, the VR camera position needs to be placed in the same position as the physical camera. This will make virtual and real-world objects appear at the same distance and on the same plane when the virtual perspective is viewing objects simultaneously with the real view. While this may cause an apparent jump in the position of virtual objects compared to purely virtual content, this does not result in any distortion of the object geometry or position; only the viewer's position has changed.

[0073] Use locator to accurately locate and correct regional position

[0074] This project uses a locator to accurately position the XR headset. After positioning is completed, a regional position correction algorithm is used to perform software correction on the deviation between the camera position and the human eye position to minimize the impact of the deviation.

[0075] Motion capture via video pass-through

[0076] The camera can capture images of the physical area of ​​the cockpit, as well as real-time images of the user's hands and legs in that area. Users can see their own movements and the physical area scenes in the cabin in real time, thus realizing motion capture capabilities.

[0077] Embodiment 3:

[0078] The virtual imaging and motion capture system mainly includes XR headsets, supporting cables, locators and supporting brackets.

[0079] XR headsets provide a highly immersive mixed reality experience with realistic visual fidelity within the maximum field of view. They can naturally blend real and virtual elements together to build an omnidirectionally visible three-dimensional space. XR headsets can capture the images in the cabin in real time as the head moves. When the user's head turns in a certain direction, the corresponding direction of the view is presented in the imaging system. By collecting the left and right visual images of the trainee's gestures, the hand activity information is fed back to the processor in real time and displayed in the XR headset view.

[0080] Set up two locators and fix them on the hardware frame of the training system. The positioning method is as follows:

[0081] Using a locator

[0082] The present invention uses HTC VIVE control locator (base station) 2.0 to accurately locate the XR head display. The locator has SteamVR compatible with the XR head display. TM The 2.0 tracking system uses a position correction algorithm after positioning is completed to perform software correction on the deviation between the camera position and the human eye position, minimize the impact of the deviation, and improve the accuracy of the XR headset's positioning system.

[0083] Modify and fix the positioner bracket

[0084] If the locator bracket is placed directly on the ground, it is easy to be bumped by passers-by and displaced. After displacement, the XR headset needs to rebuild the positioning calibration. The present invention fixes the locator bracket at the left rear and right front positions of the cockpit after modification to avoid the bracket displacement caused by people bumping into it, thereby reducing the number of XR headset positioning calibration times.

[0085] Seat depicts the eyeglass calibration reference position outline

[0086] The outline of the glasses calibration reference position is engraved on the seat to ensure the uniqueness of the calibration reference position. Before calibrating the glasses, place the XR headset at the corresponding position of the seat and perform the calibration operation to improve the accuracy of the XR headset calibration.

[0087] The physical cockpit and the virtual cockpit are the same size

[0088] The virtual cockpit model is three-dimensionally modeled according to the corresponding area size of the real cockpit in a 1:1 ratio, ensuring the matching of the virtual and real cockpit, making the virtual and real cockpit more realistic and improving the immersion of flight training.

[0089] QR code positioning correction technology

[0090] If an offset is found between the virtual and real cockpit positions, position correction can be performed. The XR headset uses a marker (QR code) position calculation method to place the same QR code pattern at the same position of the real and virtual cockpits. The QR code overlap matching technology is used to perfectly match the virtual and real cockpits and correct positioning errors.

[0091] The computer components include a high-performance computer, display, keyboard and mouse that support the operation of the system. The computer can drive a high-performance XR headset, which is a high-configuration computer that supports the operation of the system software. The display can realistically display various devices, instruments, buttons, etc. on the instrument, and the keyboard and mouse are commonly used office keyboard and mouse sets on the market.

[0092] In summary, by constructing the above-mentioned flight skill enhancement training support platform including XR headsets and computer components, the usage requirements can be met.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A flight skill enhancement training support platform integrating virtual imaging and motion capture, characterized in that: It includes a physical cockpit and virtual equipment. The physical cockpit provides the physical part of the cockpit for aircraft training, including the cockpit body, flight seats and control components. The virtual equipment is arranged in the cockpit body to provide flight trainees with a virtual cockpit based on virtual vision. The virtual cockpit includes simulated flight images and cockpit scenes that are not the physical parts of the cockpit.

2. The flight skill enhancement training support platform integrating virtual imaging and motion capture according to claim 1, characterized in that: The virtual equipment includes an XR headset and a locator. The XR headset is worn on the flight trainee's head, and the locator is located at the left and right diagonals of the front and back sides of the physical cockpit.

3. The flight skill enhancement training support platform integrating virtual imaging and motion capture according to claim 1, characterized in that: The physical camera of the XR headset is located in the same position as the virtual camera used for XR headset imaging.

4. The flight skill enhancement training support platform integrating virtual imaging and motion capture according to claim 3 is characterized in that: Use a locator to accurately position the XR headset. After positioning is completed, use the regional position correction algorithm to perform software correction on the deviation between the camera position and the human eye position to minimize the impact of the deviation.

5. The flight skill enhancement training support platform integrating virtual imaging and motion capture according to claim 4 is characterized in that: The specific regional position correction algorithm is as follows: The XR headset uses a marker position calculation method to place the same QR code pattern at the same position in the physical cockpit and the virtual cockpit, and uses QR code coincidence matching technology to match the physical cockpit and correct positioning errors.

6. The flight skill enhancement training support platform integrating virtual imaging and motion capture according to claim 3 is characterized in that: The physical camera of the XR headset captures the area image of the physical cockpit, as well as the real-time movement images of the flight trainee's hands and legs in the physical cockpit to achieve motion capture.

7. The flight skill enhancement training support platform integrating virtual imaging and motion capture according to claim 6, characterized in that: The method of combining the physical cockpit and the virtual cockpit into a virtual cockpit is: The high-definition image is acquired through the physical camera of the XR headset, cropped, and the mixed area with the real cockpit is selected. The objects in the mixed area are modeled in 1:1 three-dimensional modeling. After the modeling is completed, the model is placed at the corresponding position of the mixed area in the virtual cockpit; then a special material is given to the mixed area, and the real image projection is used to replace the virtual cockpit of the mixed area to retain the model.

8. The flight skill enhancement training support platform integrating virtual imaging and motion capture according to claim 7, characterized in that: When flight trainees operate the physical cockpit equipment, the physical camera captures real-time images of the user's hands and legs operating the equipment in the physical area, and splices them with the virtual cockpit, presenting a complete virtual and real cockpit in the glasses. The status of the operated equipment in the virtual cockpit is consistent with the status of the external real hardware equipment.

9. The flight skill enhancement training support platform integrating virtual imaging and motion capture according to claim 7, characterized in that: The virtual-reality cockpit receives the status of instruments and indicator lights sent by the flight simulation software, and drives the virtual-reality combined cockpit instruments and indicator lights to display and indicate correctly, simulating the display effects of the cockpit during the day and at night, and simulating the equipment functions of the aircraft shell, including elevators, rudders, ailerons, landing gear, flaps, and landing gear position indicator rods.

Images