A flight simulator based on mixed reality

By using a mixed reality helmet and front camera to display the real scene in the flight simulator, the problem of low clarity of the projection ring curtain and the inability to see the physical environment of VR devices is solved, and the virtual environment and physical environment seamless in omnidirectional field of view is achieved Connection improves the visual effect and manipulation accuracy of training.

CN115273601BActive Publication Date: 2024-11-01BEIHANG UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202211014562.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-23
Publication Date
2024-11-01
Estimated Expiration
2042-08-23

AI Technical Summary

Technical Problem

In existing flight simulators, the projection ring curtain cannot provide omnidirectional field of view, resulting in low clarity and lack of depth of field, and the VR equipment cannot see the physical environment during training, affecting the driver's maneuverability accuracy.

Method used

Using a flight simulator based on mixed reality, the real scene is displayed in a VR virtual environment through a mixed reality helmet and front camera, so as to simultaneously see the inside and operation of the physical cockpit in a virtual environment with omnidirectional field of view.

Benefits of technology

It can clearly see the interior and operation of the physical cockpit in the VR virtual environment, improve the visual effect and operation accuracy of training, and reduce maintenance costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115273601B_ABST
    Figure CN115273601B_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of flight simulators, and discloses a flight simulator based on mixed reality. The invention comprises a flight simulator body, a mixed reality helmet, a cockpit seat, a base and a cockpit shell, wherein a slide rail is provided in the middle of the upper surface of the base, the cockpit seat is slidably connected to the inner surface of the slide rail through a pulley provided on the lower surface, a cockpit instrument visor is fixedly connected to the upper surface of the cockpit shell, a switch control panel and a joystick are provided on the upper surfaces of both sides of the cockpit instrument visor close to the slide rail, an instrument panel is installed on the upper surface of one end of the cockpit instrument visor close to the slide rail, and a camera is provided on the front end of the outer surface of the mixed reality helmet; by displaying a real scene picture at a required position in a VR virtual environment, so as to achieve a virtual environment with an omnidirectional field of view, the interior of a physical cockpit and the operation of a driver can be seen at the same time, thereby solving the problem that a projection ring screen cannot meet the omnidirectional field of view in a flight simulator.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of flight simulators, and in particular relates to a flight simulator based on mixed reality. Background Art

[0002] With the development of the aviation industry and the advancement of aviation technology, the performance of aircraft is becoming more and more superior, and the training of pilots is becoming more and more important. How to reduce the cost of training pilots while achieving better results has become a problem that countries around the world are urgently trying to solve.

[0003] The birth of flight simulators has greatly reduced the cost of pilot training. Through flight simulator training, special dangerous situations that are difficult to encounter in reality can be simulated, reducing the risk of personnel training and reducing operating costs.

[0004] At present, the visual display of mainstream flight simulators is mainly projected onto a ring screen or a ball screen through a projector. Although it has been used for many years and the technology is relatively mature, its shortcomings are also obvious. First of all, the clarity is low. Compared with the display, the clarity of the projection combined with the screen is much lower, which is very different from the visual contrast of the actual flight. Secondly, when the projection and the screen are used as visual displays, the distance of the target is mainly achieved by zooming in and out of the image, lacking a visual sense of depth, which is particularly obvious when flying in formation. Finally, the cost is high. To achieve a satisfactory field of view, at least three projections are required. If a fighter may need a ball screen, more projection equipment is required, and the cost of a projection equipment ranges from tens of thousands to hundreds of thousands. In addition, the bulb of the projection equipment is a consumable item with a short life and high maintenance cost. Even if a ball screen is used with multiple projections, its field of view may be difficult to meet the needs of fighter training.

[0005] For this reason, some air forces of some countries and some units of the Chinese Air Force have gradually begun to use VR devices as display devices for flight simulators; the biggest advantage of using VR head-mounted display devices as visual displays is the 360° field of view without blind spots, and its clarity is equivalent to that of a display, which is incomparable to projection. At the same time, VR has its own 3D effect, that is, the distance of objects is not magnified or reduced, but has a depth of field effect. In addition, the VR display system occupies a small space, has few devices, and is relatively simple to maintain; however, the biggest disadvantage of using VR devices as display systems is that the real environment cannot be seen. The biggest problem is that the pilot cannot see his own hands after wearing the VR display device, resulting in the inability to accurately operate the switch buttons on the control panel in the cockpit during training, which will encounter great difficulties during training operations. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the projection ring screen cannot meet the omnidirectional field of view in the flight simulator, and the present application provides a flight simulator based on mixed reality.

[0007] The technical solution adopted by the present invention is as follows:

[0008] A flight simulator based on mixed reality comprises a flight simulator body and a mixed reality helmet, wherein the flight simulator body comprises a cockpit seat and a base, wherein the upper surface of the base is fixedly connected to a cockpit shell, a slide rail is provided in the middle of the upper surface of the base, the cockpit seat is slidably connected to the inner surface of the slide rail via a pulley provided on the lower surface, the upper surface of the cockpit shell is fixedly connected to a cockpit instrument visor, the upper surfaces of both sides of the cockpit instrument visor close to the slide rail are provided with switch control panels and joysticks, the upper surface of one end of the cockpit instrument visor close to the slide rail is installed with an instrument panel, and a camera is provided at the front end of the outer surface of the mixed reality helmet.

[0009] In a preferred embodiment of the invention, the flight simulation software inside the flight simulator body uses Lockheed Martin's Prepar3d as a platform.

[0010] In a preferred embodiment of the invention, the simulated instrument program inside the instrument panel uses GLstudio software to construct the instrument display content of the simulator, and the input end of the instrument panel is connected to the prepar3d through a UDP interface.

[0011] In a preferred embodiment of the invention, the overall structure of the flight simulator body is based on the mainstream cockpit layout of the most advanced fifth-generation fighter jets.

[0012] In a preferred embodiment of the invention, the creation steps are as follows:

[0013] S1. Connect all hardware devices and turn on the front camera of the mixed reality helmet through the setting option of steamVR.

[0014] S2. Open prepar3d and enter the main program.

[0015] S3. Open the virtual reality tab on the toolbar, select enable virtual reality, select the steamVR option. At this time, the main program enters VR mode. Open the virtual reality tab on the toolbar, select the protalsets option, and select the Eye view option. This option is to open the area that needs perspective.

[0016] S4. Open the virtual reality tab in the toolbar, select the passthrough cameras option, and select openVR camera. At this time, the camera on the mixed reality helmet is turned on, and the image captured by the front camera is displayed in the area opened in S4.

[0017] S5. Open the VR settings option in the virtual reality tab of the toolbar. Select the EyeView option, open Portal1, set the relevant parameters accordingly, and adjust the position, size, angle, etc. of the real scene display area. At the same time, you can create multiple real scene display area windows to suit the actual display area.

[0018] S6. After adjusting all parameters, you need to select an adjustment plan for the virtual cockpit. The plan is as follows:

[0019] 1. In prepar3d, you can choose not to display the virtual cockpit by right-clicking the mouse and selecting the cockpit option to choose not to display the virtual cockpit.

[0020] 2. Import the digital model of the physical cockpit into the Prepar3d program as a virtual cockpit display. The method is: first install the plug-in in the Prepar3d SDK in 3dmax, use 3dmax to import the digital model built by CATIA, export the model as an *.x file, and then use the xtomdl tool in the SDK to convert the *.x file to an *.mdl format file, and import it into the model folder of the corresponding model aircraft in the Prepar3d program.

[0021] S7. After all adjustments are made, the virtual cockpit in the virtual environment and the physical cockpit are fully connected in the VR display mode to achieve a display effect.

[0022] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0023] In the present invention, a VR head mounted display device is connected to a flight simulation software through steamVR, and the front camera of the VR head mounted display device is turned on, and a real scene display area is added in the flight simulation software to display the image returned by the front camera of the VR head mounted display device. In this way, the real scene picture is displayed at a required position in the VR virtual environment, so that the interior of the physical cockpit and the operation of the driver can be seen in a virtual environment with an omnidirectional field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A perspective view of the present invention;

[0025] Figure 2 A top view of the present invention;

[0026] Figure 3 is a side view of the mixed reality helmet of the present invention;

[0027] Figure 4 is a side view of the present invention;

[0028] Figure 5 It is the overall framework diagram of the present invention.

[0029] Markings in the figure: 1-flight simulator body, 2-mixed reality helmet, 3-cockpit instrument visor, 4-cockpit seat, 5-slide rail, 6-switch control panel, 7-joystick, 8-instrument panel, 9-cockpit shell, 10-camera, 11-base. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The following will be combined Figure 1-Figure 5 A mixed reality-based flight simulator according to an embodiment of the present invention is described in detail.

[0032] Example:

[0033] Reference Figure 1 A flight simulator based on mixed reality includes a flight simulator body 1 and a mixed reality helmet 2. The flight simulation software inside the flight simulator body 1 uses Lockheed Martin's prepar3d as a platform. The biggest advantage of this platform is that it is highly open and can be easily developed for secondary development. The overall structure of the flight simulator body 1 is based on the mainstream cockpit layout of the most advanced fifth-generation fighter jets, and the positions of the operating equipment and control switch buttons are based on the layout of the current mainstream fifth-generation aircraft, making the simulated flight simulator body 1 more realistic.

[0034] Reference Figure 1-Figure 4The flight simulator body 1 includes a cockpit seat 4 and a base 11. The upper surface of the base 11 is fixedly connected to a cockpit shell 9. A slide rail 5 is provided in the middle of the upper surface of the base 11. The cockpit seat 4 is slidably connected to the inner surface of the slide rail 5 through a pulley provided on the lower surface. The upper surface of the cockpit shell 9 is fixedly connected to a cockpit instrument visor 3. The upper surfaces of the cockpit instrument visor 3 on both sides close to the slide rail 5 are provided with a switch control panel 6 and a joystick 7. The cockpit seat 4 can slide on the inner surface of the slide rail 5 through the pulley so that the user can maintain a comfortable posture with the switch control panel 6 and the joystick 7. An instrument panel 8 is installed on the upper surface of one end of the cockpit instrument visor 3 close to the slide rail 5, and a camera 10 is set at the front end of the outer surface of the mixed reality helmet 2. The analog instrument program inside the instrument panel 8 uses GLstudio software to build the instrument display content of the simulator, and the input end of the instrument panel 8 is connected to prepar3d through a UDP interface. Prepar3d sends flight parameters, engine parameters, navigation parameters and other data information to GLstudio through the UDP interface. After receiving the data, GLstudio assigns it to the corresponding instrument to display the corresponding content.

[0035] Reference Figure 1-Figure 5 The steps for creating a flight simulator based on mixed reality are as follows: connect all hardware devices, turn on the front camera 10 of the mixed reality helmet 2 through the setting option of steamVR, then open prepar3d, enter the main program, then open the virtual reality tab in the toolbar, select enable virtual reality, select the steamVR option, then the main program enters VR mode, open the virtual reality tab in the toolbar, select the protal sets option, select the Eye view option, this option is to open the area that needs to be viewed through, then open the virtual reality tab in the toolbar, select the passthrough cameras option, select openVR camera, then turn on the camera 10 on the mixed reality helmet 2, and display the picture taken by the front camera 10 in the area opened in S4.

[0036] Reference Figure 1-Figure 5, then open the VR settings option in the virtual reality tab of the toolbar; select the Eye View option, open Portal1, set the relevant parameters accordingly, and adjust the position, size, angle, etc. of the real-life display area. At the same time, you can create multiple real-life display area windows to display the area according to actual needs. After adjusting all the parameters, you need to choose an adjustment plan for the virtual cockpit. You can choose not to display the virtual cockpit in prepar3d. You can right-click and select the cockpit option to choose not to display the virtual cockpit; you can also choose to import the digital model of the physical cockpit into the prepar3d program as a virtual cockpit display. The method is: first install prepar3d in 3dmax Use the plug-in in the SDK to import the digital model created by CATIA with 3dmax, export the model as an *.x file, and then use the xtomdl tool in the SDK to convert the *.x file into an *.mdl format file, and import it into the model folder of the corresponding model aircraft in the prepar3d program; connect the VR head-mounted display device to the flight simulation software through steamVR, and turn on the front camera 10 of the VR head-mounted display device, add a real-scene display area in the flight simulation software to display the image returned by the front camera 10 of the VR head-mounted display device, in this way, the real-scene picture can be displayed at the required position in the VR virtual environment, so as to achieve a virtual environment with an omnidirectional field of view while seeing the inside of the physical cockpit and the driver's operation.

[0037] The implementation principle of a mixed reality-based flight simulator embodiment of the present application is as follows:

[0038] When using this simulator, connect all hardware devices, turn on the front camera 10 of the mixed reality helmet 2 through the steamVR setting option, then open prepar3d, enter the main program, then open the virtualreality tab in the toolbar, select enable virtual reality, select the steamVR option. At this time, the main program enters VR mode, open the virtual reality tab in the toolbar, select the protal sets option, select the Eye view option, this option is to open the area that needs to be seen through, then open the virtual reality tab in the toolbar, select the passthroughcameras option, select openVR camera, then turn on the camera 10 on the mixed reality helmet 2, and display the picture taken by the front camera 10 in the area opened in S4.

[0039] Then open the VR settings option in the virtual reality tab of the toolbar. Select the EyeView option, open Portal1, set the relevant parameters accordingly, and adjust the position, size, angle, etc. of the real-life display area. At the same time, you can create multiple real-life display area windows to display the area according to actual needs. After adjusting all the parameters, you need to choose an adjustment plan for the virtual cockpit. You can choose not to display the virtual cockpit in prepar3d. You can right-click the mouse to select the cockpit option and choose not to display the virtual cockpit; you can also choose to import the digital model of the physical cockpit into the prepar3d program as a virtual cockpit display. The method is: first install the plug-in in the prepar3d sdk in 3dmax, use 3dmax to import the digital model created by CATIA, export the model as a *.x file, and then use the xtomdl tool in the SDK to convert the *.x file into a *.mdl format file, and import it into the model folder of the corresponding model aircraft in the prepar3d program.

[0040] At the same time, you can also import the cockpit digital model you drew into prepar3d, and then in the VR settings option, add 6 faces to wrap the cockpit model. At this time, the default cockpit digital model is a virtual image, and the 6 faces are real images. At this time, through the inverted portals option in the passthrough mode tab, the display content is reversed, so that the display content of the added 6 faces that wrap the cockpit can be a virtual scene, and the position of the cockpit digital model is a real scene display. Through this method, the designed cockpit can be seamlessly connected with the virtual environment. After all adjustments are made, the virtual cockpit in the virtual environment in the VR display mode is fully connected with the physical cockpit. The display effect; connect the VR head-mounted display device to the flight simulation software through steamVR, and turn on the front camera 10 of the VR head-mounted display device, and add a real scene display area in the flight simulation software to display the image returned by the front camera 10 of the VR head-mounted display device. In this way, the real scene picture is displayed at the required position in the VR virtual environment, so that the inside of the physical cockpit and the driver's operation can be seen in the virtual environment that can achieve omnidirectional vision.

[0041] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0042] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A flight simulator based on mixed reality, comprising a flight simulator body (1) and a mixed reality helmet (2), characterized in that: The flight simulator body (1) comprises a cockpit seat (4) and a base (11); the upper surface of the base (11) is fixedly connected to a cockpit shell (9); a slide rail (5) is provided in the middle of the upper surface of the base (11); the cockpit seat (4) is slidably connected to the inner surface of the slide rail (5) via a pulley provided on the lower surface; the upper surface of the cockpit shell (9) is fixedly connected to a cockpit instrument visor (3); and the upper surfaces of both sides of the cockpit instrument visor (3) close to the slide rail (5) are provided with a switch control panel (6) and a joystick (7). 7), an instrument panel (8) is installed on the upper surface of one end of the cockpit instrument visor (3) close to the slide rail (5), a camera (10) is arranged on the front end of the outer surface of the mixed reality helmet (2), the flight simulation software inside the flight simulator body (1) uses Lockheed Martin's prepar3d as a platform, the simulation instrument program inside the instrument panel (8) uses GLstudio software to construct the instrument display content of the simulator, and the input end of the instrument panel (8) is connected to the prepar3d via a UDP interface; The steps for creating the flight simulator are as follows: S1. Connect all hardware devices and turn on the front camera (10) of the mixed reality helmet (2) through the setting options of steamVR; S2. Open prepar3d and enter the main program; S3. Open the Virtual Reality tab on the toolbar, select Enable Virtual Reality, and select the SteamVR option. The main program now enters VR mode. Open the Virtual Reality tab on the toolbar, select the Portal Sets option, and select the Eye View option. This option is to open the area that needs perspective. S4. Open the virtual reality tab in the toolbar, select the passthrough cameras option, and select openVRcamera. At this time, the camera (10) on the mixed reality helmet (2) is turned on, and the image captured by the front camera (10) is displayed in the area opened in S4. S5. Open the VR settings option in the virtual reality tab of the toolbar; select the Eye View option, open Portal1, set the relevant parameters accordingly, and adjust the position, size, and angle of the real-view display area; at the same time, multiple real-view display area windows can be created to display the area according to actual needs; S6. After adjusting all parameters, you need to select an adjustment plan for the virtual cockpit. The plan is as follows: In Prepar3D, you can choose not to display the virtual cockpit by right-clicking the mouse and selecting the Cockpit option. Import the digital model of the physical cockpit into the Prepar3d program as a virtual cockpit display. The method is as follows: first install the plug-in in the Prepar3d SDK in 3dmax, use 3dmax to import the digital model built by CATIA, export the model as a *.x file, then use the xtomdl tool in the SDK to convert the *.x file to a *.mdl format file, and import it into the model folder of the corresponding model aircraft in the Prepar3d program; S7. After all adjustments are made, the virtual cockpit in the virtual environment and the physical cockpit are fully connected in the VR display mode. The display effect of the virtual cockpit and the physical cockpit being fully connected is achieved through the following operations: after importing the cockpit digital model you drew into prepar3d, in the VR settings option, add 6 faces to wrap the cockpit model. At this time, the default cockpit digital model is a virtual image, and the 6 faces are real-life images. At this time, through the inverted portals option in the passthrough mode tab, the display content can be inverted, that is, the display content of the 6 faces added to wrap the cockpit can be realized as a virtual scene, while the position of the cockpit digital model is displayed in real life.

Citation Information

Cited By

  • Solid-state battery safety performance practical operation training platform

    CN120071701A