A pilot cockpit training system

Abstract

The application discloses a kind of pilot cockpit training systems, including hemispherical green screen, training cockpit and VR glasses, camera, computing unit, gyro module, storage unit, liquid crystal display and convex lens are integrated in the VR glasses;Hemispherical green screen cover is buckled in the periphery of training cockpit;Training cockpit sends the control information in cockpit to computing unit;Camera, gyro module, storage unit and liquid crystal display are respectively connected with computing unit communication;Camera collects the real driving scene image in training cockpit;Gyro module is used for monitoring the motion data of VR glasses;Virtual driving scene image data is pre-stored in storage unit;Computing unit fuses the real driving scene image data in training cockpit with the virtual driving scene image data in storage unit and generates mixed reality scene image, and mixed reality scene image is transmitted to liquid crystal display and carries out real-time picture display.

Description

Technical Field

[0001] This invention belongs to the technical field of pilot flight simulation training systems, and specifically relates to a pilot cockpit training system. Background Technology

[0002] With the development of computer technology, virtual reality (VR) and mixed reality (MR) technologies are showing broad application prospects in various fields. Existing VR technology uses computer technology to simulate a three-dimensional virtual world that provides users with sensory experiences such as sight, hearing, and touch. Users interact naturally with this virtual world using devices such as controllers. This technology primarily relies on the computing power of computers. Its characteristics include a display field of view that closely approximates the real field of vision of the human eye, largely isolating the influence of surrounding audio and video, thus providing users with a strong sense of immersion. The implementation method is relatively simple, and the hardware cost is low.

[0003] The currently used mixed reality technology is a method of merging and displaying the real world and the virtual world. Its characteristic is that it can effectively achieve human-computer interaction between humans and both the real and virtual worlds. Traditional methods typically use holographic projection based on the principles of interference and diffraction to project virtual driving scene images into the real world. This method has drawbacks such as a narrow field of view, complex structure, high cost, high power consumption, and insufficient immersive experience. Summary of the Invention

[0004] To address the problems existing in current pilot training that uses only virtual reality or mixed reality technologies, this invention provides a pilot cockpit training system that integrates virtual reality and mixed reality technologies to achieve a highly immersive experience, a wider field of view, a simpler system structure, lower cost, and a wider range of applications with greater versatility.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A pilot cockpit training system includes a hemispherical green screen, a training cockpit, and VR glasses. The VR glasses integrate a camera, a computing unit, a gyroscope module, a storage unit, an LCD screen, and a convex lens. The hemispherical green screen is mounted around the training cockpit. Trainees wear the VR glasses and perform operational training within the training cockpit. The training cockpit is communicatively connected to the computing unit of the VR glasses, sending the trainee's control information within the cockpit to the computing unit. The camera, gyroscope module, storage unit, and LCD screen are each communicatively connected to the computing unit. The camera is used to capture real-time data of the training within the training cockpit. The system transmits real-world driving scene image data to the computing unit; the gyroscope module monitors the real-time motion data of the VR glasses and transmits the motion data to the computing unit; the storage unit pre-stores virtual driving scene image data corresponding to different models and tasks; the computing unit processes the real-world driving scene image data in the training cockpit and the virtual driving scene image data integrated in the storage unit to create a mixed reality scene image, which is then transmitted to the LCD screen for real-time display; the image displayed on the LCD screen is projected into the trainee's field of vision after being corrected by a convex lens.

[0007] Furthermore, the training cockpit is equipped with a joystick, accelerator, brake, power switch, buttons, and an instrument panel.

[0008] Furthermore, the control information of the trainee in the cockpit includes the trainee's control information on the joystick, accelerator, brake, ignition switch, and buttons.

[0009] Furthermore, the liquid crystal display screen is provided with two, left and right, and the convex lens is provided with two, left and right; the left and right liquid crystal display screens are respectively positioned in front of the left and right convex lenses, and the convex lenses are positioned in front of the eyes of the person.

[0010] Furthermore, the cameras are a pair, located at the very front of the VR glasses, and the distance between the cameras is adjustable.

[0011] Furthermore, the computing unit is used to retrieve the virtual driving scene data corresponding to the current model task built into the storage unit, and to comprehensively process the real driving scene image data transmitted by the camera, the VR glasses motion data transmitted by the gyroscope module, and the control information data transmitted by the training cockpit, and obtain mixed reality scene images through mixed reality scene fusion.

[0012] Furthermore, the mixed reality scene fusion process of the computing unit includes:

[0013] S1. Position calibration of VR glasses and virtual driving scene;

[0014] S2. Virtual Driving Scene Image Processing:

[0015] The virtual driving scene image display field is determined based on the position and angle of the VR glasses calibrated in step S1, that is, the virtual driving scene image to be displayed;

[0016] Rearview mirror image imaging in the cockpit: The size and position of the green screen on the rearview mirror are captured by VR glasses. Taking the plane of the rearview mirror as the symmetrical plane, according to the principle of plane reflection imaging, the virtual scene image of the corresponding green screen position of the rearview mirror in the virtual driving scene behind the cockpit in the storage unit is calculated in real time and retrieved. The virtual scene image is then transmitted to the LCD screen in real time.

[0017] S3. Mixed Reality Image Fusion Display:

[0018] The computing unit performs image fusion on the real scene images inside the cockpit obtained by the camera, the virtual driving scene images obtained through calculation, and the virtual scene images to be displayed in the rearview mirror.

[0019] S4. Gyroscope error correction.

[0020] Further, step S1 includes:

[0021] S11. Fixed coordinate system in virtual driving scenario: Calculate the motion direction, speed and position information of the training cockpit in the virtual driving scenario;

[0022] S12. Position calibration of VR glasses and training cockpit: Establish a coordinate system with a certain point in the training cockpit as the origin, and convert the motion data monitored by the gyroscope module in the VR glasses to the cockpit coordinate system.

[0023] S13. Transform the motion data of the VR glasses in the cockpit coordinate system to the virtual driving scene coordinate system, and finally determine the position and angle information of the VR glasses in the virtual driving scene.

[0024] Furthermore, in step S3, the image fusion includes: first aligning the three types of image frames; in the real scene image frame obtained by the camera, the computing unit replaces the detected green pixel positions with the corresponding virtual driving scene image frame and the rearview mirror virtual scene image frame respectively; finally, the image fused mixed reality scene image is transmitted to the LCD screen of the VR glasses; each frame is processed in sequence according to the above steps.

[0025] Furthermore, step S4 includes: setting marker points in a hemispherical green screen or training cockpit, and correcting the deviation generated by the gyroscope by using the position of the marker points obtained by the camera at certain time intervals.

[0026] The present invention has the following beneficial effects:

[0027] 1. Wider viewing angle:

[0028] Traditional mixed reality glasses are based on holographic projection technology, and the field of view of the displayed virtual image is no more than 40°. The field of view and angle of the displayed virtual image of the present invention are larger and closer to the field of view of the real human eye.

[0029] 2. Excellent visual experience:

[0030] Existing flight simulators use flat or curved displays, which have limited screen area and lack a sense of depth and distance. This invention can give users a more realistic sense of depth, with a 360-degree screen area just like the real world, and can simulate the real visual effects and impact of the entire flight process.

[0031] Existing flight simulators that use ordinary VR glasses prevent users from seeing the outside world and allow them to only hold the throttle and control sticks, preventing them from performing other operations. This invention allows users to see the cockpit and their hands, thus enabling them to perform all normal operations within the cockpit.

[0032] Existing flight simulators using AR glasses employ holographic projection technology to project the virtual world, resulting in a narrow field of view, transparent images, poor user experience, and a lack of realism. Furthermore, the technology is complex and has a short operating time. This invention can achieve a 120-degree wide field of view, close to the range of normal human eyes, with high image quality and a strong sense of realism.

[0033] 3. Simple structure and low cost:

[0034] Most existing mixed reality technologies employ holographic projection imaging, which involves complex manufacturing processes, high costs, and difficulties in mass production. This invention, based on virtual reality technology, uses two cameras (left and right), a gyroscope module, and a computing unit positioned in front to transform real-world scenes into realistic images. These images are then fused with virtual driving scene images and displayed on left and right LCD screens and convex lenses within the viewer's field of vision. This approach achieves a mixed reality effect while simplifying the structure and manufacturing process.

[0035] 4. Wider range of applications and stronger versatility:

[0036] This invention can be used as virtual reality glasses, and its functions can be expanded to replace traditional mixed reality glasses. Through computer processing technology, it overlays real-world scenes with virtual scenes and projects them onto an LCD screen. Therefore, it integrates the advantages of both virtual reality and mixed reality technologies. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the content of the embodiments of the present invention and these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram illustrating the structural principle of a pilot cockpit training system according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram illustrating the principle of image magnification and imaging via a convex lens for liquid crystal display as described in an embodiment of the present invention;

[0040] In the picture:

[0041] 1. Hemispherical green screen; 2. Cockpit; 3. Camera; 4. Computing unit; 5. Gyroscope module; 6. Storage unit; 7. LCD screen; 8. Convex lens. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0043] This embodiment is a pilot cockpit training system, including a hemispherical green screen 1, a training cockpit 2, and VR glasses. The VR glasses integrate a pair of cameras 3, a computing unit 4, a gyroscope module 5, a storage unit 6, two left and right liquid crystal displays 7, and two left and right convex lenses 8.

[0044] A hemispherical green screen 1 covers the perimeter of the training cockpit, providing the view through the cockpit windows. The training cockpit 2 is equipped with input devices such as joysticks, throttle, brakes, power switches, and buttons, as well as output devices such as an instrument panel (digital display screen) to display aircraft status data. Trainees wear VR glasses to perform operational training within the training cockpit. The training cockpit 2 communicates with the VR glasses' computing unit 4, sending the trainee's control information within the cockpit, including their actions on the joysticks, throttle, brakes, power switches, and buttons.

[0045] Camera 3, gyroscope module 5, storage unit 6, and LCD screen 7 are all communicatively connected to computing unit 4. Camera 3 is used to collect real-time images of the actual driving scene during training in the training cockpit 2 and transmit the collected real-time driving scene image data to computing unit 4. Gyroscope module 5 is used to monitor the real-time motion data of the VR glasses themselves and transmit the motion data to computing unit 4, achieving positioning between the real-time driving scene image and the virtual driving scene image through gyroscope module 5. Storage unit 6 pre-stores virtual driving scene image data corresponding to different models and tasks. Computing unit 4 is used to process the real-time driving scene image data in the training cockpit and the virtual driving scene image data integrated in storage unit 6. Data processing is performed to fuse mixed reality scene images: the computing unit 4 retrieves the virtual driving scene data corresponding to the current model task pre-stored in the storage unit 6, and performs comprehensive processing on the real driving scene image data transmitted by the camera 3, the VR glasses motion data transmitted by the gyroscope module 5, and the control information data transmitted by the training cockpit 2. The mixed reality scene image is obtained through mixed reality scene fusion and transmitted to the LCD screen 7 for real-time display. The left and right LCD screens 7 are respectively set in front of the left and right convex lenses 8, which are positioned in front of the eyes of the trainee. The display images of the left and right LCD screens are projected into the field of vision of the trainee after being corrected by the left and right convex lenses.

[0046] A pair of cameras 3 are located at the very front of the VR glasses. The distance between the cameras 3 is adjustable. When the distance between the cameras is the same as the distance between the eyes, the sense of space presented by the VR glasses is closest to that of the space when viewed with the naked eye.

[0047] The training cockpit 2 can be replaced depending on the specific aircraft type and mission.

[0048] The mixed reality scene fusion process of the computing unit is as follows:

[0049] S1. Positioning of VR glasses and virtual driving scene:

[0050] S11. Fixed coordinate system in virtual driving scenario: Calculate information such as the direction of motion, speed and position of the training cockpit in the virtual driving scenario;

[0051] S12. Position calibration of VR glasses and training cockpit: Establish a coordinate system with a certain point in the training cockpit as the origin, and convert the motion data monitored by the gyroscope module in the VR glasses to the cockpit coordinate system.

[0052] S13. Transform the motion data of the VR glasses in the cockpit coordinate system to the virtual driving scene coordinate system, and finally determine the position and angle of the VR glasses in the virtual driving scene.

[0053] S2. Virtual Driving Scene Image Processing:

[0054] The virtual driving scene image display field is determined based on the position and angle of the VR glasses calibrated in step S1, that is, the virtual driving scene image to be displayed;

[0055] Rearview mirror imagery in the cockpit: The size and position of the green screen on the rearview mirror are captured by VR glasses. Taking the plane of the rearview mirror as the plane of symmetry, according to the principle of planar reflection imaging, the virtual scene image corresponding to the green screen position of the rearview mirror in the virtual driving scene behind the cockpit in the storage unit is calculated in real time and retrieved. The virtual scene image is then transmitted to the LCD screen in real time.

[0056] S3. Mixed Reality Image Fusion Display:

[0057] The computing unit fuses the real-world scene image inside the cockpit obtained by the camera, the virtual driving scene image obtained through calculation, and the virtual scene image to be displayed in the rearview mirror: First, the three image frames are aligned. In the real-world scene image obtained by the camera, the computing unit replaces the detected green pixel positions with the corresponding virtual driving scene image and the virtual scene image from the rearview mirror, respectively. Finally, the fused mixed reality scene image is transmitted to the LCD screen of the VR glasses. Each frame is processed sequentially according to the above steps.

[0058] S4. Gyroscope error correction:

[0059] The gyroscope module is integrated into the VR glasses to monitor and track head movements. To account for gyroscope errors, the VR glasses are repositioned at regular intervals. This repositioning involves adding markers to a green screen or within the cockpit, and at regular intervals, the positions of these markers, obtained from the camera, are used to correct any deviations caused by the gyroscope.

[0060] The working principle of this embodiment is briefly described below:

[0061] During operation, the VR glasses' cameras first capture real-time images of the actual driving scenario for trainees in the training cockpit, as well as the visual environment including the hemispherical green screen, creating a real-world driving scene image. This image is then transmitted in real-time to the VR glasses' computing unit. Simultaneously, the computing unit accesses virtual driving scene image data from the storage unit and, through techniques such as position calibration and image fusion, merges the real-world image with the virtual driving scene. The merged mixed reality image is then displayed on an LCD screen. The image on the LCD screen is then magnified by a convex lens and directly observed by the human eye.

[0062] This embodiment uses virtual reality technology to display virtual driving scene images. By setting up two cameras on the left and right sides, one gyroscope module, and one computing unit in front, the real scene of the cockpit captured by the cameras is converted into a real image. This image is then fused with the virtual driving scene image pre-stored in the VR glasses' storage unit to create a mixed reality scene image, which is sent to the LCD screen of the VR glasses. After being corrected by a convex lens, it is projected into the trainee's field of vision.

[0063] In this embodiment, the VR glasses can be used as virtual reality glasses when the camera is off, possessing all the functions of virtual reality glasses. When the camera is on, they can be used as mixed reality glasses, where the real-world scene image and the virtual scene image are superimposed and fused through calculations by the computing unit, and then projected onto the left and right LCD screens. Therefore, this method integrates the advantages of both virtual reality technology and traditional mixed reality technology.

[0064] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A pilot cockpit training system, characterized in that, It includes a hemispherical green screen, a training cockpit, and VR glasses. The VR glasses integrate a camera, a computing unit, a gyroscope module, a storage unit, an LCD screen, and a convex lens. The hemispherical green screen is attached to the outside of the training cockpit. Trainees wear VR glasses to conduct operation training in a training cockpit. The training cockpit is connected to the computing unit of the VR glasses and sends the trainee's control information in the cockpit to the computing unit. The camera, gyroscope module, storage unit, and LCD screen are all communicatively connected to the computing unit. The camera is used to collect real-time images of the actual driving scene in the training cockpit and transmit the collected real-time driving scene image data to the computing unit. The gyroscope module is used to monitor the real-time motion data of the VR glasses themselves and transmit the motion data to the computing unit. The storage unit pre-stores virtual driving scene image data corresponding to different models and tasks. The computing unit processes the real-time driving scene image data in the training cockpit and the virtual driving scene image data integrated in the storage unit to create a mixed reality scene image, which is then transmitted to the LCD screen for real-time display. The image displayed on the LCD screen is projected into the field of vision of the trainee after being corrected by a convex lens. The mixed reality scene fusion process of the computing unit includes: S1. Positioning of VR glasses and virtual driving scene; Step S1 includes: S11. Fixed coordinate system in virtual driving scenario: Calculate the motion direction, speed and position information of the training cockpit in the virtual driving scenario; S12. Position calibration of VR glasses and training cockpit: Establish a coordinate system with a certain point in the training cockpit as the origin, and convert the motion data monitored by the gyroscope module in the VR glasses to the cockpit coordinate system. S13. Transform the motion data of the VR glasses in the cockpit coordinate system from the cockpit coordinate system to the virtual driving scene coordinate system, and finally determine the position and angle information of the VR glasses in the virtual driving scene; S2. Virtual Driving Scene Image Processing: The virtual driving scene image display field is determined based on the position and angle of the VR glasses calibrated in step S1, that is, the virtual driving scene image to be displayed; Rearview mirror image imaging in the cockpit: The size and position of the green screen on the rearview mirror are captured by VR glasses. Taking the plane of the rearview mirror as the symmetrical plane, according to the principle of plane reflection imaging, the virtual scene image of the corresponding green screen position of the rearview mirror in the virtual driving scene behind the cockpit in the storage unit is calculated in real time and retrieved. The virtual scene image is then transmitted to the LCD screen in real time. S3. Mixed Reality Image Fusion Display: The computing unit fuses the real-world scene image inside the cockpit obtained by the camera, the virtual driving scene image obtained through calculation, and the virtual scene image to be displayed in the rearview mirror. The image fusion includes: first, aligning the three types of images; in the real-world scene image obtained by the camera, the computing unit replaces the detected green pixel positions with the corresponding virtual driving scene image and the virtual scene image of the rearview mirror; finally, the fused mixed reality scene image is transmitted to the LCD screen of the VR glasses; each frame is processed in sequence according to the above steps. S4. Gyroscope error correction.

2. The pilot cockpit training system as described in claim 1, characterized in that, The training cockpit is equipped with a joystick, accelerator, brake, power switch, buttons, and an instrument panel.

3. The pilot cockpit training system as described in claim 2, characterized in that, The control information of the trainee in the cockpit includes the trainee's control information on the joystick, accelerator, brake, ignition switch and buttons.

4. The pilot cockpit training system as described in claim 1, characterized in that, The liquid crystal display screen is provided with two, left and right, and the convex lens is provided with two, left and right; the left and right liquid crystal display screens are respectively positioned in front of the left and right convex lenses, and the convex lenses are positioned in front of the eyes of the person.

5. A pilot cockpit training system as described in claim 1, characterized in that, The cameras are a pair, located at the very front of the VR glasses, and the distance between the cameras is adjustable.

6. A pilot cockpit training system as described in claim 1, characterized in that, The computing unit is used to retrieve the virtual driving scene data corresponding to the current model task built into the storage unit, and to comprehensively process the real driving scene image data transmitted by the camera, the VR glasses motion data transmitted by the gyroscope module, and the control information data transmitted by the training cockpit, and obtain mixed reality scene images through mixed reality scene fusion.

7. A pilot cockpit training system as described in claim 1, characterized in that, Step S4 includes: setting marker points in a hemispherical green screen or training cockpit, and correcting the deviation generated by the gyroscope by using the position of the marker points obtained by the camera at certain time intervals.

Images