System and method for augmented reality windshield projection of aircraft cockpit
By using an augmented reality windshield projection system in the cockpit of the aircraft, the problem of single information display and insufficient external perception in the prior art is solved, providing a vivid head-up display, improving the pilot's external environment perception and flight safety.
Patent Information
- Application Number
- CN202411259305.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-11-19
AI Technical Summary
The head-up display equipment in the cockpit of the existing aircraft occupies the head space of the crew. The information display method is single and monochromatic, and the means of perception of the external environment are limited, making it difficult to meet the pilot's need to quickly obtain external information during complex operations.
Provide an augmented reality windshield projection system, which obtains the aircraft's own state and external environment information through the perception module, generates an augmented reality image, and projects it onto the airplane cockpit, and uses an optical projection module to project it in the light hood to provide a head-up view. Display interface to adapt to different flight stages and pilot mission needs.
It improves the pilot's perception of the external environment and provides vivid information display, reducing the difficulty of obtaining information in the pilot's complex operation process, improving flight safety, and avoiding the risk of display equipment colliding with the pilot's head.
Smart Images

Figure CN118977859A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a human-machine interactive display system and method for an aircraft cockpit, and in particular to a system and method for augmented reality windshield projection for an aircraft cockpit. Background Art
[0002] The cockpit of an aircraft (such as a large civil aircraft) is the main place for pilots to perform tasks. It integrates a large amount of information in a limited space and has complex interactive scenes. For example, in critical stages such as takeoff and approach of an aircraft (especially a civil aircraft), the operation process is complex, the reaction time is short, and the pilot needs to pay attention to a lot of information. The aircraft itself has many parameters and instruments with a large distribution range, making it difficult for the crew to take into account both the external field of view and the instrument data at the same time.
[0003] However, existing aircraft have cockpit head-up display devices installed above the crew's head space, which occupies the crew's head space, has a single information display mode, monochrome display is not vivid enough, occupies the crew's head space, and may cause personal injury in turbulent environments, which urgently need to be improved. In addition, existing equipment only obtains internal information of the aircraft through the aircraft's internal indication and recording system, and has limited means of perceiving the external environment.
[0004] The present disclosure has been improved with respect to but not limited to the above-mentioned factors. Summary of the invention
[0005] To this end, the present disclosure provides an augmented reality windshield projection system and method for an aircraft cockpit, which can perceive the external environment in real time during various flight phases of the aircraft (such as ground taxiing and flight), and project key information of the pilot's existence perception needs onto the cockpit windshield, so that the pilot can easily look straight ahead.
[0006] According to a first aspect of the present disclosure, a system for augmented reality windshield projection in an aircraft cockpit is provided, comprising: a perception module, the perception module being configured to sense state information of the aircraft itself and information of the aircraft's external environment; an augmented reality image generation module, the augmented reality image generation module being configured to determine the flight phase of the aircraft based on the aircraft's own state information, and to generate an augmented reality image based on the flight phase and using information from the perception module; and an optical projection module, the optical projection module being arranged in a front sunshade of the aircraft cockpit and being configured to project the augmented reality image onto the windshield of the aircraft.
[0007] According to one embodiment, the perception module includes: an on-board indication and recording device for sensing the aircraft's own status information; a visual sensing device arranged outside the cockpit, the visual sensing device collects the image in front of the aircraft within the pilot's field of view and includes a visual camera and an infrared camera; and a radar detection device, the radar detection device includes a laser radar and / or a millimeter-wave radar for detecting terrain information and obstacles in the aircraft's surrounding environment during the operation of the aircraft, wherein the aircraft's own status information includes at least one of the aircraft's airspeed, flight altitude, global navigation satellite system (GNSS) positioning, the configuration status of each aircraft system, and warning information.
[0008] According to another embodiment, using information from the perception module to generate an augmented reality image includes: integrating the information from the perception module to obtain the picture content of the augmented reality image to be projected; and eliminating the distortion of the augmented reality image when it is projected onto the windshield according to the curvature of the windshield.
[0009] According to another embodiment, the perception module further includes: a cockpit detection device, the cockpit detection device including a camera arranged in the cockpit and a collection device for obtaining the crew's operations on the side stick, the joystick plate and / or the pedals from the flight control system, the camera collecting information related to the crew's head and gaze direction.
[0010] According to another embodiment, the flight crew of the aircraft includes two pilots, and using the information from the perception module to generate an augmented reality image includes: determining the pilot at the stick and the pilot not at the stick based on the crew's manipulation of the side stick, the joystick plate and / or the pedals collected from the flight control system; using the acquired aircraft status information and environmental information to generate a first augmented reality image and a second augmented reality image to be projected for the pilot at the stick and the non-pilot respectively; and projecting the first augmented reality image onto the windshield portion in front of the pilot at the stick, and projecting the second augmented reality image onto the windshield portion in front of the non-pilot at the stick.
[0011] According to yet another embodiment, the augmented reality image generation module is further configured to suspend or dim the display of the second augmented reality image in response to the ballast pilot turning his head toward the non-ballast pilot.
[0012] According to yet another embodiment, the augmented reality image generation module is further configured to change the content of the projected second augmented reality image in response to an input instruction from the non-coaching pilot.
[0013] According to another embodiment, the flight phases include ground taxiing, takeoff, climb / cruise / descent, approach, and landing, and wherein: in the ground taxiing phase, the augmented reality image includes at least one of the following: the current configuration settings of the aircraft, a map of the airport where it is located, the current destination, path guidance to the current destination, tower traffic control instructions, collision warnings, images of the direction of travel ahead of the aircraft, and surrounding objects that the aircraft may collide with during movement; in the takeoff phase, the augmented reality image includes at least one of the following: information related to the aircraft attitude, sideslip information, airspeed information, flight altitude information, direction to the next waypoint, and warnings that are not suppressed during the takeoff phase; in climb / cruise / descent, the augmented reality image Including at least one of the following: the current flight status of the aircraft, the next waypoint location information, as well as weather information and collision warning; in the approach phase, the augmented reality image includes at least one of the following: the current flight status, the current accurate position of the aircraft, the current flight plan and the next waypoint location, the tower traffic control instructions, the target point and path to be traveled to; the image of the aircraft's forward direction of travel, the aircraft's surrounding terrain environment information, the glide path route, and the landing runway contour; and in the landing phase, the augmented reality image includes at least one of the following: the current aircraft landing configuration setting, the current accurate position of the aircraft, the current deceleration trend of the aircraft; the tower traffic control instructions, the runway departure point location, the image of the aircraft's forward direction of travel, and the aircraft's surrounding terrain environment information.
[0014] According to yet another embodiment, the windshield is a multi-layer structure, and each layer of the windshield is filled with a wedge-shaped interlayer.
[0015] According to another embodiment, the projection position of the center of the augmented reality image on the windshield of the aircraft is the gaze position of the pilot of the aircraft when sitting in the cockpit and looking straight at the windshield, so that the augmented reality image is covered on the corresponding real object in the pilot's field of view.
[0016] According to a second aspect of the present disclosure, a method for augmented reality windshield projection in an aircraft cockpit is provided, comprising: determining the current flight phase of the aircraft based on information obtained from an onboard indication and recording device; determining aircraft status information and environmental information to be displayed based on the determined flight phase; collecting the aircraft status information and environmental information to be displayed from the onboard indication and recording system and various environmental sensors arranged on the aircraft; using the acquired aircraft status information and environmental information to generate an augmented reality image to be projected; and projecting the augmented reality image onto the front windshield of the cockpit of the aircraft.
[0017] According to one embodiment, the flight crew of the aircraft includes two pilots, and the method further includes: collecting the crew's manipulations of the side stick, the joystick plate and / or the pedals from the flight control system of the aircraft; determining the pilot at the stick and the pilot not at the stick based on the collected manipulations; using the acquired aircraft status information and environmental information to generate a first augmented reality image and a second augmented reality image to be projected for the pilot at the stick and the pilot not at the stick, respectively; and projecting the first augmented reality image onto the front windshield portion of the cockpit in front of the pilot at the stick, and projecting the second augmented reality image onto the front windshield portion of the cockpit in front of the pilot not at the stick.
[0018] According to a third aspect of the present disclosure, an aircraft is provided, comprising the system for augmented reality windshield projection in an aircraft cockpit according to the first aspect of the present disclosure.
[0019] Aspects generally include methods, apparatus, systems, computer program products, and processing systems substantially as described herein with reference to and as illustrated by the accompanying figures.
[0020] The foregoing has broadly outlined the features and technical advantages of examples according to the present disclosure so that the following detailed description can be better understood. Additional features and advantages will be described hereinafter. The disclosed concepts and specific examples can be easily used as the basis for modifying or designing other structures for implementing the same purpose as the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein, both in terms of their organization and method of operation, and the associated advantages will be better understood by considering the following description in conjunction with the accompanying drawings. Each of the accompanying drawings is provided for illustration and description purposes and does not define limitations on the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to understand in detail the manner in which the above-stated features of the present disclosure are used, the above briefly summarized content may be described in more detail with reference to various aspects, some of which are illustrated in the accompanying drawings. However, it should be noted that the accompanying drawings illustrate only certain typical aspects of the present disclosure and should not be considered to limit its scope, as the description may allow for other equally effective aspects. The same reference numerals in different drawings may identify the same or similar elements.
[0022] Figure 1 A schematic diagram of a system for augmented reality windshield projection in an aircraft cockpit according to an example embodiment of the present disclosure is shown;
[0023] Figure 2 A schematic diagram of a system for augmented reality windshield projection in an aircraft cockpit according to a specific example of the present disclosure is shown;
[0024] Figure 3 A schematic diagram showing a comparison of the imaging of a windshield with a wedge-shaped interlayer and a windshield without a wedge-shaped interlayer;
[0025] Figure 4 A projection head-up display effect diagram of a system for augmented reality windshield projection according to an example embodiment of the present disclosure is shown;
[0026] Figure 5 A flow chart showing a method for augmented reality windshield projection for an aircraft cockpit according to an example embodiment of the present disclosure; and
[0027] Figure 6 is a schematic diagram illustrating an example aircraft according to aspects of the present disclosure. DETAILED DESCRIPTION
[0028] The inventors realize that existing aircrafts have cockpit head-up display devices installed above the crew's head space, which occupies the crew's head space, has a single information display mode, monochrome display is not vivid enough, occupies the crew's head space, and may cause personal injury in a turbulent environment, which urgently needs to be improved. In addition, the existing equipment only obtains the internal information of the aircraft through the aircraft's internal indication and recording system, and has limited means of perceiving the external environment.
[0029] To this end, the present disclosure provides an augmented reality windshield projection method and system for an aircraft cockpit. The system of the present disclosure determines the display information that needs to be provided to the pilot according to the current flight phase of the aircraft and the pilot's work tasks, obtains the aircraft's surrounding environment data through sensors such as cameras and laser radars, obtains the aircraft status data through the flight control system, generates an augmented reality image after data processing through a data fusion method, and then projects the augmented reality image on the front windshield of the aircraft cockpit to provide the pilot with the required information, enhance the flight crew's situational awareness, and improve flight safety.
[0030] The method and system of the present disclosure can provide a head-up display interface to the crew members by placing the projection device in the sunshade at the front of the cockpit and using the windshield as the display screen. In particular, when providing the display interface, the method and system of the present disclosure can identify the eye position and gaze direction of the crew members through the camera in the cockpit for a two-pilot crew, and determine the crew division of labor at the current stage in combination with the crew input information obtained from the flight control system (or the settings input by the pilot), so as to display different augmented reality images to different pilots, and make these augmented reality images not interfere with each other.
[0031] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. This detailed description includes specific details to provide a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details.
[0032] refer to Figure 1 , which shows a schematic diagram of a system 100 for augmented reality windshield projection in an aircraft cockpit according to an exemplary embodiment of the present disclosure.
[0033] like Figure 1 As shown, the system 100 for augmented reality windshield projection in an aircraft cockpit may include a perception module 101 , an augmented reality image generation module 103 , and an optical projection module 105 .
[0034] In one embodiment of the present disclosure, the perception module 101 may be configured to sense the state information of the aircraft itself and the information of the external environment of the aircraft. In one example, the perception module 101 may include an onboard indication and recording device for sensing the state information of the aircraft itself: a visual sensing device arranged outside the cockpit, the visual sensing device collects the image of the front of the aircraft in the pilot's field of vision and includes a visual camera and an infrared camera; and a radar detection device, the radar detection device includes a laser radar and / or a millimeter wave radar for detecting terrain information and obstacles of the surrounding environment of the aircraft during the operation of the aircraft (especially in scenes such as the ground, take-off and approach landing field). In another example, the perception module 101 may also include a cockpit detection device, which may include a camera arranged in the cockpit and a collection device for obtaining the crew's manipulation at the side stick, the joystick plate and / or the pedal from the flight control system, wherein the camera collects information related to the crew's head and gaze direction. According to this example, the state information of the aircraft itself may include at least one of the aircraft airspeed, flight altitude, global navigation satellite system (GNSS) positioning, the configuration status of each system of the aircraft, and warning information, or any other suitable information. Further according to this example, the onboard indication and recording device can be the original structure of the aircraft, which is responsible for providing the aircraft's own status information to the optical projection system; the visible camera captures the scene image within the pilot's field of view in front of the aircraft for subsequent image recognition processing; the infrared camera can especially shoot in dark external environments to provide pictures for enhancing the visual system and image recognition.
[0035] refer to Figure 2 , which shows a schematic diagram of a system 200 for augmented reality windshield projection in an aircraft cockpit according to a specific example of the present disclosure. Figure 2 As can be seen in FIG. 2 , the system 200 may include a perception module 201, an augmented reality image generation module 203, and an optical projection module 205. Figure 2 As shown, the perception module 201 includes an (on-board) indication and recording system (such as an atmospheric data and inertial reference system for providing airspeed and attitude information, a GNSS for providing position information, and any other suitable device for providing other flight information), a radar detection module (such as a lidar for providing environmental object information, an airborne millimeter-wave radar), a visual sensing device (such as a visual camera for providing forward video images, an infrared camera for providing vision enhancement information), and a cockpit detection module (such as a cockpit camera for providing crew eye position information, and a crew flight control acquisition module for providing crew flight task division information).
[0036] In one embodiment of the present disclosure, the augmented reality image generation module 103 may be configured to determine the flight phase of the aircraft based on the aircraft's own state information, and generate an augmented reality image based on the determined flight phase using information from the perception module 101. In one example, determining the flight phase of the aircraft may be based on manual input by the pilot, the aircraft's current flight altitude, airspeed, landing gear and flap settings, etc., which will not be described in detail here. Further according to this embodiment, using information from the perception module 101 to generate an augmented reality image may include integrating information from the perception module 101 to obtain the image content of the augmented reality image to be projected; and eliminating the distortion of the augmented reality image when it is projected onto the windshield according to the curvature of the windshield.
[0037] like Figure 2 As shown in the specific example in , the augmented reality image generation module 203 may include a data fusion unit and an image generation unit, wherein the data fusion unit may integrate the data from the indication recording system and the radar detection module, and coordinate the data formats of different information to obtain information content that can be directly projected on the windshield for display and send it to the image generation unit, while the image generation unit may receive information from the data fusion unit and information from the visual sensor device and the cockpit detection module to determine the content of the picture to be projected and displayed, and eliminate the distortion of the picture when it is projected onto the windshield according to the curvature of the windshield.
[0038] In one embodiment of the present disclosure, the optical projection module 105 may be arranged in the front sunshade of the aircraft cockpit and configured to project the augmented reality image from the augmented reality image generation module 103 onto the windshield of the aircraft. In one example, the optical projection module 105 may include an optical device and a corresponding optical path for projection of an augmented reality image.
[0039] like Figure 2As shown in the specific example in, the optical projection module 205 may include an optical device and a corresponding optical path for projecting the augmented reality image from the augmented reality image generation module 203 onto the windshield. In this example, the optical device can receive the generated augmented reality image, and the image generation unit is controlled by the driver board to generate an optical image. The generated optical image is projected onto the windshield of the aircraft cockpit by the optical path, reflected into the eyes of the pilot, and a virtual image of the display content is formed in front of the aircraft cockpit. Further according to this embodiment, the position of the image projection can also be adjusted according to the crew eye position and gaze direction information provided by the perception module 201, so that the crew can read the display information at all times, and in the case of a dual-pilot crew, avoid the simultaneous appearance of projection images on both sides in the field of vision of one pilot to prevent his field of vision from being disturbed. In addition, for vibrations that may exist during flight, the optical projection module 205 can also implement an anti-shake algorithm to eliminate and reduce jitter.
[0040] In another embodiment of the present disclosure, when the flight crew of the aircraft includes two pilots, the augmented reality image generation module 103 may also be configured to: determine the pilot at the yoke and the pilot not at the yoke based on the crew members' manipulations at the side stick, the joystick plate and / or the pedals collected from the flight control system; use the acquired aircraft state information and environmental information to generate the first augmented reality image and the second augmented reality image to be projected for the pilot at the yoke and the pilot not at the yoke, respectively; and project the first augmented reality image onto the windshield portion in front of the pilot at the yoke, and project the second augmented reality image onto the windshield portion in front of the pilot not at the yoke. In this embodiment, considering that the division of tasks between the pilot at the yoke and the pilot not at the yoke is different, and thus the content requirements for the display screen may also be different, the content of the first and second augmented reality images may be different.
[0041] In order to prevent the augmented reality image from affecting the field of vision of the pilots (especially the stick-steering pilot), in a preferred embodiment, the augmented reality image generation module 103 may also be configured to terminate or dim the display of the second augmented reality image for the non-stick-steering pilot in response to the stick-steering pilot turning his head in the direction of the non-stick-steering pilot.
[0042] In order to enable pilots (especially non-coach pilots) to conveniently access various required information, the augmented reality image generation module 103 may also be configured to change the content of the projected second augmented reality image in response to input instructions from non-coach pilots. For example, the augmented reality image generation module 103 may generate multiple second augmented reality images, and display a corresponding second augmented reality image in response to a request from a non-coach pilot. Alternatively, the augmented reality image generation module 103 may also be configured to generate a new second augmented reality image in real time in response to a request from a non-coach pilot.
[0043] In another embodiment of the present disclosure, the flight phases of an aircraft may include ground taxiing, takeoff, climb / cruise / descent, approach, landing, etc., and in the ground taxiing phase, the augmented reality image may include at least one of the following: the aircraft's current configuration settings, a map of the airport where it is located, the current destination, path guidance to the current destination, tower traffic control instructions, collision warnings, images of the aircraft's forward direction of travel, and surrounding objects that the aircraft may collide with during movement; in the takeoff phase, the augmented reality image may include at least one of the following: information related to the aircraft's attitude, sideslip information, airspeed information, flight altitude information, direction to the next waypoint, and warnings that are not suppressed during the takeoff phase; in climb / cruise / descent ... side slip information, airspeed information, flight altitude information, direction to the next waypoint, and warnings that are not suppressed during the takeoff phase; in The image may include at least one of the following: the current flight status of the aircraft, the location information of the next waypoint, as well as weather information and collision warning; in the approach phase, the augmented reality image may include at least one of the following: the current flight status, the current accurate location of the aircraft, the current flight plan and the location of the next waypoint, the tower traffic control instructions, the target point and path to be traveled to; the image of the direction of travel in front of the aircraft, the terrain environment information around the aircraft, the glide path route, and the landing runway contour; and in the landing phase, the augmented reality image may include at least one of the following: the current aircraft landing configuration setting, the current accurate location of the aircraft, the current deceleration trend of the aircraft; the tower traffic control instructions, the location of the runway departure point, the image of the direction of travel in front of the aircraft, and the terrain environment information around the aircraft. The content requirements of the augmented reality image in each flight phase are described in a specific example below.
[0044] During the ground taxiing phase (before takeoff and after landing), the system 100 can: read the current aircraft configuration settings through the flight control system; obtain the current accurate position of the aircraft through the satellite positioning system; obtain the current airport map information through the aircraft information database; obtain the target point and path to be traveled by reading the traffic control instructions of the tower; obtain the image of the direction of travel in front of the aircraft through the visual camera; and sense the surrounding objects that may collide with the aircraft during movement through the millimeter wave radar. After obtaining the above information, the system 100 can provide the above information combination to the pilot. For example, the aircraft positioning, airport map, travel target and travel route are combined into a map path guidance; the travel target, travel route, objects that may collide with each other and the image information in front of the aircraft captured by the visual camera are fused and processed, and the travel route, destination and objects that may collide with each other are covered on the corresponding objects in the pilot's field of vision in the form of highlighted marks through augmented reality technology. When the brightness of the image captured by the visible camera is too dark or the light intensity is lower than a predetermined threshold, the enhanced vision system (EVS) image captured by the infrared camera with the same field of view as the visible camera can be used as a substitute or supplement for the image captured by the visible camera.
[0045] When the aircraft is in the take-off phase, in order to ensure that the pilot's field of vision is not interfered with, the system 100 may only provide the pilot (especially the pilot at the stick) with flight information related to the current aircraft, such as: information related to the aircraft attitude; sideslip information; airspeed related information; altitude related information; direction of the next waypoint; and warnings that are not suppressed during the take-off phase. To this end, the system 100 can read information related to the current flight status of the aircraft through the flight control system; obtain the current accurate position of the aircraft through the satellite positioning system; read the aircraft flight plan through the flight control system, etc. Alternatively, the pilot can manually turn on the vision enhancement function or when the brightness of the picture taken by the visible camera is too dark or the light intensity is lower than a predetermined threshold, the enhanced vision system (EVS) picture taken by the infrared camera with the same field of view as the visible camera can be used as a substitute or supplement to the image taken by the visible camera to enhance the pilot's perception of the external environment.
[0046] During the climb / cruise / descent phases, the system 100 can mainly provide the pilot with the current aircraft flight status, the next waypoint location information, and other important information that may affect the flight process (such as weather, collision warning, etc.). To this end, the system 100 can: read the current aircraft flight status related information through the flight control system; obtain the current aircraft accurate location through the satellite positioning system; and obtain the current flight plan and the next waypoint through the aircraft information database.
[0047] During the flight phase when the aircraft is in the approach scenario, the system 100 mainly provides the pilot with information such as the current aircraft flight configuration information, the current waypoint position, the approach route, and the target airport position. To this end, the system 100 can: read the current flight status data settings through the flight control system; obtain the current accurate position of the aircraft through the satellite positioning system; obtain the current flight plan and the next waypoint position through the aircraft information database; obtain the target point and path to be traveled by reading the traffic control instructions of the tower; obtain the image of the direction of travel in front of the aircraft through the camera; and perceive the terrain environment information around the aircraft through the laser radar. After that, the system 100 can provide the above information combination to the pilot, for example, combining the aircraft positioning, airport location, and glide path into a map path guidance; performing image recognition on the image information in front of the aircraft captured by the visual camera, and obtaining the runway position expected to land through visual recognition; and covering the glide path route, landing runway contour, etc. on the corresponding objects in the pilot's field of vision in the form of highlighted marks through augmented reality technology. Alternatively, the pilot can manually turn on the vision enhancement function or when the brightness of the image captured by the visible camera is too dark or the light intensity is lower than a predetermined threshold, the enhanced vision system (EVS) image captured by the infrared camera with the same field of view as the visible camera can be used as a substitute or supplement for the image captured by the visible camera to enhance the pilot's perception of the external environment. Alternatively, when both the visible camera and the infrared camera cannot obtain a clear image of the external environment, the pilot can manually set the image using the aircraft satellite positioning information, the map information stored in the aircraft's onboard database, and the surrounding environment information obtained by the onboard radar detection to construct the image, and synthesize it with the image captured by the infrared camera to project it onto the windshield.
[0048] During the flight phase when the aircraft is in the landing scene, the system 100 mainly provides the pilot with information such as the current aircraft flight configuration information, braking distance prompts, and runway departure position prompts. To this end, the system 100 can: read the current aircraft landing configuration settings through the flight control system; obtain the current accurate position of the aircraft through the satellite positioning system; obtain the current deceleration trend of the aircraft through the braking system; obtain the runway departure point position by reading the traffic control instructions of the tower; obtain the image of the aircraft's forward direction through the visual camera; and sense the terrain environment information around the aircraft through the laser radar. After that, the system 100 can provide the pilot with the above information combination: perform image recognition on the image information in front of the aircraft captured by the camera, and obtain the aircraft braking distance prompt in conjunction with the aircraft deceleration trend; and use augmented reality technology to cover the braking distance and the expected departure taxiway in the form of highlighted marks on the corresponding objects in the pilot's field of vision.
[0049] Preferably, in an example, in the case of a two-pilot crew, in the flight phase other than the ground taxiing phase, the system 100 can determine the current crew work division (i.e., the pilot holding the stick and the pilot not holding the stick) according to the crew's inputs on the side stick / control wheel stick and pedals (or manual settings by the pilot). Preferably, according to the different divisions of labor, the projection display on the pilot holding the stick side will emphasize more flight-related data than the projection display on the pilot not holding the stick side, and the display screen on the pilot holding the stick side will have a higher display priority.
[0050] In yet another embodiment of the present disclosure, the windshield of an aircraft may be a multi-layer structure, and each layer of the windshield may be filled with a wedge-shaped interlayer to alleviate the ghost imaging problem caused by the multi-layer glass. Figure 3 FIG. 2 shows a comparative schematic diagram of the imaging of a windshield with a wedge-shaped interlayer and a windshield without a wedge-shaped interlayer. Figure 3 As shown at (a), a windshield without a wedge-shaped interlayer will cause the double image and the virtual image to not overlap, causing trouble to the pilot; while as shown at (b), a windshield with a wedge-shaped interlayer will cause the double image and the virtual image to overlap, thereby alleviating the imaging double image problem caused by the multi-layer glass structure of the windshield.
[0051] In addition, since the optical projection system is arranged at the front side of the aircraft cockpit, considering the complex electromagnetic environment at the nose position during flight, the system 100 can also be equipped with corresponding electromagnetic shielding measures to prevent the environment from damaging the system 100.
[0052] In an example of the present disclosure, the projection position of the center of the augmented reality image on the windshield of the aircraft is the gaze position of the pilot of the aircraft when sitting in the cockpit and looking straight at the windshield, so that the augmented reality image is covered on the corresponding real object in the pilot's field of vision. Figure 4 , which shows a projection head-up display effect diagram of an augmented reality windshield projection system according to an example embodiment of the present disclosure.
[0053] Reference below Figure 5 , which shows a flow chart of a method 500 for augmented reality windshield projection in an aircraft cockpit according to an example embodiment of the present disclosure.
[0054] Method 500 may include, at block 510, determining the current flight phase of the aircraft based on information obtained from an onboard indication recording device. In one example, determining the current flight phase of the aircraft may be based on manual input by the pilot, the current flight altitude, airspeed, landing gear and flap settings of the aircraft, etc., which will not be described in detail herein.
[0055] In box 520, method 500 may include determining the aircraft status information and environmental information that need to be displayed based on the determined flight phase. For example, the flight phase of the aircraft may include ground taxiing, takeoff, climb / cruise / descent, approach, landing, etc., and in the ground taxiing phase, the augmented reality image may include at least one of the following: the current configuration settings of the aircraft, a map of the airport where it is located, the current destination, path guidance to the current destination, tower traffic control instructions, collision warnings, images of the direction of the aircraft's forward movement, and surrounding objects that the aircraft may collide with during movement; in the takeoff phase, the augmented reality image may include at least one of the following: information related to the aircraft's attitude, sideslip information, airspeed information, flight altitude information, direction to the next waypoint, and warnings that are not suppressed during the takeoff phase; in climb / cruise / descent, the augmented reality image may include The augmented reality image may include at least one of the following: the current flight status of the aircraft, the location information of the next waypoint, as well as weather information and collision warning; in the approach phase, the augmented reality image may include at least one of the following: the current flight status, the current accurate location of the aircraft, the current flight plan and the location of the next waypoint, the tower traffic control instructions, the target point and path to be traveled to; the image of the direction of travel in front of the aircraft, the terrain environment information around the aircraft, the glide path route, and the landing runway contour; and in the landing phase, the augmented reality image may include at least one of the following: the current aircraft landing configuration setting, the current accurate location of the aircraft, the current deceleration trend of the aircraft; the tower traffic control instructions, the location of the runway departure point, the image of the direction of travel in front of the aircraft, and the terrain environment information around the aircraft.
[0056] In box 530, method 500 may include collecting aircraft status information and environmental information to be displayed from the onboard indication recording system and various environmental sensors disposed on the aircraft, and in box 540, method 500 may include using the acquired aircraft status information and environmental information to generate an augmented reality image to be projected. For example, method 500 may integrate data from the indication recording system and the radar detection module, coordinate the data formats of different information, to obtain information content that can be directly projected on the windshield for display, and method 500 may also receive information from the visual sensor device and the cockpit detection module to determine the content of the image to be projected and displayed, and eliminate the distortion of the image when it is projected on the windshield according to the curvature of the windshield.
[0057] At block 550 , method 500 may include projecting an augmented reality image onto a cockpit front windshield of the aircraft.
[0058] In another embodiment of the present disclosure, the flight crew of the aircraft includes two pilots, and method 500 may also include collecting the crew's manipulations of the side stick, the joystick plate and / or the pedals from the aircraft's flight control system; determining the steering pilot and the non-steering pilot based on the collected manipulations; using the acquired aircraft state information and environmental information to generate a first augmented reality image and a second augmented reality image to be projected for the steering pilot and the non-steering pilot, respectively; and projecting the first augmented reality image onto a front windshield portion of the cockpit in front of the steering pilot, and projecting the second augmented reality image onto a front windshield portion of the cockpit in front of the non-steering pilot.
[0059] refer to Figure 6 , which shows a schematic diagram of an aircraft 600 according to an exemplary embodiment of the present disclosure. In this embodiment, the aircraft 600 may include an augmented reality windshield projection system according to various embodiments of the present disclosure, such as for Figure 1 A system 100 for augmented reality windshield projection in an aircraft cockpit is described.
[0060] As described above, the present invention provides an equivalent or even better head-up display effect. By arranging a projection device in the sun visor at the front of the aircraft cockpit, the pilot's head space is freed up, and there is no need to arrange a display screen in front of the pilot's head, thereby avoiding the risk of the pilot colliding with the display screen under turbulent conditions.
[0061] The disclosed system and method also improve the crew's situational awareness and ability to perceive the external environment during each flight process. Based on the crew's different requirements for flight missions at different flight stages, the disclosed system and method innovatively proposes a cockpit head-up display system and method that provides augmented reality functions, provides augmented reality display images related to the pilot's current mission, enhances the pilot's situational awareness, and improves the aircraft's operational capabilities.
[0062] In view of the dual-pilot crew cockpit feature, the present invention also utilizes the in-cabin camera to obtain the crew's eye position and gaze direction, and uses the crew input to obtain each pilot's current work task, so as to display augmented reality images to different pilots respectively, and also enables preventing augmented reality images for one pilot from appearing in the field of view of another pilot at the same time.
[0063] The above detailed description includes references to the accompanying drawings, which form part of the detailed description. The accompanying drawings illustrate specific embodiments that can be practiced by way of illustration. These embodiments are also referred to herein as "examples". Such examples may include elements other than those shown or described. However, examples including the elements shown or described are also contemplated. In addition, examples using any combination or arrangement of those elements shown or described are also contemplated, or with reference to the specific examples (or one or more aspects thereof) shown or described herein, or with reference to other examples (or one or more aspects thereof) shown or described herein.
[0064] In the appended claims, the terms "including" and "comprising" are open ended, that is, systems, devices, articles, or processes having elements other than those elements listed after such terms in a claim are still deemed to fall within the scope of that claim. Furthermore, in the appended claims, the terms "first," "second," and "third," etc. are used merely as labels and are not intended to indicate a numerical order to their objects.
[0065] In addition, the order of each operation explained in this specification is exemplary. In alternative embodiments, each operation can be performed in a different order than that shown in the drawings, and each operation can be combined into a single operation or split into more operations.
[0066] The above description is intended to be illustrative, not restrictive. For example, the examples described above (or one or more aspects thereof) may be used in conjunction with other embodiments. Other embodiments may be used, such as by a person of ordinary skill in the art, after reviewing the above description. The abstract allows the reader to quickly determine the nature of the present technology disclosure. The abstract is submitted, and it is understood that the abstract will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above specific embodiments, various features may be grouped together to make the disclosure fluent. However, the claims may not state every feature disclosed herein, because the embodiments may characterize a subset of the features. In addition, the embodiments may include fewer features than those disclosed in a particular example. Therefore, the attached claims are thus incorporated into the specific embodiments, and a claim exists independently as a separate embodiment. The scope of the embodiments disclosed herein should be determined with reference to the attached claims and the full scope of equivalents to which such claims are entitled.
Claims
1. A system for augmented reality windshield projection in an aircraft cockpit, characterized in that: include: A perception module, wherein the perception module is configured to sense the state information of the aircraft itself and the information of the external environment of the aircraft; an augmented reality image generation module, the augmented reality image generation module being configured to determine the flight phase of the aircraft based on the aircraft's own state information, and to generate an augmented reality image based on the flight phase and using the information from the perception module; as well as An optical projection module is disposed in a front sunshade of an aircraft cockpit and is configured to project the augmented reality image onto a windshield of the aircraft.
2. The system according to claim 1, characterized in that The perception module comprises: On-board indication and recording device for sensing the aircraft's own status information: A visual sensor device disposed outside the cockpit, the visual sensor device collects images of the front of the aircraft within the pilot's field of view and includes a visual camera and an infrared camera; and A radar detection device, wherein the radar detection device includes a laser radar and / or a millimeter wave radar for detecting terrain information and obstacles in the aircraft's surrounding environment during the operation of the aircraft. The aircraft's own status information includes at least one of the aircraft's airspeed, flight altitude, global navigation satellite system (GNSS) positioning, configuration status of each aircraft system, and warning information.
3. The system according to claim 2, characterized in that Using information from the perception module to generate an augmented reality image includes: Integrating information from the perception module to obtain the picture content of the augmented reality image to be projected; and The distortion of the augmented reality image when projected onto the windshield is eliminated according to the curvature of the windshield.
4. The system according to claim 2, characterized in that The perception module also includes: A cockpit detection device, the cockpit detection device includes a camera arranged in the cockpit and a collection device for obtaining the crew's operations on the side stick, the control stick plate and / or the pedals from the flight control system, and the camera collects information related to the crew's head and gaze direction.
5. The system according to claim 4, characterized in that The flight crew of the aircraft includes two pilots, and using information from the perception module to generate an augmented reality image includes: Determine the pilot who holds the stick and the pilot who does not hold the stick based on the flight crew's manipulation of the side stick, control panel and / or pedals collected from the flight control system; Using the acquired aircraft status information and environmental information to generate a first augmented reality image and a second augmented reality image to be projected for the pilot at the helm and the pilot not at the helm, respectively; and The first augmented reality image is projected onto a portion of the windshield in front of the stick pilot, and the second augmented reality image is projected onto a portion of the windshield in front of the non-stick pilot.
6. The system according to claim 5, characterized in that The augmented reality image generation module is further configured to stop or dim the display of the second augmented reality image in response to the ballast pilot turning his head toward the non-ballast pilot.
7. The system according to claim 5, characterized in that The augmented reality image generation module is further configured to change the content of the projected second augmented reality image in response to input instructions from the non-coaching pilot.
8. The system according to claim 1, characterized in that The flight phases include ground taxi, take-off, climb / cruise / descent, approach, landing, and among them: During the ground taxiing phase, the augmented reality image includes at least one of the following: the current configuration settings of the aircraft, a map of the airport where the aircraft is located, the current destination, a path guide to the current destination, a tower traffic control instruction, a collision warning, an image of the aircraft's forward direction of travel, and surrounding objects that the aircraft may collide with during movement; During the take-off phase, the augmented reality image includes at least one of the following: information related to aircraft attitude, side slip information, airspeed information, flight altitude information, direction to the next waypoint, and warnings that are not suppressed during the take-off phase; During climb / cruise / descent, the augmented reality image includes at least one of the following: current aircraft flight status, next waypoint location information, weather information, and collision warning; During the approach phase, the augmented reality image includes at least one of the following: current flight status, current accurate aircraft position, current flight plan and next waypoint position, tower traffic control instructions, target point and path to be traveled; image of the aircraft's forward direction, terrain environment information around the aircraft, glide path route, and landing runway contour; and During the landing phase, the augmented reality image includes at least one of the following: the current aircraft landing configuration setting, the current accurate position of the aircraft, the current deceleration trend of the aircraft; tower traffic control instructions, the runway departure point position, an image of the aircraft's forward direction of travel, and terrain environment information around the aircraft.
9. The system according to claim 1, characterized in that The windshield is a multi-layer structure, and each layer of the windshield is filled with a wedge-shaped interlayer.
10. The system according to claim 1, characterized in that The projection position of the center of the augmented reality image on the windshield of the aircraft is the gaze position of the pilot of the aircraft when sitting in the cockpit and looking straight at the windshield, so that the augmented reality image covers the corresponding real object in the pilot's field of view.
11. A method for augmented reality windshield projection in an aircraft cockpit, characterized in that: include: Determine the current flight phase of the aircraft based on the information obtained from the onboard indication and recording device; Determine the aircraft status information and environmental information to be displayed according to the determined flight phase; Collecting the aircraft status information and environmental information required for display from the onboard indication and recording system and various environmental sensors arranged on the aircraft; using the acquired aircraft status information and environmental information to generate an augmented reality image to be projected; as well as The augmented reality image is projected onto a front windshield of a cockpit of the aircraft.
12. The method according to claim 11, characterized in that The flight crew of the aircraft includes two pilots, and the method further includes: collecting the crew's manipulations on the side stick, control column and / or pedals from the flight control system of the aircraft; Determining a stick-holding pilot and a non-stick-holding pilot based on the collected maneuvers; Using the acquired aircraft status information and environmental information to generate a first augmented reality image and a second augmented reality image to be projected for the pilot at the helm and the pilot not at the helm, respectively; and The first augmented reality image is projected onto a front windshield portion of the cockpit in front of the pilot at the yoke, and the second augmented reality image is projected onto a front windshield portion of the cockpit in front of the non-pilot at the yoke.
13. An aircraft comprising a system according to any one of claims 1-10.
Citation Information
Patent Citations
Aircraft augmented reality system and corresponding method of operating
CN111292575A
Image generation method, image generation device and vehicle-mounted head-up display system
CN113112413A
Systems and methods for a heads-up display for an electric aircraft
US11598960B1
Augmented Reality System for Pilot and Passengers
US20180251230A1
