A synthetic visual correction method and system based on a head-up display

Abstract

This invention belongs to the field of aircraft navigation and positioning technology, specifically relating to a method and system for correcting synthetic visual scenes based on a head-up display (HUD). The method involves acquiring airport environment images from airborne visual sensors based on the aircraft's current operational phase; identifying airport ground marking features from these images and matching them with elements in an airport ground feature database; calculating the aircraft's current geographical location using the geographical location information of the matched elements and their relative position to the aircraft's current location; using this geographical location information to correct errors in the airborne positioning equipment; transmitting the corrected image to an airborne synthetic visual scene system; and acquiring aircraft attitude data from airborne sensors to generate a synthetic airport visual scene image from the aircraft operator's perspective, at an angle equal to the external airport road. This invention improves the display accuracy of HUD-based synthetic visual scenes and increases their usability during aircraft operational phases.

Description

Technical Field

[0001] This invention belongs to the field of aircraft navigation and positioning technology, specifically relating to a synthetic visual correction method and system based on a head-up display. Background Technology

[0002] A head-up display (HUD) is a crucial component of an airborne display system. Through processing and calculation of information from various sensors, the HUD displays flight guidance symbols and composite visual images. These symbols and images are projected onto a transparent composite mirror directly in front of the pilot and focused at infinity. The guidance symbols and composite visual images are equiangular relative to the external view, providing the pilot with the most intuitive flight guidance information unmatched by any other display device.

[0003] Currently, when displaying composite visuals on the ground during surface operations, civil aircraft primarily rely on airborne GNSS (Global Navigation Satellite System) equipment for positioning data. However, the positioning accuracy of typical aircraft GNSS equipment is only 5-10 meters. During surface operations, even meter-level positioning errors can cause significant deviations between the airport runway lines, taxiway lines, etc., displayed on the head-up display (HUD) and the actual external scenery. This prevents pilots from obtaining accurate, angular runway and taxiway lines. The resulting display errors reduce the usability of HUD-based airborne composite visuals during surface operations. While China is introducing Ground Based Augmentation Systems (GBAS) to improve positioning accuracy to sub-meter levels, only a very small number of airports in China have implemented GBAS. Considering the need for most airports without GBAS to use HUDs for surface guidance during operations, further research is needed on HUD-based airborne composite visual correction methods and systems. Summary of the Invention

[0004] In view of this, the present invention provides a synthetic visual scene correction method and system based on a head-up display. When the aircraft is in the field operation phase, the airborne visual sensor acquires images of the airport environment around the aircraft, extracts airport ground marker features from the images and maps them to corresponding elements in the airport ground feature database, calculates the current geographical location of the aircraft using the geographical location information of the element in the database and its relative position information in the image, corrects the aircraft's positioning error using this information, and then transmits the position information to the airborne synthetic visual scene system to draw an isometric field guidance synthetic visual scene image displayed on the head-up display, thereby improving the accuracy of the synthetic visual scene guidance on the head-up display during the field operation phase and enhancing the pilot's situational awareness during the field operation phase.

[0005] The technical solution of this invention is as follows:

[0006] A synthetic scene correction method based on a head-up display includes the following steps:

[0007] Step 1: Determine if the aircraft is currently in the surface operation phase;

[0008] Step 2: When the aircraft is in the field operation phase, acquire images of the airport environment around the aircraft from the airborne visual sensors.

[0009] Step 3: Perform image processing on the acquired airport environment image to identify airport ground marking features;

[0010] Step 4: Based on the aircraft's current position output by the airborne positioning equipment, search for the area for feature matching from the airport ground marker feature database;

[0011] Step 5: Match the elements of this area in the airport ground marker feature database with the airport ground marker features in the airport environment image to find the database element corresponding to the airport ground marker features in the database;

[0012] Step 6: Calculate the relative position of the aircraft with respect to the airport ground marker features using the airport ground marker feature information in the airport environment image;

[0013] Step 7: Calculate the current geographical location of the aircraft by using the geographical location information of the corresponding elements of the airport ground features and their relative position information with the current location of the aircraft.

[0014] Step 8: Compare the calculated current geographical location information of the aircraft with the geographical location information of the aircraft output by the airborne positioning equipment to determine whether the aircraft positioning is accurate and obtain the aircraft positioning error.

[0015] Step 9: When the positioning error is within the allowable range, the calculated current geographical location information of the aircraft is used to correct the aircraft positioning error;

[0016] If the positioning error exceeds the allowable range, a positioning error alarm is generated;

[0017] Step 10: Transmit the corrected aircraft positioning information to the airborne synthetic vision system, acquire aircraft attitude data from airborne sensors, generate a synthetic airport vision image from the perspective of the aircraft operator at the same angle as the external airport road, and display the synthetic vision image on the head-up display.

[0018] Furthermore, in step 2, the airport environment image is acquired based on one or more airborne vision sensors, and the airport environment image is an airport image in front of or to the side of the aircraft.

[0019] Furthermore, the visual sensor is a sensor in the visible light band and / or a sensor in the infrared band.

[0020] Furthermore, it was identified that the airport ground marking features contain at least three feature points.

[0021] Furthermore, the area for feature matching is searched from the airport ground feature database and determined based on the positioning information output by the airborne positioning equipment and the maximum positioning error of the positioning equipment.

[0022] Furthermore, the relative position information of the aircraft relative to the airport ground marking features is calculated based on the relative position of the airport ground marking features in the airport environment image, the installation position and angle information of the visual sensor relative to the aircraft reference point, and the visual sensor imaging parameters. The relative position information includes distance and direction.

[0023] Furthermore, the synthesized visual image is based on reading an airport map database, using the aircraft's current position as a base point, and is drawn according to the aircraft's attitude and altitude information; the drawn elements include at least images of the road edge and centerline in front of the aircraft, and the name of the road the aircraft is currently on, the name of the target road, and the turning instructions are displayed in a prominent position in the synthesized visual image.

[0024] Furthermore, the synthesized visual image also includes flight parameter symbols for ground speed, attitude, and altitude generated based on the aircraft's parameters, and these flight parameter symbols are superimposed on the synthesized visual image.

[0025] The present invention also proposes a head-up display-based synthetic scene correction system for implementing the above-mentioned head-up display-based synthetic scene correction method, comprising:

[0026] The data interface module connects to the visual sensor, airborne positioning equipment and airport data link to acquire information on the aircraft’s current flight phase, positioning data and airport environment images.

[0027] The data storage module is used to store the airport ground feature database, airport map database, visual sensor installation and imaging parameters, and system logs;

[0028] The visual correction module, connected to the data interface module, is used to process airport environment images collected by one or more visual sensors, identify airport ground marking features, and read the airport ground feature database to find the area for feature matching. It matches the elements of the area in the database with the airport ground marking features in the image, finds the corresponding database element, and calculates the geographical location information of the aircraft's current position through the geographical location information of the element and its relative position information with the current position of the aircraft, thereby correcting the aircraft's positioning error.

[0029] The graphics processing module, connected to the visual correction module, is used to receive corrected aircraft positioning data, read airport map database, generate composite airport visual images, draw road edges and centerlines that are at the same angle as the external scene, and draw the name of the road where the aircraft is currently located, the name of the target road, and turning instructions.

[0030] The head-up display module, connected to the graphics processing module, serves as a carrier for displaying the synthetic visual image directly in front of the aircraft operator.

[0031] Furthermore, the visual correction module is also used to output a positioning error alarm message when the difference between the calculated current geographical location information of the aircraft and the geographical location information of the aircraft output by the airborne positioning device exceeds the allowable range;

[0032] The graphics processing module is also used to receive positioning error alarm information output by the visual correction module, and draw error alarm symbols at prominent positions in the synthesized visual image to prompt the aircraft operator that the current aircraft positioning is incorrect.

[0033] The graphics processing module is also used to receive aircraft parameters, generate ground speed, attitude, and altitude flight parameter symbols, and overlay them onto the synthesized visual image.

[0034] The beneficial effects of this invention are:

[0035] 1) Given that the current aircraft positioning accuracy at most airports does not meet the requirements for isometric display of airborne synthetic visuals on head-up displays for scene guidance, this study breaks through the method of improving aircraft positioning accuracy during the field operation phase, improves the display accuracy of synthetic visuals based on head-up displays, and increases the availability of synthetic visuals based on head-up displays during the aircraft field operation phase.

[0036] 2) The method and system of this application are compatible with current airport facilities and airborne equipment, and can utilize existing airborne augmented vision systems to carry out airport environmental image acquisition without increasing aircraft modification costs or requiring additional airport facilities. They are applicable to most airports in China.

[0037] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present application. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a diagram illustrating the architecture of a synthetic visual correction system based on a head-up display, according to an embodiment of this application.

[0040] Figure 2 This is a schematic diagram illustrating airport ground marker feature recognition in an airport environment image according to an embodiment of this application.

[0041] Figure 3 This is a diagram showing the correspondence between airport ground marker features in an airport environment image and elements in an airport ground feature database, as described in an embodiment of this application. Detailed Implementation

[0042] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0043] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0044] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0045] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0046] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0047] This embodiment of a synthetic visual correction method based on a head-up display includes the following steps:

[0048] Step 1: Determine if the aircraft is currently in the surface operation phase;

[0049] Step 2: When the aircraft is in the field operation phase, acquire images of the airport environment around the aircraft from the airborne visual sensors.

[0050] Step 3: Perform image processing on the acquired airport environment images to identify airport ground marking features, such as... Figure 2 As shown;

[0051] Step 4: Based on the aircraft's current position output by the airborne positioning equipment, search for areas for feature matching in the airport's ground feature database, such as... Figure 3 As shown in S3;

[0052] Step 5: Match the elements of this area in the airport ground feature database with the airport ground marker features in the airport environment image, and find the database elements corresponding to the airport ground marker features in the database. The correspondence is as follows: Figure 3 As shown in A1;

[0053] Step 6: Calculate the relative position of the aircraft with respect to the airport ground marker features using the airport ground marker feature information in the airport environment image;

[0054] Step 7: Calculate the geographical location information of the aircraft's current location by using the geographical location information of the corresponding elements of the airport ground features and their relative position information with respect to the aircraft's current location.

[0055] Step 8: Compare the calculated current geographical location information of the aircraft with the geographical location information of the aircraft output by the airborne positioning equipment to determine whether the aircraft positioning is accurate.

[0056] Step 9: If the difference between the calculated current geographical location information of the aircraft and the geographical location information of the aircraft output by the airborne positioning equipment is within the allowable range, then the calculated current geographical location information of the aircraft is used to correct the aircraft positioning error.

[0057] If the difference between the calculated current geographical location information of the aircraft and the geographical location information output by the airborne positioning equipment exceeds the allowable range, an aircraft positioning error alarm will be displayed on the head-up display.

[0058] Step 10: Transmit the corrected aircraft positioning information to the airborne synthetic vision system, acquire aircraft attitude data from airborne sensors, generate a synthetic airport vision image from the perspective of the aircraft operator at the same angle as the external airport road, and display the synthetic vision image on the head-up display.

[0059] In this embodiment, the determination of whether the aircraft is currently in the operational phase is made by comprehensively using airborne inertial navigation and atmospheric sensor data, wheel-borne signals, and airport air traffic control (ATC) data to determine the current flight phase of the aircraft.

[0060] In this embodiment, the acquisition of airport environment images around the aircraft from airborne visual sensors can be achieved using one or more visual sensors to acquire airport environment images in front of or to the side of the aircraft.

[0061] In this embodiment, the visual sensor can be a sensor in the visible light band or a sensor in the infrared band.

[0062] In this embodiment, the identification of airport ground marking features includes at least three feature points.

[0063] In this embodiment, the area for feature matching is searched from the airport ground feature database and determined based on the longitude and latitude information output by the airborne positioning device and the maximum positioning error of the device.

[0064] In this embodiment, the calculation of the relative position information of the aircraft relative to the airport ground marking features requires the use of information such as the relative position of the airport ground marking features in the airport environment image, the installation position and angle information of the visual sensor relative to the aircraft reference point, and the imaging parameters of the visual sensor. The calculated relative position information includes information such as distance and direction.

[0065] In this embodiment, the synthesized visual image is displayed on the head-up display, the airport map database is read, and based on the aircraft's current position as the base point and the aircraft's attitude and altitude information, at least the information including the road edge line and center line in front of the aircraft is drawn. The name of the road where the aircraft is currently located, the name of the target road, and the turning instructions are displayed in a prominent position on the image.

[0066] In this embodiment, when the synthesized visual image is displayed on the head-up display, flight parameter symbols such as ground speed, attitude, and altitude are also generated based on the aircraft's parameters and superimposed on the synthesized visual image.

[0067] In one embodiment, the present invention also proposes a synthetic scene correction system for implementing the synthetic scene correction method based on a head-up display in the above embodiments, comprising:

[0068] The data interface module connects to the visual sensor, airborne positioning equipment and airport data link to acquire the aircraft’s current flight phase, positioning data and airport environment images.

[0069] The data storage module is used to store the airport ground feature database, airport map database, visual sensor installation and imaging parameters, and system logs;

[0070] The visual correction module, connected to the data interface module, is used to process airport environment images collected by one or more visual sensors, identify airport ground marking features, and read the airport ground feature database to find the area for feature matching. It matches the elements of the area in the database with the airport ground marking features in the image, finds the corresponding database element, and calculates the geographical location information of the aircraft's current position through the geographical location information of the element and its relative position information with the current position of the aircraft, thereby correcting the aircraft's positioning error.

[0071] The graphics processing module, connected to the visual correction module, is used to receive corrected aircraft positioning data, read the airport map database, generate a composite visual image of the airport, draw road edges and centerlines that are at the same angle as the external scene, and draw the name of the road where the aircraft is currently located, the name of the target road, and the turning instructions.

[0072] The head-up display module, connected to the graphics processing module, serves as a carrier for displaying the synthetic visual image directly in front of the aircraft operator.

[0073] In this embodiment, the visual correction module is also used to output a positioning error alarm message when the difference between the calculated current geographical location information of the aircraft and the geographical location information of the aircraft output by the airborne positioning device exceeds the allowable range.

[0074] In this embodiment, the graphics processing module is also used to receive positioning error alarm information output by the visual correction module, and draw an error alarm symbol at a prominent position in the synthesized visual image to prompt the aircraft operator that the current aircraft positioning is incorrect.

[0075] In this embodiment, the graphics processing module is also used to receive aircraft parameters, generate flight parameter symbols such as ground speed, attitude, and altitude, and overlay them onto the synthesized visual image.

[0076] In one embodiment, the present invention also provides a synthetic visual correction system based on a head-up display, installed inside an aircraft, such as... Figure 1 As shown, it includes a data interface module, a view correction module, a graphics processing module, a data storage module, and a head-up display module.

[0077] Specifically, the data interface module is connected to the aircraft's inertial navigation, ground speed, radio altitude, and atmospheric sensors, and receives aircraft positioning data through GNSS positioning equipment. It also receives images of the airport environment surrounding the aircraft through an enhanced visual system. The visual correction module is connected to the data interface module, data storage module, and graphics processing module. The graphics processing module is connected to the data storage module and the aircraft cockpit head-up display module.

[0078] The aircraft acquires images of the surrounding airport through an enhanced visual system. Then, the synthetic visual correction system based on a head-up display in this example performs feature recognition and matching to complete the aircraft's positioning correction, and generates and displays a synthetic visual scene guidance image based on the head-up display.

[0079] The data interface module receives avionics data from sensors such as inertial navigation, radio altimeter, GNSS, and atmospheric instruments to determine whether the aircraft is in the field operation phase. It then transmits the aircraft parameters to the vision correction module and the graphics processing module. It also receives airport environment images output by the enhanced vision system and transmits them to the vision correction module for correction of the synthesized vision.

[0080] When the aircraft is in the field operation phase, the vision correction module receives the airport environment image output by the enhanced vision system, identifies the airport ground features in the image, and reads the airport ground feature database to find the area for feature matching. It matches the elements of the area in the database with the airport ground marker features in the image to find the corresponding database element. Through the geographical location information of the element and its relative position information with the current position of the aircraft, the geographical location information of the aircraft is calculated.

[0081] If the difference between the calculated current geographical location information of the aircraft and the geographical location information of the aircraft output by the airborne positioning equipment is within the allowable range, the calculated current geographical location information of the aircraft is used to correct the aircraft positioning error.

[0082] If the difference between the calculated current geographical location information of the aircraft and the geographical location information of the aircraft output by the airborne positioning equipment exceeds the allowable range, an aircraft positioning error alarm message will be output to the graphics processing module.

[0083] The graphics processing module receives aircraft parameters and corrected aircraft positioning information, reads the airport map database, and, based on the aircraft's current longitude, latitude, altitude, and the three-dimensional digital model of the earth, draws a composite visual image from the perspective of the aircraft operator at the same angle as the external airport road, including the road edge line and center line, and draws the name of the road where the aircraft is currently located, the name of the target road, and the turning instructions.

[0084] The head-up display module receives composite visual images and displays them directly in front of the aircraft operator.

[0085] Specifically, when generating the composite visual image, the graphics processing module also generates flight parameter symbols such as ground speed, attitude, and altitude based on the aircraft parameters, and overlays them onto the composite visual image.

[0086] Specifically, after receiving the aircraft positioning error alarm information output by the visual correction module, the graphics processing module draws an error alarm symbol in a prominent position in the synthesized visual image to alert the aircraft operator that the current aircraft positioning is incorrect.

[0087] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for synthetic scene correction based on a head-up display, characterized in that, Includes the following steps: Step 1: Determine if the aircraft is currently in the surface operation phase; Step 2: When the aircraft is in the field operation phase, acquire images of the airport environment around the aircraft from the airborne visual sensors. Step 3: Perform image processing on the acquired airport environment image to identify airport ground marking features; Step 4: Based on the aircraft's current position output by the airborne positioning equipment, search for the area for feature matching from the airport ground marker feature database; Step 5: Match the elements of this area in the airport ground marker feature database with the airport ground marker features in the airport environment image to find the database element corresponding to the airport ground marker features in the database; Step 6: Calculate the relative position of the aircraft with respect to the airport ground marker features using the airport ground marker feature information in the airport environment image; Step 7: Calculate the current geographical location of the aircraft by using the geographical location information of the corresponding elements of the airport ground features and their relative position information with the current location of the aircraft. Step 8: Compare the calculated current geographical location information of the aircraft with the geographical location information of the aircraft output by the airborne positioning equipment to determine whether the aircraft positioning is accurate and obtain the aircraft positioning error. Step 9: When the positioning error is within the allowable range, the calculated current geographical location information of the aircraft is used to correct the aircraft positioning error; If the positioning error exceeds the allowable range, a positioning error alarm is generated; Step 10: Transmit the corrected aircraft positioning information to the airborne synthetic vision system, acquire aircraft attitude data from airborne sensors, generate a synthetic airport vision image from the perspective of the aircraft operator at the same angle as the external airport road, and display the synthetic vision image on the head-up display.

2. The synthetic visual correction method according to claim 1, characterized in that, In step 2, the airport environment image is acquired based on one or more airborne vision sensors, and the airport environment image is an airport image in front of or to the side of the aircraft.

3. The synthetic visual correction method according to claim 2, characterized in that, The visual sensor is a sensor in the visible light band and / or a sensor in the infrared band.

4. The method for correcting synthesized visual scenes according to claim 1, characterized in that, The airport ground marking features are identified to contain at least three feature points.

5. The synthetic visual correction method according to claim 1, characterized in that, The region for feature matching is searched from the airport ground feature database and determined based on the positioning information output by the airborne positioning equipment and the maximum positioning error of the positioning equipment.

6. The synthetic visual correction method according to claim 1, characterized in that, The relative position information of the aircraft relative to airport ground marking features is calculated based on the relative position of airport ground marking features in the airport environment image, the installation position and angle information of the visual sensor relative to the aircraft reference point, and the imaging parameters of the visual sensor. The relative position information includes distance and direction.

7. The synthetic visual correction method according to claim 1, characterized in that, The synthesized visual image is based on reading the airport map database, using the aircraft's current position as the base point, and is drawn according to the aircraft's attitude and altitude information; the drawn elements include at least the images of the road edge and centerline in front of the aircraft, and the name of the road where the aircraft is currently located, the name of the target road, and the turning instructions are displayed in a prominent position in the synthesized visual image.

8. The synthetic visual correction method according to claim 7, characterized in that, The synthesized visual image also includes flight parameter symbols for ground speed, attitude, and altitude generated based on the aircraft's parameters, which are superimposed on the synthesized visual image.

9. A head-up display-based synthetic scene correction system for implementing the synthetic scene correction method based on any one of claims 1-8, characterized in that, include: The data interface module connects to the visual sensor, airborne positioning equipment and airport data link to acquire information on the aircraft’s current flight phase, positioning data and airport environment images. The data storage module is used to store the airport ground feature database, airport map database, visual sensor installation and imaging parameters, and system logs; The visual correction module, connected to the data interface module, is used to process airport environment images collected by one or more visual sensors, identify airport ground marking features, and read the airport ground feature database to find the area for feature matching. It matches the elements of the area in the database with the airport ground marking features in the image, finds the corresponding database element, and calculates the geographical location information of the aircraft's current position through the geographical location information of the element and its relative position information with the current position of the aircraft, thereby correcting the aircraft's positioning error. The graphics processing module, connected to the visual correction module, is used to receive corrected aircraft positioning data, read airport map database, generate composite airport visual images, draw road edges and centerlines that are at the same angle as the external scene, and draw the name of the road where the aircraft is currently located, the name of the target road, and turning instructions. The head-up display module, connected to the graphics processing module, serves as a carrier for displaying the synthetic visual image directly in front of the aircraft operator.

10. The synthetic visual correction system according to claim 9, characterized in that, The vision correction module is also used to output a positioning error alarm message when the difference between the calculated current geographical location information of the aircraft and the geographical location information of the aircraft output by the airborne positioning device exceeds the allowable range. The graphics processing module is also used to receive positioning error alarm information output by the visual correction module, and draw error alarm symbols at prominent positions in the synthesized visual image to prompt the aircraft operator that the current aircraft positioning is incorrect. The graphics processing module is also used to receive aircraft parameters, generate ground speed, attitude, and altitude flight parameter symbols, and overlay them onto the synthesized visual image.

Images