Airborne Enhanced Vision Method

Abstract

The method of airborne enhanced vision of the present invention belongs to the technical field of vision display methods, and avoids safety accidents when the aircraft takes off or lands. It includes S101: acquiring first graphic data, second data and third data; S102: simulating the second graphic data corresponding to the first data in real time according to the second data and the third data; S103: determining the third graphic data in the first graphic data; S104: determining the fourth image data in the first graphic data; S105: performing image equalization processing on the fourth image data and identifying abnormal pixels; S106: treating the abnormal pixels as intruders, highlighting and / or marking the pixel interval of the intruder; S107: superimposing or merging the highlighted and / or marked pixels on the first graphic data, generating the fifth graphic data and transmitting it to the display device for display or display. The present invention is used to improve the function of the airborne display interface.

Description

Technical Field

[0001] The invention belongs to the technical field of visual display methods and relates to an airborne enhanced visual method. Background Art

[0002] A runway incursion is any incident where an aircraft, vehicle or pedestrian mistakenly appears on the takeoff or landing runway. Currently, airports use ground monitoring equipment and airport activity area safety systems to monitor runway incursions and send information to air traffic controllers, who then take necessary measures to remove foreign objects from the runway to avoid affecting aircraft safety. However, it is difficult for aircraft and pilots to detect runway incursions. If an emergency occurs, such as a failure of the runway incursion ground monitoring equipment or a problem with air traffic control, it is difficult for aircraft to proactively identify and avoid runway incursions.

[0003] At present, the enhanced vision system, referred to as EVS, is widely used on aircraft. It is mainly used in the aircraft approach, landing and taxiing phases to facilitate pilots to observe the situation around the airport, especially to help pilots observe the target landing runway more clearly. A typical EVS system includes imaging equipment, including but not limited to visible low-light television cameras, infrared cameras, or any other suitable light detection system capable of detecting light or electromagnetic radiation within or outside the visible spectrum. EVS receives light from the external vision and forms an EVS image, which is then displayed on the driver's display. It is used to enhance the pilot's situational awareness and improve flight safety.

[0004] Currently, the resolution of EVS is low and it is easily affected by fog and haze, resulting in the runway not being prominent in the EVS image. It is difficult for pilots to identify the runway location from the image and it is difficult to detect runway intrusion events by visual inspection.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a method for airborne enhanced vision to avoid safety accidents when the aircraft takes off or lands. The technical solution of this case has many technical benefits, as described below:

[0007] A method for airborne enhanced vision, the method comprising:

[0008] S101: respectively acquiring first graphic data captured in real time by a vision enhancement system on the airborne platform, second data of electronic sensor status information, and third data of an airport runway in a memory on the airborne platform;

[0009] S102: simulating second graphic data corresponding to the first data in real time according to the second data and the third data;

[0010] S103: Determine third graphic data in the first graphic data according to the second graphic data;

[0011] S104: determining fourth image data in the first image data according to the third image data;

[0012] S105: performing image equalization processing on the fourth image data and identifying abnormal pixels;

[0013] S106: treating the abnormal pixel points as intrusion objects, and highlighting and / or marking the pixel point intervals of the intrusion objects;

[0014] S107: The highlighted and / or marked pixel points are superimposed or merged on the first graphic data to generate fifth graphic data which is transmitted to a display device for presentation or display.

[0015] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0016] Generate an image of the runway and select a range with a smaller grayscale number based on the grayscale histogram of the runway image to form an abnormally prominent grayscale range on the runway, indicating that there are other objects different from the runway in the range, that is, intruders. Determine the abnormal range, mark the range in a prominent manner, and identify it in the enhanced vision original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 Enhanced vision system components diagram.

[0019] Figure 2 Runway enhancement display data processing diagram.

[0020] Figure 3 Example of EVS original image.

[0021] Figure 4 Example diagram of runway location ranges.

[0022] Figure 5 Example of runway location range in EVS image.

[0023] Figure 6 An example of the precise runway location calculated from the runway location range in the enhanced vision image.

[0024] Figure 7 Comparison of the original image and the runway enhanced display image;

[0025] Figure 8 Runways and their histograms;

[0026] Fig. 9 This is the screen displayed after correction. DETAILED DESCRIPTION

[0027] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0028] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present invention, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0029] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0030] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that aspects can be practiced without these specific details. In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise stated, "multiple" means two or more.

[0031] like Figure 3 As shown in FIG. 1 , since EVS is easily affected by external weather, the runway information in the image is blurred, and the pilot cannot find the runway from the image, which will cause the aircraft to make a go-around or fail to land safely.

[0032] like Figures 1 to 9 The method for airborne enhanced vision shown in the figure comprises:

[0033] S101: respectively acquiring first graphic data captured in real time by a vision enhancement system on the airborne platform, second data of electronic sensor status information, and third data of an airport runway in a memory on the airborne platform;

[0034] First graphical data, acquired by the enhanced vision system, EVS 1, based on the first signal, including range, angle and first data (aircraft position error), the first graphical data being onboard known field of view data, the EVS 1 comprising one or more sensors adapted to be mounted to the aircraft and configured to detect light signals external to the aircraft. The sensors may include visible low-light television cameras, infrared cameras, millimeter wave (MMW) cameras, or any other light sensing device capable of detecting light within or outside the visible spectrum, determining a horizontal viewing angle △h, a vertical viewing angle △v and a system and aircraft position error δ;

[0035] Second data, avionics sensors, such as the global positioning system (GPS), inertial navigation system, etc., obtain the aircraft's altitude, attitude and GPS position based on the second signal. Determine the aircraft's position in the spatial coordinate system ((fx1, fy1, fz1), data error is ε), attitude (roll r, pitch p) and heading (h) information;

[0036] The third data, obtained by the onboard data storage device 3 according to the third signal, such as runway related information,

[0037] Determine the position of the runway in the spatial coordinate system according to the third signal (the four vertex positions of the runway boundary, if it is a relative position, it needs to be converted into vertex position data: (△x1,△y1,△z1), (△x2,△y2,△z2), (△x3,△y3,△z3), (△x4,△y4,△z4), the data error is σ;

[0038] The processor 4 processes the above data. In the data processing stage, the display range of the runway in the EVS screen is calculated according to the camera imaging principle. The specific steps are as follows:

[0039] S102: Figure 4 As shown, second graphic data corresponding to the first data in real time is simulated according to the second data and the third data, and the second graphic data is, the simulated runway;

[0040] a) determining the display angle of the camera as the horizontal angle △h and the vertical angle △v according to the viewing angle (horizontal angle △h and vertical angle △v) of the EVS image in the first signal;

[0041] b) Determine the position of the camera in the three-dimensional space coordinate system as (fx1, fy1, fz1) according to the position information of the aircraft in the second signal (fx1, fy1, fz1);

[0042] c) Determine the camera rotation angle r and pitch angle p according to the aircraft attitude (roll r, pitch p) in the second signal;

[0043] d) determining the camera direction as h according to the heading (h) of the aircraft in the second signal;

[0044] In the spatial coordinate system, according to the spatial position information of the runway of the third signal ((△x1,△y1,△z1), (△x2,△y2,△z2), (△x3,△y3,△z3), (△x4,△y4,△z4)), and the aircraft position error ε, the runway relative position error σ, and the EVS installation error δ, the position range of the runway in the spatial coordinate system is determined to be ((△x1+ξ,△y1+ξ,△z1+ξ), (△x2+ξ,△y2+ξ,△z2+ξ), (△x3+ξ,△y3+ξ,△z3+ξ), (△x4+ξ,△y4+ξ,△z4+ξ)), where ξ=ε+data error σ+EVS installation error δ;

[0045] S103: Figure 4 and Figure 5 As shown, the third graphic data in the first graphic data is determined according to the second graphic data. The range image of the actual runway is determined, the field of view of the runway in the second image data is simulated, and the field of view of the actual runway in the first graphic data is marked. The field of view is a range of visualization.

[0046] The runway position range in the enhanced visual image is calculated based on the data in S101: (△x1 / , △y1 / ), (△x2 / ,△y2 / ), (△x3 / ,△y3 / ), (△x4 / ,△y4 / );

[0047] S104: Determine fourth image data in the first image data according to the third image data; the fourth image data is image data of a real runway, such as Figure 4 The area of ​​the runway boundary line in the middle. The generated runway position image is consistent with the EVS image display range. Traverse each pixel in the runway position image (quadrilateral) to find the four vertex positions of the quadrilateral (the boundary vertices of the four sides) as the position range of the runway in the EVS image (△x1 / , △y1 / ), (△x2 / ,△y2 / ), (△x3 / ,△y3 / ), (△x4 / ,△y4 / ). This range can ensure that the image of the runway in the EVS image is within this range, such as Figure 6 The area of ​​the runway boundary line is in the middle. Next, you just need to find the runway line in this range;

[0048] S105: Performing image equalization processing on the fourth image data and identifying abnormal pixels.

[0049] The runway obtained in S104 is within the position range in the EVS image. Since the runway lights are brighter than other areas within the range and there is no interference from other luminous lines near the runway, the runway boundary can be found by using methods including but not limited to Hough transform or Canny edge detection algorithm in image processing. After finding the runway boundary line in the EVS image, the runway is enhanced and drawn according to the position of the boundary line, such as increasing the intensity, color, shape or a combination thereof, so that it stands out in the EVS image, such as Figure 7 The source image is compared with the enhanced image.

[0050] After the runway is identified, histogram equalization is performed on the runway range image and runway intrusion is identified. The specific steps are as follows:

[0051] S106: treating the abnormal pixel points as intrusions, and highlighting and / or marking the pixel point intervals of the intrusions;

[0052] like Figure 8 As shown, an image within the runway range is obtained, the grayscale value of each pixel of the runway image is counted, and a grayscale histogram of the runway image is formed;

[0053] like Fig. 9As shown, the fourth image data is grayed out to 0-255, the pixel grayscale interval of the runway of the fourth image data is determined, and the abnormal interval is determined according to the grayscale interval as the pixel interval of the intruder. According to the grayscale histogram of the runway image, a range with a larger grayscale number is selected to form a clearer runway image, and the peak grayscale range of 97-255 is selected in the figure;

[0054] According to the grayscale histogram of the runway image, a range with a smaller grayscale number is selected to form an abnormally prominent grayscale range on the runway (the peak grayscale range of 141-255 is selected in the figure), indicating that there are other objects different from the runway in this range, that is, intruders. The abnormal range is determined and marked in a prominent way, and marked in the enhanced vision original image;

[0055] S107: The highlighted and / or marked pixel points are superimposed or merged on the first graphic data to generate fifth graphic data which is transmitted to a display device for presentation or display.

[0056] After the runway identification is completed, the generated enhanced visual runway and runway incursion highlight image are sent to the airborne display for display. Usually, the runway incursion is highlighted in a flashing manner, such as displaying / canceling the runway incursion highlight box at intervals of 0.5s.

[0057] The method provided by the present invention is described in detail above. Specific examples are used herein to illustrate the principle and implementation of the present invention, and the description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of invention, the invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the invention claims.

Claims

1. An airborne enhanced vision method, suitable for an airborne enhanced vision system to identify an airport runway, wherein the airborne device is equipped with a data storage device, an avionics sensor, an electronic sensor, a camera and a controller, and wherein: The data storage device stores the boundary information of the runway and the installation error δ of the camera, and the method comprises: S101: respectively obtaining first graphic data captured in real time by a visual enhancement system on the airborne platform, second data of electronic sensor status information, and third data of an airport runway in a memory on the airborne platform, wherein the controller determines first data according to data fed back by the electronic sensor, the first data being a position error ε between the visual enhancement system and the carrier aircraft, and the controller determines position information, roll, pitch, and heading of the carrier aircraft in a spatial coordinate system according to data fed back by the avionics sensor; boundary information of the runway includes (△x1, △y1, △z1), (△x2, △y2, △z2), (△x3, △y3, △z3), (△x4, △y4, △z4), and data error σ; S102: simulating second graphic data corresponding to the first data in real time according to the second data and the third data, and using the second graphic data as a simulated runway, wherein: Determine, according to the first image data, that the camera display angle is a horizontal angle △h and a vertical angle △v; Determine the position (fx1, fy1, fz1) of the camera in the three-dimensional space coordinate system according to the position information, determine the rotation angle r and the pitch angle p of the camera according to the roll and pitch, and determine the direction h of the camera according to the heading; The position range of the second graphic data in the space coordinate system is expressed as: (△x2+ξ,△y2+ξ,△z2+ξ),(△x3+ξ,△y3+ξ,△z3+ξ),(△x4+ξ, △y4+ξ, △z4+ξ)), (△x1+ξ, △y1+ξ, △z1+ξ), where the error ξ=ε+σ+δ; S103: the second graphic data is calibrated or marked in the first image data to determine the range image of the actual runway in the first image data, and the range image of the actual runway in the first image data is used as the third graphic data; S104: determining a range of real runway image data in the first graphic data according to the third graphic data, and using the range of the real runway image data as fourth image data, wherein the range of the real runway image data is determined in the first graphic data by a traversal algorithm using the third graphic data; S105: performing image equalization processing on the fourth image data and identifying abnormal pixels; S106: treating the abnormal pixel points as intrusion objects, and highlighting and / or marking the pixel point intervals of the intrusion objects; S107: The highlighted and / or marked pixel points are superimposed or merged on the first graphic data to generate fifth graphic data which is transmitted to a display device for presentation or display.

2. The method according to claim 1, characterized in that The method described in S103 includes: The model information of the carrier aircraft is obtained and an error value is determined, and the third graphic data is corrected according to the error value.

3. The method according to claim 1, characterized in that The method described in S104 includes: the third graphic data is transformed into the image data of the real runway in the first graphic data as the fourth image data through the Houng transformation method.

4. The method according to claim 1, characterized in that The method described in S105 includes: performing 0-255 grayscale processing on the fourth image data, determining the pixel grayscale interval of the fourth image data track, and determining the abnormal interval according to the grayscale interval as the intruder pixel point interval.

5. The method according to claim 1, characterized in that The method described in S104 includes: the third graphic data uses the Canny edge detection method to use the image data of the real runway in the first graphic data as the fourth image data.

Images