Avionics Display Systems

Abstract

The present invention is entitled “AVIATION ELECTRONICS DISPLAY SYSTEM.” The avionics display system generally relates to techniques for displaying an image to a pilot by generating a semi-transparent image to be overlaid on top of the pilot's view; determining whether the pilot's view includes one or more locations of stored visual references that include a source of color-based information; overlaying the semi-transparent image on top of the pilot's view; and modifying the semi-transparent image to convey the color-based information to the pilot.

Description

Technical Field

[0001] The present disclosure relates generally to avionics display systems, and more particularly to an avionics display system including one or more visual systems. Background Art

[0002] Aircraft display systems are used in many modern aircraft to assist pilots in performing various tasks. Modern displays include near-eye (NTE) displays, such as head-mounted displays (HMDs), which may form part of the pilot's helmet, head-up displays (HUDs), and other types of displays in which the pilot is able to view the surrounding environment through the display itself.

[0003] The display system may present "enhanced" information to the pilot using a visual system (VS). More specifically, a visual system allows information to be presented to the pilot that would not otherwise be easily visible to the pilot. One such example of a visual system is an enhanced vision system (EVS). An enhanced vision system typically overlays images onto the view of the environment seen by the pilot in order to present information to the pilot that would otherwise not be visible to the pilot due to, for example, low light conditions or adverse weather conditions. Typically, the EVS overlays information onto the view of the environment outside the aircraft that the pilot sees through the NTE display. Some examples of the types of information that may be presented to the pilot via the EVS include a night vision image stream captured using an infrared (IR) camera or a thermal image stream captured using a thermal imaging camera. The captured image stream of the EVS is overlaid on top of the view of the environment that the pilot sees through the NTE display.

[0004] Another example of a visual system is a synthetic vision system (SVS). A synthetic vision system may overlay computer-generated images onto a pilot's view of the environment. These computer-generated images may be semi-transparent, enabling the pilot to view the external environment through the overlay image while still having the benefit of the information presented by the computer-generated image. For example, a computer-generated image may present information to the pilot about airport features (such as runway locations, aircraft building locations) or terrain features (such as hazardous or overhead terrain) that the pilot might not otherwise have been able to see due to, for example, poor visibility conditions. Other types of visual systems are known, such as combined EVS / SVS visual systems.

[0005] In conventional vision systems, the overlaid images are typically monochrome. For example, in the case of an overlaid EVS image presenting a night vision image to a pilot, a green monochrome night vision image may be overlaid on top of the environment view seen by the pilot through the NTE display.

[0006] In some cases, color-based information shown in the pilot's environment view may be used by the pilot to make decisions (e.g., allowing the pilot to distinguish between a runway illuminated by white lights and a taxiway illuminated by blue and / or green lights). In these cases, the monochrome image superimposed by the visual system may obscure the color-based information presented to the pilot from these color-based information sources, and the pilot may have to turn off the visual system in order to obtain the color-based information.

[0007] Therefore, it is desirable to reduce the likelihood that color-based information will be obscured by the vision system displayed on an aircraft display.Other desirable features and characteristics will become apparent from the ensuing detailed description and appended claims. Summary of the invention

[0008] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description.

[0009] In an exemplary embodiment, an avionics display system is provided. The avionics display system includes a display module for displaying an image superimposed on a pilot's view. The avionics display system also includes a visual system that is operably in communication with the display module, the visual system being configured to generate a semi-transparent image and causing the display to superimpose the generated semi-transparent image on the pilot's view. The avionics display system also includes a navigation database that is operably in communication with the visual system, the navigation database storing information related to the position and boundaries of a visual reference in the pilot's view that would otherwise be obscured by the superimposed image, the visual reference including a source of color-based information. The visual system is configured to modify the generated semi-transparent image in the pilot's view using the information related to the position, boundaries, and / or color of the visual reference to transmit the color-based information to the pilot.

[0010] In another exemplary embodiment, a method of displaying an image to a pilot is provided, comprising: generating, using a processor, a semi-transparent image to be overlaid on top of a pilot's view; determining, using a processor, whether the pilot's view includes one or more locations of stored visual references, the visual references including a source of color-based information; overlaying, using a processor, the semi-transparent image on top of the pilot's view; and modifying the semi-transparent image to convey the color-based information to the pilot.

[0011] In another exemplary embodiment, a method for displaying an image to a pilot is provided. The method includes the step of using a processor to generate a semi-transparent image to be superimposed on top of the pilot's view. The method also includes the step of using a processor to identify one or more visual references that are sources of color-based information in the pilot's view. The method also includes the step of using a processor to determine the location of the visual reference identified in the pilot's view. The method also includes the step of using a processor to superimpose the semi-transparent image on top of the pilot's view. The method also includes the step of modifying the semi-transparent image to transmit the color-based information to the pilot. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] A more complete understanding of the subject matter may be derived from the following detailed description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like elements, and wherein:

[0013] Figure 1 A conventional image generated by a vision system is shown;

[0014] Figure 2 shows a schematic diagram of an aircraft display system according to various embodiments;

[0015] Figure 3 shows representative images displayed by an aircraft display system according to various embodiments;

[0016] Figure 4A and Figure 4B shows a comparison between a conventional image generated by a vision system and an image generated by a vision system of an aircraft display system according to an exemplary embodiment;

[0017] Figure 5A and Figure 5B shows another comparison between a conventional image generated by a vision system and an image generated by a vision system of an aircraft display system according to an exemplary embodiment;

[0018] Figure 6 A flow chart showing a method according to an exemplary embodiment; and

[0019] Figure 7 Another flow chart of a method according to an exemplary embodiment is shown. DETAILED DESCRIPTION

[0020] The following specific embodiments are merely illustrative in nature and are not intended to limit the embodiments of the subject matter or the application and use of such embodiments. As used herein, the word "exemplary" means "used as an example, instance or illustration". Therefore, any embodiment described herein as "exemplary" is not necessarily understood to be preferred or advantageous over other embodiments. All embodiments described herein are exemplary embodiments provided to enable those skilled in the art to manufacture or use the systems and methods defined by the claims. As used herein, the term "module" refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device, either individually or in any combination, including but not limited to: application specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or grouped) and memories that execute one or more software or firmware programs, combinational logic circuits and / or other suitable components that provide the functionality. It is not intended to be bound by any express or implied theory presented in the aforementioned technical field, background technology, summary of the invention or the following specific embodiments.

[0021] For the sake of simplicity, conventional techniques and components may not be described in detail herein. In addition, any connecting lines and arrows shown in the various figures included herein are intended to represent example functional relationships and / or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in the embodiments of the present disclosure.

[0022] Figure 1 An exemplary image 100 generated by a conventional aircraft display system in conjunction with a vision system is shown. In image 100, a monochrome image, such as an IR "night vision" image, is generated by a vision system, such as an EVS, and then superimposed on top of the pilot's view of the environment as seen through the NTE display. The monochrome image is semi-transparent so that the pilot can see the terrain beneath the superimposed image as well as the information presented by the monochrome image.

[0023] like Figure 1 As can be seen in FIG. 1 , the pilot's environment view includes the runway 110 and one or more visual references including color-based information, which are shown in FIG. Figure 1exemplified by runway lighting 120 in FIG. In practice, airport runway lighting 120 has a white color to indicate the boundary of the runway to the pilot, and also to indicate to the pilot that the depicted boundary is the runway and not (for example) a taxiway, which may instead be indicated using green or blue lighting. However, because the monochrome semi-transparent image from the visual system is overlaid on top of the environmental view, the pilot cannot view the "white" color information of the runway lighting 120. Therefore, the pilot may have difficulty distinguishing whether the lights depict a runway (with white lighting); a taxiway (typically delineated with green or blue lighting); or a pedestrian walkway lighting (typically red). In order to obtain color-based information from the visual reference of the airport lighting, the pilot must deactivate the visual system to view this color-based information, thereby increasing pilot workload at a time when pilot workload is already high (landing).

[0024] It should be appreciated that the above-described issues may be experienced using any type of vision system imagery that is overlaid on top of visual references in the pilot's view in a manner that obscures color-based information for the pilot.

[0025] Figure 2 A functional schematic diagram of an avionics display system 200 is shown in accordance with various embodiments. The outline arrows in the figure are used to illustrate the operable connections between the various elements shown in the figure.

[0026] In an exemplary embodiment, the avionics display system 200 includes a navigation database 202. The navigation database 202 is configured to store information related to the location of one or more visual references that can provide color-based information to the pilot. The information stored in the navigation database includes information related to one or more of the location of each visual reference, the boundaries of the range that define the location of each respective visual reference, and the color of each respective visual reference. In an exemplary embodiment, the navigation database 202 stores the location, boundaries, and colors of airport runway lighting. For example, the navigation database 202 may store the location and color (typically white) associated with runway lighting at a particular airport. In addition and / or as an alternative, the navigation database 202 is configured to store the location, boundaries, and colors of taxiway lighting; airport lighting in walkways; and other sources of color-based information.

[0027] In an exemplary embodiment, navigation database 202 is configured to be updated when a new visual reference of interest is constructed. Updates to navigation database 202 may be performed manually by a pilot or technician, or may be performed automatically when a list of visual references is updated in a central server and a data transmission connection is formed between navigation database 202 and the central server. For example, when a new visual reference of interest is constructed (such as constructing new airport lighting at a particular airport), the location, boundaries, and color of the new visual reference are included in navigation database 202 when navigation database 202 is updated.

[0028] The avionics display system 200 also includes at least one visual system 204. In an exemplary embodiment, the at least one visual system 204 includes at least one of an enhanced vision system (EVS) or a synthetic vision system (SVS) or a combination of an enhanced vision system and a synthetic vision system. The at least one visual system 204 includes one or more processors, such as a graphics processor. The at least one visual system 204 is configured to generate an image to be displayed to the pilot in an overlaid manner on the pilot's view, as will be described in further detail below.

[0029] The avionics display system 200 also includes a display module 206 configured to display images generated by the at least one visual system 204. In an exemplary embodiment, the display module 206 includes a near-eye (NTE) display, such as a head-mounted display (HMD) or a head-up display (HUD), and the image presented to the pilot is superimposed on top of the environmental view seen by the pilot through the NTE display. The color of the superimposed image is typically monochrome. In another exemplary embodiment, the display module 206 includes a head-down display (HDD).

[0030] In an exemplary embodiment, the at least one vision system 204 is configured to be independent of Figure 2The at least one visual system 204 may generate images to be displayed on the display module 206 based on other systems and modules shown. For example, the at least one visual system 204 may generate images based on sensors 208, such as night vision or thermal cameras mounted on the aircraft. In additional or alternative exemplary embodiments, the at least one visual system 204 may generate images based on information obtained from the navigation database 202. For example, the at least one visual system 204 may obtain information related to a visual reference location, such as an airport runway. The at least one visual system 204 is then configured to perform an object recognition algorithm based on the stored visual reference and the location of the image source (e.g., an image obtained from a "low light" IR camera) to determine the location of the airport feature in the pilot's view through the display. The at least one visual system 204 is then configured to overlay a semi-transparent computer-generated image representing the airport feature on the pilot's view on the display module in the form of an "augmented reality" view to highlight the location of the airport feature to the pilot. In this way, even in low light or inclement weather conditions, the pilot is able to locate the airport feature with the help of the image displayed on the display module 206.

[0031] In an exemplary embodiment, the overlaid imagery may be automatically adjusted based on weather condition information obtained from an external source, such as a ground station, via receiver module 210. Specifically, at least one vision system 204 is configured to vary the translucency of the overlaid imagery based on the received weather conditions, with a less translucent (and therefore more recognizable) image being overlaid on the pilot's view in poor visibility conditions (e.g., during rain or fog), and a more translucent image being overlaid in better visibility conditions.

[0032] In use, after generating an image to be overlaid on the pilot's view via the display module 206, the at least one visual system 204 is configured to cross-reference the features detected by the object recognition algorithm with information stored in the database relating to the location of the visual reference of the color-based information source to determine whether any color-based information is obscured in the pilot's view due to the overlaid image. The at least one visual system 204 is then configured to modify the generated image in some manner to convey the color-based information to the pilot.

[0033] In an exemplary embodiment, the at least one visual system 204 is configured to identify any visual references in the pilot's view that include color-based information and then modify the generated image by disabling overlaying of semi-transparent images over the locations of those visual references in the pilot's view on the display module 206. The areas where overlay images may be disabled may be defined by location and boundary information associated with the identified visual references stored in the navigation database.

[0034] In an alternative embodiment, the navigation database 202 does not include stored information related to the location of one or more visual references that can provide color-based information to the pilot. In an alternative exemplary embodiment, the at least one visual system does not cross-reference the features detected by the object recognition algorithm with any stored information. Instead, the object recognition algorithm determines color-based information based on the recognition of the object, and the at least one visual system 204 is configured to modify the overlay image based on the recognition of the object. In this way, it can be ensured that color-based information from dynamic objects (such as other aircraft) is not obscured in the pilot's view. For example, the object recognition algorithm can identify the object in the pilot's view as runway lighting. Based on this recognition, the at least one visual system 204 is configured to modify the overlay image to transmit color-based information associated with the runway lighting to the pilot without cross-referencing the location of the identified object with the location and boundary information stored in the navigation database 202. Instead, the overlay image can be modified based on information obtained from the airport ground system and / or based on information obtained from onboard sensors.

[0035] Figure 3 A representative image 300 generated by an aerial vision system according to various exemplary embodiments is shown in FIG. Figure 3 As can be seen in the representative image 300, the runway 310, the runway lighting 320, and the taxiway lighting 330 are each identified by at least one visual system as constituting a visual reference, which includes color-based information based on the corresponding visual reference position information stored in the navigation database, and thus the area of ​​the superimposed image is selectively omitted. Specifically, defined areas 340 and 350 are omitted in the image, and these areas 340, 350 are defined by the position and boundary information associated with the runway lighting 320 and the taxiway lighting 330 stored in the navigation database, respectively.

[0036] The image is modified by selectively omitting regions 340 and 350 in the image overlaid on the pilot's view so that the pilot is able to view the color-based information presented by the visual references 320, 330 without these visual references being obscured by the overlaid image and without disabling the display of the rest of the overlaid image. In this way, the pilot is able to benefit from the color-based information presented by the visual references 320, 330 (in this case, the runway lighting and the taxiway lighting) as well as the information shown in the overlaid image generated by the at least one vision system 204. The pilot is therefore able to make an informed decision on how to operate the aircraft based on these two sources of information without increasing his workload.

[0037] Figure 4A and Figure 4BA comparison between a conventional image 400 generated by a conventional aircraft display system and an image 450 generated by an aircraft display system according to an exemplary embodiment is shown. Figure 4A As can be seen in the conventional image 400, any visual reference showing color-based information is obscured by the overlay image generated by the vision system. Figure 4B In an image 450 generated by an aircraft display system according to an exemplary embodiment, locations 460, 470 of visual references including color-based information are identified by an object recognition algorithm, and at least one vision system 204 omits image overlays in regions 480, 490 of the pilot's view associated with boundaries defined around these identified locations 460, 470 of visual references.

[0038] Additionally or alternatively, in an exemplary embodiment, the overlay image may be modified to convey color-based information in another manner. In an exemplary embodiment, the at least one visual system 204 is configured to present a text label with the overlay image in the area, the text label based on color information associated with the visual reference stored in the navigation database 202.

[0039] Figure 5A and Figure 5B A comparison between a conventional image 500 generated by a conventional aircraft display system and an image 550 generated by an aircraft display system according to an exemplary embodiment is shown. Figure 5A As can be seen in the conventional image 500, any visual reference showing color-based information is obscured by the overlay image generated by the vision system. Figure 5B In an image 550 generated by an aircraft display system according to an exemplary embodiment, a location 560, 570 of a visual reference including color-based information is detected, and the vision system 204 modifies the image 550 by displaying textual indicia 580, 590 adjacent to the boundaries of the identified visual reference. Figure 5A and Figure 5B In the example of FIG. 5 , a text label “BLUE” is presented next to visual reference position 560 stored in the navigation database corresponding to the color blue, and a text label “RED” is presented next to visual reference position 570 stored in the navigation database corresponding to the color red. In alternative exemplary embodiments, other forms of labels (such as numbers predetermined to correspond to specific colors) may be displayed on the generated image. In alternative exemplary embodiments, an audio description of the color-based information may be transmitted to the pilot.

[0040] In an alternative embodiment, information of the type of color and visual reference of the color-based information may be data linked to the aircraft using an appropriate communication mechanism from a ground-based information source. The at least one visual system 204 is configured to decode the uplink message and modify the overlay image to convey the information to the pilot.

[0041] See now Figure 6 , which shows a flow chart of a method S600 for displaying an image to a pilot.

[0042] In step S601, data for generating an image is obtained. In an exemplary embodiment, the data is obtained from a camera, such as a night vision camera or a thermal camera. In an exemplary embodiment, the data is obtained from a combination of sources, such as terrain and / or feature location information obtained from a navigation database and an image obtained from a night vision or thermal camera. After obtaining the data, the method proceeds to step S602.

[0043] At step S602, a semi-transparent image is generated using one or more visual systems to be displayed above the pilot's view. In an exemplary embodiment, the one or more visual systems may include an enhanced vision system (EVS); a synthetic vision system (SVS); and / or a combined EVS / SVS. After the image is generated, the method proceeds to step S603.

[0044] At step S603, one or more visual systems and position information stored in the navigation database is used to determine whether the pilot view includes one or more locations of visual references that include color-based information. This determination can be performed using an object recognition algorithm based on the expected locations of the stored visual references and the image data obtained from the camera. Exemplary visual references that include color-based information include runway lighting, taxiway lighting, and other types of airport lighting. After determining whether the pilot view includes one or more locations of visual references that include color-based information, the method proceeds to step S604.

[0045] At step S604, the generated image is superimposed on the pilot's view using a display module. In an exemplary embodiment, the display module is a near-eye (NTE) display, such as a head-up display (HUD) or a head-mounted display (HMD). The method then proceeds to step S605.

[0046] At step S605, the semi-transparent image is modified to convey the color-based information of the visual reference to the pilot. In an exemplary embodiment, the semi-transparent image is modified by omitting an area of ​​the pilot's view from the semi-transparent image, the omitted area corresponding to a predetermined boundary of one or more locations of a source of the color-based information associated with the visual reference in the pilot's view. In this way, the pilot is able to view the color-based information sources without being obscured by the overlaid semi-transparent image generated by the one or more visual systems.

[0047] Go to Figure 7 , which shows a flow chart of a method S700 for displaying an image to a pilot.

[0048] In step S701, data for generating an image is obtained. In an exemplary embodiment, the data is obtained from a camera, such as a night vision camera or a thermal camera. In an exemplary embodiment, the data is obtained from a combination of sources, such as terrain and / or feature location information obtained from a navigation database and an image obtained from a night vision or thermal camera. After obtaining the data, the method proceeds to step S702.

[0049] At step S702, an image is generated using one or more visual systems to be displayed above the pilot's view. In an exemplary embodiment, the one or more visual systems include an enhanced vision system (EVS); a synthetic vision system (SVS); or a combined EVS / SVS. After the image is generated, the method proceeds to step S703.

[0050] At step S703, one or more visual systems and position information stored in the navigation database are used to determine whether the pilot view includes one or more locations of visual references that include color-based information. Exemplary visual references that include color-based information include runway lighting, taxiway lighting, and other types of airport lighting. After determining whether the pilot view includes one or more locations of visual references that include color-based information, the method proceeds to step S704.

[0051] At step S704, the generated image is superimposed on the pilot's view using a display module. In an exemplary embodiment, the display module is a near-eye (NTE) display, such as a head-up display or a head-mounted display. The method then proceeds to step S705.

[0052] At step S705, the semi-transparent image is modified to convey the color-based information to the pilot. In step S705, the semi-transparent image is modified by displaying a marker using a display module proximate to the visual reference location. The displayed marker indicates to the pilot the color of the color-based information associated with the visual reference. In an exemplary embodiment, the displayed marker also indicates to the pilot the type of visual reference identified, such as runway lighting, taxiway lighting, etc.

[0053] Although at least one exemplary embodiment has been presented in the foregoing specific embodiments, it should be understood that there are a large number of variations. Specifically, there are other technologies, such as audio messages, etc., that present to the pilot an indication of color-based information about stored visual references. It should also be understood that one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of the present disclosure in any way. On the contrary, the foregoing detailed description will provide a convenient roadmap for implementing the exemplary one or more embodiments for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope of the present disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. An avionics display system, comprising: a display module, the display module being used to display an image superimposed on the pilot's view, the display module comprising a near-eye display; a visual system in operable communication with the display module, the visual system configured to generate a monochrome semi-transparent image and to cause the display module to overlay the generated monochrome semi-transparent image on the pilot view; and a navigation database in operable communication with the vision system, the navigation database configured to store information relating to the location, boundaries, and colors of visual references in the pilot's view that would otherwise be obscured by the overlaid monochrome semi-transparent image, the visual references comprising a source of color-based information; Wherein the visual system is configured to modify the generated monochrome semi-transparent image in the pilot's view using the stored information relating to the position, the boundaries and the color of the visual reference so as to convey the color-based information to the pilot.

2. The avionics display system of claim 1 , wherein the visual system is configured to modify the generated monochrome semi-transparent image in the pilot view to convey the color-based information to the pilot by selectively omitting areas of the generated monochrome semi-transparent image in the pilot view, the omitted areas being selected based on the position of the visual reference and the boundary information.

3. The avionics display system of claim 1 , wherein the visual system is configured to modify the generated monochrome semi-transparent image in the pilot's view to convey the color-based information to the pilot by displaying a marker in the pilot's view proximate to the location of the visual reference, wherein the form of the marker is based on the color information of the visual reference in the pilot's view.

4. The avionics display system of claim 1, wherein the vision system comprises a vision system selected from the group consisting of: an enhanced vision system; and a combined vision system comprising an enhanced vision system and a synthetic vision system. 5 . The avionics display system of claim 1 , wherein the vision system is configured to vary a translucency of the generated monochromatic translucent image based on the determined visibility condition.

6. A method of displaying an image to a pilot, comprising: generating, using a processor, a monochrome semi-transparent image to be overlaid on top of the pilot's view on the near-eye display; determining, using a processor, whether the pilot view includes one or more locations of a stored visual reference, the visual reference including a source of color-based information; overlaying the monochrome semi-transparent image on top of the pilot's view using a processor; as well as The monochrome semi-transparent image is modified to convey the color-based information to the pilot.

7. A method according to claim 6, wherein the step of modifying the monochrome semi-transparent image includes omitting one or more selected areas of the monochrome semi-transparent image, and the one or more selected areas of the monochrome semi-transparent image are determined based on stored position and boundary information associated with the one or more positions of the stored visual reference.

8. A method according to claim 6, wherein the step of modifying the monochrome semi-transparent image includes displaying one or more marks next to the one or more positions of the stored visual references, respectively, and the marks are based on the stored position and color information associated with the one or more visual references.

Images