Visual perception enhancement system

Abstract

This application relates to a visual perception augmentation system. In some embodiments, a system within a cockpit of an aerial vehicle determines a field of view (FOV) of an aerial vehicle operator based on line-of-sight data associated with the aerial vehicle operator stationed within the cockpit of the aerial vehicle. The system determines display configuration information based on a set of one or more visual references derived from the FOV of the aerial vehicle operator. The system updates a projection display component positioned within the cockpit and within the FOV of the aerial vehicle operator to display at least visual representation information associated with the set of one or more visual references based on the display configuration information.

Description

Technical Field

[0001] This disclosure generally relates to a visual perception enhancement system and a visual perception enhancement system in the cockpit of an aircraft. Background Technology

[0002] Projection displays (such as head-up displays (HUDs) or windshields) present information on a transparent surface. Such displays can be included in aircraft to provide flight information to the aircraft operator, thereby reducing the need for the operator to view other instruments or displays within the aircraft. Summary of the Invention

[0003] One embodiment of this disclosure provides a visual perception enhancement system, comprising a first projection display component configured to be within the field of view (FOV) of a first aircraft operator; a second projection display component configured to be within the FOV of a second aircraft operator; a camera system configured to observe the line of sight of the first aircraft operator and the line of sight of the second aircraft operator; and one or more processors configured to: determine first display configuration information based on relative orientation information derived from aircraft data and based on a first set of one or more visual references derived from the FOV of the first aircraft operator; determine second display configuration information based on the relative orientation information and based on a second set of one or more visual references derived from the FOV of the second aircraft operator; update the first projection display component based on the first display configuration information to at least display first visual representation information associated with the first set of one or more visual references; and update the second projection display component based on the second display configuration information to at least display second visual representation information associated with the second set of one or more visual references.

[0004] Another embodiment of this disclosure provides a non-transitory computer-readable medium storing a set of instructions, including one or more instructions that, when executed by one or more processors of a system within the cockpit of an aircraft, cause the system to: determine the aircraft operator's field of view (FOV) based on line-of-sight data associated with the aircraft operator; determine display configuration information based on a set of one or more visual references derived from the aircraft operator's FOV, based on relative orientation information; and update projection display components positioned within the cockpit and the aircraft operator's FOV based on the display configuration information to at least display visual representation information associated with the set of one or more visual references.

[0005] Another embodiment of this disclosure provides a method performed by a system within the cockpit of an aircraft, comprising: determining the field of view (FOV) of the aircraft operator based on line-of-sight data associated with the aircraft operator; determining display configuration information based on a set of one or more visual references derived from the FOV of the aircraft operator; and updating a projection display component positioned within the cockpit and the FOV of the aircraft operator based on the display configuration information to at least display visual representation information associated with the set of one or more visual references.

[0006] The features, functions, and advantages already discussed can be implemented independently in various embodiments or combined in other embodiments, and further details of these embodiments can be seen with reference to the following description and figures. Attached Figure Description

[0007] Figures 1A-1I This is a diagram of an example implementation of a visual perception enhancement system.

[0008] Figure 2 This is a diagram of an example implementation of a visual perception enhancement system.

[0009] Figure 3 This is a diagram of an example environment in which the systems and / or methods described herein are implemented.

[0010] Figure 4 This is a diagram of an example component of a device associated with a visual perception enhancement system.

[0011] Figure 5 This is a flowchart of an example process associated with a visual perception enhancement system. Detailed Implementation

[0012] The following detailed description of the exemplary embodiments is with reference to the accompanying drawings. The same reference numerals in different drawings may identify the same or similar elements.

[0013] Aircraft operators (such as captains and first officers) can observe the external environment through one or more windshields while stationed in the cockpit to control the aircraft's flight operations. This enhances the operator's situational awareness, enabling informed flight control decisions. To further assist, many aircraft include projection displays positioned in front of the windshield in the cockpit, providing flight information that the operator can view while facing one or more windshields to observe the external environment.

[0014] In many cases, projection displays provide information about visual references in the external environment, but the placement of this information does not take into account the different fields of view (FOV) of the aircraft operator. For example, visual representations (such as icons) associated with visual references appear in the same relative orientation on each projection display to indicate the proximity of the visual reference to the aircraft. However, as seen by a particular aircraft operator, the positioning of the visual representation on a particular projection display is often not aligned with the orientation of the visual reference actually observed by that particular aircraft operator through one or more windshields of the aircraft.

[0015] This misalignment between the projected visual representation and the actual observed orientation of the visual reference can lead to confusion or misjudgment by the aircraft operator. When the displayed information does not accurately correspond to the external environment, the operator may misinterpret the proximity or orientation of the visual reference, which can affect situational awareness. This discrepancy can result in inaccurate or unnecessary flight maneuvers regarding navigation, positioning, alignment, or other types of aircraft actions related to the visual reference. For example, the operator might cause the aircraft to deviate from its flight path associated with the visual reference, after which the aircraft performs a corrective maneuver to realign itself with the path. This increases the computational load on the Flight Management System (FMS) and other guidance systems (such as the Flight Guidance System (FGS) or another guidance system) to calculate and implement the corrective maneuver, and may increase the aircraft's fuel consumption to perform the maneuver.

[0016] Some embodiments described herein include a visual perception enhancement system. The visual perception enhancement system is located within the cockpit of an aircraft. A first aircraft operator and a second aircraft operator may be stationed within the cockpit (e.g., within corresponding sections of the cockpit). The visual perception enhancement system includes a first projection display component located within the cockpit (e.g., within a first section of the cockpit), configured within the first aircraft operator's field of view (FOV) (e.g., when the first aircraft operator is stationed within the cockpit (e.g., at a first operator's station)), and the visual perception enhancement system also includes a second projection display component located within the cockpit (e.g., within a second section of the cockpit) and configured within the second aircraft operator's FOV (e.g., when the second aircraft operator is stationed within the cockpit (e.g., at a second operator's station)). The visual perception enhancement system includes a camera system comprising multiple cameras and located within the cockpit. The camera system is configured to observe the line of sight of a first aircraft operator (e.g., when the first aircraft operator is stationed in the cockpit) (this line of sight originates from the first aircraft operator's point of view (POV) (e.g., the apex of the first aircraft operator's FOV)) and the line of sight of a second aircraft operator (e.g., when the second aircraft operator is stationed in the cockpit) (this line of sight originates from the second aircraft operator's POV (e.g., the apex of the second aircraft operator's FOV)). The visual perception enhancement system includes one or more processors communicatively connected to the first projection display component, the second projection display component, and the camera system.

[0017] In some implementations, a visual perception enhancement system (e.g., using one or more processors) acquires first line-of-sight data captured by a camera system to determine the field of view (FOV) of a first aircraft operator. The visual perception enhancement system (e.g., using one or more processors) additionally determines first display configuration information associated with the visual reference and the FOV of the first aircraft operator, based on relative orientation information indicating an estimated relative orientation of the visual reference relative to the aircraft. The first display configuration information indicates a first display orientation for the visual reference. Therefore, the visual perception enhancement system (e.g., using one or more processors) updates a first projection display component based on the first display configuration information to at least display first visual representation information associated with the visual reference. This causes the first projection display component to display visual representation elements associated with the visual reference (as indicated by the first visual representation information), which will be positioned at the first display orientation (as indicated by the first display configuration information).

[0018] Furthermore, in some embodiments, the visual perception enhancement system (e.g., using one or more processors) acquires second line-of-sight data captured by a camera system to determine the field of view (FOV) of the second aircraft operator. The visual perception enhancement system (e.g., using one or more processors) additionally determines second display configuration information associated with the visual reference and the FOV of the second aircraft operator based on relative orientation information. The second display configuration information indicates a second display orientation for the visual reference. Therefore, the visual perception enhancement system (e.g., using one or more processors) updates a second projection display component based on the second display configuration information to at least display second visual representation information associated with the visual reference. This results in the second projection display component displaying visual representation elements associated with the visual reference (as indicated by the second visual representation information), which will be positioned at the second display orientation (as indicated by the second display configuration information).

[0019] In this way, visual representation elements associated with the visual reference are displayed on a first projection display component, such as appearing to a first aircraft operator aligned with the orientation of the visual reference actually observed by the first aircraft operator through one or more windshields of the aircraft. Furthermore, visual representation elements associated with the visual reference are displayed on a second projection display component, such as appearing to a second aircraft operator aligned with the orientation of the visual reference actually observed by the second aircraft operator through one or more windshields of the aircraft. This alignment between the displayed visual representation elements and the corresponding observed orientation of the visual reference reduces the likelihood of confusion or misjudgment by the aircraft operator. Therefore, the aircraft operator is more likely to accurately assess the proximity or direction of the visual reference, which improves the aircraft operator's situational awareness.

[0020] Therefore, visual perception enhancement systems can reduce many inaccurate or unnecessary flight maneuvers related to navigation, positioning, alignment, or other types of aircraft operations relative to visual references. For example, due to improved situational awareness, aircraft operators are less likely to cause the aircraft to deviate from its flight path associated with the visual reference. Consequently, corrective maneuvers to realign the aircraft with the flight path are less necessary. This reduces the computational load on the aircraft's FMS and other guidance systems (such as FGS or another guidance system) to calculate and implement any corrective maneuvers, and the aircraft's fuel efficiency is improved by performing fewer or no corrective maneuvers.

[0021] Figures 1A-1I This is a diagram of an example implementation 100 associated with a visual perception enhancement system. (See diagram below.) Figures 1A-1I As shown, Example Implementation 100 includes an aircraft that includes a visual perception enhancement system, an FMS, and an FGS. The following description, in conjunction with... Figure 2 and Figure 3Describe these systems and devices in more detail.

[0022] like Figure 1A As shown, the visual perception enhancement system is located within the cockpit of the aircraft. As further described herein, the visual perception enhancement system is configured to enhance the situational awareness of the aircraft's first operator (e.g., the aircraft's captain), such as when the first operator is stationed in the cockpit (e.g., acting as the first-in-command regarding flight operations of the aircraft) and / or the situational awareness of the aircraft's second operator (e.g., the aircraft's first officer), such as when the second operator is stationed in the cockpit (e.g., acting as a second-in-command regarding flight operations of the aircraft). Figure 1A As further shown, the visual perception enhancement system includes a first projection display component, a second projection display component, a camera system including multiple cameras, and / or one or more processors.

[0023] In some embodiments, a first projection display unit is located within the cockpit and configured to be within the first aircraft operator's field of view (FOV) when the first aircraft operator is stationed in the cockpit, and a second projection display unit is located within the cockpit and configured to be within the second aircraft operator's FOV when the second aircraft operator is stationed in the cockpit. Each projection display unit includes at least one of the following: a head-up display (HUD), a projection screen, and / or a projection component (e.g., connected to or integrated into another component of the aircraft, such as the aircraft's windshield). In some embodiments, the first and second projection display units are separate components, and the first projection display unit is configured to display information relevant to the first aircraft operator, and the second projection display unit is configured to display information relevant to the second aircraft operator.

[0024] In some embodiments, the camera system is positioned within the cockpit and configured to observe (e.g., using multiple cameras of the camera system) the line of sight of a first aircraft operator (originating from the first aircraft operator's POV) when the first aircraft operator is stationed in the cockpit, and to observe the line of sight of a second aircraft operator (originating from the second aircraft operator's POV) when a second aircraft operator is stationed in the cockpit. The camera system includes eye-tracking cameras, gaze detection cameras, and / or another type of camera configured to capture line-of-sight data of a person within the camera system's FOV. Line-of-sight data indicates, for example, the person's POV, which is the apex of the person's FOV. In some embodiments, the POV is the orientation of at least one of the person's eyes.

[0025] In some implementations, one or more processors are configured to control the visual perception enhancement system. For example, one or more processors are configured to control a first projection display component, a second projection display component, and / or a camera system. One or more processors are configured to perform the functions described herein. Figures 1A-1I One or more other operations as described above.

[0026] like Figure 1A As further illustrated, and via reference numeral 102, a camera system (e.g., using multiple cameras) captures first line-of-sight data and provides this data to one or more processors. The camera system captures the first line-of-sight data when the first aircraft operator is stationed in the cockpit (e.g., when the aircraft is in flight and the first aircraft operator is monitoring and / or controlling the aircraft's flight operations). Therefore, the first line-of-sight data indicates the POV (e.g., the orientation of at least one eye of the first aircraft operator, which is the apex of the first aircraft operator's FOV) for the first aircraft operator (e.g., when the first aircraft operator is stationed in the cockpit).

[0027] In some implementations, one or more processors cause the camera system to capture first line-of-sight data and provide the first line-of-sight data to the one or more processors. For example, one or more processors send control information to the camera system via a communication connection between the one or more processors and the camera system, and the camera system captures the first line-of-sight data in response to the control information. Thus, the camera system sends (e.g., in real-time or near real-time) the first line-of-sight data to one or more processors via the communication connection. In this way, one or more processors obtain the first line-of-sight data.

[0028] As shown in reference numeral 104, a camera system (e.g., using multiple cameras) captures second line-of-sight data and provides it to one or more processors. The camera system captures the second line-of-sight data when the second aircraft operator is stationed in the cockpit (e.g., when the aircraft is in flight and the second aircraft operator is monitoring and / or controlling the aircraft's flight operations). Therefore, the second line-of-sight data indicates the second aircraft operator's point of view (e.g., the orientation of at least one of the second aircraft operator's eyes, which is the apex of the second aircraft operator's field of view), for example, when the second aircraft operator is stationed in the cockpit.

[0029] In some implementations, one or more processors cause the camera system to capture second line-of-sight data and provide the second line-of-sight data to the one or more processors. For example, one or more processors send control information to the camera system via a communication connection between the one or more processors and the camera system, and the camera system captures second line-of-sight data in response to the control information. Thus, the camera system sends (e.g., in real-time or near real-time) second line-of-sight data to one or more processors via the communication connection. In this way, one or more processors obtain the second line-of-sight data.

[0030] As shown in reference numeral 106, the FMS (and optionally the FGS) provides aircraft data to one or more processors. Aircraft data includes navigation data, guidance data, or control data. Navigation data indicates the aircraft's current dynamic state (e.g., navigation state) (e.g., derived from data obtained from one or more sensors and / or navigation aids of the aircraft), as well as other static navigation information (such as that relating to terrain, airports, waypoints, and / or routes) obtained from one or more data structures (e.g., included in and / or accessible to the FMS) and / or other dynamic navigation information wirelessly transmitted to the aircraft (e.g., relating to the positioning of other aircraft or other moving objects). Guidance data indicates the flight path the aircraft is to follow (e.g., determined by the FMS, for example, based on a flight plan). Control data indicates the required inputs (e.g., inputs to the FMS or other control systems of the aircraft) that enable the aircraft to align with the flight path (e.g., as indicated by the guidance data).

[0031] In some implementations, one or more processors cause the FMS to provide aircraft data to one or more processors. For example, one or more processors send control information to the FMS via a communication connection between the processors and the FMS, and the FMS, in response to the control information, sends (e.g., in real-time or near real-time) aircraft data to one or more processors via the communication connection. In this way, one or more processors obtain aircraft data.

[0032] In some implementations, such as Figure 1BAs shown, and via reference numeral 108, a visual perception enhancement system (e.g., using one or more processors) determines the FOV of the first aircraft operator (e.g., based on first line-of-sight data). For example, the visual perception enhancement system processes the first line-of-sight data (e.g., using one or more analysis techniques, such as analysis using a Kalman filter or another time-series analysis technique) to determine the estimated azimuth (and optional azimuth error) and / or estimated orientation (and optional orientation error) of the first aircraft operator's POV. Therefore, the visual perception enhancement system determines the FOV of the first aircraft operator based on the estimated azimuth (and optional azimuth error) and / or estimated orientation (and optional orientation error) of the first aircraft operator's POV. That is, the visual perception enhancement system determines the FOV of the first aircraft operator based on a representative range of human visual angles, a visual origin point (e.g., as indicated by the estimated azimuth of the first aircraft operator's POV), and a visual direction (e.g., as indicated by the estimated orientation of the first aircraft operator's POV).

[0033] In some implementations, as shown by reference numeral 110, a visual perception enhancement system (e.g., using one or more processors) determines the FOV of a second aircraft operator, for example, based on second line-of-sight data. For instance, the visual perception enhancement system processes the second line-of-sight data (e.g., using one or more analysis techniques, such as analysis using a Kalman filter or another time-series analysis technique) to determine the estimated azimuth (and optionally, the estimated azimuth error) and / or the estimated orientation (and optionally, the estimated orientation error) of the second aircraft operator's POV. Therefore, the visual perception enhancement system determines the FOV of the second aircraft operator based on the estimated azimuth (and optionally, the estimated azimuth error) and / or the estimated orientation (and optionally, the estimated orientation error) of the second aircraft operator's POV. That is, the visual perception enhancement system determines the FOV of the second aircraft operator based on a representative range of human visual angles, a visual origin point (e.g., as indicated by the estimated azimuth of the second aircraft operator's POV), and a visual direction (e.g., as indicated by the estimated orientation of the second aircraft operator's POV).

[0034] In some implementations, such as Figure 1CAs shown, and by reference to numeral 112, the visual perception enhancement system (e.g., using one or more processors) determines relative orientation information associated with multiple visual references. As further described herein, the multiple visual references include a first group of one or more visual references and / or a second group of one or more visual references. Each visual reference is either a real-world visual reference or a virtual visual reference. A real-world visual reference corresponds to an object or feature in the environment in which the aircraft is flying, such as a horizon, runway, airport, terrain, or another aircraft. A virtual visual reference does not directly correspond to a physical entity in the environment but is associated with other information relevant to the flying aircraft, such as the aircraft's flight path, waypoint orientation, or other information associated with the aircraft's navigation, guidance, or control. The relative orientation information indicates the estimated relative orientation of each of the multiple visual references relative to the aircraft.

[0035] Relative bearing information is derived from aircraft data (e.g., aircraft data obtained from the FMS by a visual perception enhancement system, as discussed in this article). Figure 1A (As described in reference numeral 106). For example, the visual perception enhancement system determines the estimated bearing of an aircraft based on aircraft data. The visual perception enhancement system processes the aircraft data (e.g., using one or more analysis techniques, such as analysis techniques using Kalman filters or another time series analysis technique) to determine the estimated bearing of the aircraft (and optionally, the estimated bearing error). Furthermore, the visual perception enhancement system determines the bearing (and optionally, the bearing error) of a visual reference among multiple visual references (e.g., as indicated by the aircraft data) based on the aircraft data. Therefore, the visual perception enhancement system determines the estimated relative bearing of the visual references with respect to the aircraft based on the estimated bearing of the aircraft and the bearings of the visual references.

[0036] In some implementations, such as Figure 1DAs shown, and via reference numeral 114, a visual perception enhancement system (e.g., using one or more processors) determines a first group of one or more visual references from a plurality of visual references. For example, the visual perception enhancement system determines a group of visual references from a plurality of visual references based on the first aircraft operator's FOV and relative orientation information. This group of visual references includes visual references aligned with the first aircraft operator's FOV (e.g., visual references that can be seen by the first aircraft operator). Furthermore, the visual perception enhancement system determines (e.g., based on aircraft data) flight operation parameters (e.g., associated with the aircraft's navigation, guidance, or control) and selects a specific group of one or more visual references from the group of visual references associated with the flight operation parameters as the first group of one or more visual references. For example, when the aircraft is in a specific flight phase (e.g., takeoff, climb, cruise, or landing, and other examples), the visual perception enhancement system determines the flight operation parameters associated with the specific flight phase and selects a specific group of one or more visual references associated with the flight operation parameters as the first group of one or more visual references. In this way, the first group of one or more visual references includes different visual references at different times.

[0037] Additionally, or alternatively, the visual perception enhancement system identifies one or more selection parameters associated with the first projection display component. For example, a first aircraft operator interacts with an input component of the first projection display component to input one or more selection parameters, and the first projection display component sends one or more selection parameters to one or more processors of the visual perception enhancement system (e.g., via a communication connection between the first projection display component and one or more processors). One or more selection parameters include, for example, selection parameters associated with flight operation parameters (e.g., indicating a specific set of flight operation parameters to be displayed); selection parameters associated with the display preferences of a first aircraft operator (e.g., indicating a set of information to be displayed); selection parameters associated with weather events related to the aircraft's flight path (e.g., indicating whether to display weather event information); selection parameters associated with the orientation of obstacles related to the aircraft's flight path (e.g., indicating whether to display obstacle orientation information); selection parameters associated with the aircraft's performance (e.g., indicating whether to display aircraft performance information); selection parameters associated with the aircraft's state (e.g., internal state and / or external state) (e.g., indicating whether to display aircraft state information); selection parameters associated with the aircraft's predicted performance (e.g., indicating whether to display predicted aircraft performance information); selection parameters associated with control information associated with the aircraft (e.g., indicating whether to display control information); selection parameters associated with the aircraft's navigation (e.g., indicating whether to display navigation information); and / or selection parameters associated with the orientation of other aircraft (e.g., indicating whether to display the orientation information of other aircraft). Therefore, the visual perception enhancement system selects a specific set of one or more visual references associated with one or more selection parameters from the visual reference group as the first set of one or more visual references.

[0038] In some implementations, the visual perception enhancement system excludes one or more visual references from a first set of one or more visual references. For example, the visual perception enhancement system excludes a visual reference from the first set of one or more visual references when the combination error (which combines two or more of the estimated azimuth error of the first operator's POV, the estimated orientation error of the first operator's POV, the estimated azimuth error of the aircraft, and / or the azimuth error of the visual reference) meets (e.g., greater than or equal to) a combination error threshold. Alternatively, the visual perception enhancement system may label or otherwise indicate that a visual reference is an "uncertain" visual reference and include it in the first set of one or more visual references instead of excluding it.

[0039] In some implementations, as shown by reference numeral 116, a visual perception enhancement system (e.g., using one or more processors) determines first visual representation information associated with a first set of one or more visual references. For example, the visual perception enhancement system determines the corresponding characteristics of the visual references in the first set of one or more visual references (e.g., whether each visual reference is a real-world visual reference or a virtual visual reference). Therefore, the visual perception enhancement system generates first visual representation information based on the corresponding characteristics of the visual references in the first set of one or more visual references, including one or more visual representation elements for each visual reference in the first set of one or more visual references. Each visual representation element is a specific shape or set of shapes (with specific shading, shading, marking, pattern, or other type of visual element). Therefore, one or more visual representation elements corresponding to a visual reference are configured to visually represent the visual reference (e.g., when the first visual representation information is displayed). When a visual reference is marked as an indeterminate visual reference, as described herein, one or more visual representation elements may include an indication that the visual reference is an indeterminate visual reference (e.g., by using a dashed pattern, a specific color, or other type of indication).

[0040] In some implementations, such as Figure 1E As shown, and via reference numeral 118, the visual perception enhancement system (e.g., using one or more processors) determines a second group of one or more visual references from a plurality of visual references (e.g., which will be displayed by a second projection display component, as further described herein). For example, the visual perception enhancement system determines a group of visual references from a plurality of visual references based on the FOV and relative orientation information of the second aircraft operator. This group of visual references includes visual references aligned with the FOV of the second aircraft operator (e.g., visual references that can be seen by the second aircraft operator). Furthermore, the visual perception enhancement system determines (e.g., based on aircraft data) flight operation parameters (e.g., associated with the aircraft's navigation, guidance, or control) and selects a specific group of one or more visual references associated with the flight operation parameters from the group of visual references as the second group of one or more visual references. For example, when the aircraft is in a specific flight phase (e.g., takeoff, climb, cruise, or landing, and other examples), the visual perception enhancement system determines the flight operation parameters associated with the specific flight phase and selects a specific group of one or more visual references associated with the flight operation parameters as the second group of one or more visual references. In this way, the second group of one or more visual references includes different visual references at different times.

[0041] Additionally, or alternatively, the visual perception enhancement system identifies one or more selection parameters associated with the second projection display component. For example, a second aircraft operator interacts with an input component of the second projection display component to input one or more selection parameters, and the second projection display component sends the one or more selection parameters to one or more processors of the visual perception enhancement system (e.g., via a communication connection between the second projection display component and one or more processors). The one or more selection parameters include those described herein. Figure 1D The same or similar selection parameters as those described in reference numeral 116. Therefore, the visual perception enhancement system selects a specific set of one or more visual references associated with one or more selection parameters from the visual reference group as a second set of one or more visual references.

[0042] In some implementations, the visual perception enhancement system excludes one or more visual references from a second set of one or more visual references. For example, the visual perception enhancement system excludes a visual reference from the second set of one or more visual references when the combination error (which combines two or more of the estimated azimuth error of the second operator's POV, the estimated orientation error of the second operator's POV, the estimated azimuth error of the aircraft, and / or the azimuth error of the visual reference) meets (e.g., greater than or equal to) a combination error threshold. Alternatively, the visual perception enhancement system may label or otherwise indicate that a visual reference is an indeterminate visual reference and include it in the second set of one or more visual references, rather than excluding it.

[0043] In some implementations, as shown by reference numeral 120, a visual perception enhancement system (e.g., using one or more processors) determines second visual representation information associated with a second set of one or more visual references. For example, the visual perception enhancement system determines the corresponding characteristics of the visual references in the second set of one or more visual references (e.g., whether each visual reference is a real-world visual reference or a virtual visual reference). Therefore, the visual perception enhancement system generates second visual representation information based on the corresponding characteristics of the visual references in the second set of one or more visual references, including one or more visual representation elements for each visual reference in the second set of one or more visual references. Each visual representation element is a specific shape or set of shapes (with specific shading, shading, marking, pattern, or other types of visual elements). Therefore, one or more visual representation elements corresponding to a visual reference are configured to visually represent the visual reference (e.g., when the second visual representation information is displayed). When a visual reference is marked as an indeterminate visual reference, as described herein, one or more visual representation elements may include an indication that the visual reference is an indeterminate visual reference (e.g., by using a dashed pattern, a specific color, or other type of indication).

[0044] In some implementations, such as Figure 1F As shown, and via reference numeral 122, the visual perception enhancement system (e.g., using one or more processors) determines first display configuration information (e.g., based on at least one of the first aircraft operator's FOV, relative orientation information, or first visual representation information). For example, the visual perception enhancement system targets a visual reference in a first set of one or more visual references and determines the display scale of that visual reference based on the relative orientation information. The display scale indicates how large a visual representation element corresponding to the visual reference will appear (e.g., when the first visual representation information is displayed), based on the distance between the visual reference and the aircraft (e.g., as indicated by the relative orientation information). Additionally, or alternatively, the visual perception enhancement system determines an alignment vector associated with the first aircraft operator and the visual reference for the visual reference based on the relative orientation information. The alignment vector indicates a line between the first aircraft operator (e.g., the first aircraft operator's POV) and the visual reference, based on the orientation of the visual reference and the aircraft (e.g., as indicated by the relative orientation information) and the first aircraft operator's FOV. Therefore, the visual perception enhancement system determines the display orientation of the visual reference (on the first projection display component) based on the alignment vector. The display azimuth is aligned with the alignment vector so that the display azimuth follows the line between the first aircraft operator and the visual reference. In this way, the display configuration information is associated with the FOV of the first set of one or more visual references and the first aircraft operator.

[0045] In some implementations, such as Figure 1G As shown, and via reference numeral 124, the visual perception enhancement system (e.g., using one or more processors) transmits first visual representation information and first display configuration information. For example, the visual perception enhancement system transmits the first visual representation information and first display configuration information to the first projection display component via a communication connection between the first projected display component and one or more processors. In some embodiments, the visual perception enhancement system transmits the first visual representation information and first display configuration information to update the first projection display component, for example, updating the first projection display component based on the first display configuration information to display the first visual representation information.

[0046] Therefore, as indicated by reference numeral 126, the first projection display component displays first visual representation information based on first display configuration information. For example, for each visual reference in a first group of one or more visual references, the first projection display component displays one or more visual representation elements associated with the visual reference (e.g., as indicated by the visual representation information) at the display scale and / or display orientation of the visual reference (e.g., as indicated by the first display configuration information). Thus, the first projection display component is updated to display one or more visual representation elements corresponding to each visual reference in that group of one or more visual references (e.g., aligned with the POV of the first aircraft operator and the relative orientation of the visual references with respect to the aircraft).

[0047] In some implementations, such as Figure 1H As shown, and via reference numeral 128, the visual perception enhancement system (e.g., using one or more processors) determines second display configuration information (e.g., based on at least one of the second aircraft operator's FOV, relative orientation information, or second visual representation information). For example, the visual perception enhancement system targets a visual reference in a second set of one or more visual references and determines the display scale of the visual reference based on the relative orientation information. The display scale indicates how large a visual representation element corresponding to the visual reference will appear (e.g., when the second visual representation information is displayed), based on the distance between the visual reference and the aircraft (e.g., as indicated by the relative orientation information). Additionally, or alternatively, the visual perception enhancement system determines an alignment vector associated with the second aircraft operator and the visual reference for the visual reference based on the relative orientation information. The alignment vector indicates a line between the second aircraft operator (e.g., the second aircraft operator's POV) and the visual reference, based on the orientation of the visual reference and the aircraft (e.g., as indicated by the relative orientation information) and the second aircraft operator's FOV. Therefore, the visual perception enhancement system determines the display orientation of the visual reference (on the second projection display component) based on the alignment vector. The display azimuth is aligned with the alignment vector so that the display azimuth follows the line between the second aircraft operator and the visual reference. In this way, the display configuration information is associated with the second set of one or more visual references and the FOV of the second aircraft operator.

[0048] In some implementations, such as Figure 1IAs shown, and via reference numeral 130, the visual perception enhancement system (e.g., using one or more processors) transmits second visual representation information and second display configuration information. For example, the visual perception enhancement system transmits the second visual representation information and second display configuration information to a second projection display component, for example, via a communication connection between one or more processors and the second projection display component. In some embodiments, the visual perception enhancement system transmits the second visual representation information and second display configuration information to update the second projection display component, for example, updating the second projection display component based on the second display configuration information to display the second visual representation information.

[0049] Therefore, as indicated by reference numeral 132, the second projection display component displays second visual representation information based on the second display configuration information. For example, for each visual reference in a second group of one or more visual references, the second projection display component displays one or more visual representation elements associated with the visual reference (e.g., as indicated by the visual representation information) at the display scale and / or display orientation of the visual reference (e.g., as indicated by the second display configuration information). Thus, the second projection display component is updated to display one or more visual representation elements corresponding to each visual reference in that group of one or more visual references (e.g., aligned with the POV of the second aircraft operator and the relative orientation of the visual references with respect to the aircraft).

[0050] In some implementations, the visual perception enhancement system iteratively performs the procedures outlined herein. Figures 1A-1I At least some of the operations described. For example, after updating the first projection display component, the visual perception enhancement system obtains updated first line-of-sight data and updated aircraft data; determines an updated field of view (FOV) for the first aircraft operator; determines updated relative orientation information; determines an updated first set of one or more visual references; determines updated first visual representation information associated with the updated first set of one or more visual references; determines updated first display configuration information; and / or updates the first projection display component based on the first display configuration information to display the updated first visual representation information. As another example, after updating the second projection display component, the visual perception enhancement system obtains updated second line-of-sight data and updated aircraft data; determines an updated FOV for the second aircraft operator; determines updated relative orientation information; determines an updated second set of one or more visual references; determines updated second visual representation information associated with the updated second set of one or more visual references; determines updated second display configuration information; and / or updates the second projection display component based on the second display configuration information to display the updated second visual representation information.

[0051] As indicated above, Figures 1A-1I Provided as an example. Other examples may be related to... Figures 1A-1I The descriptions are different.

[0052] Figure 2 This is a diagram of an example implementation 200 associated with a visual perception enhancement system. (See diagram for example.) Figure 2 As shown, the visual perception enhancement system is located within the cockpit of the aircraft. A first aircraft operator and a second aircraft operator are housed within the cockpit. The visual perception enhancement system includes a first projection display unit located within the cockpit and configured within the first aircraft operator's field of view (FOV), and a second projection display unit located within the cockpit and configured within the second aircraft operator's FOV. The visual perception enhancement system includes a camera system located within the cockpit and including a first camera configured to observe the line of sight of the first aircraft operator and a second camera configured to observe the line of sight of the second aircraft operator. The visual perception enhancement system includes one or more processors communicatively connected to the first projection display unit, the second projection display unit, at least one first camera, and at least one second camera.

[0053] like Figure 2 As further illustrated, visual references are objects or features in the environment in which the aircraft is flying. For example... Figure 2 As further shown, the visual reference appears (or will appear) to be in different orientations of the respective FOVs of the first and second aircraft operators (e.g., due to the different orientations of the first and second aircraft operators within the cockpit of the aircraft).

[0054] As this article is about Figures 1A-1I The processor acquires first line-of-sight data captured by a first camera of a camera system to determine the field of view (FOV) of a first aircraft operator; determines first display configuration information associated with the visual reference and the FOV of the first aircraft operator based on relative orientation information indicating an estimated relative orientation of the visual reference relative to the aircraft; and updates a first projection display component based on the first display configuration information to display first visual representation information associated with the visual reference. This causes the first projection display component to display one or more visual representation elements associated with the visual reference (as indicated by the first visual representation information), which will be positioned at the first display orientation (as indicated by the first display configuration information).

[0055] In addition, as this article discusses Figures 1A-1IFurthermore, one or more processors obtain second line-of-sight data captured by a second camera of the camera system to determine the field of view (FOV) of the second aircraft operator; based on relative orientation information, determine second display configuration information associated with the visual reference and the FOV of the second aircraft operator; and update a second projection display component based on the second display configuration information to display second visual representation information associated with the visual reference. This causes the second projection display component to display one or more visual representation elements associated with the visual reference (as indicated by the second visual representation information), which will be positioned at a second display orientation as indicated by the second display configuration information.

[0056] Therefore, as Figure 2 As further shown, the first display orientation is aligned with a first alignment vector between the first aircraft operator and the visual reference, and the second display orientation is aligned with a second alignment vector between the second aircraft operator and the visual reference. In this way, the visual representation elements corresponding to the visual reference appear visually aligned to both the first and second aircraft operators.

[0057] As indicated above, Figure 2 This is provided as an example. Other examples may be related to... Figure 2 The content described is different.

[0058] Figure 3 This is a diagram of an example environment 300 in which the systems and / or methods described herein are implemented. Figure 3 As shown, environment 300 includes aircraft 310, visual perception enhancement system 320, FMS 330, and / or FGS 340. The devices in environment 300 are interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0059] Aircraft 310 includes any suitable vehicle and / or equipment capable of flight. Aircraft 310 includes, for example, aircraft (e.g., jet aircraft, propeller aircraft, gliders, etc.), helicopters, rocket ships, spacecraft, space shuttles, airships, or inflatable airships, as well as other examples of airborne vehicles and / or airborne equipment capable of flight. Aircraft 310 includes a cockpit, in which at least one of a first or second aircraft operator will be housed, as described herein.

[0060] The visual perception enhancement system 320 includes one or more devices capable of receiving, generating, storing, transmitting, processing, and / or providing information, as described elsewhere herein. The visual perception enhancement system 320 includes at least one of a first projection display component, a second projection display component, a camera system including multiple cameras, or one or more processors, as described herein. In some embodiments, the visual perception enhancement system 320 includes a computing device, such as a wireless communication device, a mobile phone, a user equipment, a laptop computer, a tablet computer, a desktop computer, a server device, or a similar type of device.

[0061] FMS 330 includes one or more devices capable of receiving, generating, storing, transmitting, processing, and / or providing information, as described elsewhere herein. FMS 330 includes one or more devices, such as a Flight Management Computer (FMC), a Control Display Unit (CDU), and / or other devices that automate one or more of the flight planning, navigation, and operational tasks of aircraft 310. FMS 330 is configured to generate, store, and / or update flight plans. FMS 330 is configured to guide or otherwise control aircraft 310 according to the flight plan.

[0062] FGS 340 includes one or more devices capable of receiving, generating, storing, transmitting, processing, and / or providing information, as described elsewhere herein. FGS 340 includes one or more devices, such as a Flight Guidance Computer (FGC), a CDU, and / or other devices that automate one or more flight guidance tasks of the aircraft 310.

[0063] Figure 3 The number and arrangement of devices and networks shown are provided as examples. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or [other arrangements]. Figure 3 The devices and / or networks shown are arranged differently. Furthermore, Figure 3 The two or more devices shown can be implemented within a single device, or Figure 3 The single device shown can be implemented as multiple distributed devices. Additionally, or alternatively, a group of devices in environment 300 (e.g., one or more devices) can perform one or more functions described as being performed by another group of devices in environment 300.

[0064] Figure 4This is a diagram of example components of device 400 associated with a visual perception enhancement system. Device 400 corresponds to aircraft 310, visual perception enhancement system 320, FMS 330, and / or FGS 340. In some embodiments, aircraft 310, visual perception enhancement system 320, FMS 330, and / or FGS 340 include one or more devices 400 and / or one or more components of device 400. Figure 4 As shown, device 400 includes bus 410, processor 420, memory 430, input unit 440, output unit 450 and / or communication unit 460.

[0065] Bus 410 includes one or more components that enable wired and / or wireless communication between components of device 400. Bus 410 will... Figure 4 Two or more components are coupled together (e.g., via operational coupling, communication coupling, electronic coupling, and / or electrical coupling). For example, bus 410 includes electrical connections (e.g., wires, traces, and / or leads) and / or wireless buses. Processor 420 includes a central processing unit, graphics processing unit, microprocessor, controller, microcontroller, digital signal processor, field-programmable gate array, application-specific integrated circuit, and / or another type of processing unit. Processor 420 is implemented in hardware, firmware, or a combination of hardware and software. In some embodiments, processor 420 includes one or more processors capable of being programmed to perform one or more operations or processes described elsewhere herein.

[0066] Memory 430 includes volatile and / or non-volatile memory. For example, memory 430 includes random access memory (RAM), read-only memory (ROM), hard disk drive, and / or another type of memory (e.g., flash memory, magnetic storage, and / or optical storage). Memory 430 includes internal memory (e.g., RAM, ROM, or hard disk drive) and / or removable memory (e.g., removable via a Universal Serial Bus connection). In some embodiments, memory 430 is a non-transitory computer-readable medium. Memory 430 stores information related to the operation of device 400, one or more instructions, and / or software (e.g., one or more software applications). In some embodiments, memory 430 includes one or more memories coupled (e.g., communication coupling) to one or more processors (e.g., processor 420), for example via bus 410. The communication coupling between processor 420 and memory 430 enables processor 420 to read and / or process information stored in memory 430 and / or store information in memory 430.

[0067] Input component 440 enables device 400 to receive input (e.g., user input and / or sensed input). For example, input component 440 includes a touchscreen, keyboard, keypad, mouse, button, microphone, switch, sensor, GPS sensor, GNSS sensor, accelerometer, gyroscope, and / or actuator. Output component 450 enables device 400 to provide output (e.g., via a display, speaker, and / or LED). Communication component 460 enables device 400 to communicate with other devices via wired and / or wireless connections. For example, communication component 460 includes a receiver, transmitter, transceiver, modem, network interface card, and / or antenna.

[0068] Device 400 performs one or more of the operations or processes described herein. For example, a non-transitory computer-readable medium (e.g., memory 430) stores a set of instructions (e.g., one or more instructions or codes) for execution by processor 420. Processor 420 executes the set of instructions to perform one or more of the operations or processes described herein. In some embodiments, execution of the set of instructions by one or more processors 420 causes one or more processors 420 and / or device 400 to perform one or more of the operations or processes described herein. In some embodiments, hardwired circuitry is used in place of or in combination with instructions to perform one or more of the operations or processes described herein. Additionally, or alternatively, processor 420 is configured to perform one or more of the operations or processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0069] Figure 4 The number and arrangement of components shown are provided as examples. Device 400 may include additional components, fewer components, different components, or components with different arrangements (as opposed to...). Figure 4 Compared to those components shown). Additionally, or alternatively, a set of components of device 400 (e.g., one or more components) performs one or more functions described as being performed by another set of components of device 400.

[0070] Figure 5 This is a flowchart of an example process 500 associated with a visual perception enhancement system. In some implementations, Figure 5 One or more process blocks are executed by systems within the cockpit of the aircraft (aircraft 310), such as the visual perception enhancement system 320. In some embodiments, Figure 5 One or more process blocks are executed by another device or group of devices that are separate from or include the system (e.g., another system of the aircraft (e.g., FMS 330 and / or FGS 340)). Additionally, or alternatively, Figure 5One or more process blocks are executed by one or more components of the device 400 (e.g., processor 420, memory 430, input component 440, output component 450 and / or communication component 460).

[0071] The system also includes a first projection display unit (located in the cockpit and configured to be within the first aircraft operator's field of view when the first aircraft operator is stationed in the cockpit), a second projection display unit (located in the cockpit and configured to be within the second aircraft operator's field of view when the second aircraft operator is stationed in the cockpit), and a camera system (configured to observe the line of sight of the first aircraft operator and the line of sight of the second aircraft operator).

[0072] like Figure 5 As shown, process 500 includes determining the FOV of the first aircraft operator based on first line-of-sight data captured by the camera system (block 510). For example, the system determines the FOV of the first aircraft operator based on first line-of-sight data captured by the camera system, as described above.

[0073] like Figure 5 As further shown, process 500 includes determining the FOV of the second aircraft operator based on second line-of-sight data captured by the camera system (block 520). For example, the system determines the FOV of the second aircraft operator based on second line-of-sight data captured by the camera system, as described above.

[0074] like Figure 5 As further shown, process 500 includes determining first display configuration information based on a first set of one or more visual references derived from the FOV of the first aircraft operator (block 530). For example, the system determines the first display configuration information based on relative orientation information derived from aircraft data, and based on a first set of one or more visual references derived from the FOV of the first aircraft operator, as described above.

[0075] like Figure 5 As further shown, process 500 includes determining second display configuration information based on a second set of one or more visual references derived from the FOV of the second aircraft operator (block 540). For example, the system determines the second display configuration information based on relative orientation information, using a second set of one or more visual references derived from the FOV of the second aircraft operator, as described above.

[0076] like Figure 5As further shown, process 500 includes updating a first projection display component based on first display configuration information to display at least first visual representation information associated with a first group of one or more visual references (block 550). For example, the system updates the first projection display component based on the first display configuration information to display at least first visual representation information associated with a first group of one or more visual references, as described above.

[0077] like Figure 5 As further shown, process 500 includes updating the second projection display component based on the second display configuration information to display at least second visual representation information associated with the second set of one or more visual references (block 560). For example, the system updates the second projection display component based on the second display configuration information to display at least second visual representation information associated with the second set of one or more visual references, as described above.

[0078] Process 500 includes additional implementations, such as any single implementation or any combination of implementations described below and / or in relation to one or more other processes described elsewhere herein.

[0079] In a first embodiment, process 500 includes deriving the first aircraft operator's FOV based on the first aircraft operator's POV.

[0080] In the second embodiment, alone or in combination with the first embodiment, the aircraft data includes navigation data, guidance data, or control data.

[0081] In the third embodiment, either alone or in combination with one or more of the first and second embodiments, the relative orientation information indicates the estimated relative orientation of each visual reference in the first set of one or more visual references and the second set of one or more visual references relative to the aircraft.

[0082] In the fourth embodiment, alone or in combination with one or more of the first to third embodiments, process 500 includes determining the estimated bearing of the aircraft and the bearing of a visual reference in a first set of one or more visual references and a second set of one or more visual references, and determining relative bearing information based on the estimated bearing of the aircraft and the bearing of the visual references to indicate the estimated relative bearing of the visual references relative to the aircraft.

[0083] In the fifth embodiment, alone or in combination with one or more of the first to fourth embodiments, process 500 includes determining a visual reference group based on a first FOV and relative orientation information of a first aircraft operator, determining flight operation parameters based on aircraft data, and selecting a first group of one or more visual references associated with the flight operation parameters from the visual reference group.

[0084] In the sixth embodiment, alone or in combination with one or more of the first to fifth embodiments, process 500 includes determining a visual reference group based on the FOV and relative orientation information of the first aircraft operator, identifying selection parameters associated with the first projection display component, and selecting one or more visual references associated with the selection parameters from the visual reference group.

[0085] In the seventh embodiment, the selection parameters, alone or in combination with one or more of the first to sixth embodiments, include selection parameters associated with flight operation parameters, selection parameters associated with the display preferences of the first aircraft operator, selection parameters associated with weather events related to the aircraft's flight path, selection parameters associated with the location of obstacles related to the aircraft's flight path, selection parameters associated with the aircraft's performance, selection parameters associated with the aircraft's state, selection parameters associated with the aircraft's predictive performance, selection parameters associated with control information associated with the aircraft, selection parameters associated with the aircraft's navigation, or selection parameters associated with the location of other aircraft.

[0086] In the eighth embodiment, alone or in combination with one or more of the first to seventh embodiments, process 500 includes determining corresponding characteristics of visual references in a first group of one or more visual references, and generating first visual representation information based on the corresponding characteristics of the visual references in the first group of one or more visual references, the information including one or more visual representation elements for each visual reference in the first group of one or more visual references.

[0087] In the ninth embodiment, alone or in combination with one or more of the first to eighth embodiments, process 500 includes determining the display scale of a visual reference in a first group of one or more visual references based on relative orientation information, determining an alignment vector associated with the POV of the first aircraft operator and the visual reference based on the relative orientation information, and determining the display orientation of the visual reference based on the alignment vector.

[0088] In the tenth embodiment, alone or in combination with one or more of the first to ninth embodiments, the first visual representation information indicates one or more visual representation elements for each visual reference in the first group of one or more visual references, the first display configuration information indicates a display scale and a display orientation for each visual reference in the first group of one or more visual references, and wherein, in order to update the first projection display component based on the first display configuration information to display at least the first visual representation information, one or more processors are configured to transmit the first visual representation information and the first display configuration information to the first projection display component to update the first projection display component, thereby displaying one or more visual representation elements at a display scale and a display orientation for each visual reference in the first group of one or more visual references.

[0089] although Figure 5 Example blocks of process 500 are shown, but in some implementations, process 500 includes additional blocks, fewer blocks, different blocks, or blocks arranged differently (as opposed to...). Figure 5 Compared to those blocks depicted. Additionally, or alternatively, two or more blocks of process 500 can be executed in parallel.

[0090] This application also includes the following examples: Example 1. A visual perception enhancement system (320) in the cockpit of an aircraft (310), comprising: The first projection display unit is configured within the field of view (FOV) of the first aircraft operator; The second projection display unit is configured within the field of view (FOV) of the second aircraft operator; A camera system configured to observe the line of sight of the first aircraft operator and the line of sight of the second aircraft operator; and One or more processors (420) are configured to: Based on relative orientation information derived from aircraft data, first display configuration information is determined based on a first set of one or more visual references derived from the FOV of the first aircraft operator. Based on the relative orientation information, second display configuration information is determined based on a second set of one or more visual references derived from the FOV of the second aircraft operator; Based on the first display configuration information, the first projection display component is updated to display at least the first visual representation information associated with the first group of one or more visual references; and Based on the second display configuration information, the second projection display component is updated to display at least the second visual representation information associated with the second group of one or more visual references.

[0091] Example 2. The visual perception enhancement system (320) according to Example 1. Wherein, the first visual representation information indicates one or more visual representation elements for each visual reference in the first group of one or more visual references. Wherein, the first display configuration information indicates the display scale and display orientation for each visual reference in the first group of one or more visual references, and In order to update the first projection display component based on the first display configuration information to display at least the first visual representation information, the one or more processors (420) are configured to: The first visual representation information and the first display configuration information are transmitted to the first projection display component to update the first projection display component, thereby displaying the one or more visual representation elements at the display scale and the display orientation for each visual reference in the first group of one or more visual references.

[0092] Example 3. A non-transitory computer-readable medium (430) storing a set of instructions, said set of instructions comprising: One or more instructions, when executed by one or more processors (420) of a system within the cockpit of the aircraft (310), cause the system to: Based on line-of-sight data associated with the aircraft operator, the aircraft operator's field of view (FOV) is determined; Display configuration information is determined based on relative orientation information and a set of one or more visual references derived from the FOV of the aircraft operator; and The projection display components located within the cockpit and the FOV of the aircraft operator are updated based on the display configuration information to display at least visual representation information associated with the set of one or more visual references.

[0093] Example 4. According to the non-transitory computer-readable medium (430) of Example 3, wherein one or more of the instructions further cause the system to: Select one or more visual references from a set of visual references that are associated with the selection parameters.

[0094] Example 5. The non-transitory computer-readable medium (430) according to Example 4, wherein the selection parameters include: Selection parameters associated with flight operation parameters; Selection parameters associated with the display preferences of the aircraft operator; Selection parameters associated with weather events related to the flight path of the aircraft (310); Selection parameters associated with the orientation of obstacles related to the flight path of the aircraft (310); Selection parameters associated with the performance of the aircraft (310); Selection parameters associated with the state of the aircraft (310); Selection parameters associated with the predicted performance of the aircraft (310); Selection parameters associated with control information related to the aircraft (310); Selection parameters associated with the navigation of the aircraft (310); or Selection parameters related to the orientation of other aircraft (310).

[0095] Example 6. A non-transitory computer-readable medium (430) according to Example 3, wherein the one or more instructions that cause the system to determine the display configuration information cause the system to: Based on the relative orientation information, an alignment vector associated with the aircraft operator and visual reference is determined; and The display orientation of the visual reference is determined based on the alignment vector.

[0096] Example 7. According to the non-transitory computer-readable medium (430) of Example 3, wherein one or more instructions causing the system to update the projection display component based on the display configuration information to at least display the visual representation information cause the system to: The visual representation information and the display configuration information are transmitted to the projection display component to update the projection display component, thereby displaying one or more visual representation elements associated with a visual reference in the set of one or more visual references, as indicated by the visual representation information, at the display orientation of the visual reference indicated by the display configuration information.

[0097] Example 8. A method performed by a system (320) within the cockpit of an aircraft (310), comprising: The field of view (FOV) of the aircraft operator is determined based on line-of-sight data associated with the aircraft operator. Based on one or more visual references derived from the FOV of the aircraft operator, display configuration information is determined; and The projection display components located within the cockpit and the FOV of the aircraft operator are updated based on the display configuration information to display at least visual representation information associated with the set of one or more visual references.

[0098] Example 9. According to the method described in Example 8, determining the display configuration information includes: Based on the estimated relative orientation of the visual reference with respect to the aircraft (310), an alignment vector associated with the aircraft operator and the visual reference is determined; and The display orientation of the visual reference is determined based on the alignment vector.

[0099] Example 10. According to the method of Example 8, wherein updating the projection display component based on the display configuration information to at least display the visual representation information includes: The projection display component is updated to display one or more visual representation elements associated with a visual reference in the set of one or more visual references, as indicated by the visual representation information, and positioned as indicated by the display configuration information.

[0100] Example 11. The method described in Example 8 further includes: Based on the updated line-of-sight data associated with the aircraft operator, the updated FOV of the aircraft operator is determined; Based on the set or more visual references derived from the updated FOV of the aircraft operator, updated display configuration information is determined; and The projection display component is updated based on the updated display configuration information to display at least the updated visual representation information associated with the set of one or more visual references.

[0101] The foregoing disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the embodiments described herein to their precise forms. Modifications and variations can be made based on the foregoing descriptions, or can be derived from practice of the embodiments described herein.

[0102] As used herein, the term "component" is intended to be interpreted broadly as hardware, firmware, and / or a combination of hardware and software. It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementations described herein. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the descriptions herein.

[0103] Even if a particular combination of features is recited in the claims and / or described in the specification, such combinations are not intended to limit the embodiments described herein. In fact, many of these features can be combined in ways not specifically recited in the claims and / or described in the specification. Although each dependent claim listed herein may depend directly on only one claim, this specification includes combinations of each dependent claim with each other claim in the claim set. As used herein, the phrase “at least one” in the list of denotating items refers to any combination of these items (including a single member). For example, “at least one of a, b, or c” is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical items.

[0104] When “component” or “one or more components” (or another element, such as “processor” or “one or more processors”) is described or claimed to perform or be configured to perform multiple operations (within a single claim or across multiple claims), this language is intended to broadly encompass a wide range of architectures and contexts. For example, unless explicitly required otherwise (e.g., by using “first component” and “second component” or other language that distinguishes components in a claim), this language is intended to cover a single component that performs or is configured to perform all operations, a group of components that collectively perform or are configured to perform all operations, a first component that performs or is configured to perform a first operation and a second component that performs or is configured to perform a second operation, or any combination of components that perform or are configured to perform these operations. For example, when a claim takes the form “one or more components configured to: perform X; perform Y; perform Z,” the claim should be interpreted as meaning “one or more components configured to perform X; one or more (possibly different) components configured to perform Y; and one or more (possibly different) components configured to perform Z.”

[0105] Unless explicitly stated otherwise, no element, action, or instruction used herein should be construed as critical or necessary. Furthermore, as used herein, the articles “a” and “an” are intended to include one or more items and are interchangeable with “one or more”. Furthermore, as used herein, the article “the” is intended to include one or more items mentioned in conjunction with the article “the” and is interchangeable with “one or more”. Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, or a combination of related and unrelated items) and is interchangeable with “one or more”. Where only one item is intended, the phrase “only one” or similar language is used. Furthermore, as used herein, the terms “has,” “have,” “having,” etc., are intended to be open-ended terms. Furthermore, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.” Furthermore, as used herein, the term “or” is intended to be inclusive when used in series and may be used interchangeably with “and / or” unless otherwise explicitly stated (e.g., if used in combination with “any” or “only one of them”).

Claims

1. A visual perception enhancement system (320) in the cockpit of an aircraft (310), comprising: The first projection display unit is configured within the field of view (FOV) of the first aircraft operator; The second projection display unit is configured within the field of view (FOV) of the second aircraft operator; A camera system configured to observe the line of sight of the first aircraft operator and the line of sight of the second aircraft operator; and One or more processors (420) are configured to: Based on relative orientation information derived from aircraft data, first display configuration information is determined based on a first set of one or more visual references derived from the FOV of the first aircraft operator. Based on the relative orientation information, second display configuration information is determined based on a second set of one or more visual references derived from the FOV of the second aircraft operator; Based on the first display configuration information, the first projection display component is updated to at least display first visual representation information associated with the first group of one or more visual references; and Based on the second display configuration information, the second projection display component is updated to display at least the second visual representation information associated with the second group of one or more visual references.

2. The visual perception enhancement system (320) according to claim 1, wherein the one or more processors (420) are further configured to: The FOV of the first aircraft operator is derived from the first aircraft operator's viewpoint.

3. The visual perception enhancement system (320) according to claim 1, wherein the aircraft data includes: Navigation data; Guiding data; or Control data.

4. The visual perception enhancement system (320) according to claim 1, wherein, The relative orientation information indicates the estimated relative orientation of each of the first group of one or more visual references and the second group of one or more visual references relative to the aircraft (310).

5. The visual perception enhancement system (320) according to claim 1, wherein the one or more processors (420) are further configured to: Determine the estimated azimuth of the aircraft (310) and the azimuth of the visual references in the first set of one or more visual references and the second set of one or more visual references; and The relative bearing information is determined based on the estimated bearing of the aircraft (310) and the bearing of the visual reference to indicate the estimated relative bearing of the visual reference relative to the aircraft (310).

6. The visual perception enhancement system (320) according to claim 1, wherein the one or more processors (420) are further configured to: Based on the first FOV of the first aircraft operator and the relative orientation information, a visual reference group is determined; Determine flight operation parameters based on the aircraft data; and Select one or more visual references from the group of visual references that are associated with the flight operation parameters.

7. The visual perception enhancement system (320) according to claim 1, wherein the one or more processors (420) are further configured to: Based on the FOV and relative orientation information of the first aircraft operator, a visual reference group is determined; Identify the selection parameters associated with the first projection display component; and Select one or more visual references from the first group that are associated with the selection parameters from the group of visual references.

8. The visual perception enhancement system (320) according to claim 7, wherein the selection parameters include: Selection parameters associated with flight operation parameters; Selection parameters associated with the display preferences of the first aircraft operator; Selection parameters associated with weather events related to the flight path of the aircraft (310); Selection parameters associated with the orientation of obstacles related to the flight path of the aircraft (310); Selection parameters associated with the performance of the aircraft (310); Selection parameters associated with the state of the aircraft (310); Selection parameters associated with the predicted performance of the aircraft (310); Selection parameters associated with control information related to the aircraft (310); Selection parameters associated with the navigation of the aircraft (310); or Selection parameters related to the orientation of other aircraft (310).

9. The visual perception enhancement system (320) according to claim 1, wherein the one or more processors (420) are further configured to: Determine the corresponding characteristics of the visual references in the first group of one or more visual references; and Based on the corresponding characteristics of the visual references in the first group of one or more visual references, the first visual representation information is generated, the first visual representation information including one or more visual representation elements for each visual reference in the first group of one or more visual references.

10. The visual perception enhancement system (320) according to claim 1, wherein, In order to determine the first display configuration information, the one or more processors (420) are configured to: Based on the relative orientation information, determine the display ratio of the visual reference in the first group of one or more visual references; Based on the relative orientation information, an alignment vector associated with the viewpoint of the operator of the first aircraft (310) and the visual reference is determined; and The display orientation of the visual reference is determined based on the alignment vector.

Images