System and method for providing enhanced view of aircraft external environment during flight

Abstract

Aircraft, systems, and methods of processing video content displayed on an aircraft during flight are disclosed herein. A disclosed embodiment of the method obtains location data corresponding to a current geographic location of the aircraft, attitude data corresponding to a current attitude of the aircraft, and video data for a real-time video feed corresponding to a view of an environment external to the aircraft. The received video data is processed into enhanced video output data based on the location data, the pose data, and geospatial data relating to at least some of the geographic features in the view of the environment. The enhanced video content is presented on a display device onboard the aircraft, where the enhanced video content corresponds to the enhanced video output data.

Description

Technical Field

[0001] Embodiments of the subject matter described herein relate generally to vehicle display and entertainment systems. More specifically, embodiments of the subject matter relate to systems onboard an aircraft that generate an enhanced display of the environment external to the aircraft during flight. Background Art

[0002] A vehicle (such as an aircraft) may include one or more cameras mounted on the body that capture a real-time video feed of the external environment. The captured video content may be presented to passengers in the aircraft cabin or to the aircraft crew. For example, a real-time view of the sky, terrain, geography, and / or landmarks may be rendered on one or more displays provided for passenger viewing. Alternatively or additionally, the captured video content may be used by an operator of the vehicle, for example, a pilot or co-pilot of the aircraft.

[0003] It is desirable to provide an improved or enhanced viewing experience for occupants of a vehicle, such as passengers of an aircraft.Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background. Summary of the invention

[0004] The following includes a description of an embodiment of an aircraft, the aircraft including: an avionics subsystem for providing position data corresponding to a current geographic location of the aircraft and attitude data corresponding to a current attitude of the aircraft; an imaging system for capturing video data corresponding to a view of an environment external to the aircraft; and at least one video processor coupled to the avionics subsystem and the imaging system. The at least one video processor is configured to process the video data captured by the imaging system into enhanced video output data, wherein the at least one video processor generates the enhanced video output data based on the position data provided by the avionics subsystem, the attitude data provided by the avionics subsystem, and geospatial data associated with at least some geographic features in the view of the environment. The aircraft also includes a display device coupled to the at least one video processor to receive the enhanced video output data, the display device being configured to display enhanced video content corresponding to the received enhanced video output data.

[0005] A method for processing video content displayed on an aircraft during flight is also disclosed herein. Embodiments of the disclosed method involve: obtaining position data corresponding to a current geographic location of the aircraft; obtaining attitude data corresponding to a current attitude of the aircraft; receiving video data for a real-time video feed corresponding to a view of an environment external to the aircraft; processing the received video data into enhanced video output data based on the obtained position data, the obtained attitude data, and geospatial data associated with at least some geographic features in the view of the environment; and causing the enhanced video content to be presented using a display device on the aircraft, wherein the enhanced video content corresponds to the enhanced video output data.

[0006] Also disclosed herein is an embodiment of a system comprising: at least one processor; and at least one processor-readable medium associated with the at least one processor, the at least one processor-readable medium storing processor-executable instructions that can be configured to be executed by the at least one processor to perform a method for processing video content for display on an aircraft during flight. An embodiment of the method involves: obtaining position data corresponding to a current geographic location of the aircraft; obtaining attitude data corresponding to a current attitude of the aircraft; receiving video data for a real-time video feed corresponding to a view of an environment external to the aircraft; processing the received video data into enhanced video output data based on the obtained position data, the obtained attitude data, and geospatial data associated with at least some geographic features in the view of the environment; and causing the enhanced video content to be presented with a display device on the aircraft, wherein the enhanced video content corresponds to the enhanced video output data.

[0007] This Summary is provided to introduce some concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] A more complete understanding of the subject matter may be derived by referring to the detailed description and claims when considered in conjunction with the following drawings, wherein like reference numerals refer to similar elements throughout.

[0009] Figure 1 is a drawing depicting an aircraft flying over a geographical feature;

[0010] Figure 2 is a simplified block diagram representation of a system on an aircraft that is configured according to certain embodiments of the present invention to process and present augmented video content during operation of the aircraft;

[0011] Figure 3is a block diagram of an exemplary embodiment of a computer-based device;

[0012] Figure 4 depicts image content of a real-time video feed captured by a forward-facing camera mounted to a vertical stabilizer of an aircraft;

[0013] Figure 5 Depicted Figure 4 An enhanced version of the image content shown in;

[0014] Figure 6 depicts enhanced image content comprising an image of a bridge captured by a camera of an aircraft in flight under a first operating condition;

[0015] Figure 7 Depicting enhanced image content comprising captured by a camera of an aircraft in flight under a second operating condition Figure 6 An image of the bridge shown in ; and

[0016] Figure 8 is a flow chart illustrating an exemplary embodiment of a method of processing video content. DETAILED DESCRIPTION

[0017] The following detailed description is merely illustrative in nature and is not intended to limit the embodiments of the subject matter or the application and uses of such embodiments. As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any implementation described herein as exemplary is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, background technology, summary of the invention, or the following detailed description.

[0018] Techniques and techniques may be described herein in terms of functions and / or logic block components, and with reference to symbolic representations of operations, processing tasks, and functions that may be performed by various computing components or devices. Such operations, tasks, and functions are sometimes referred to as computer-implemented, computerized, software-implemented, or computer-implemented. It should be understood that the various block components shown in the figures may be implemented by any number of hardware, software, and / or firmware components configured to perform a specified function. For example, embodiments of a system or component may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which may perform various functions under the control of one or more microprocessors or other control devices.

[0019] When implemented in software or the like, the various elements of the systems and devices described herein are essentially code segments or instructions that cause one or more processor devices to perform various tasks. In some embodiments, the program or code segment is stored in at least one tangible processor-readable medium, which may include any medium capable of storing or transmitting information. Examples of non-transitory and processor-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, etc.

[0020] For the sake of brevity, conventional technologies related to other functional aspects of vehicle control and navigation systems, vehicle display systems (including aircraft cabin display systems), vehicle instrument systems, image data processing, virtual and augmented reality systems and methods, and systems (and the various operating components of the systems) may not be described in detail herein. In addition, the connecting lines shown in the various figures included herein are intended to represent exemplary functional relationships and / or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in the embodiments of the subject matter.

[0021] Disclosed herein are vehicle-based display systems and associated methods of operation. According to certain non-limiting embodiments, the system is deployed on an aircraft such as an airplane. However, it should be understood that embodiments of the disclosed system may be used in other vehicle applications, including but not limited to: a train; a helicopter; an automobile; a boat; a monorail; an amusement park ride; a transportation system; a ski lift; or the like.

[0022] Figure 1 is a diagram of an exemplary embodiment of an aircraft 100 depicting flight over a geographic feature 102, which may be natural terrain, a man-made structure, a landmark, or anything that can be seen from the vantage point of the aircraft 100. The aircraft 100 includes or cooperates with one or more onboard aircraft systems 104, which may include, but are not limited to, any of the following in combination and in any number of iterations or multiples: a flight control system; a navigation system; an avionics subsystem; an instrument system; a display device; an imaging system including one or more cameras; an image / video processing system; an alert system; a messaging or notification system for the flight crew; a global positioning system (GPS) receiver; and various sensor devices or components. In certain embodiments, the onboard aircraft systems 104 include one or more components or subsystems that cooperate to process and display enhanced video content. For example, the onboard aircraft systems 104 may include Figure 2 The exemplary system configurations depicted in the drawings may include or cooperate with items included in or in conjunction with. Figure 1The various onboard aircraft systems 104 are shown as a single block, but it is understood that embodiments of the aircraft 100 will utilize a variety of different physical, logical, and computer-implemented components to implement the onboard aircraft systems 104 .

[0023] The depicted embodiment of the aircraft 100 includes an external structure 112 that defines the exterior of the aircraft 100. The aircraft 100 includes an imaging system having any number of cameras mounted to or integrated with the external structure 112. Each camera is mounted, configured, and oriented in a particular manner so that it can capture video data corresponding to a view of an environment external to the aircraft 100. In this regard, the depicted embodiment of the aircraft includes a camera 120 that can be operated to capture an unobstructed view of the environment 106 located below the nose of the aircraft 100. Figure 1 Camera 120 is schematically shown capturing image data that includes area 122 beneath exterior structure 112. Area 122 captured by camera 120 may include some of geographic features 102.

[0024] The aircraft 100 may include any number of additional or alternative cameras 120 mounted in any desired location. For example, embodiments of the aircraft 100 may include any or all of the following, but are not limited to: a camera 120 located on the vertical stabilizer 130 and oriented generally in a forward direction; a camera 120 located on the vertical stabilizer 130 and oriented generally in a tail direction; a camera 120 located on the belly 132 and oriented generally in a downward and forward direction; a camera 120 located on the belly 132 and oriented generally in a downward and rearward direction; a camera 120 positioned to capture a “pilot’s eye” view from the perspective of the windshield 134; a wingtip camera 120 positioned at or near the wingtip of the aircraft and oriented generally forward and toward the fuselage of the aircraft. Systems on the aircraft 100 (such as a cabin management system, a mobile application, a passenger entertainment system, etc.) may be used to select or configure parameters associated with the camera 120. For example, a user on board the aircraft may interact with the appropriate system or application to: select a specific camera view; select a multiple-camera display; control camera options (where supported, e.g., zoom settings, tilt control, pan, wide-angle lens settings, etc.); enable / disable camera displays.

[0025] The camera 120 used for the systems and methods described herein can be fixed in one position corresponding to a single viewing angle, or it can be movable in one or more dimensions (e.g., tilt, pan, roll, altitude) and can be controlled by a user on the aircraft 100. The camera 120 used for the systems and methods described herein can be configured to capture a single view, or it can be configured as a multi-element or multi-lens camera to capture multiple different views, multiple different focal points, etc. In this regard, the camera 120 can be appropriately configured to capture stereo images to support three-dimensional rendering effects. In addition, the camera 120 can be equipped with a fixed lens or an adjustable zoom lens, which can be controlled by a user on the aircraft 100. The options and settings described herein, as well as other features used by modern cameras, can be used in the system described herein.

[0026] In practice, the aircraft 100 may include several externally facing cameras 120 that provide views of the aircraft, terrain, and surrounding environment. While such real-time camera feeds may be entertaining and informative in certain circumstances (e.g., during takeoff and landing, when passing interesting landmarks, etc.), they may not be as useful or interesting at night, when the aircraft 100 is flying at high altitudes, or when the aircraft 100 is flying over the ocean, through clouds, or over uninteresting scenery. Through the application of modern 3D graphics combined with real-time position and attitude information of the aircraft 100, dynamic computer-generated images can be synthesized into the content provided by the real-time cameras. Thus, a system of the type described herein can add informative and entertaining visual elements that appear as part of the environment captured by the camera 120. For example, the system can generate and render identifiers that float above landmarks, generate and render 3D information maps that move with and around the surface of the host aircraft 100 and / or with and around features present in the surrounding environment (e.g., buildings, terrain features, roads, trees, etc.).

[0027] Figure 2 is a simplified block diagram representation of system 200 on an aircraft. According to certain embodiments of the present invention, system 200 is configured to process and present augmented video content during operation of an aircraft. Figure 2 Describes the Figure 1 In fact, Figure 1 The onboard aircraft systems 104 shown in FIG. 1 may include Figure 21. The embodiment of the system 200 shown includes, but is not limited to: an avionics subsystem 202; an imaging system including at least one camera 120; at least one video processor 204; and a display device 206. A deployed implementation of the system 200 may include some or all of these components, additional systems and components (as needed or desired), and / or alternative systems and components (as needed or desired). Figure 2 Illustrative examples only, depict certain systems and components that support the inventive subject matter described herein.

[0028] The avionics subsystem 202 is suitably configured and operated to provide status information or data related to the current real-time operating conditions of the host aircraft. According to certain embodiments, the avionics subsystem 202 provides position data 210 corresponding to the current geographic location of the aircraft, and provides attitude data 212 corresponding to the current attitude of the aircraft. The avionics subsystem 202 may include or cooperate with a suitably configured navigation, geolocation, or tracking system that generates the position data 210. In certain implementations, the position data 210 may be provided using GPS or an equivalent geolocation system. The attitude data 212 includes data associated with the pitch, roll, and yaw states of the aircraft 100. In this regard, the avionics subsystem 202 may include or cooperate with an inertial reference system on the aircraft 100, which may utilize any number of sensors, inertial measurement units, etc. As Figure 2 As schematically depicted in , position data 210 and posture data 212 are provided to video processor 204 on a continuous basis or otherwise as needed.

[0029] As previously described, the system 200 may include at least one camera 120 that captures and provides video data 216 corresponding to a view of the environment outside the aircraft. For simplicity and ease of description, Figure 2Only one camera 120 is shown, which provides corresponding video data 216 to the video processor 204 in a continuous manner or otherwise as needed. The raw video data 216 may be processed and modified into enhanced video output data 218 in a manner described in more detail below. The display device 206 is coupled to the video processor 204 (or may be integrated with the video processor 204) to receive the enhanced video output data 218. The display device 206 functions in a conventional manner to display enhanced video content corresponding to the received enhanced video output data 218. The display device 206 may be implemented and deployed in a variety of ways on the aircraft. To this end, the display device 206 may be implemented as: a common stand-alone monitor (mounted to an interior panel, bulkhead, fixture, or cabinet); a display integrated into a seat back or headrest; a retractable drop-down or slide-out display element; a transparent or translucent film display; or the like.

[0030] At least one video processor 204 is coupled to the avionics subsystem 202 to obtain position data 210 and attitude data 212, and is coupled to the camera 120 to obtain video data 216. The video processor 204 is configured to process the input video data 216 (captured by the camera 120) into enhanced video output data 218, wherein the enhanced video output data 218 represents a modified, supplemented, or enhanced version of the original video data 216. More specifically, the video processor 204 generates the enhanced video output data 218 based on at least the following information: the position data 210 provided by the avionics subsystem 202; the attitude data 212 provided by the avionics subsystem 202; geospatial data 220 associated with at least some geographic features present in a current view of the environment (as captured by the camera 120); and calibration data 224 of the imaging system or camera 120 used to capture the original video data 216.

[0031] The geospatial data 220 defines or represents visual or graphical content that is associated in some way with features, items, or elements conveyed by the captured video data 216. The geospatial content conveyed by the geospatial data 220 may be used to modify, enhance, or augment the original video data 216 such that the resulting enhanced video output data 218 contains visual content from the scene captured by the camera 120 mixed with visual content corresponding to at least some of the geospatial data 220. In some embodiments, the geospatial data 220 includes metadata or is implemented as metadata such that the enhanced video content presented by the display device 206 includes visual content corresponding to at least some of the metadata. Although the type, contextual meaning, and characteristics of metadata may vary from one implementation to another, exemplary embodiments may be compatible with any or all of the following metadata, but not limited to: data identifying, describing, or characterizing geographic features appearing in a view of an environment; data identifying, describing, or characterizing architectural features appearing in a view of an environment; data identifying, describing, or characterizing weather associated with a view of an environment; data identifying, describing, or characterizing religious information associated with a view of an environment; and / or data identifying, describing, or characterizing individuals, sports teams, organizations, companies, businesses, objects, entities, geopolitical boundaries, historically significant locations, or culturally significant locations appearing in a view of an environment.

[0032] The camera calibration data 224 is associated with the image capture characteristics, optical characteristics, and / or configuration settings of the camera 120 used as the source of the raw video data 216. The video processor 204 utilizes the calibration data 224 to create enhanced image / video content that accurately matches or blends with the appearance of the video content conveyed by the raw video data 216. To this end, camera calibration data 224 may include or be implemented as any of the following, but is not limited to: the current (or fixed) position of camera 120 relative to a reference point, plane, or location on the host aircraft; the current (or fixed) orientation of camera 120 relative to a reference point, plane, or location on the host aircraft; the current (or fixed) "line of sight" of camera 120, which may be represented by any coordinate system relative to a reference point, plane, or location on the host aircraft; measurements or metrics related to the quality or clarity of the lens used by camera 120; measurements or metrics related to the presence of visual artifacts or distortion introduced by camera 120 and / or the lens of camera 120; measurements or metrics of color introduced by camera 120; measurements or metrics related to the spectrum of light captured (e.g., to accommodate infrared camera implementations).

[0033] At least one video processor 204 may include any combination of software, firmware, hardware, etc. For example, video processor 204 may include an application specific integrated circuit (ASIC), an electronic circuit, a microprocessor device and memory that executes one or more software or firmware programs, a combinational logic circuit, and / or other suitable components that provide the described functionality. According to certain embodiments, at least one video processor 204 performs tasks associated with image transformation and enhanced content creation 230 to generate supplemental video content 232 in response to real-time position data 210, real-time pose data 212, camera calibration data 224, and corresponding geospatial data 220 related to the external environment. At least one video processor 204 also performs tasks associated with video synthesis 234, which combines at least some content from the original video data 216 with at least some content in the supplemental video content 232 to obtain enhanced video output data 218. Although in Figure 2 Not separately depicted in FIG. 2 , the at least one video processor 204 may be suitably configured and operated to perform additional or alternative image or video processing routines or tasks on the raw video data 216 and / or content associated with the geospatial data 220 .

[0034] According to some embodiments, Figure 1 The aircraft system 104 and Figure 2 One or more of the items depicted in the can be implemented as at least one computer-based or processor-based device, system or device component. In this regard, Figure 3 is a simplified block diagram representation of an exemplary embodiment of a computer-based device 300 , which may be used to implement certain devices or systems onboard aircraft 100 .

[0035] The device 300 generally includes, but is not limited to, at least one processor 302; at least one memory storage device, storage medium, or memory element 304; a display 306; at least one communication (network) interface 308; and input and output (I / O) devices 310, such as an input interface, one or more output devices, one or more human / machine interface elements, etc. In practice, the device 300 may include additional components, elements, and functionality that may be conventional in nature or irrelevant to the specific applications and methods described herein.

[0036] For example, the processor 302 may be a central processing unit (CPU), a field programmable gate array (FPGA), a microcontroller, an application specific integrated circuit (ASIC), or any other logic device or combination thereof. One or more memory elements 304 are communicatively coupled to at least one processor 302 and may be implemented with any combination of volatile and non-volatile memory. The memory element 304 has non-temporary machine-readable and computer-executable instructions (program code) 312 stored thereon, wherein the instructions 312 may be configured to be executed by at least one processor 302 as required. When executed by at least one processor 302, the instructions 312 cause at least one processor 302 to perform the associated tasks, processes, and operations defined by the instructions 312. Of course, the memory element 304 may also include instructions associated with the file system of the main device 300 and instructions associated with other applications or programs. In addition, the memory element 304 may be used as a data storage unit of the main device 300. For example, the memory element 304 may provide storage 314 for aircraft data, navigation data, sensor data, measurements, image and / or video content, settings or configuration data of the aircraft, etc.

[0037] Display 306, if deployed with a particular embodiment of device 300, may be integrated with device 300 or communicatively coupled to device 300 as a peripheral or accessory component. The shape, size, resolution, and technology of display 306 will be appropriate to the particular implementation of device 300. Display 306 may be implemented as a monitor, a touch screen, or another conventional electronic display capable of graphically presenting data and / or information provided by device 300.

[0038] The communication interface 308 represents the hardware, software, and processing logic that enable the device 300 to support data communications with other devices. In practice, the communication interface 308 can be appropriately configured to support wireless and / or wired data communication protocols suitable for a specific embodiment. For example, the communication interface 308 can be designed to support aircraft network protocols, cellular communication protocols, short-range wireless protocols (such as Bluetooth communication protocols), and / or WLAN protocols. As another example, if the device 300 is a desktop or laptop computer, the communication interface can be designed to support Bluetooth communication protocols, WLAN protocols, and LAN communication protocols (e.g., Ethernet). According to certain aircraft applications, the communication interface 308 is designed and configured to support one or more airborne network protocols, which are used for information communication between equipment, components, and subsystems of the aircraft 100.

[0039] I / O devices 310 enable a user of device 300 to interact with device 300 as desired. In practice, I / O devices 310 may include, but are not limited to: an input interface that receives data for processing by device 300; a speaker, an audio transducer, or other audio feedback component; a tactile feedback device; a microphone; a mouse or other pointing device; a touch screen or touchpad device; a keyboard; a joystick; a biometric sensor or reader (such as a fingerprint reader, a retinal or iris scanner, a palm print or palm vein reader, etc.); a camera; a lidar sensor; or any conventional peripheral device. In this context, touch screen display 306 may be classified as I / O device 310. In addition, touch screen display 306 may include or be controlled to function as a fingerprint or palm print scanner. Tactile feedback devices may be controlled to generate variable amounts of tactile or physical feedback, such as vibration, force, a sense of knocking or collision, detectable motion, etc. Tactile feedback devices and related control schemes are well known and, therefore, will not be described in detail herein.

[0040] As described above, the system 200 is configured and operated in an appropriate manner to process and generate enhanced video content corresponding to the current view of the host aircraft's external environment. Figure 4 Depicted is image content 400 of a real-time video feed captured by a forward-facing camera mounted to a vertical stabilizer of an aircraft 402 (a portion of which appears in the field of view). Image content 400 corresponds to a particular moment captured by the real-time video feed, as conveyed by raw video data obtained by the forward-facing camera. In contrast, Figure 5 An enhanced image content 500 is depicted that represents an enhanced version of the image content 400. In this example, the enhanced image content 500 includes all of the original image content captured by the front-facing camera, combined with at least one visible supplemental feature 502. Figure 2 As explained, certain visual characteristics (e.g., shape, size, contextual meaning, orientation, alignment, perspective, location within the field of view) of the at least one visible supplemental feature 502 are influenced by or determined by the position data 210, pose data 212, geospatial data 220, and camera calibration data 224. For this specific non-limiting example, the at least one visible supplemental feature 502 is an icon, graphic, or text label that describes or identifies a landmark visible within the field of view of the original image content 400. For simplicity, Figure 5 The augmented image content 500 depicted in FIG. 5 contains only one visible supplemental feature 502 (which identifies a state flag). In practice, any number of discrete supplemental features 502 may be shown as augmented content.

[0041] As another non-limiting example, Figure 6Depicted is enhanced image content 602 that includes an image of a bridge 604 captured by a camera of an aircraft in flight under a first operating condition. Figure 7 Augmented image content 702 is depicted that includes an image of a bridge 604 captured by a camera of an aircraft in flight under a second operating condition. Augmented image content 602 includes a visible supplemental feature 606 in the form of an identification label that is linked to bridge 604: GOLDEN GATE. Augmented image content 702 also includes a visible supplemental feature 706 in the form of an identification label that is linked to bridge 604: GOLDEN GATE. Although the textual content and contextual meaning of supplemental features 606, 706 are the same, the visual appearance and displayed features are distinguishable due to the different operating conditions of the host aircraft (which results in different perspectives of the aircraft camera). For example, the shape, size, and 3D perspective of the letters used with visible supplemental feature 606 are visually distinguishable from the letters used with visible supplemental feature 706.

[0042] In some embodiments, descriptive labels, icons, or other supplemental visual content may be removed from the augmented video content at appropriate times, for example, when an associated landmark or feature becomes too small to be distinguished, when a more popular or important visible feature takes precedence, or when an associated landmark or feature begins to leave the field of view. The systems and methods described herein dynamically update the visual appearance of such augmented image content in a continuous manner in response to changes in camera orientation, where such changes are affected by the current position and attitude of the aircraft. Thus, visible supplemental features may be generated in an appropriate manner for inclusion in the augmented video content, and in a manner that maintains visual consistency with the original video content captured by the camera.

[0043] Figure 5-Figure 7It is intended to demonstrate how supplementary or additional graphical content can be combined with original video content to generate enhanced video content for presentation to users on an aircraft. In some embodiments, the supplementary information corresponds to geospatial data linked to features, elements, landmarks, and / or environmental conditions that appear in the original video feed. Therefore, the enhanced video content may include graphics associated with: geographic features (e.g., natural or artificial lakes, waterways, hills, mountains, parks, landscapes, trees, beaches); architectural features (e.g., buildings, roads, bridges, equipment); weather conditions, weather patterns, or environmental conditions (e.g., clouds, storm fronts, hurricanes, rain, snow, tidal features). The enhanced video content may also include graphics associated with people, sports teams, organizations, businesses, companies, objects, entities, geopolitical boundaries, historical or culturally important locations, religious information such as prayer directions, or anything that may have a contextual relationship with the geospatial content. In some embodiments, the enhanced video content includes or is rendered as three-dimensional graphical content corresponding to at least some of the retrieved geospatial data (alternatively or additionally, two-dimensional content may also be placed into the rendered scene). The enhanced video content may also include graphics, text, or other graphical content associated with the flight status, operating condition, or performance of the primary aircraft, including but not limited to: airspeed; acceleration; altitude; geographic location (latitude, longitude); flight plan; flight trajectory; bank angle; destination airport; waypoints; or current operating status data of the aircraft.

[0044] Figure 8 800 is a flow chart illustrating an exemplary embodiment of a method 800 for processing video content. The various tasks performed in conjunction with the method 800 may be performed by software, hardware, firmware, or any combination thereof. For purposes of illustration, the following description of the method 800 may refer to the above description of the method 800 in conjunction with the Figure 1-Figure 3 In practice, portions of method 800 may be performed by different elements of the described system, such as camera 120, avionics subsystem 202, video processor 204, or display device 206. It should be understood that method 800 may include any number of additional or alternative tasks, Figure 8 The tasks shown in the method 800 need not be performed in the order shown, and the method 800 may be incorporated into a more comprehensive process or process having additional functionality not described in detail herein. Furthermore, as long as the intended overall functionality remains intact, Figure 8 One or more of the tasks shown may be omitted from the disclosed embodiment of method 800 .

[0045] Iterations of method 800 may be performed at any desired refresh rate (e.g., 30 frames per second) to process video content to be displayed on an aircraft during flight or during ground-based operations. The illustrated embodiment of method 800 obtains position data corresponding to the current geographic location of the host aircraft (task 802), and obtains attitude data corresponding to the current attitude of the host aircraft (task 804). For example, the position data may include GPS data indicating longitude, latitude, and altitude information. The attitude data may include sensor data indicating the pitch, roll, and yaw of the aircraft. Method 800 also accesses or receives video data for a real-time video feed corresponding to a view of the external environment of the host aircraft (task 806). As described above, the video data may be captured and provided by a camera appropriately arranged and oriented to the structure of the aircraft.

[0046] Method 800 may continue by accessing or retrieving context-relevant geospatial data associated with one or more currently visible geographic features (task 808). In this regard, the context-relevant geographic features of interest appear in a real-time video feed captured by the camera. Thus, method 800 may consider at least some of the position data and / or at least some of the pose data to identify or specify the particular geospatial data processed by task 808. Method 800 may continue by performing an image transformation on at least some of the retrieved / accessed geospatial data (task 810). The image transformation associated with task 810 produces transformed image content corresponding to at least a portion of the geospatial data of interest. Task 810 is performed to manipulate or modify the original geospatial data so that it corresponds to an actual view of the surrounding environment conveyed by the captured video content. Figure 1 For example, image transformation processing can generate an image having an actual visual image of the environment represented by the captured video data. Figure 1 In some embodiments, the image transformation routine is based on the obtained position data (task 802) and the obtained pose data (task 804). The image transformation routine may generate the transformed image content based on camera calibration data for the imaging system, as described above with reference to Figure 2 Explained.

[0047] The method 800 continues by performing additional image / video processing to generate enhanced video output data corresponding to the original video data (task 812). More specifically, the method 800 processes the received video data using the transformed image content to generate enhanced video output data. As described above, the enhanced video output data is ultimately based on the original video data, the obtained aircraft position data, the obtained aircraft attitude data, the identified geospatial data, and any applicable camera calibration data. The generated enhanced video output data is provided to at least one display device (task 814), which is operated to display enhanced video content corresponding to the enhanced video output data (task 816). For the exemplary embodiments described herein, at least one video processor 204 causes the enhanced video content to be presented on a display device on the host aircraft.

[0048] If the video presentation is terminated, a different camera view is selected, or otherwise causes the method 800 to end (the "yes" branch of query task 818), the method 800 exits. Otherwise, the method 800 returns to task 802 so that the described routine can be repeated to refresh the augmented display. In this way, the augmented video content is updated and refreshed in a continuous manner to maintain consistency with the real-time video feed from the camera.

[0049] Although at least one exemplary embodiment has been presented in the foregoing detailed description, it should be understood that there are a large number of variations. It should also be understood that one or more exemplary embodiments described herein are not intended to limit the scope, applicability, or configuration of the claimed subject matter in any way. On the contrary, the foregoing detailed description will provide a convenient roadmap for implementing the described embodiment or embodiments for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope defined by the claims, and the scope includes equivalents and foreseeable equivalents known at the time of filing this patent application.

Claims

1. An aircraft, comprising: an avionics subsystem for providing position data corresponding to a current geographic location of the aircraft and attitude data corresponding to a current attitude of the aircraft; an imaging system for capturing video data corresponding to a view of an environment external to the aircraft; at least one video processor coupled to the avionics subsystem and the imaging system, the at least one video processor configured to process the video data captured by the imaging system into enhanced video output data, wherein the at least one video processor generates the enhanced video output data based on the position data provided by the avionics subsystem, the attitude data provided by the avionics subsystem, and geospatial data associated with at least some geographic features in the view of the environment; as well as A display device is coupled to the at least one video processor to receive the enhanced video output data, the display device being configured to display enhanced video content corresponding to the received enhanced video output data.

2. The aircraft according to claim 1, wherein: The geospatial data includes metadata; The enhanced video content includes visual content corresponding to at least some of the metadata; as well as The metadata includes: data identifying, describing, or characterizing geographic features that appear in the view of the environment; data identifying, describing, or characterizing architectural features that appear in the view of the environment; data identifying, describing, or characterizing weather associated with the view of the environment; data identifying, describing, or characterizing religious information associated with the view of the environment; and / or data identifying, describing, or characterizing individuals, sports teams, organizations, companies, businesses, objects, entities, geopolitical boundaries, historically significant locations, or culturally significant locations that appear in the view of the environment.

3. The aircraft according to claim 1, wherein: The imaging system includes a camera mounted to or integrated with an external structure of the aircraft.

4. The aircraft according to claim 1, wherein: The at least one video processor performs an image transformation on at least some of the geospatial data to obtain transformed image content; and The image transformation is based on the position data and the pose data.

5. The aircraft according to claim 4, wherein: The image transformation generates the transformed image content having an alignment, orientation, perspective, and scale that is consistent with the view of the environment represented by the captured video data.

6. The aircraft according to claim 4, wherein: The image transformation generates the transformed image content based on calibration data of the imaging system.

7. The aircraft according to claim 1, wherein: The augmented video content includes three-dimensional graphical content corresponding to at least some of the geospatial data.

8. The aircraft according to claim 1, wherein: The augmented video content includes graphical content corresponding to current operational status data of the aircraft.

9. A method for processing video content displayed on an aircraft during flight, the method comprising: obtaining position data corresponding to a current geographic location of the aircraft; Obtaining attitude data corresponding to a current attitude of the aircraft; receiving video data for a real-time video feed corresponding to a view of an environment external to the aircraft; processing the received video data into enhanced video output data based on the obtained position data, the obtained pose data, and geospatial data associated with at least some geographic features in the view of the environment; as well as The enhanced video content is caused to be presented using a display device on the aircraft, wherein the enhanced video content corresponds to the enhanced video output data.

10. The method according to claim 9, wherein: The geospatial data includes metadata; The enhanced video content includes visual content corresponding to at least some of the metadata; as well as The metadata includes: data identifying, describing, or characterizing geographic features that appear in the view of the environment; data identifying, describing, or characterizing architectural features that appear in the view of the environment; data identifying, describing, or characterizing weather associated with the view of the environment; data identifying, describing, or characterizing religious information associated with the view of the environment; and / or data identifying, describing, or characterizing individuals, sports teams, organizations, companies, businesses, objects, entities, geopolitical boundaries, historically significant locations, or culturally significant locations that appear in the view of the environment.

11. The method according to claim 9, wherein: The processing step includes performing an image transformation on at least some of the geospatial data to obtain transformed image content; and The image transformation is based on the position data and the pose data.

12. The method according to claim 11, wherein: The image transformation generates the transformed image content having an alignment, orientation, perspective, and scale consistent with the view of the environment represented by the received video data.

13. The method according to claim 11, wherein: The image transformation generates the transformed image content based on calibration data of an imaging system that captured the video data.

14. The method according to claim 9, wherein: The augmented video content includes three-dimensional graphical content corresponding to at least some of the geospatial data.

15. The method according to claim 9, wherein: The augmented video content includes graphical content corresponding to current operational status data of the aircraft.

16. A system comprising: at least one processor; as well as At least one processor-readable medium associated with the at least one processor, the at least one processor-readable medium storing processor-executable instructions configurable to be executed by the at least one processor to perform a method of processing video content for display on an aircraft during flight, the method comprising: obtaining position data corresponding to a current geographic location of the aircraft; Obtaining attitude data corresponding to a current attitude of the aircraft; receiving video data for a real-time video feed corresponding to a view of an environment external to the aircraft; processing the received video data into enhanced video output data based on the obtained position data, the obtained pose data, and geospatial data associated with at least some geographic features in the view of the environment; and The enhanced video content is caused to be presented using a display device on the aircraft, wherein the enhanced video content corresponds to the enhanced video output data.

17. The system of claim 16, wherein: The geospatial data includes metadata; The enhanced video content includes visual content corresponding to at least some of the metadata; as well as The metadata includes: data identifying, describing, or characterizing geographic features that appear in the view of the environment; data identifying, describing, or characterizing architectural features that appear in the view of the environment; data identifying, describing, or characterizing weather associated with the view of the environment; data identifying, describing, or characterizing religious information associated with the view of the environment; and / or data identifying, describing, or characterizing individuals, sports teams, organizations, companies, businesses, objects, entities, geopolitical boundaries, historically significant locations, or culturally significant locations that appear in the view of the environment.

18. The system of claim 16, wherein: The processing step includes performing an image transformation on at least some of the geospatial data to obtain transformed image content; and The image transformation is based on the position data and the pose data.

19. The system of claim 18, wherein: The image transformation generates the transformed image content having an alignment, orientation, perspective, and scale consistent with the view of the environment represented by the received video data.

20. The system of claim 18, wherein: The image transformation generates the transformed image content based on calibration data of an imaging system that captured the video data.

Images