System and method for video display

Abstract

Systems and methods for video display, embodiments provide an aircraft synthetic vision system ("SVS") that provides passengers with an external view. Omitting windows reduces cost, reduces weight, and simplifies the design and manufacture of hypersonic aircraft. Unlike modern SVS, there is no lag between the external view and the actual aircraft motion. Passengers do not feel motion sickness associated with a mismatch in visual and vestibular system feedback. The lag can be eliminated by (a) predicting the internal motion of the aircraft based on sensor feedback, and (b) displaying a camera image that is transformed to match the predicted aircraft orientation when the image is delayed by the display system. Embodiments predict the aircraft orientation and pre-transform using dead reckoning to adjust the video signal and the image captured from the camera based on the sensed aircraft dynamics and the aircraft electronic model to match the orientation when the image is displayed. Compared to traditional SVS, this method is more robust, cheaper and more effective in providing a motion sickness-free view outside the aircraft.

Description

Background Art

[0001] Passengers in moving vehicles (including ground transportation, marine vessels, submersibles, and aircraft) often suffer from various forms of motion sickness. In aircraft in particular, providing passengers with an outside view significantly reduces claustrophobia and motion sickness. Adding traditional windows increases aircraft manufacturing and maintenance costs, and also increases aircraft weight. Windows made of conventional transparent materials may also be thermally infeasible for various hypersonic aircraft now being developed, at least due to the temperature limitations of such materials.

[0002] Conventional contemporary electronic video displays (sometimes called "synthetic vision systems" or "SVS") used in place of windows on aircraft avoid the disadvantages of windows, but also directly contribute to and exacerbate the severity of motion sickness because the displayed external view lags behind the actual motion of the aircraft, even when state-of-the-art low-latency camera and display systems are used. The presence of any lag or delay, even low lag or delay, is sufficient to create a mismatch between the passenger's visual and vestibular system feedback, resulting in motion sickness. Summary of the invention

[0003] The present application describes a display system. The display system includes an inertial measurement system (IMS) configured to sense dynamic characteristics of an aircraft; a camera configured to image an external scene; a video display system inside the aircraft; and an image processing subsystem. The image processing subsystem includes a video processor. The image processing subsystem is configured to access an electronic model of the aircraft and compare the dynamic characteristics sensed by the IMS with the electronic model to generate a predicted movement of the aircraft, estimate the lag between the scene and the external scene imaged by the camera based on the predicted movement, and generate a video signal pre-compensated for the lag to drive the video processor to display a lag-compensated video image of the external scene on the video display system.

[0004] A method for video display is also described. The method includes sensing the dynamic characteristics of an aircraft using an inertial measurement system (IMS); imaging an external scene using a camera; displaying the external scene imaged by the camera using a video display system inside the aircraft; and configuring an image processing subsystem including a video processor to access an electronic model of the aircraft and compare the dynamic characteristics sensed by the IMS with the electronic model. The comparison produces a predicted movement of the aircraft; estimating the lag between the scene and the external scene imaged by the camera based on the predicted movement; and generating a video signal pre-compensated for the lag to drive the video processor to display a lag-compensated video image of the external scene on the video display system.

[0005] One or more non-transitory computer-readable media are also described, having computer-executable instructions stored thereon for video display, which when executed by a computer cause the computer to perform operations. The operations include sensing dynamic characteristics of an aircraft; imaging an external scene using a camera; displaying the external scene imaged by the camera using a video display system inside the aircraft; configuring an image processing subsystem including a video processor to access an electronic model of the aircraft and compare the dynamic characteristics sensed by the IMS with the electronic model. The comparison produces a predicted movement of the aircraft; estimating a lag between the scene and the external scene imaged by the camera based on the predicted movement; and generating a video signal pre-compensated for the lag to drive the video processor to display a lag-compensated video image of the external scene on the video display system.

[0006] The present invention summary is provided to introduce some concepts in a simplified form, and these concepts will be further described in the following detailed description. When read in conjunction with the accompanying drawings, the foregoing invention summary and the following specific description of certain embodiments will be better understood. The present invention summary is neither intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] These and other features, aspects and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings, in which:

[0008] Figure 1 is a block diagram illustrating an embodiment of a display system according to one embodiment.

[0009] Figure 2 is a flow chart illustrating an implementation of a display system using a rigid body model as an electronic model according to one implementation.

[0010] Figure 3 is a flow chart illustrating an embodiment of a display system using a structural dynamics model as an electronic model and using different display images according to one embodiment.

[0011] Figure 4 It is a graphical representation of the forces, movements, and rotations of an aircraft using an aerodynamic model as an electronic model to implement the display system.

[0012] Figure 5 is a diagram of an external scene when imaged by a camera according to one embodiment.

[0013] Figure 6 is an illustration of an external scene as the image is displayed to passenger(s) on a synthetic vision system (SVS) according to one embodiment.

[0014] Figure 7 is an illustration of a mismatch between an exterior scene as imaged by a camera and the exterior scene as the image is displayed to passenger(s) on an SVS, according to one embodiment.

[0015] Figure 8 is an illustration of a transformation of an imaged external scene according to one embodiment.

[0016] Fig. 9 is an illustration of a transformed imaged external scene according to an embodiment such that the transformed external scene matches the actual external scene.

[0017] Fig.10 is a flow chart illustrating a method for video display according to one embodiment.

[0018] Fig.11 is a flow chart illustrating a method for video display according to one embodiment, the method also including performing dead reckoning to adjust the video signal.

[0019] Fig.12 is a flow chart illustrating a method for video display according to one embodiment, the method also comprising transforming the camera output using an inertial measurement system (IMS) output and the camera output with an electronic model to compensate for lag.

[0020] Fig.13 is a schematic perspective view of an aircraft according to one embodiment.

[0021] Fig.14 is an illustrative perspective view of a portion of an aircraft interior according to one embodiment.

[0022] Fig.15 is a block diagram illustrating an operating environment according to one embodiment.

[0023] Corresponding reference numerals indicate corresponding parts throughout the drawings according to the embodiments. DETAILED DESCRIPTION

[0024] Referring to the accompanying drawings, embodiments of the present disclosure include systems and methods for video display that provide passengers in a moving vehicle, particularly an aircraft, with an outside view that reduces claustrophobia and motion sickness. Utilizing such a video display eliminates the need for conventional windows. Omitting such conventional windows in aircraft construction can reduce aircraft manufacturing and maintenance costs, reduce aircraft weight, and make fuselage structure penetrations unnecessary for the purpose of providing an outside view. Fewer structural penetrations enhance and extend the useful life of the structure. In addition, not using such conventional windows simplifies the design and construction of hypersonic aircraft by eliminating or substantially reducing the need for conventional windows made of temperature-limiting materials that may not be thermally feasible.

[0025] Systems and methods for electronic video displays (synthetic vision systems or "SVS") are disclosed that are characterized by no lag (or reduced lag) between the displayed external view and the actual motion of the aircraft, so that passengers do not experience discomfort due to lag or delays that create a mismatch between passenger vision and vestibular system feedback. Thus, the visual-vestibular mismatch vector for motion sickness is eliminated.

[0026] In various embodiments, the elements described herein operate in an unconventional manner to provide a lag-free video display by (a) predicting the motion of the interior of an aircraft based on sensor feedback representing aircraft dynamics (e.g., the orientation of control surfaces and accelerometer measurements), and (b) displaying a camera image that is rotated or translated to match the predicted orientation of the aircraft to compensate for the delay of the display system. Thus, the present disclosure is superior to conventional embodiments of synthetic vision systems that do not or do not adequately eliminate lag and therefore have no protection (or unsatisfactory protection) against motion sickness caused by a mismatch between passenger vision and vestibular system feedback. The performance of the embodiments of the video display disclosed herein (measured by the ability to provide an apparently lag-free and apparently live video stream of the external environment of a moving vehicle) is substantially equal to and sometimes exceeds conventional existing contemporary video display / synthetic vision systems with designs that introduce inherent and unavoidable lag.

[0027] Furthermore, unlike other conventional synthetic vision systems, the present disclosure does not attempt to reduce latency in any camera or display system. Such reduction is technically complex and (often prohibitively) expensive. Instead, the disclosed display systems and methods predict the orientation of a vehicle (particularly an aircraft) and pre-transform images captured from the camera to match that orientation when the images are displayed to the passengers. This transformation is accomplished by using dead reckoning to generate predicted movement of the vehicle. Some embodiments use dead reckoning to adjust the video signal based on Euler angles and an electronic model of the aircraft.

[0028] Thus, the present disclosure is mechanically more robust and cost-effective to implement, and is more effective than conventional synthetic vision systems in providing an apparent real-time view of the vehicle's external environment while avoiding inducing motion sickness and / or airsickness.

[0029] Reference again Figure 1 , a block diagram illustrates an embodiment of a display system 100 according to one embodiment. The display system 100 includes an inertial measurement system (IMS) 180 configured to sense dynamic characteristics 182 of an aircraft 112. In some embodiments, the IMS 180 includes at least one sensor 102, and in other embodiments, the dynamic characteristics 182 include at least one of: force, torque, rotation rate, or moment 150. In some embodiments including the sensor 102, illustrated are sensor 102a, sensor 102b, sensor 102c, and sensor 102d. The display system 100 also includes a camera 104 configured to image an external scene 152 and a video display system 106 for the interior of the aircraft 112. In some embodiments, the video display system 106 is configured to display the external scene 152 imaged by the camera 104. The display system 100 also includes an image processing subsystem 108, which includes a video processor 110. In some embodiments, the video processor 110 includes a frame buffer to store and subsequently process each video frame, ensuring display at a constant frame rate and avoiding overcompensation. In some embodiments of the display system 100, the camera 104 is outward facing and has a known field of view, a known position, and a known delay. Likewise, in some embodiments of the display system 100, the video display system 106 includes a plurality of display screens 116 sized and shaped to be aircraft windows.

[0030] Image processing subsystem 108 is configured to access electronic model 120 of aircraft 112 and compare dynamic characteristics 182 sensed by IMS 180 with electronic model 120 to generate predicted movement 184 of aircraft 112, estimate lag 122 between scene 124 and external scene 152 imaged by camera 104 based on predicted movement 184, and generate video signal 126 that has been pre-compensated for lag 122 to drive video processor 110 to display delay-compensated video image 128 of external scene 152 on video display system 106. Scene 124 may represent an image that would be displayed in the absence of lag 122.

[0031] In some embodiments, the hysteresis 122 of the display system 100 is a known constant. In such embodiments, the one or more stored hysteresis 122 is determined prior to use of the display system 100, including but not limited to during: initial engineering or design of the display system 100; measurement or testing of a pre-production prototype of the display system 100; or measurement or testing of the display system 100 before or after the display system 100 is installed in the aircraft 112. The one or more stored hysteresis 122 is stored as described elsewhere herein.

[0032] In some embodiments, the electronic model 120 (e.g., a motion model) calculates the response of the interior of the aircraft 112 over time to the measured and / or predicted dynamic characteristics 182. In some embodiments of the display system 100, the lag 122 is related to (i) a delay time 132 between sensing an image 154 corresponding to the external scene 152 at the camera 104 and outputting a video signal 156 from the camera 104, (ii) a processor lag time 134 of the video processor 110, and (iii) a lag time 136 between outputting a video signal 126 pre-compensated for the delay 122 and displaying the external scene 152 corresponding to the video signal 126 on the video display system 106. In some such embodiments, at least one of the camera 104, the video display system 106, and the video processor 110 has a known delay 158, such that any detected change in the known delay 158 with respect to at least one of the camera 104, the video display system 106, or the video processor 110 indicates at least one of a failure or a disturbance in at least one of the camera 104, the video display system 106, or the video processor 110.

[0033] In some other embodiments of the display system 100, the electronic model 120 of the aircraft 112 includes (i) one or more models that predict how the aircraft will move in response to control inputs 140 (e.g., engine thrust, etc.), and (ii) one or more models 142 that define the amount of turning of the aircraft when one or more forces, torques, rotation rates, or moments are applied to the aircraft. Various other embodiments include rigid and / or dynamic models as described in more detail in this disclosure.

[0034] The image processing subsystem 108 is also configured to generate a video signal 126 pre-compensated for the lag 122 to drive the video processor 110 to display a lag-compensated video image 128 of the external scene 152 on the video display system 106. In some embodiments of the display system 100, the image processing subsystem 108 is also configured to perform dead reckoning to generate a predicted movement of the aircraft. In some embodiments, the dead reckoning is performed based on the Euler angles 130 of the aircraft 112 and the electronic model 120.

[0035] In some embodiments of the display system 100, the video processor 110 receives outputs from the inertial measurement system 186 (IMS outputs) from the IMS 180 and camera outputs 192 from the camera 104, and transforms the camera outputs 192 using the IMS outputs 186 and the electronic model 120 to compensate for the lag 122. In such embodiments, the transformation includes translating or rotating at least one of the external scenes 152 imaged by the camera 104. The translation and rotation aligns the image output 194 corresponding to the external scene 152 imaged by the camera 104 at a first time with a predicted orientation 196 of the aircraft 112 at a second time when the external scene 152 is to be displayed on the video display system 106. In some embodiments, the IMS outputs 186 include at least one sensor output 190 of at least one of the sensors 102, including but not limited to the sensors 102a, 102b, 102c, and 102d. For example, the sensor 102a may be a system configured to measure the deflection of a control surface such as an elevator of the aircraft. In some embodiments, the sensor(s) 102 may be physical sensors attached to control surfaces. In other embodiments, the output of the sensor(s) 102 may be calculated or estimated values ​​(e.g., commanded or measured positions of control surfaces). The motion model may use the sensor outputs to predict the motion of the aircraft 112.

[0036] Some embodiments of the display system 100 also include a plurality of video processors 160 (e.g., video processors 110) and a plurality of video display systems 162 (e.g., video display system 106). In such embodiments, the electronic model 120 includes responses 114 of the seat groups 164 to the dynamic characteristics 182 of the aircraft 112, and wherein each seat group 164 is associated with at least one video processor 166 (e.g., video processor 110) and at least one video display system 168 (e.g., video display system 106). The at least one video processor 166 and the at least one video display system 168 associated with each seat group 164 communicate with each other.

[0037] Various embodiments use one or more motion models, as described in more detail below. For example, one or more embodiments of the present disclosure use one or more of a kinematic model, a dynamic model, a structural model, and the like.

[0038] Figure 2 is a flow chart illustrating an embodiment of a display system using a rigid body model 200 as an electronic model according to an embodiment. In some embodiments, the electronic model is Figure 1 The electronic model 120 of the display system using the rigid body model 200 is applicable to a display system (e.g., Figure 1 100) of a rigid body vehicle (e.g., Figure 1 The rigid body model 200 defines how the system behaves given a known initial motion and known application dynamics (e.g., Figure 1 The IMS (e.g., Figure 1 The IMS 180 measures the rate of change of vehicle orientation along up to six axes (x, y, z, pitch, roll and yaw). The number of axes depends on what is appropriate for the specific application.

[0039] The calculations performed by the rigid body model 200 begin with image capture at time t0 at operation 202. At operation 204, when t≌t0, (one or more) angular rates and (one or more) accelerations are measured at time t. At operation 206, a kinematic model (e.g., a six-degree-of-freedom (6DoF) model) provides a predicted lag-free image view position and angle at time t0+Δt. Such a kinematic model predicts how a rigid body moves over time when the motion of other points over time is known. In some embodiments, Δt is the sum of: (i) the lag time between when an image corresponding to an external scene is sensed at a camera and when a video signal is output from the camera, (ii) the processor lag time of a video processor, and (iii) the lag time between the output of a video signal pre-compensated for lag and the display of an external scene corresponding to the video signal on a video display system. At operation 208, the image is transformed to the predicted position and angle. The transformed image is displayed at operation 210. Thereafter, the work of the rigid body model 200 is completed.

[0040] In an assumption of a rigid vehicle with a constant rotation rate (e.g., the vehicle pitches up at a constant 2 degrees per second), where the total system lag of the display system is 0.5 seconds, the vehicle is imaged in the external lens (e.g., Figure 1 The external scene 152) is viewed from a camera (e.g., Figure 1 camera 104) to a video display system (e.g., Figure 1 To compensate, the display system rotates the image displayed (e.g., Figure 1 A 1 degree rotation correction is introduced in the scene 124).

[0041] In another assumption of a rigid aircraft with a varying rate of rotation (i.e., subject to angular acceleration) across and through the aircraft, kinematics is used to introduce appropriate corrections by multiplying the acceleration by the lag time. Such calculations use at least the rate of rotation and acceleration measurements from the IMS, but in some embodiments are also refined using third and fourth order derivatives of this data. In some embodiments where the aircraft is traveling at very high speeds and close to the ground, clouds, or other airborne obstacles, linear translation is used to perform the corrections. In other words, the display system detects, for example, that the aircraft is flying through clouds and determines that the clouds were at the leading edge of the aircraft's wing when the camera took the last image. By the time the image is displayed, the clouds will need to be at the trailing edge of the wing, so an appropriate corrective transformation will need to be performed.

[0042] Figure 3 is a diagram showing the use of the structural dynamics model 300 as an electronic model (eg, Figure 1120) and a flowchart of an embodiment of a display system for displaying different images. The display system implemented using the structural dynamics model 300 is suitable for a display system (e.g., Figure 1 Applications of non-rigid (i.e., structural dynamics) vehicles, such as the display system 100 Figure 1 112 in certain embodiments. A structural dynamics vehicle does not remain rigid while in motion. Rather, in a structural dynamics vehicle, when the vehicle is already moving in a set direction at a set speed while controlling the moving parts of the surface, the vehicle will change course and / or speed accordingly. The structural dynamics model 300 defines how the system behaves in a moving structural dynamics vehicle, where given a known initial motion and known applied dynamics (e.g., Figure 1 The dynamic characteristics of the vehicle 182) are as time passes, and each point on the vehicle moves as time passes. IMS (e.g., Figure 1 The IMS 180 measures the rate of change of the vehicle's orientation along up to six axes (x, y, z, pitch, roll and yaw). The number of axes depends on what is appropriate for the particular application. The structural dynamics model 300 is particularly important in any application where the inability to compensate for different rotation rates across the vehicle could exacerbate motion sickness.

[0043] In operation 302, the calculations performed by the structural dynamics model 300 begin with image capture at time t0. In operation 304, the structural dynamics model 300 estimates forces and moments within a time range starting from t0 and continuing to time t0+Δt. In some embodiments, Δt is the sum of: (i) the lag time between when an image corresponding to an external scene is sensed at a camera and a video signal is output from the camera, (ii) the processor lag time of a video processor, and (iii) the lag time between the output of a video signal pre-compensated for lag and the display of an external scene corresponding to the video signal on a video display system. In operation 306, the structural dynamics model 300 provides a predicted lag-free image view position and angle at time t0+Δt. In operation 308, the image is transformed to the predicted position and angle. In operation 310, the transformed image is displayed. Thereafter, the work of the structural dynamics model 300 is completed.

[0044] In some embodiments of the structural dynamics model 300, forces are measured with strain gauges on the vehicle, and angular moments and / or angular rates are measured with rate gyroscopes. Some other embodiments measure accelerations of various parts of the structural dynamics vehicle and use the structural dynamics model 300 to estimate the forces that produce those accelerations. Still other embodiments measure the deflections of aerodynamic control surfaces (e.g., ailerons, elevators, etc.) and use the structural dynamics model 300 to estimate the forces produced by the control surfaces.

[0045] In other embodiments of the structural dynamics model 300, the deflection of the aerodynamic control surfaces is predicted by interrogating the flight control system and using the aerodynamic model to estimate the forces generated by the control surfaces over time. In some such embodiments, the flight control system is a proportional integral derivative (PID) controller that moves each control surface to a known target position at a known rate. PID controllers are instruments used in industrial control applications to regulate temperature, flow, pressure, velocity, and other process variables. Such embodiments can well predict and model the effects of forces that will occur even after the image transformation is completed. Similarly, certain predictive embodiments of the structural dynamics model 300 easily predict the aerodynamic forces from external impactors (such as updrafts or side gusts) by measuring acceleration, or such impactors are generated until time t0 and using a gust model (e.g., based on previous statistical models encountering similar gusts) to predict the forces that the impactor will generate in the next Δt interval. Therefore, the effects of forces can be predicted and modeled even after the image transformation is implemented.

[0046] Some embodiments of the structural dynamics model 300 combine any number of the aforementioned calculations and predictions, in whole or in part, to accomplish the image transformation.

[0047] Figure 4 is a graphical representation of the forces, movements, and rotations of an aircraft using an aerodynamic model 400 as an electronic model to implement a display system. In some embodiments, the electronic model is Figure 1 The electronic model 120 of the embodiment of the display system using the aerodynamic model 400 is suitable for the display system (e.g., Figure 1 The aerodynamic model 400 defines the aerodynamic sounds of any vehicle (e.g., an aircraft) required for the display system 100. Figure 1 The dynamic characteristics generated at different locations on the aircraft 112 (e.g., Figure 1 The aircraft is described in detail in terms of dynamic characteristics 182) of an aircraft moving through air of known density at a known speed and in a known orientation, wherein the known orientation includes measured or estimated positions of control surfaces (e.g., ailerons, elevators, etc.).

[0048] Assuming that aircraft 450 uses aerodynamic model 400 as the electronic model to implement the display system, elevator 406 deflects a measured angle 402. Given the current airspeed, the deflection of elevator 406 produces an upward force F on elevator 406. z_tail 404. Given the moment arm x from the center of gravity 410 ("CG") tail 408, the force generated by the equation t y=(F z_tail )(X tail ) defines the torque t y 412. Torque t y 412 pushes the nose of the aircraft 450 downward at an angular acceleration α (not shown). The angular acceleration α is defined as α=t y / I y , where t y t y 412, and I y (not shown) is the current moment of inertia of the vehicle 450 about the y-axis (not shown) through the CG 410. In some embodiments, as the vehicle 450 consumes fuel or pumps fuel from one fuel tank (not shown) to another, I y Changes over time.

[0049] In the camera 422 (e.g. Figure 1 At the moment when the camera 104 captures the image, the nose of the aircraft 450 is pointed at an angle θ0 414 (not shown) at an angular velocity ω0 (not shown). Given the angular acceleration α, the aircraft 450 rotates by the formula Δθ=θ0+ω0Δt+1 / 2αΔt 2 , where Δθ is Δ(θ414); and in certain embodiments, Δt has the value Figures 2 to 3 424 (e.g., Figure 1 The display system rotates the image through the same angle Δθ so that the image is displayed at an angle θ. t 418 is positioned so as to match the angle of the aircraft 450 when displayed, thereby preventing motion sickness, as described elsewhere in this disclosure. In some embodiments, the aerodynamic model 400 may take into account the torque t y 412 is the effect of the curvature of aircraft 450. Thus, aerodynamic model 400 may adjust the predicted motion based on offset 420 from center of gravity 410 based on the curvature of the airframe.

[0050] Figure 5-7 Together, there is shown the external scene as perceived by a passenger traveling in a vehicle equipped with an SVS that is unable to correct for the lag time introduced between the time an image of the external scene outside the vehicle is captured and the time such external scene is displayed to the passenger. Figure 5 is a diagram of an external scene 500 as imaged by a camera according to one embodiment. Figure 6is an illustration of an external scene 600 as images are displayed to a passenger on an SVS, according to one embodiment. Figure 7 is an illustration 700 of a mismatch between an external scene 500 as imaged by a camera and an external scene 600 as the image is displayed to a passenger on an SVS, wherein the external scene 600 is presented as a wireframe overlay on the external scene 500, according to one embodiment. Figure 7 The mismatch between the external scene 500 and the external scene 600 shown in the diagram 700 may cause motion sickness in passengers using such SVS implementations. The display systems and methods disclosed herein (including but not limited to the display system 100, the method 1000, the method 1100 and the method 1200) solve the problem caused by Figures 5 to 7 Sources of motion sickness shown.

[0051] Figure 8-9 The operation of the SVS incorporating the display systems and / or methods disclosed herein (including but not limited to display system 100, method 1000, method 1100, and method 1200) is collectively shown to prevent the following: Figure 7 The mismatch between the external scene when the camera is imaging and the external scene when the image is displayed to the passengers on the SVS is shown. Figure 8 is an illustration 800 of a transformation of an imaged external scene, according to an embodiment. Fig. 9 is an illustration 900 of a transformed imaged external scene according to an embodiment such that the transformed external scene matches the actual external scene.

[0052] Fig.10 is a flow chart illustrating a method 1000 for video display according to one embodiment. In some embodiments, Fig.10 The process shown in is performed by the display system, e.g. Figure 1 The process uses an inertial measurement system (IMS) to sense dynamic characteristics of an aircraft at operation 1002; uses a camera to image an external scene at operation 1004; uses a video display system inside the aircraft to display the external scene imaged by the camera at operation 1006; configures an image processing subsystem including a video processor to access an electronic model of the aircraft at 1008; and compares the dynamic characteristics sensed by the IMS with the electronic model to generate a predicted movement of the aircraft at operation 1010, estimates a lag between the scene and the external scene imaged by the camera based on the predicted movement, and generates a video signal pre-compensated for the lag to drive the video processor to display a lag-compensated video image of the external scene on the video display system.

[0053] Some embodiments of method 1000 also include: imaging an external scene using a plurality of cameras (e.g., at operation 1004); and using a plurality of video display systems to display the external scene imaged by the plurality of cameras (e.g., at operation 1010). In such embodiments, the electronic model includes responses of groups of seats within the aircraft to dynamic characteristics of the aircraft, and each group of seats is associated with at least one video processor and at least one video display system. The at least one video processor and at least one video display system are associated with each group of seats in communication with each other.

[0054] exist Fig.10 In some embodiments of the method 1000 shown in , the lag involves (i) a lag time between when an image corresponding to an external scene is sensed at a camera and a video signal is output from the camera, (ii) a processor lag time of a video processor, and (iii) a lag time between the output of a video signal pre-compensated for lag and the display of an external scene corresponding to the video signal on a video display system. In some such embodiments, at least one of the camera, the video display system, and the video processor has a known delay, such that any detected change in the known delay of at least one of the camera, the video display system, or the video processor indicates at least one of a failure or interference in at least one of the camera, the video display system, or the video processor. For example, the lag may represent a time difference between a scene seen by an observer looking out a window and a scene displayed on a screen. Thereafter, the process is complete. Although Fig.10 The operations shown are performed by the display system, but aspects of the present disclosure contemplate performance of the operations by other entities.In some implementations, a cloud service performs one or more operations.

[0055] exist Fig.10 In some other embodiments of the method 1000 shown in , the video display system includes a plurality of display screens sized and shaped to resemble aircraft windows. Fig.10 In other embodiments of the method 1000 shown in , the electronic model of the aircraft includes (i) one or more models that predict how the aircraft moves in response to control inputs, and (ii) one or more models that define an amount of bending of the aircraft when one or more forces or moments are applied to the aircraft.

[0056] Fig.11 is a flow chart illustrating a method for video display 1100 according to one embodiment, further comprising performing dead reckoning to adjust the video signal. In some embodiments, Fig.11 The method shown in Figure 1 Operations 1102, 1104, 1106, 1108, and 1110 are similar to Fig.10 Operations 1002, 1004, 1006, 1008, and 1010 of method 1000 are described in detail in detail in the present specification and therefore will not be repeated in this specification. At operation 1112, the image processing subsystem performs dead reckoning to generate a predicted movement of the aircraft. Thereafter, the process is complete. Fig.11 The operations shown are performed by the display system, but aspects of the present disclosure contemplate performance of the operations by other entities.In some implementations, a cloud service performs one or more operations.

[0057] Fig.12 1 is a flow chart showing a method 1200 for video display according to one embodiment, the method also includes transforming the camera output using an output from an inertial measurement system (IMS output) and a camera output with an electronic model to compensate for lag. In some embodiments, Fig.12 The method shown in Figure 1 Operations 1202, 1204, 1206, 1208, and 1210 are similar to Fig.10 Operations 1002, 1004, 1006, 1008, and 1010 of method 1000 are described in detail in detail in conjunction with the description of the method 1000 and therefore will not be repeated in this specification. At operation 1212, method 1200 receives, using a video processor, an output from an inertial measurement system (IMS output) and a camera output from a camera. The process uses the IMS output and the electronic model to transform the camera output to compensate for lag at operation 1214. The transformation includes translating or rotating at least one of the external scenes imaged by the camera. The translation and rotation align the image output corresponding to the external scene imaged by the camera at a first time with the predicted orientation of the aircraft at a second time when the external scene is to be displayed on the video display system. Thereafter, the process is complete. Although Fig.12 The operations shown are performed by the display system, but aspects of the present disclosure contemplate performance of the operations by other entities.In some implementations, a cloud service performs one or more operations.

[0058] Additional Examples

[0059] Some embodiments of the display systems disclosed herein include a video processor (e.g., Figure 1 The video processor 110 has a known delay and: receives the video of the camera (e.g., Figure 1 video output from camera 104); using sensor data (e.g., from IMS 180 or Figure 1 of the at least one sensor 102) and a computer model (e.g., Figure 1 electronic model 120) to predict the interior of the aircraft (e.g., Fig.14 The orientation of the interior of the aircraft 1400) will be the same as the orientation of the image as it is captured by the camera (e.g., each frame of the video output) as it appears on the display (e.g., Figure 1 The camera initially captures an image of the world outside the aircraft at a time t0 when the aircraft is in a first position and orientation. The sensor data and the model predict the position and orientation of the aircraft at a future time t1, where t1 is t later than t0. Δ The amount, t Δ Equal to the sum of the delays in the camera, display, and video processor.

[0060] Any reference herein to "force, torque, rotation rate, or moment" is intended to mean any combination of at least one of force, torque, rotation rate, or moment.

[0061] In some embodiments, a hysteresis (such as Figure 1 The hysteresis 122) is stored on a computer-readable medium accessible to the system or method disclosed herein. In some such embodiments, the operating environment includes such a computer-readable medium. Fig.15 The related discussion in this paper realizes such an operating environment.

[0062] The video processor transforms the image captured by the camera at t0 into the image that the passenger will see when looking out the window at t1. The delays of the video processor, camera, and display add up so that the transformed image appears on the screen at t1. Therefore, the passenger perceives no lag in the image relative to the motion of the aircraft.

[0063] Unless otherwise noted, any embodiments described herein as being incorporated into or used with a particular type of vehicle (e.g., an aircraft) should be understood to be installable into and used with any other type of vehicle (e.g., a train, a submersible, a tank, an armored personnel carrier, a watercraft, etc.) Specifically, for military applications, the embodiments disclosed herein are well suited for vehicle firing positions and other such stations, allowing the vehicle operator(s) to operate well away from the potentially dangerous exterior of the vehicle (where local hostile forces may be targeting) without having to worry about motion sickness.

[0064] The present disclosure is directed to a video display system and method that reduces motion sickness and increases the accuracy of video in a synthetic vision system used on a moving vehicle. Some embodiments of such a synthetic vision system include, but are not limited to, an external camera mounted near the exterior of the vehicle, wherein associated equipment inside the vehicle is configured to display video from the camera. Some embodiments disclosed herein use any or all of the kinematic model(s), rigid body model(s), structural dynamics model(s), and / or aerodynamic model(s) disclosed herein, to suit the intended application.

[0065] In some embodiments of any of the display systems and methods disclosed herein: (1) any (one or more) cameras (e.g., Figure 1 cameras 104) are outward facing and each have a known field of view, a known position, and a known delay; and (2) any display system (e.g., Figure 1 The video display system 106 of FIG. 1 has a known delay. Such a display system and method is configured to display a field of view that is smaller than the field of view of the camera(s). By utilizing a field of view that is smaller than the camera(s), the display system can effectively inspect portions of the external scene that are not yet visible on the display system but have been captured by the camera(s), and can perform corresponding transformations to achieve zero-lag display.

[0066] Some embodiments use a digital scene generator ("DSG") (not shown) instead of a camera (e.g., Figure 1 The scene generator uses a geographic information system ("GIS") to render images similar to what a camera would produce from a known location on a vehicle equipped with a display system and / or employing the methods of the present disclosure. Hypersonic vehicles are particularly good candidates for use with embodiments incorporating DSG, as the high temperatures of traveling at hypersonic speeds not only make windows impractical, but also often make cameras impractical or expensive.

[0067] In some embodiments, a video display system (e.g., Figure 1The video display system 106 of the present invention is a two-dimensional display. In other embodiments, the video display system is a three-dimensional display configured to provide stereoscopic binocular vision (e.g., by using two cameras, each camera is offset relative to one eye of the passenger). Any type of camera can be used with the present disclosure, including but not limited to a two-dimensional camera, a three-dimensional camera, a plurality of cameras used together or in combination of any type, a camera whose output is stitched together, a depth camera, etc. Ideally, the camera used in the embodiments disclosed herein has a fixed frame rate, providing a fixed delay that is easy to compensate, but the embodiments of the present disclosure can also operate at a variable frame rate. The closer the passenger sits next to the camera, the closer he will feel to the external scene. However, in some embodiments, it is useful to place the camera away from the passenger so as to provide an unobstructed view of the external scene to the passenger whose field of view will be blocked in the presence of a window (e.g., a passenger sitting above the wing of an aircraft).

[0068] In the context of the present disclosure, "no lag," "zero lag," "no lag," and functionally equivalent descriptors should be understood to refer to both actual lag-free implementations and implementations in which lag is negligible to the passenger without a sufficient mismatch between their visual input and their vestibular system to cause motion sickness. Whether a particular implementation is actually lag-free or has such negligible lag depends on the available resources and requirements of the particular application. In all implementations of the present disclosure, the video processing used to transform the displayed scene (e.g., Figure 1 The lead interval of the electronic model of the aircraft (the scene 124 or the lag compensated video image 128) is in whole embodiments equal to the sum of the delays or lags, or nearly equal to the sum of the delays or lags, so that the difference is not sufficient to cause motion sickness due to the mismatch between the passenger's vision and the vestibular system feedback.

[0069] In some embodiments, a group of seats (e.g. Figure 1 164) includes a plurality of adjacent rows of seats. Based on an aircraft (e.g., Figure 1 A number of such groups of seats may be selected based on the flexibility of the interior and / or fuselage of the aircraft 112 in the aircraft. If the fuselage of the aircraft is extremely rigid, then any force, torque, or moment (e.g., aileron deflection) will change the orientation of all seats in the same way at the same time, so that one group of seats is sufficient. If the fuselage of the aircraft is less rigid, then the same force, torque, or moment will change the orientation of some seats before other seats, and in different directions and to different degrees than other seats, or any combination of the foregoing. In such a case, over time t ΔThe predicted change in orientation is sometimes correct for some seat groups and incorrect for others, which requires a separate video processor to be dedicated to each seat group. The video processor associated with each seat group uses the kinematic data and an electronic model (e.g. Figure 1 104) to predict the movement of the group of seats, transform the output of a camera (e.g., camera 104) to match the predicted movement, and send the transformed output of the camera to (one or more) displays associated with the group of seats (e.g., Figure 1 Some such embodiments use smaller seating groups (or even single seats) in premium sections of the aircraft interior to provide a better match between the displayed image and the exterior scene.

[0070] This article includes aircraft (e.g., Figure 1 In some embodiments of the aircraft 112, the IMS (e.g., Figure 1 The IMS 180 also includes, but is not limited to, accelerometers on the nose, tail, or wing(s) and / or encoders for measuring the orientation of flight control surfaces. In some embodiments disclosed herein, the IMS measures accelerations on six axes (x, y, z, pitch, roll, and yaw). Many modern aircraft contain factory-installed sensors or factory-installed IMS suitable for use with the disclosure herein when paired with a structural dynamics model of the specific aircraft to ensure proper interpretation of the sensor or IMS data.

[0071] Dead reckoning is an open-loop navigation method based on determinations that do not rely on celestial navigation based on records of routes traveled or flown, distance traveled (e.g., based on speed estimates), a known starting point, and a known or estimated drifting position of a ship or aircraft. Dead reckoning relies on measurements of heading and distance traveled without any external reference (e.g., landmarks) to verify or correct the resulting prediction of the current position. Certain embodiments of dead reckoning incorporate compass-based navigation techniques (e.g., traveling from A to B, where B is known to deviate from true north by a certain angle). Euler angles are three angles used to describe the orientation of a rigid body relative to a fixed coordinate system. Euler angles also represent the orientation of a physically movable reference system, or represent the orientation of a general basis in three-dimensional linear algebra. Here, Euler angles represent pitch, roll, and yaw. Given a particular reference system, Euler angles determine how to measure the orientation of an object (or part of an object) relative to the reference system. In some implementations herein, dead reckoning refers to an open-loop method that estimates changes in orientation based on elapsed time and measured or predicted rates of change of one or more Euler angles.

[0072] As used herein, a material or component described using the terms "transparent" or "translucent" means that light can be transmitted through the material and emitted from the other side of the material. The term "transparent" indicates a greater amount of light transmittance than the term "translucent", such that a transparent material has less light distortion, diffusion, and / or attenuation than a translucent material. In the present disclosure, unless explicitly stated, the use of the term "translucent" to describe a material or component is not intended to exclude materials that are also transparent. In some embodiments, a material or component described as "translucent" means that the material or component is at least translucent and may also be (but not always) transparent.

[0073] In the context of the present disclosure, "imaging" when used as a verb includes but is not limited to forming a visual representation of an object by scanning it with a detector or electromagnetic beam. "Camera" as used herein includes but is not limited to any image capture device used for electronic motion picture acquisition. "External scene" refers to the portion of the real world around which the systems and methods disclosed herein receive information in order to provide a lag-free display.

[0074] In some embodiments, at least part of the functions of each element in the drawings is performed by other elements in the drawings and / or entities (eg, computers) not shown in the drawings.

[0075] In some embodiments, Fig.10 , Fig.11 and Fig.12 The operations shown in the figure are performed by a single person, a group of people, a fully or partially automated video display system, or any combination of the foregoing. As an illustration, in some embodiments, the camera and IMS are provided by different suppliers to a completely independent assembler who couples the camera and IMS to the aircraft.

[0076] Although aspects of the disclosure have been described in terms of various embodiments and their associated operations, those skilled in the art will appreciate that combinations of operations from any number of different embodiments are also within the scope of aspects of the disclosure.

[0077] Although aspects of the present disclosure do not track personally identifiable information, embodiments have been described with reference to data monitored and / or collected from users. In some embodiments, the user is provided with notification of data collection (e.g., via a dialog box or preference setting), and is given an opportunity to give or deny consent for monitoring and / or collection. Permission sometimes takes the form of opt-in consent or opt-out consent.

[0078] Exemplary operating environment

[0079] The present disclosure may be used in a variety of applications in a variety of terrestrial and off-terrestrial environments. For illustrative purposes only, and not intended to limit the possible operating environments in which embodiments of the present disclosure may operate, the following exemplary operating environments are presented. Fig.13 In some embodiments of the aircraft 1300, the present disclosure may be operated within an aircraft operating environment. Embodiments of the aircraft 1300 include, but are not limited to, an aircraft body 1302 having at least one camera 1316 in the vicinity, a plurality of high-level systems 1304, and an interior 1306. In some embodiments, the at least one camera 1316 is Figure 1 Some embodiments of the aircraft 1300 incorporate high-level systems 1304, including, but not limited to, one or more of a propulsion system 1308, an electrical system 1310, a hydraulic system 1312, and an environmental system 1314. Any number of other systems may be included in embodiments of the aircraft 1300. Although an aviation embodiment is shown, the principles are applicable to other industries, such as the automotive and marine industries.

[0080] Embodiments of the present disclosure also include aircraft (e.g., Fig.13 The exact elements and configuration of the aircraft interior 1400 will vary depending in part on the elements and configuration of the associated aircraft 1300. For purposes of illustration only, and not intended to limit the possible aircraft interiors 1400 encompassed by embodiments of the present disclosure, the following exemplary aircraft interior 1400 is provided. Fig.14 1400, the present disclosure may be operated within an aircraft interior operating environment. Embodiments of aircraft interior 1400 include, but are not limited to, at least one passenger seat 1402 and an associated at least one interior video display 1404. In various embodiments of the present disclosure, at least one passenger seat 1402 may be configured as a chair, a bench, or any other device suitable for allowing an aircraft passenger to sit safely and securely during aircraft operations. In some embodiments, the associated at least one interior video display 1404 is a video display system 106 or Figure 1 1400. Any number of other elements may be included in embodiments of aircraft interior 1400. Although embodiments are shown, the principles are also applicable to other industries, such as the automotive and marine industries.

[0081] The present disclosure may be used in accordance with Fig.151500. In such an embodiment, the components of the computing device 1518 may be implemented as part of an electronic device according to one or more embodiments described in this specification. The computing device 1518 includes one or more processors 1519, which may be microprocessors, controllers, or any other suitable type of processor for processing computer executable instructions to control the operation of an electronic device. Platform software including an operating system 1520 or any other suitable platform software may be provided on the device 1518 to enable application software 1521 to execute on the device. According to one embodiment, the video display described herein may be implemented by software.

[0082] Computer executable instructions may be provided using any computer readable medium accessible to computing device 1518. Computer readable media may include, but are not limited to, computer storage media and communication media such as memory 1522. Computer storage media such as memory 1522 include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules, etc. Computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage devices, cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by a computing device. In contrast, communication media may embody computer readable instructions, data structures, program modules, etc. in a modulated data signal such as a carrier wave or other transmission mechanism. As defined herein, computer storage media do not include communication media. Therefore, computer storage media themselves should not be interpreted as propagating signals. The propagated signal itself is not an example of a computer storage medium. Although computer storage media (memory 1522) is shown within computing device 1518, those skilled in the art will appreciate that the storage may be distributed or located remotely and may be accessed via a network or other communications link (eg, using communications interface 1523).

[0083] The computing device 1518 may include an input / output controller 1524, which is configured to output information to one or more output devices 1525, which in some embodiments may be a display or speaker that can be separated from or integrated into an electronic device. The input / output controller 1524 may also be configured to receive and process input from one or more input devices 1526, which in some embodiments may be a keyboard, microphone, or touch pad. In one embodiment, the output device 1525 may also serve as an input device. A touch-sensitive display is such a device. The input / output controller 1524 may also output data to a device other than the output device, such as a locally connected printing device. In some embodiments, a user may provide input to (one or more) input devices 1526 and / or receive output from (one or more) output devices 1525.

[0084] The functions described herein are at least partially performed by one or more hardware logic components. According to one embodiment, when the program code is executed by the processor 1519, the computing device 1518 is configured by the program code to perform the described operations and functional embodiments. Alternatively or in addition, the functions described herein are at least partially performed by one or more hardware logic components. Without limitation, illustrative types of hardware logic components available include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), specific program standard products (ASSPs), systems with systems on chip (SOCs), complex programmable logic devices (CPLDs), and graphics processing units (GPUs).

[0085] Thus, various embodiments include systems and methods for video display that use an inertial measurement system (IMS) to sense dynamic characteristics of an aircraft; use a camera to image an external scene; use a video display system inside the aircraft to display the external scene imaged by the camera; configure an image processing subsystem including a video processor to access an electronic model of the aircraft and compare the dynamic characteristics sensed by the IMS with the electronic model to generate a predicted movement of the aircraft, estimate the lag between the scene and the external scene imaged by the camera based on the predicted movement, and generate a video signal pre-compensated for the lag to drive the video processor to display a lag-compensated video image of the external scene on the video display system.

[0086] As described herein, the present disclosure provides systems and methods for lag-free video display. The systems and methods herein are effectively and efficiently constructed and deployed in lag-free video displays suitable for use in many mobile vehicles (including but not limited to the above exemplary operating environments).

[0087] Although various spatial and directional terms may be used to describe the present disclosure, such as top, bottom, lower, middle, side, horizontal, vertical, front, etc., it should be understood that these terms are used only with respect to the orientation shown in the drawings. The orientation may be inverted, rotated, or otherwise changed so that up is down and vice versa, horizontal becomes vertical, etc.

[0088] As used herein, a structure, limitation, or element "configured to" perform a task or operation is specifically structurally formed, constructed, or adjusted in a manner corresponding to the task or operation. For clarity and avoidance of doubt, an object that can merely be modified to perform a task or operation is not "configured to" perform a task or operation as used herein.

[0089] As will be apparent to one skilled in the art, any range or value given herein may be expanded or altered without losing the effect sought.

[0090] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as exemplary forms of implementing the claims.

[0091] It will be understood that the above benefits and advantages may relate to one embodiment or may relate to several embodiments. These embodiments are not limited to embodiments that solve every problem discussed in the background technology of this article, nor are they limited to embodiments that have any or all of the benefits and advantages.

[0092] The implementations illustrated and described herein, as well as implementations not specifically described herein but within the scope of aspects of the claims, constitute exemplary means for video display.

[0093] Unless otherwise noted, the execution order or the order of carrying out the operations in the embodiments of the present disclosure illustrated and described herein are not essential. That is, unless otherwise noted, the operations can be carried out in any order, and the examples of the present disclosure may include more or less operations than the operations disclosed herein. As an illustration, it is also within the scope of the various aspects of the present disclosure to expect to carry out or carry out a specific operation before, at the same time, or after another operation.

[0094] When introducing elements of aspects of the present disclosure or embodiments thereof, the articles "a," "an," "the," and "said" are intended to indicate that there are one or more elements. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements in addition to the listed elements. The term "exemplary" is intended to mean "example." The phrase "one or more of: A, B, and C" means "at least one of A and / or at least one of B and / or at least one of C."

[0095] Having described various aspects of the disclosure in detail, it will be apparent that modifications and variations may be made without departing from the scope of the various aspects of the disclosure as defined in the appended claims. As various changes may be made to the above constructions, products and methods without departing from the scope of the various aspects of the disclosure, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

[0096] It should be understood that the above description is intended to illustrate rather than limit. As an illustration, the above embodiments (and / or aspects thereof) can be used in combination with each other. In addition, without departing from its scope, many modifications are feasible to adapt specific situations or materials to the teachings of various embodiments of the present disclosure. Although the size and type of materials described herein are intended to define the parameters of various embodiments of the present disclosure, these embodiments are by no means restrictive, but exemplary embodiments. After browsing the above description, many other embodiments will be obvious to those of ordinary skill in the art. Therefore, the scope of various embodiments of the present disclosure should be determined with reference to the attached claims and the full range of equivalents enjoyed by these claims. In the attached claims, the terms "including" and "wherein" are used as the plain English equivalents of the corresponding terms "including" and "wherein". In addition, the terms "first", "second" and "third" and the like are used only as labels and are not intended to impose numerical requirements on their objects. In addition, the limitations of the following claims are not written in the format of device plus function, nor are they intended to be interpreted based on 35 U.S.C. §112 (f), unless such claim limitations explicitly use the phrase "means for ..." and then make a functional description without further structure.

[0097] This written description uses examples to disclose various embodiments of the present disclosure, including the best mode, and also enables a person of ordinary skill in the art to practice the various embodiments of the present disclosure, including making and using any device or system and performing any incorporated method. The scope of patent protection for the various embodiments of the present disclosure is defined by the claims, and includes other examples that occur to a person of ordinary skill in the art. If these examples have structural elements that do not differ from the literal language of the claims, or if these examples include equivalent structural elements that do not differ substantially from the literal language of the claims, such other examples are intended to be within the scope of the claims.

[0098] The following clauses describe further aspects:

[0099] 1. A display system (100), comprising:

[0100] an inertial measurement system (IMS) (180) configured to sense dynamic characteristics (182) of the aircraft (112);

[0101] a camera (104) configured to image an external scene (152);

[0102] a video display system (106) inside the aircraft (112); and

[0103] An image processing subsystem (108) includes a video processor (110) configured to access an electronic model (120) of an aircraft (112) and compare dynamic characteristics (182) sensed by an IMS (180) with the electronic model (120) to generate a predicted movement (184) of the aircraft (112), estimate a lag (122) between a scene (124) and an external scene (152) imaged by a camera (104) based on the predicted movement (184), and generate a video signal (126) pre-compensated for the lag (122) to drive the video processor (110) to display a lag-compensated video image (128) of the external scene (152) on a video display system (106).

[0104] 2. The display system (100) of clause 1, wherein the image processing subsystem (108) is further configured to perform dead reckoning to generate a predicted movement (184) of the aircraft (112).

[0105] 3. A display system (100) according to any of the preceding clauses, wherein the lag (122) relates to (i) a lag time (132) between sensing an image (154) corresponding to the external scene (152) at the camera (104) and outputting a video signal (156) from the camera (104), (ii) a processor lag time (134) of the video processor (110), and (iii) a lag time (136) between outputting a video signal (126) pre-compensated for the lag (122) and displaying an external scene (152) corresponding to the video signal (126) on the video display system (106).

[0106] 4. The display system (100) of clause 3, wherein at least one of the camera (104), the video display system (106), and the video processor (110) has a known hysteresis (158);

[0107] Any detected change in the known hysteresis (158) from at least one of the camera (104), the video display system (106), or the video processor (110) is indicative of at least one of a malfunction or a disturbance in at least one of the camera (104), the video display system (106), or the video processor (110).

[0108] 5. A display system (100) according to any of the preceding clauses, wherein the video processor (110) receives an output (186) from the inertial measurement system (180) and a camera output (192) from the camera (104), and transforms the camera output (192) using the IMS output (186) and an electronic model (120) to compensate for the lag (122);

[0109] wherein the transformation comprises at least one of translating or rotating an external scene (152) imaged by the camera (104); and

[0110] wherein the translation and rotation align an image output (194) corresponding to an external scene (152) imaged by a camera (104) at a first time with a predicted orientation (196) of the aircraft (112) at a second time when the external scene (152) will be displayed on a video display system (106).

[0111] 6. A display system (100) according to any of the preceding clauses, further comprising a plurality of video processors (160) and a plurality of video display systems (162), wherein the electronic model (120) comprises responses (114) of seat groups (164) in the aircraft (112) to dynamic characteristics (182) of the aircraft (112), and wherein each seat group (164) is associated with at least one video processor (166) and at least one video display system (168), and at least one video processor (166) and at least one video display system (168) are associated with each seat group (164) that communicate with each other.

[0112] 7. The display system (100) of any one of the preceding clauses, wherein the video display system (106) comprises a plurality of display screens (116) sized and shaped to be aircraft windows.

[0113] 8. A display system (100) according to any of the preceding clauses, wherein the electronic model (120) of the aircraft (112) includes (i) one or more models that predict how the aircraft moves in response to control inputs (140); and (ii) one or more models (142) that define an amount of bending of the aircraft when one or more forces or moments are applied to the aircraft.

[0114] 9. A display system (100) according to any of the preceding clauses, wherein the dynamic characteristic (182) comprises at least one of a force, a torque, a rotation rate or a moment (150).

[0115] 10. A method (1000, 1100, 1200) for video display, the method comprising:

[0116] sensing (1002) dynamic characteristics of the aircraft using an inertial measurement system (IMS);

[0117] Imaging an external scene using a camera (1004);

[0118] displaying (1006) the external scene imaged by the camera using a video display system inside the aircraft;

[0119] configuring (1008) an image processing subsystem including a video processor to access an electronic model of the aircraft, and

[0120] comparing (1010) the dynamic characteristics sensed by the IMS with the electronic model to generate a predicted movement of the aircraft;

[0121] estimating (1010) a lag between a scene and the external scene imaged by the camera based on the predicted movement; and

[0122] A video signal is generated (1010), the video signal being pre-compensated for the lag to drive the video processor to display a lag-compensated video image of the external scene on the video display system.

[0123] 11. The method (1000, 1100, 1200) of clause 10, further comprising performing dead reckoning using (1112) the image processing subsystem to generate the predicted movement of the aircraft.

[0124] 12. A method (1000, 1100, 1200) according to any of the preceding clauses, wherein the lag relates to (i) a lag time between sensing an image corresponding to the external scene at a camera and outputting a video signal from the camera, (ii) a processor lag time of a video processor, and (iii) a lag time between outputting a video signal pre-compensated for lag and displaying an external scene corresponding to the video signal on a video display system.

[0125] 13. The method (1000, 1100, 1200) of clause 12, wherein at least one of the camera, the video display system, and the video processor has a known delay;

[0126] Any detected change in the known delay from at least one of the camera, the video display system, or the video processor is caused to indicate at least one of a malfunction or a disturbance in at least one of the camera, the video display system, or the video processor.

[0127] 14. The method (1000, 1100, 1200) according to any of the preceding clauses, further comprising:

[0128] receiving (1212) with a video processor an output from an inertial measurement system (IMS output) and a camera output from a camera, and

[0129] transforming the camera output to compensate for lag using the (1214) IMS output and the electronic model;

[0130] wherein the transformation comprises at least one of translating or rotating an external scene imaged by the camera; and

[0131] Wherein the translation and rotation align an image output corresponding to an external scene imaged by the camera at a first time with a predicted orientation of the aircraft at a second time when the external scene will be displayed on the video display system.

[0132] 15. The method (1000, 1100, 1200) according to any of the preceding clauses, further comprising:

[0133] Imaging an external scene (152) using a plurality of cameras (104);

[0134] Using a plurality of video display systems (106) to display an external scene (152) imaged by a plurality of cameras (104);

[0135] wherein the electronic model includes responses of seating groups within the aircraft to aircraft dynamics, and wherein each seating group is associated with at least one video processor and at least one video display system, the at least one video processor and the at least one video display system being associated with each seating group in communication with one another.

[0136] 16. The method (1000, 1100, 1200) of any of the preceding clauses, wherein the video display system comprises a plurality of display screens sized and shaped as aircraft windows.

[0137] 17. A method (1000, 1100, 1200) according to any of the preceding clauses, wherein the electronic model of the aircraft includes (i) one or more models that predict how the aircraft moves in response to control inputs; and (ii) one or more models that define the amount of bending of the aircraft when one or more forces or moments are applied to the aircraft.

[0138] 18. A method (1000, 1100, 1200) according to any of the preceding clauses, wherein the dynamic characteristic comprises at least one of a force, a torque, a rotation rate or a moment.

[0139] 19. One or more non-transitory computer-readable media (1522) for video display having computer-executable instructions stored thereon, which when executed by a computer (1518) cause the computer (1518) to perform operations comprising:

[0140] sensing (1002) dynamic characteristics (182) of the aircraft (112) using an inertial measurement system (IMS) (180);

[0141] Imaging (1004) an external scene (152) using a camera (104);

[0142] displaying (1006) an external scene (152) imaged by the camera (104) using a video display system (106) inside the aircraft (112);

[0143] configuring (1008) an image processing subsystem (108) including a video processor (110) to access an electronic model (120) of an aircraft (112), and

[0144] comparing (1010) the dynamic characteristics (182) sensed by the IMS (180) with the electronic model (120) to generate a predicted movement (184) of the aircraft (112);

[0145] estimating (1010) a lag (122) between the scene (124) and an external scene (152) imaged by the camera (104) based on the predicted movement (184); and

[0146] A video signal (126) is generated (1010) that is pre-compensated for lag (122) to drive a video processor (110) to display a lag-compensated video image (128) of an external scene (152) on a video display system (106).

[0147] 20. The one or more non-transitory computer-readable media (1522) having computer-executable instructions stored thereon for video display according to clause 19 further comprising using (1112) the image processing subsystem (108) to perform dead reckoning to generate predicted movement (184) of the aircraft (112).

[0148] 21. One or more non-transitory computer-readable media (1522) according to any of the preceding clauses, wherein the lag (122) relates to (i) a lag time (132) between sensing an image (154) corresponding to the external scene (152) at the camera (104) and outputting a video signal (156) from the camera (104), (ii) a processor lag time (134) of a video processor (110), and (iii) a lag time (136) between outputting a video signal (126) pre-compensated for the lag (122) and displaying an external scene (152) corresponding to the video signal (126) on a video display system (106).

[0149] 22. The one or more non-transitory computer-readable media (1522) of clause 21, wherein at least one of the camera (104), the video display system (106), and the video processor (110) has a known delay (158);

[0150] Any detected change in the known delay (158) from at least one of the camera (104), the video display system (106), or the video processor (110) is indicative of at least one of a malfunction or a disturbance in at least one of the camera (104), the video display system (106), or the video processor (110).

[0151] 23. The one or more non-transitory computer-readable media (1522) for video display having computer-executable instructions stored thereon according to any preceding clause further comprising:

[0152] receiving (1212) with a video processor (110) an output (186) from an inertial measurement system (IMS output (186)) and a camera output (192) from a camera (104), and

[0153] using (1214) the IMS output (186) and the electronic model (120) to transform the camera output (192) to compensate for the lag (122);

[0154] wherein the transformation comprises at least one of translating or rotating an external scene (152) imaged by the camera (104); and

[0155] wherein the translation and rotation align an image output (194) corresponding to an external scene (152) imaged by a camera (104) at a first time with a predicted orientation (196) of the aircraft (112) at a second time when the external scene (152) will be displayed on a video display system (106).

[0156] 24. One or more non-transitory computer-readable media (1522) having computer-executable instructions stored thereon for video display according to any of the preceding clauses further comprising:

[0157] Imaging an external scene (152) using a plurality of cameras (104);

[0158] Using a plurality of video display systems (162) to display an external scene (152) imaged by a plurality of cameras (104);

[0159] wherein the electronic model (120) includes responses of seating groups (164) in the aircraft (112) to dynamic characteristics (182) of the aircraft (112), and wherein each seating group (164) is associated with at least one video processor (166) and at least one video display system (168), the at least one video processor (166) and at least one video display system (168) being associated with each seating group (164) in communication with each other.

[0160] 25. The one or more non-transitory computer-readable media (1522) of any of the preceding clauses, wherein the video display system (106) comprises a plurality of display screens (116) sized and shaped as aircraft windows.

[0161] 26. One or more non-transitory computer-readable media (1522) according to any of the preceding clauses, wherein the electronic model (120) of the aircraft (112) includes (i) one or more models that predict how the aircraft moves in response to control inputs (140) and (ii) one or more models (142) that define an amount of bending of the aircraft when one or more forces or moments are applied to the aircraft.

[0162] 27. One or more non-transitory computer-readable media (1522) according to any of the preceding clauses, wherein the dynamic characteristic (182) includes at least one of a force, a torque, a rotation rate, or a moment (150).

Claims

1. A display system (100), comprising: an inertial measurement system or IMS (180) configured to sense dynamic characteristics (182) of the aircraft (112); a camera (104) configured to image an external scene (152); a video display system (106) inside the aircraft (112); and An image processing subsystem (108) comprising a video processor (110), wherein the image processing subsystem (108) is configured to: accessing an electronic model (120) of the aircraft (112) and comparing the dynamic characteristics (182) sensed by the IMS (180) to the electronic model (120) to calculate a response to the sensed dynamic characteristics over time, using the calculated response to predict movement (184) of the vehicle (112), the predicted movement of the vehicle comprising a prediction of a change in orientation of the vehicle interior between a first time when the camera acquires an image frame imaged by the camera and a second time when the image frame appears on the video display system, estimating a lag (122) between a scene (124) and the external scene (152) imaged by the camera (104) based on the predicted movement (184), and generating a video signal (126) including the image frame, the video signal (126) having been pre-compensated for the lag (122) to drive the video processor (110) to display a lag-compensated video image (128) of the external scene (152) on the video display system (106), the lag-compensated video image of the external scene: is transformed by the video processor to match the predicted movement of the aircraft from the first time to the second time, and and displayed on the video display system at a second time.

2. The display system (100) of claim 1, wherein the image processing subsystem (108) is further configured to perform dead reckoning to generate the predicted movement (184) of the aircraft (112).

3. The display system (100) according to any one of claims 1-2, wherein the lag (122) involves (i) a lag time (132) between sensing an image (154) corresponding to the external scene (152) at the camera (104) and outputting a video signal (156) from the camera (104), (ii) a processor lag time (134) of the video processor (110), and (iii) a lag time (136) between outputting the video signal (126) pre-compensated for the lag (122) and displaying the external scene (152) corresponding to the video signal (126) on the video display system (106).

4. The display system (100) of claim 3, wherein at least one of the camera (104), the video display system (106), and the video processor (110) has a known delay (158); Such that any detected change in the known delay (158) from at least one of the camera (104), the video display system (106), or the video processor (110) indicates at least one of a failure or a disturbance in at least one of the camera (104), the video display system (106), or the video processor (110).

5. The display system (100) of any one of claims 1-4, wherein the video processor (110) receives an output (186) from the inertial measurement system (180) and a camera output (192) from the camera (104), and transforms the camera output (192) using the IMS output (186) and the electronic model (120) to compensate for the lag (122); wherein the transformation comprises at least one of translating or rotating the external scene (152) imaged by the camera (104); and wherein the translation and rotation align an image output (194) corresponding to the external scene (152) imaged by the camera (104) at a first time with a predicted orientation (196) of the aircraft (112) at a second time when the external scene (152) is to be displayed on the video display system (106).

6. The display system (100) according to any one of claims 1-5, further comprising a plurality of video processors (160) and a plurality of video display systems (106), wherein the electronic model (120) comprises responses (114) of seat groups (164) in the aircraft (112) to the dynamic characteristics (182) of the aircraft (112), and wherein each seat group (164) is associated with at least one video processor (166) and at least one video display system (106), and the at least one video processor (166) and the at least one video display system (106) are associated with each seat group (164) that communicate with each other.

7. The display system (100) of any one of claims 1-6, wherein the video display system (106) includes a plurality of display screens (116) sized and shaped as aircraft windows.

8. A display system (100) according to any one of claims 1-7, wherein the electronic model (120) of the aircraft (112) includes (i) one or more models (140) that predict how the aircraft moves in response to control inputs; and (ii) one or more models (142) that define an amount of bending of the aircraft when one or more forces or moments are applied to the aircraft.

9. The display system (100) of any one of claims 1-8, wherein the dynamic characteristic (182) comprises at least one of a force, a torque, a rotation rate, or a moment (150).

10. A method (1000, 1100, 1200) for video display, comprising: sensing (1002) dynamic characteristics of the aircraft using an inertial measurement system (IMS); Imaging an external scene using a camera (1004); displaying (1006) the external scene imaged by the camera using a video display system inside the aircraft; configuring (1008) an image processing subsystem including a video processor to access an electronic model of the aircraft, and comparing (1010) the dynamic property sensed by the IMS to the electronic model to calculate a response to the sensed dynamic property over time; using the calculated response to predict movement of the aircraft, the predicted movement of the aircraft comprising a prediction of a change in orientation of an interior of the aircraft between a first time when the camera acquires an image frame imaged by the camera and a second time when the image frame appears on the video display system; estimating a lag between a scene and the external scene imaged by the camera based on the predicted movement; and generating a video signal including the image frame, the video signal being pre-compensated for the lag to drive the video processor to display a lag-compensated video image of the external scene on the video display system, the lag-compensated video image of the external scene: is transformed by the video processor to match the predicted movement of the aircraft from the first time to the second time, and and displayed on the video display system at a second time.

11. The method (1000, 1100, 1200) of claim 10, further comprising performing dead reckoning using (1112) the image processing subsystem to generate the predicted movement of the aircraft.

12. The method (1000, 1100, 1200) according to any one of claims 10-11, wherein the lag relates to (i) a lag time between when an image corresponding to the external scene is sensed at the camera and a video signal is output from the camera, (ii) a processor lag time of the video processor, and (iii) a lag time between outputting the video signal pre-compensated for the lag and displaying the external scene corresponding to the video signal on the video display system, and wherein at least one of the camera, the video display system, and the video processor has a known delay; Such that any detected change in the known delay from at least one of the camera, the video display system, or the video processor indicates at least one of a malfunction or a disturbance in at least one of the camera, the video display system, or the video processor.

13. The method (1000, 1100, 1200) according to any one of claims 10-12, further comprising: receiving (1212) with the video processor an output from the inertial measurement system, i.e., an IMS output, and a camera output from the camera, and transforming the camera output to compensate for the lag using (1214) the IMS output and the electronic model; wherein the transformation comprises at least one of translating or rotating the external scene imaged by the camera; and Wherein the translation and rotation align an image output corresponding to the external scene imaged by the camera at a first time with a predicted orientation of the aircraft at a second time when the external scene is to be displayed on the video display system.

14. The method (1000, 1100, 1200) according to claim 10, further comprising: Imaging the external scene (152) using a plurality of cameras (104); Using a plurality of video display systems (106) to display the external scene (152) imaged by the plurality of cameras (104); wherein the electronic model includes responses of seating groups within the aircraft to dynamic characteristics of the aircraft, and wherein each seating group is associated with at least one video processor and at least one video display system, the at least one video processor and the at least one video display system being associated with each seating group in communication with each other.

15. A method (1000, 1100, 1200) according to any one of claims 10-14, wherein the video display system includes a plurality of display screens sized and shaped as windows of an aircraft, wherein the electronic model of the aircraft includes (i) one or more models that predict how the aircraft moves in response to control inputs; and (ii) one or more models that define an amount of bending of the aircraft when one or more forces or moments are applied to the aircraft, and wherein the dynamic characteristics include at least one of force, torque, rotation rate, or moment.

Images