WIDE FIELD OF VIEW GEOREFERENCED SYNTHETIC IMAGING SYSTEM

Abstract

An imaging system for an aircraft is provided. A plurality of image sensors are attached, fixed, or secured to the aircraft. Each image sensor is configured to generate sensor-generated pixels based on an environment surrounding the aircraft. Each of the sensor-generated pixels is associated with respective pixel data including position data, intensity data, acquisition time data, sensor type data, pointing angle data, latitude data, and longitude data. A controller generates a buffer image including synthetic layer pixels, maps the sensor-generated pixels onto the synthetic layer pixels in the buffer image, fills a plurality of regions of the buffer image with the sensor-generated pixels, and presents the buffer image on a head-mounted display (HMD) to a user of the aircraft. Figure 1

Description

Title of the invention: WIDE FIELD OF VIEW GEOREFERENCED SYNTHETIC IMAGING SYSTEM

[0001] An imaging system for an aircraft system according to one or more illustrative embodiments of the present disclosure is provided. In one illustrative embodiment, the system includes a plurality of image sensors attached, fixed, or secured to the aircraft, each image sensor configured to generate sensor-generated pixels based on an environment surrounding the aircraft, wherein each of the sensor-generated pixels is associated with respective pixel data including: position data, intensity data, acquisition time data, sensor type data, pointing angle data, and latitude data and longitude data derived from the pointing angle data. In another illustrative embodiment, the system includes a controller communicatively coupled to the plurality of image sensors,configured to: generate a buffer image comprising synthetic layer pixels, each of the synthetic layer pixels being associated with latitude data and longitude data derived from a synthetic vision database; map the sensor-generated pixels to the synthetic layer pixels in the buffer image by matching the latitude data and the longitude data derived from the pointing angle data with the latitude data and the longitude data derived from the synthetic vision database; fill a plurality of regions of the buffer image with the sensor-generated pixels, each of the respective regions of the buffer image being respectively filled by the sensor-generated pixels that are generated by a respective image sensor; and present the buffer image on a head-mounted display (HMD) to a user of the aircraft.

[0002] Advantageously, the pointing angle data is generated by a respective position encoder integrated into or communicatively coupled to each of the image sensors.

[0003] Advantageously, the pointing angle data is generated by a respective inertial measurement unit (IMU) integrated with or communicatively coupled to each of the image sensors.

[0004] Advantageously, the pointing angle data is generated by a respective attitude and heading reference system (AHRS) integrated or communicatively coupled to each of the image sensors.

[0005] Advantageously, the plurality of image sensors comprises at least one sensor still image and at least one rotating image sensor.

[0006] Advantageously, the aircraft comprises a winch attached, fixed or secured thereto, wherein the winch comprises a winch cable extendable in a first direction, and the at least one fixed image sensor comprises a winch image sensor configured to capture the environment in the first direction.

[0007] Advantageously, the aircraft comprises a rotating projector, and the at least one rotating image sensor comprises a projector image sensor configured to rotate in conjunction with the rotating projector.

[0008] Advantageously, the at least one rotating image sensor comprises a turret image sensor.

[0009] Advantageously, at least one of the plurality of image sensors is configured to detect visible light or infrared light.

[0010] Advantageously, at least one of the plurality of image sensors comprises a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) device.

[0011] Advantageously, at least one of the plurality of image sensors comprises an imaging radar configured to detect radiofrequency radiation.

[0012] It is understood that the foregoing general description and the following detailed description are merely exemplary and explanatory and are not intended to limit the invention as claimed. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the invention and, together with the general description, serve to explain the principles of the invention. Brief description of the drawings

[0013] The numerous advantages of the disclosure will be better understood by those skilled in the art by referring to the appended figures in which:

[0014] [Fig.l] is a side view of an imaging system for an aircraft, according to one or more embodiments of the present disclosure.

[0015] [Fig.2] is a georeferenced buffer image constructed by sensor fusion, according to one or more embodiments of the present disclosure.

[0016] FIGURES 3-4 are isometric views of a head-mounted display (HMD), according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Before explaining in detail at least one embodiment of the inventive concepts described herein, it should be understood that the inventive concepts are not limited in their application to the construction details and arrangement of the components or steps or methodologies presented in the following description. or illustrated in the drawings. In the following detailed description of the embodiments of the present disclosure, numerous specific details are presented to enable a more thorough understanding of the inventive concepts. However, it will be apparent to those skilled in the art having the benefit of this disclosure that the inventive concepts described herein may be practiced without these specific details. In other instances, well-known features may not be described in detail to avoid unnecessarily complicating this disclosure. The inventive concepts described herein are susceptible of other embodiments or of being practiced or carried out in various ways. Furthermore, it should be understood that the phraseology and terminology employed herein are for the purpose of description and should not be construed as limiting.

[0018] As used herein, a letter following a reference numeral is intended to refer to an embodiment of the feature or element that may be similar, but not necessarily identical, to a previously described element or feature having the same reference numeral (e.g., 1, 1a, 1b). These shorthand notations are used for convenience only and should not be construed as limiting in any manner the inventive concepts described herein unless expressly stated otherwise. Furthermore, unless expressly stated otherwise, "or" refers to an inclusive or and not an exclusive or. For example, a condition A or B is satisfied by any of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0019] Finally, as used herein, any reference to "an embodiment" or "certain embodiments" means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the inventive concepts described herein.The occurrences of the phrase "in certain embodiments" at various places in the description do not necessarily all refer to the same embodiment, and embodiments of the described inventive concepts may include one or more of the features expressly described or inherently present herein, or any combination or subcombination of two or more of these features, as well as any other features not necessarily expressly described or inherently present in the present disclosure.

[0020] Real-time imaging systems are used to provide situational awareness to pilots in adverse visibility conditions (e.g., during helicopter search and rescue missions). Real-time imaging systems Real-time imaging systems may include, for example, a head-mounted display (HMD) configured to receive and display images from one or more cameras mounted externally on an aircraft or other vehicle. One limitation of these real-time imaging systems is a small field of view (FOV) that may be inadequate for tasks such as searching for a stranded person during a search and rescue (SAR) operation. This limitation is sometimes overcome by distributed aperture systems that replicate a camera multiple times to increase the field of view. However, this solution is expensive due to the significant hardware and technical requirements (mounting and integrating multiple high-definition cameras on an aircraft).

[0021] Several platforms (military and civilian) are already equipped with imaging systems capable of displaying a wider field of view (FOV) achieved by rotating a swivel turret camera whose FOV is limited. When these systems are coupled to HMDs on a user's head (e.g., a pilot or SAR operator), the time required to rotate the camera in the direction of the user's view can be unacceptably slow. Thus, a solution that uses cameras commonly found on aircraft imaging systems is desirable.

[0022] Embodiments of the present invention provide an imaging system that introduces metadata into a video stream from multiple image sensors (i.e., cameras) on a per-pixel basis, including time, sensor type, camera-specific information (pointing angle, magnification settings, position encoder readings, etc.), and georeferencing information. In some embodiments, the video stream may include an underlying synthetic terrain representation. The metadata is then used to generate a wraparound image that appears to be a real-time image. The present imaging system generates a real-time, wide-field image for pilot consumption on a see-through display by leveraging the image sensors already deployed on many platforms, without the need to install multiple additional image sensors.

[0023] [Fig.l] is a side view of an imaging system 100, according to one or more embodiments of the present disclosure. The imaging system 100 may be substantially integrated with a vehicle 102. As illustrated, the vehicle 102 may be a helicopter, although the present disclosure is not limited thereto. For example, the vehicle 102 may include a land vehicle (e.g., a car or truck), a marine vehicle (e.g., a ship or boat), or another aerial vehicle (e.g., an airplane). The vehicle 102 may be occupied by one or more users 150 (e.g., a pilot 150 operating the vehicle, and a search and rescue operator 150 monitoring the environment for stranded evacuees). The vehicle 102 may include a winch 110 with cable to lower rescue teams and free evacuees while the vehicle 102 hovers above (thereby avoiding obstacles or ocean troughs). In some embodiments, the winch 110 may incorporate one or more cameras to monitor winching operations.

[0024] The imaging system 100 may include a still image sensor 104, a projector image sensor 106, a turret image sensor 108, and a winch image sensor 112. It should be noted that each of the image sensors 104, 106, 108, and 112 may be an image detector (e.g., a focal plane array) configured to detect visible and / or infrared light, and each may include a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) device.

[0025] The fixed image sensor 104 may be attached, fixed, or secured to a front portion of the vehicle 102 (e.g., at the nose of the helicopter) and may capture images of the environment surrounding the vehicle 102 (e.g., at the front of the vehicle). The fixed image sensor 104 may be configured to be static (e.g., fixed in place) while the vehicle 102 is moving to provide stable images. The searchlight image sensor 106 may be attached, fixed, or secured to a movable searchlight (illumination projector) attached to a front portion of the vehicle 102. The searchlight image sensor 106 may be configured to rotate in conjunction with the user-operated searchlight 150 to capture images of the environment surrounding the vehicle 102 in the direction of a beam emitted by the searchlight.The turret image sensor 108 may be attached, fixed, or secured to a front portion of the vehicle 102, and may be a ball-and-socket type image sensor capable of rotation (e.g., in two degrees of freedom—horizontal and vertical rotation). The turret image sensor 108 may capture images of the environment surrounding the vehicle 102 (e.g., in front of the vehicle 102, to the sides of the vehicle 102, and to the rear of the vehicle 102). The winch image sensor 112 may be configured to capture images of the environment beneath the vehicle 102 (e.g., in the direction of extension of a winch cable). Thus, each of the image sensors 104, 106, 108, and 112 may be configured to separately generate pixels for a georeferenced buffer image.

[0026] In some embodiments, the image sensors 104, 106, 108, and / or 112 may include one or more imaging radars (e.g., in addition to or instead of CCD or CMOS cameras) configured to capture images of the environment surrounding the vehicle 102 by detecting radio frequency (RF) radiation. The one or more imaging radars may transmit an RF signal and record the intensity of an RF signal. reflected to determine an amount of scattered light. The recorded scattering can then be mapped onto a two-dimensional plane (e.g., assigning a brighter color to points with higher reflectivity) to generate images.

[0027] [Fig. 2] is a georeferenced buffer image 200 constructed by sensor fusion, according to one or more embodiments of the present disclosure. The georeferenced buffer image 200 may be generated using a controller (e.g., a computer or computing device) onboard the vehicle 102 or communicatively coupled to an HMD worn by the user(s) 150.

[0028] The buffer image 200 may comprise a pixel array with a plurality of pixel columns and a plurality of pixel rows. Each respective image sensor 104, 106, 108, and 112 may independently generate images for generating the pixels in respective regions of the buffer image 200 at a respective fill rate. For example, the still image sensor 104 may generate images for the pixels in region 204, the projector image sensor 106 may generate images for the pixels in region 208, the turret image sensor 108 may generate images for the pixels in region 206, and the winch image sensor 112 may generate images for the pixels in region 210.

[0029] In some embodiments, regions 204, 206, 208, and 210 of buffer image 200 are separate from each other and do not overlap. In other embodiments, regions 204, 206, 208, and 210 of buffer image 200 may overlap (partially or substantially). If the regions overlap, then images from one of image sensors 104, 106, 108, and 112 may be prioritized for generating pixels in the overlapping area between the regions. For example, if region 204 overlaps region 208, then images from projector image sensor 106 may be prioritized over images from still image sensor 104 for generating pixels in the overlapping area.

[0030] In addition to position data (e.g., the xy address in the image matrix) and intensity data (e.g., one value for grayscale images and three values ​​for color images), each pixel in the image buffer 200 may be associated with metadata, including, but not limited to, acquisition time data (e.g., a timestamp indicating the time each pixel is captured), sensor type data (e.g., the type of image sensor used to capture the pixel), and pointing angle data (e.g., indicating the direction the respective sensor is facing when the pixel is captured). The pointing angle data of the respective image sensor 104, 106, 108, and 112 may be used to generate longitude data and latitude data for each generated pixel.

[0031] The pointing angle data may be generated from a position encoder, an inertial measurement unit (IMU), or an attitude and heading reference system (AHRS) integrated with (or communicatively coupled to) the respective image sensor 104, 106, 108, and / or 112. The pointing angle data as presented herein may be advantageous in that conventional image sensors currently provide pointing angle data in relatively large steps (e.g., 1° or 2° of rotation at a time) whereas the present system 100 may provide more precise pointing angle data (e.g., to the milliradian level) using a position encoder, an IMU, or an AHRS.

[0032] In some embodiments, the controller may generate a synthetic layer in the buffer image 200. The synthetic layer may include synthetic images for generating pixels in the regions 204, 206, 208, and / or 210 using a database of synthetic images (e.g., from a synthetic vision system [SVS] or a combined vision system [CVS] of an aircraft). The synthetic layer may feature synthetic terrain (e.g., 3D polygonal representations of the earth's surface) and obstacles (e.g., building icons, radio towers, etc.), and may be textured using artificially generated textures or satellite imagery.

[0033] The synthetic layer may provide a base magnification (e.g., zoom) and horizon line for the buffer image 200. The pixels generated by the image sensors 104, 106, 108, and 112 may be mapped to the pixels generated by the synthetic layer (e.g., by matching latitude and longitude values ​​derived from the pointing angle metadata associated with the sensor-generated pixels with latitude and longitude values ​​derived from a synthetic vision database). When the image sensors are operational, the sensor-generated pixels may replace the pixels in the synthetic layer, such that the sensor-generated pixels are presented to the user 150 on the HMD.However, when one of the image sensors is not functioning (e.g., it stops transmitting image data / generating pixels), the synthetic layer pixels may replace the sensor-generated pixels in the respective region of the buffer image 200, such that the synthetic layer pixels are presented to the user 150 on the HMD. In some embodiments, the system 100 may further include one or more imaging radars that generate radar pixels in regions 204, 206, 208, and / or 210 of the buffer image 200.

[0034] In some embodiments, the buffer image 200 may be generated as a multi-plane digital image buffer. In other embodiments, the buffer 200 can be constructed using a more compact representation of the data (such as a list of "pixel vectors" containing only pixel values ​​and other metadata for the data generated from the environment captured by the image sensors).

[0035] [Fig. 3] is an isometric view of an HMD 300 configured to be worn by the user(s) 150, according to one or more embodiments of the present disclosure. The horizontal line 310 that is parallel to the top of the frame of the HMD 300 may serve as a reference to indicate that the head of the user 150 is level. The horizontal line 310 may be derived from a synthetic vision system database and may provide a base level of magnification. As illustrated, the HMD 300 may be a pair of glasses, but the present disclosure is not limited thereto, and the HMD 300 may be a headset, glasses, etc. The buffer image 200 may be presented (e.g., by projection or using LEDs) on a display portion of the HMD 300. In some embodiments, the HMD 300 presents a stereoscopic pair of buffer images 200 (e.g., one image 200 for each eye of the user 150).In some embodiments, only the portion of the buffer image 200, which is visible in the optical field of view of the user 150, is presented on the HMD 300.

[0036] [Fig. 4] is an isometric view of the HMD 300 shown in [Fig. 3]. The horizontal line 310 is offset, indicating that the user 150 has moved his or her head. Thus, a new region of the buffer 200 is shown on the HMD 300.

[0037] It is noted herein that the controller (e.g., a computer or computing device) may include one or more processors and memory. For purposes of this disclosure, the term "processor" or "processing element" may be broadly defined to encompass any device having one or more processing or logic elements, e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more microprocessor devices, one or more application-specific integrated circuit (ASIC) devices, one or more field programmable gate arrays (FPGAs), or one or more digital signal processors (DSPs).In this sense, the processor(s) may include any device configured to execute algorithms and / or instructions (e.g., program instructions stored in memory), and may be configured to perform the method steps described in the present disclosure (e.g., generating a buffer image comprising synthetic layer pixels, mapping the sensor-generated pixels to the synthetic layer pixels in the buffer image, filling a plurality of regions of the buffer image with the sensor-generated pixels, and presenting the buffer image on a head-mounted display (HMD) to a user of the aircraft). The support of . Memory may include any storage medium known in the art suitable for storing program instructions executable by the associated processor(s). For example, the memory medium may include, but is not limited to, read-only memory (ROM), random access memory (RAM), a magnetic or optical memory device (e.g., a hard disk drive), magnetic tape, a solid-state drive, etc.

[0038] It is believed that the present disclosure and many of its related advantages will be understood from the foregoing description, and that various modifications may be made in the form, construction, and arrangement of components without departing from the disclosed subject matter or sacrificing all of its material advantages. The form disclosed is merely illustrative, and it is the intent of the following claims to encompass and include such modifications. Further, it is to be understood that the invention is defined by the appended claims.

Claims

Claims

1. An imaging system (100) for an aircraft comprising a head-mounted display (300) and: a plurality of image sensors (104, 106, 108, 112) attached, fixed, or secured to the aircraft, wherein each image sensor is configured to generate sensor-generated pixels based on an environment surrounding the aircraft, wherein each of the sensor-generated pixels is associated with respective pixel data comprising: position data, intensity data, acquisition time data, sensor type data, pointing angle data, and latitude data and longitude data derived from the pointing angle data; and a controller communicatively coupled to the plurality of image sensors, configured to: generate a buffer image (200) comprising synthetic layer pixels,wherein each of the synthetic layer pixels is associated with latitude data and longitude data derived from a synthetic vision database, mapping the sensor-generated pixels onto the synthetic layer pixels in the buffer image by matching the latitude data and longitude data derived from the pointing angle data with the latitude data and longitude data derived from the synthetic vision database, filling a plurality of regions (204, 208, 206, 210) of the buffer image (200) with the sensor-generated pixels, wherein each of the respective regions of the buffer image is respectively filled by the sensor-generated pixels that are generated by a respective image sensor, presenting the buffer image (200) on said head-mounted display (300) to a user of the aircraft.,

2. The imaging system (100) of claim 1, wherein the pointing angle data is generated by a respective position encoder integrated with or communicatively coupled to each of the image sensors.

3. The imaging system (100) of claim 1, wherein the pointing angle data is generated by a respective inertial measurement unit (IMU) integrated with or communicatively coupled to each of the image sensors (104, 106, 108, 112).

4. The imaging system (100) of claim 1, wherein the pointing angle data is generated by a respective attitude and heading reference system (AHRS) integrated or communicatively coupled to each of the image sensors (104, 106, 108, 112).

5. The imaging system (100) of claim 1, wherein the plurality of image sensors comprises at least one fixed image sensor (104, 112) and at least one rotating image sensor (108, 106).

6. The imaging system (100) of claim 5, wherein the aircraft comprises a winch attached, fixed or secured thereto, wherein the winch (110) comprises a winch cable extendable in a first direction, and the at least one fixed image sensor (104, 112) comprises a winch image sensor (112) configured to capture the environment in the first direction.

7. The imaging system (100) of claim 5, wherein the aircraft comprises a rotating projector (106), and the at least one rotating image sensor (108) comprises a projector image sensor configured to rotate in conjunction with the rotating projector (106).

8. The imaging system (100) of claim 5, wherein the at least one rotating image sensor comprises a turret image sensor (108).

9. The imaging system (100) of claim 1, wherein at least one of the plurality of image sensors is configured to detect visible light or infrared light.

10. The imaging system (100) of claim 1, wherein at least one of the plurality of image sensors comprises a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) device.

11. The imaging system (100) of claim 1, wherein at least one of the plurality of image sensors comprises an imaging radar configured to detect radio frequency radiation.

Images