Radiometric thermal imaging improvements for navigation systems and methods

Abstract

The present invention provides thermal imaging and navigation systems and related technologies to improve the operation of manned or unmanned mobile platforms (including passenger vehicles). A system includes a thermal imaging device configured to be mounted on a vehicle. The thermal imaging device is configured to: when mounted on the vehicle, capture a first image of a scene containing a portion of the vehicle, and capture a second image of the scene associated with a reflection from the portion of the vehicle. The system also includes a logic device configured to communicate with the thermal imaging device and determine a disparity map based on the first image and the second image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 167,644, filed on March 29, 2021, entitled “RADIOMETRIC THERMAL IMAGING IMPROVEMENTS FOR NAGIVATION SYSTEMS AND METHODS,” the entire contents of which are incorporated herein by reference.

[0003] This application is a continuation-in-part of International Patent Application No. PCT / US2021 / 012554, filed on January 7, 2021, entitled “VEHICULAR RADIOMETRIC CALIBRATION SYSTEMS AND METHODS,” which claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 959,602, filed on January 10, 2020, entitled “VEHICULAR RADIOMETRIC CALIBRATION SYSTEMS AND METHODS,” and the entire contents of the above two applications are incorporated herein by reference.

[0004] This application is related to U.S. patent application No. 17 / 685,349, entitled “THERMAL IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” filed on March 2, 2022, which claims priority to U.S. Provisional Patent Application No. 63 / 159,444, entitled “THERMAL IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” filed on March 10, 2021, and U.S. Provisional Patent Application No. 63 / 167,632, entitled “THERMAL IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” filed on March 29, 2021, and the entire contents of the above applications are incorporated herein by reference.

[0005] This application is related to U.S. patent application No. 17 / 636,841 filed on February 18, 2022, which is a U.S. national phase patent application under 35 USC 371 of International Patent Application No. PCT / US2020 / 048450, entitled “MULTISPECTRAL IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” filed on August 28, 2020, which claims priority and benefits to U.S. Provisional Patent Application No. 62 / 894,544, entitled “MULTISPECTRAL IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” filed on August 30, 2019. The entire contents of the above two applications are incorporated herein by reference. Technical Field

[0006] The present invention relates generally to multispectral imaging and, more particularly, to systems and methods of multispectral imaging for navigation of mobile platforms. Background Art

[0007] Modern mobile platforms, such as assisted or autonomous manned and unmanned land vehicles and aircraft, including unmanned aerial vehicles (UAVs), remotely operated underwater vehicles (ROVs), unmanned (water) surface vehicles (USVs), and unmanned ground vehicles (UGVs), any of which can be configured as unmanned sensor platforms, capable of operating over long distances and in all environments (rural, urban, and even underwater). The operation of such a system may include real-time feedback to the pilot and / or wireless transmission between the unmanned platform and a remote base station, which typically includes a display to efficiently transmit telemetry, imagery, and other sensor data captured by the platform to the operator. The operator can typically monitor the automatic or assisted navigation of the manned or unmanned mobile platform throughout the mission, relying solely on image feedback or data received from the mobile platform, and pilot or otherwise control the manned or unmanned mobile platform as necessary.

[0008] Conventional imaging systems are often too expensive and bulky, or lack sufficient contrast in relatively common environmental conditions to be used for reliable and safe autonomous or assisted navigation of vehicles or other mobile platforms. Therefore, there is a need for compact imaging systems and related technologies to provide reliable scene assessment for navigation of mobile platforms. Summary of the invention

[0009] The present disclosure provides a multispectral navigation system and related technologies to improve the operation of manned or unmanned mobile platforms (including manned vehicles and unmanned sensors or survey platforms with assisted or automatic navigation). One or more embodiments of the described multispectral navigation system can advantageously include a multispectral imaging system, which includes a multispectral imaging module, a communication module configured to establish a wireless communication link with a base station associated with the mobile platform, an orientation and / or position sensor configured to measure the orientation and / or position of the multispectral imaging system and / or the coupled mobile platform, a controller, and one or more additional sensors for measuring and providing sensor data corresponding to the manipulation and / or other operation of the mobile platform, wherein the controller is used to control the operation of the communication module, the orientation and / or position sensor and / or the mobile platform.

[0010] In various embodiments, such additional sensors may include a remote sensor system configured to capture sensor data of the survey area, from which a two-dimensional and / or three-dimensional spatial map of the survey area may be generated. For example, the navigation system may include one or more visible spectrum, infrared and / or ultraviolet cameras and / or other remote sensor systems coupled to the mobile platform. The mobile platform may typically be a flying platform (e.g., a manned aircraft, a UAS, and / or other flying platform), a land platform (e.g., a mobile vehicle), or a waterborne platform (e.g., a ship or submarine). More generally, a multispectral imaging system for a multispectral navigation system may be implemented as a multispectral autonomous vehicle imaging system (e.g., MAVIS, for various autonomous or automatically piloted mobile platforms or vehicles).

[0011] In one embodiment, a system includes a multispectral imaging system, the multispectral imaging system including a multispectral imaging module configured to provide multispectral image data corresponding to an expected route of a mobile platform, and a logic device configured to communicate with the multispectral imaging system. The logic device can be configured to receive the multispectral image data corresponding to the expected route; receive orientation and / or position data corresponding to the multispectral image data; and generate maneuvering obstacle information corresponding to the expected route based at least in part on a combination of the orientation and / or position data and the multispectral image data.

[0012] In another embodiment, a method includes receiving multispectral image data from a multispectral imaging system, the multispectral imaging system including a multispectral imaging module configured to provide multispectral image data corresponding to an expected route of a mobile platform; receiving orientation and / or position data corresponding to the multispectral image data; and generating maneuvering obstacle information corresponding to the expected route based at least in part on a combination of the orientation and / or position data and the multispectral image data.

[0013] In one embodiment, a system includes a thermal imaging system including a thermal imaging module configured to provide thermal image data corresponding to an expected route of a mobile platform, and a logic device configured to communicate with the thermal imaging system. The logic device can be configured to receive the thermal image data corresponding to the expected route; and generate maneuver obstacle information corresponding to the expected route based at least in part on the thermal image data.

[0014] In another embodiment, a method includes receiving thermal image data from a thermal imaging system including a thermal imaging module configured to provide thermal image data corresponding to an expected path of a mobile platform; and generating maneuvering obstacle information corresponding to the expected path based at least in part on the thermal image data.

[0015] The scope of the present invention is defined by the claims, which are incorporated into this section by reference. By considering the following detailed description of one or more embodiments, those skilled in the art will more fully understand the embodiments of the present invention and realize the additional advantages thereof. Reference will be made to the accompanying drawings which will first be briefly described. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A diagram showing a multi-spectral navigation system according to an embodiment of the present disclosure is shown.

[0017] Figure 2 A diagram showing a mobile platform utilizing a multispectral navigation system according to an embodiment of the present disclosure is shown.

[0018] Figure 3 A diagram showing a multispectral imaging system for a multispectral navigation system according to an embodiment of the present disclosure is shown.

[0019] Figure 4-6 A display view including an image generated by a multispectral imaging system for a multispectral navigation system according to an embodiment of the present disclosure is shown.

[0020] Figure 7 A flow chart illustrating various operations for providing assisted or automatic pilot navigation using a multi-spectral navigation system according to an embodiment of the present disclosure.

[0021] Figure 8-10 A display view including an image generated by a multispectral imaging system for a multispectral navigation system according to an embodiment of the present disclosure is shown.

[0022] Fig.11 is a diagram illustrating functional benefits associated with a thermal imaging navigation system according to an embodiment of the present disclosure.

[0023] Fig.12 A diagram showing a mobile platform utilizing a thermal imaging navigation system according to an embodiment of the present disclosure is shown.

[0024] Fig.13A A data flow diagram of a mobile platform utilizing a thermal imaging navigation system according to an embodiment of the present disclosure is shown.

[0025] Fig. 13B A block diagram of an update system for a mobile platform utilizing a thermal imaging navigation system according to an embodiment of the present disclosure is shown.

[0026] Fig.14A -B shows a display view including an image generated by a thermal imaging system for a thermal imaging navigation system according to an embodiment of the present disclosure.

[0027] Fig.15 A flow chart illustrating various operations for providing assisted or automatic pilot navigation including automatic emergency braking using a thermal imaging navigation system according to an embodiment of the present disclosure.

[0028] Fig.16A -F shows a display view including an image generated by a thermal imaging system of a thermal imaging navigation system according to an embodiment of the present disclosure.

[0029] Fig.17A A diagram showing a model for atmospheric temperature compensation of a thermal imaging navigation system according to an embodiment of the present disclosure.

[0030] Fig. 17B A graph showing atmospheric transmission versus range for a thermal imaging navigation system according to an embodiment of the present disclosure is shown.

[0031] Fig. 17C A graph showing atmospheric transmission versus wavelength for a thermal imaging navigation system according to an embodiment of the present disclosure is shown.

[0032] Fig.18 A diagram showing ambient air and humidity sensors of a thermal imaging navigation system according to an embodiment of the present disclosure.

[0033] Fig.19 A display view including an image generated by a thermal imaging system for a thermal imaging navigation system according to an embodiment of the present disclosure is shown.

[0034] Fig. 20 A diagram showing a monocular range sensor for range determination using a thermal imaging navigation system in accordance with an embodiment of the present disclosure.

[0035] Fig.21 A diagram illustrating range determination using a thermal imaging navigation system in accordance with an embodiment of the present disclosure.

[0036] Fig. 22A flow chart illustrating various operations for generating a range map using a thermal imaging navigation system according to an embodiment of the present disclosure.

[0037] Fig.23A -C shows a display view including an image generated by a thermal imaging system for a thermal imaging navigation system according to an embodiment of the present disclosure.

[0038] Embodiments of the present invention and their advantages are best understood by referring to the following detailed description.It should be appreciated that like reference numerals are used to identify like elements shown in one or more of the Figures. DETAILED DESCRIPTION

[0039] Multispectral navigation systems and related technologies are provided to improve the operational flexibility and reliability of mobile platforms, including unmanned mobile sensor platforms. Imaging systems for advanced driver assistance systems (ADAS) typically acquire real-time video images of the immediate environment around a vehicle or mobile platform. Such images can assist a navigator (e.g., a human or an autopilot) in making decisions regarding navigation of the mobile platform, such as braking or evasive maneuvers. Contemporary commercial ADAS cameras produce images with limited or no spectral content: images are either RGB color or monochrome visible bands. Such limited spectral content means that under many common environmental conditions (e.g., heading and / or time of day), physical objects may have little or no contrast against the sky or other distant backgrounds.

[0040] A real-world example might be a situation where the ADAS fails to detect a featureless ground obstruction (e.g., an enclosed trailer) that is relatively large relative to the sky / horizon. Such a deficiency may be caused by limited visible differences in the visible spectrum (e.g., color, brightness). Such a deficiency is not uncommon: due to diffuse reflections from the sky "dome," white objects under an overcast sky will tend to have a brightness very similar to the sky. Embodiments of the multi-spectral navigation system described herein are much less likely to fail under similar conditions because it is extremely unlikely that a physical object would emit or reflect light similar to that in the sky over a combination of spectral bands, which may include the visible spectrum and spectrums outside of the visible spectrum, such as the infrared and / or ultraviolet spectrum.

[0041] Scenes presented to imaging navigation systems often include regions with very different near infrared (NIR), visible light (VIS), and long wave UV (LWUV) spectral content. Multispectral imaging systems that are sensitive to light spectrums beyond the visible can more reliably determine the composition of scene content, including, for example, being able to reliably distinguish the sky from other vehicles, trees, bushes, buildings, or roads. As a result, multispectral imaging systems can provide imaging navigation systems with a much more nuanced data stream.

[0042] For example, simultaneous measurements in selected bands provide a rough reflectance / emission spectrum of surface materials in the scene. Materials such as vegetation reflect sunlight with unique spectral signatures, while the sky emits a different spectrum. Vehicles and road surfaces also behave differently in such selected bands than they do in the visible bands. Using a small number (e.g., 2, 3, 4, or 5) of relatively wide spectral bands for a multispectral camera, as opposed to many closely spaced, relatively narrow spectral channels, has several advantages. Wider spectral bands (e.g., typically associated with wide bandpass filters) generally mean that more scene flux will reach the detector and result in better exposure in low light conditions. Wider spectral bands also allow for shorter integration times, thereby reducing motion blur, a particular problem associated with the edges of images captured by forward-looking systems mounted in fast-moving vehicles with relatively high angular rates of motion.

[0043] Such multispectral imaging capture and processing techniques can also be used for aircraft, including unmanned aerial systems. For example, a UAV with autonomous operation capabilities can be implemented with an imaging system that can help humans or autopilots make decisions about what to do in different flight situations (including during takeoff, landing, and evasive maneuvers). A multispectral imaging system with image analysis capabilities can provide processed sensor information about the physical environment to help navigators avoid obstacles and turn, such as a white blank billboard, which may have the same visible band radiation as the cloudy sky behind the billboard. Multispectral images can also help determine the location of vegetation, which can help mobile platforms avoid landing on trees. Multispectral images can also help mobile platforms know the location of the sky in the imaging scene because the sky usually has obvious multispectral features. Multispectral images can also help UAS reliably and accurately locate other UAS in the sky, which is very useful for aerial manipulation of UAS clusters, such as UAS clusters.

[0044] In addition to the above, embodiments can be made relatively compact, thereby reducing size, weight and power requirements (relative to conventional systems), and are therefore suitable for deployment in a variety of applications (e.g., relatively small unmanned land vehicles and aircraft systems). Modern manned and unmanned mobile platforms (including unmanned sensor platforms, such as unmanned aerial vehicles (UAVs), remotely operated underwater vehicles (ROVs), unmanned (water) surface vehicles (USVs), and unmanned ground vehicles (UGVs)) are capable of operating over long distances and in all environments. Such systems typically rely on portable power sources that may limit their range of travel. The embodiments described herein provide a relatively lightweight, compact and distinctive multi-spectral navigation system that generally increases the achievable range of such mobile platforms (including unmanned sensor platforms), which may be particularly useful when attempting to navigate relatively quickly and exhaustively within a survey area.

[0045] In various embodiments, multispectral images and / or other sensor data can be transmitted to a base station in real time or after the survey, which can be configured to combine the sensor data with a map or floor plan of the survey area to present the sensor data in the survey map within the spatial range of the map or floor plan. Such a map or floor plan can be two-dimensional or three-dimensional. The survey map can be stored at the base station and, if the base station includes a display, presented to the operator / user in real time as a graphical overlay map. For example, during operation, such a map can provide insight for navigating a mobile platform or positioning a mobile platform for stationary observation, or, if operations are to be performed in the same area at a future time, such a map can provide information for route planning for future operations.

[0046] Figure 1 A block diagram of a multispectral navigation system 100 according to an embodiment of the present disclosure is shown. In some embodiments, the system 100 can be configured to fly over a scene, fly over a structure, or approach a target and image or sense the scene, structure, or target or a portion thereof using a gimbal system 122 to aim a multispectral imaging system / sensor payload 140 at the scene, structure, or target or a portion thereof and / or using a sensor cradle 128 to aim an environmental sensor 160 at the scene, structure, or target or a portion thereof. The resulting images and / or other sensor data can be processed (e.g., by the sensor payload 140, the platform 110, and / or the base station 130) and displayed to a user using a user interface 132 (e.g., one or more displays, such as a multifunction display (MFD), a portable electronic device, such as a tablet computer, laptop computer, or smart phone, or other appropriate interface) and / or stored in a memory for later viewing and / or analysis. In some embodiments, system 100 may be configured to use such images and / or other sensor data to control operation of platform 110, sensor payload 140, and / or environmental sensor 160, as described herein, such as controlling gimbal system 122 to aim sensor payload 140 in a particular direction or controlling propulsion system 124 to move and / or orient platform 110 to a desired position / orientation in a scene or structure or relative to a target.

[0047] In further embodiments, the system 100 may be configured to use the platform 110 and / or the sensor cradle 128 to position and / or orient the environmental sensor 160 at or relative to a scene, structure, or object, or a portion thereof. The generated sensor data may be processed (e.g., by the environmental sensor 160, the platform 110, and / or the base station 130) and displayed to a user using a user interface 132 (e.g., one or more displays, such as a multifunction display (MFD), a portable electronic device, such as a tablet computer, a laptop computer, or a smart phone, or other appropriate interface) and / or stored in a memory for later viewing and / or analysis. In some embodiments, the system 100 may be configured to use such sensor data to control the operation of the platform 110 and / or the environmental sensor 160, as described herein, such as controlling the propulsion system 124 to move and / or orient the platform 110 to a desired position / orientation in a scene or structure or relative to a desired position / orientation of an object.

[0048] exist Figure 1 In the illustrated embodiment, the multispectral navigation system 100 includes a platform 110, an optional base station 130, and at least one multispectral imaging system 140. The platform 110 may be a mobile platform configured to move or fly and position the multispectral imaging system 140 and / or environmental sensor 160 (e.g., relative to a designated or detected target). Figure 1 As shown, platform 110 may include one or more of a controller 112, an orientation sensor 114, a gyroscope / accelerometer 116, a global navigation satellite system (GNSS) 118, a communication module 120, a gimbal system 122, a propulsion system 124, a sensor cradle 128, and other modules 126. Operation of platform 110 may be substantially autonomous and / or partially or completely controlled by an optional base station 130, which may include one or more of a user interface 132, a communication module 134, and other modules 136. In other embodiments, platform 110 may include one or more elements of base station 130, such as for various types of manned aircraft, land vehicles, and / or surface or underwater watercraft.

[0049] Sensor payload 140 and / or environmental sensor 160 may be physically coupled to platform 110 and configured to capture sensor data (e.g., visible spectrum images, infrared images, ultraviolet images, narrow aperture radar data, analyte sensor data, directional radiation data, and / or other sensor data) of a target location, area, and / or object selected and / or framed by operation of platform 110 and / or base station 130. In some embodiments, one or more elements of system 100 may be implemented in a combined housing or structure that can be coupled to or within platform 110 and / or held or carried by a user of system 100.

[0050] Controller 112 may be implemented as any appropriate logic device (e.g., a processing device, a microcontroller, a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a memory storage device, a memory reader, or other device or combination of devices) that may be adapted to execute, store, and / or receive appropriate instructions, such as software instructions that implement control loops for, for example, controlling various operations of platform 110 and / or other elements of system 100. Such software instructions may also implement methods for processing infrared images and / or other sensor signals, determining sensor information, providing user feedback (e.g., via user interface 132), querying a device for operating parameters, selecting operating parameters for a device, or performing any of the various operations described herein (e.g., operations performed by logic devices of various devices of system 100).

[0051] In addition, a non-transitory medium may be provided to store machine-readable instructions for loading into and executing by the controller 112. In these and other embodiments, the controller 112 may be implemented with other components, such as volatile memory, non-volatile memory, one or more interfaces, and / or various analog and / or digital components for interfacing with devices of the system 100, where appropriate. For example, the controller 112 may be adapted to store, for example, sensor signals, sensor information, parameters for coordinate system transformations, calibration parameters, calibration point sets, and / or other operating parameters over time, and to provide such stored data to a user using the user interface 132. In some embodiments, for example, the controller 112 may be integrated with one or more other elements of the platform 110, or distributed as multiple logical devices within the platform 110, the base station 130, and / or the sensor payload 140.

[0052] In some embodiments, controller 112 may be configured to substantially continuously monitor and / or store status and / or sensor data provided by one or more elements of platform 110, sensor payload 140, environmental sensor 160, and / or base station 130, such as, for example, the position and / or orientation of platform 110, sensor payload 140, and / or base station 130, and the status of a communication link established between platform 110, sensor payload 140, environmental sensor 160, and / or base station 130. Such a communication link may be configured to substantially continuously establish and then transmit data between elements of system 100 throughout operation of system 100, wherein such data may include various types of sensor data, control parameters, and / or other data.

[0053] The orientation sensor 114 may be implemented as one or more of a compass, a float, an accelerometer, and / or other device capable of measuring the orientation of the platform 110 (e.g., the magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations (e.g., gravity and / or magnetic north)), the gimbal system 122, the imaging system / sensor payload 140, and / or other elements of the system 100 and providing such measurements as sensor signals and / or data that may be communicated to various devices of the system 100. The gyroscope / accelerometer 116 may be implemented as one or more electronic sextants, semiconductor devices, integrated chips, accelerometer sensors, accelerometer sensor systems, or other devices capable of measuring the angular velocity / acceleration and / or linear acceleration (e.g., direction and magnitude) of the platform 110 and / or other elements of the system 100 and providing such measurements as sensor signals and / or data that may be communicated to other devices of the system 100 (e.g., user interface 132, controller 112).

[0054] GNSS 118 may be implemented in accordance with any global navigation satellite system, including GPS, GLONASS, and / or Galileo-based receivers and / or other devices capable of determining an absolute and / or relative position of platform 110 (e.g., or elements of platform 110) based on wireless signals received from airborne and / or terrestrial sources (e.g., eLoran and / or other at least partially terrestrial systems), and capable of providing measurements such as sensor signals and / or data (e.g., coordinates) that may be communicated to various devices of system 100. In some embodiments, for example, GNSS 118 may include an altimeter, or may be used to provide an absolute altitude.

[0055] The communication module 120 may be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of the system 100. For example, the communication module 120 may be configured to receive flight control signals and / or data from the base station 130 and provide them to the controller 112 and / or the propulsion system 124. In other embodiments, the communication module 120 may be configured to receive images and / or other sensor information (e.g., visible spectrum, infrared, and / or ultraviolet still images or video images) from the sensor payload 140 and relay the sensor data to the controller 112 and / or the base station 130. In other embodiments, the communication module 120 may be configured to receive sensor data and / or other sensor information from the environmental sensor 160 and relay the sensor data to the controller 112 and / or the base station 130. In some embodiments, for example, the communication module 120 may be configured to support spread spectrum transmissions, and / or multiple simultaneous communication channels between elements of the system 100. The wireless communication link may include one or more analog and / or digital radio communication links, such as WiFi, etc., as described herein, and may, for example, be a direct communication link established between elements of the system 100, or may be relayed through one or more wireless relay stations configured to receive and retransmit wireless communications.

[0056] In some embodiments, the communication module 120 can be configured to monitor the status of the communication links established between the platform 110, the sensor payload 140, and / or the base station 130. Such status information can be provided to, for example, the controller 112, or transmitted to other elements of the system 100 for monitoring, storage, or further processing, as described herein. The communication links established by the communication module 120 can be configured to transmit data between elements of the system 100 substantially continuously throughout the operation of the system 100, wherein such data includes various types of sensor data, control parameters, and / or other data, as described herein.

[0057] In some embodiments, the gimbal system 122 may be implemented as an actuated gimbal mount that may be controlled by the controller 112, for example, to stabilize the sensor payload 140 relative to a target or aim the sensor payload 140 according to a desired orientation and / or relative position. Thus, the gimbal system 122 may be configured to provide the relative orientation (e.g., orientation relative to the platform 110) of the sensor payload 140 to the controller 112 and / or the communication module 120 (e.g., the gimbal system 122 may include its own orientation sensor 114). In other embodiments, the gimbal system 122 may be implemented as a gravity-driven mount (e.g., non-actuated). In various embodiments, the gimbal system 122 may be configured to provide power, support wired communications, and / or otherwise facilitate operation of the articulated sensor / sensor payload 140. In other embodiments, the gimbal system 122 may be configured to couple to a laser pointer, rangefinder, and / or other device, for example, to support, stabilize, power, and / or aim multiple devices (e.g., the sensor payload 140 and one or more other devices) substantially simultaneously. In an alternative embodiment, the multi-spectral imaging system / sensor payload 140 may be secured to the mobile platform 110 such that the gimbal system 122 is implemented as a fixed view mounting system for the sensor payload 140 .

[0058] The propulsion system 124 may be implemented as one or more propellers, turbines, or other thrust-based propulsion systems, and / or other types of propulsion systems that may be used to provide power and / or lift to the platform 110 and / or to steer the platform 110. In some embodiments, the propulsion system 124 may include a plurality of propellers (e.g., three, four, six, eight, or other types of "rotors") that may be controlled (e.g., by the controller 112) to provide lift and motion to the platform 110 and to provide orientation to the platform 110. In other embodiments, the propulsion system 124 may be configured to primarily provide thrust, while other structures of the platform 110 provide lift, such as in fixed-wing embodiments (e.g., where the wings provide lift) and / or aerostat embodiments (e.g., balloons, airships, hybrid aerostats).

[0059] In various embodiments, the propulsion system 124 may be implemented with a portable power source, such as a battery and / or an internal combustion engine / generator and a fuel source, for example, which may be coupled to the propulsion system 124 and / or a transmission and / or drive train of the platform 110. In other embodiments, the propulsion system 124 may be implemented with a braking system 125, for example, which may be used or controlled to inhibit or eliminate movement of the platform 110, such as an electromechanically controlled disc or drum based braking system for a ground vehicle (including a passenger vehicle).

[0060] For example, other modules 126 may include other and / or additional sensors, actuators, communication modules / nodes, and / or user interfaces / interface devices, and may be used to provide, for example, additional environmental information relevant to the operation of platform 110. In some embodiments, other modules 126 may include humidity sensors, wind and / or water temperature sensors, barometers, altimeters, analyte detection systems, radar systems, proximity sensors, visible spectrum cameras or infrared / thermal cameras (with additional mounts), irradiance detectors, and / or other environmental sensors that provide measurements and / or other sensor signals that may be displayed to a user and / or used by other devices of system 100 (e.g., controller 112) to provide operational control of platform 110 and / or system 100.

[0061] In some embodiments, other modules 126 may include one or more actuated and / or articulated devices (e.g., multi-spectral active illuminators, visible light and / or infrared cameras, radar, sonar, and / or other actuated devices) coupled to platform 110, wherein each actuated device includes one or more actuators adapted to adjust the orientation of the device relative to platform 110 in response to one or more control signals (e.g., provided by controller 112). In particular, other modules 126 may include a stereo vision system configured to provide image data that may be used to calculate or estimate, for example, the position of platform 110 or to calculate or estimate the relative position of navigation hazards proximate to platform 110. In various embodiments, controller 130 may be configured to use such proximity and / or position information to help safely navigate platform 110 and / or monitor communication link quality, as described herein.

[0062] The ranging sensor system 127 may be implemented as a radar, sonar, lidar, and / or other ranging sensor system that is fixed relative to the platform 110, the imaging system 140, and / or the environmental sensor 160, and is configured to provide two-dimensional and / or three-dimensional ranging sensor data corresponding to a depth map that overlaps with the field of view of the imaging system 140 and / or the environmental sensor 160 and / or is substantially centered on the optical axis of the imaging system 140 and / or the environmental sensor 160.

[0063] In embodiments where the ranging sensor system 127 is implemented as a radar system, the ranging sensor system 127 may be implemented as one or more electrically and / or mechanically coupled controllers, transmitters, receivers, transceivers, signal processing logic devices, various electrical components, antenna elements of various shapes and sizes, multi-channel antennas / antenna modules, radar assemblies, assembly supports, and / or various actuators suitable for adjusting the orientation of any component of the ranging sensor system 127, as described herein. For example, in various embodiments, the ranging sensor system 127 may be implemented according to various radar system arrangements that may be used to detect, for example, features of a land surface or a body of water surface and objects thereon or above, and / or their relative velocities (e.g., their Doppler velocities).

[0064] More generally, the ranging sensor system 127 may be configured to transmit one, multiple, or a series of radar beams (e.g., long-range sensor beams), receive corresponding radar returns (e.g., long-range sensor returns), and convert the radar returns into radar data and / or images (e.g., long-range sensor image data), such as one or more intensity maps and / or aggregations of intensity maps indicating the relative position, orientation, and / or other characteristics of structures, weather phenomena, waves, other moving structures, surface boundaries, and / or other maneuvering obstacles and / or objects that reflected the radar beam back to the ranging sensor system 127. The ranging sensor system 127 may be configured to provide such data and / or images, for example, to a user interface of the platform 110 and / or base station 130 for display to a user, or to the controller 112 for additional processing, as described herein. In addition, such data may be used to generate one or more graphs corresponding to AIS data, ARPA data, MARPA data, and / or one or more other target tracking and / or identification protocols.

[0065] In some embodiments, ranging sensor system 127 may be implemented using a compact design where multiple radar antennas, sensors, and / or associated processing equipment are located within a single radar assembly housing that is configured to interface with the rest of system 100 via a single cable that provides power and communications to and from ranging sensor system 127. In some embodiments, ranging sensor system 127 may include orientation and / or position sensors configured to help provide two-dimensional or three-dimensional waypoints, improve radar data and / or image quality, and / or provide highly accurate radar image data, as described herein.

[0066] Conventional radar systems can be expensive and cumbersome, and are generally not designed to provide relatively accurate and / or distortion-free radar image data. Embodiments of the ranging sensor system 127 include low-cost single-channel, dual-channel, and / or multi-channel (e.g., synthetic aperture) radar systems that can be configured to produce detailed two-dimensional and three-dimensional radar data and / or images. In some embodiments, the ranging sensor system 127 can, for example, integrate the electronics and transducer into a single waterproof package to reduce size and cost, and can be implemented with a single connection to other devices of the system 100 (e.g., via an Ethernet cable with Power over Ethernet, an integrated power cable, and / or other communication and / or power transmission conduits integrated into a single interface cable).

[0067] In various embodiments, the ranging sensor system 127 can be implemented with its own dedicated orientation and / or position sensor (e.g., similar to the orientation sensor 114, the gyroscope / accelerometer 116, and / or the GNSS 118), which can be incorporated within the radar assembly housing to provide a three-dimensional orientation and / or position of the radar assembly and / or antenna for use when processing or post-processing radar data for display. The sensor information can be used, for example, to correct for movement of the radar assembly during and / or between beam transmissions, to provide improved alignment of corresponding radar returns / samples, and / or to generate images based on the measured orientation and / or position of the radar assembly / antenna. In other embodiments, an external orientation and / or position sensor can be used alone or in combination with one or more integrated sensors.

[0068] In embodiments where the ranging sensor system 127 is implemented with one or more position sensors, the ranging sensor system 127 can be configured to provide a variety of radar data and / or image enhancements. For example, the ranging sensor system 127 can be configured to provide radar data and / or precise positioning and / or orientation of user-defined waypoints away from the platform 110. Similarly, the ranging sensor system 127 can be configured to provide accurate two-dimensional and / or three-dimensional aggregation and / or display of a series of radar data; if there is no orientation data or position data to help determine the track or heading, the radar system will typically assume a straight track, which may result in image artifacts and / or other inaccuracies in the corresponding radar data and / or images. In addition, when implemented with a position sensor, the ranging sensor system 127 can be configured to generate accurate and detailed intensity maps of maneuvering obstacles without the use of a magnetometer.

[0069] In embodiments where the ranging sensor system 127 is implemented with an orientation and / or position sensor, the ranging sensor system 127 may be configured to store such position / location information as well as other sensor information (radar returns, temperature measurements, text descriptions, altitude, platform speed, and / or other sensor and / or control information) available to the system 100. In some embodiments, the controller 112 may be configured to generate a lookup table so that a user can select a desired configuration of the ranging sensor system 127 for a particular location or coordinate some other sensor information. Alternatively, an automatic adjustment algorithm may be used to select the best configuration based on the sensor information.

[0070] In various embodiments, the ranging sensor system 127 may also be implemented with a co-located imaging system (e.g., imaging system 140), which may include one or more different types of imaging modules that may be incorporated into the radar assembly housing to provide image data substantially simultaneously with the radar data for use in processing or post-processing the radar data for display. The image data may be used to improve the operator's understanding of the radar data and increase the overall functionality of the system 100. For example, an embodiment may include one or more imaging modules such that the imaging module rotates with the radar antenna of the ranging sensor system 127 to generate a panoramic view corresponding to a radar plane position indicator (PPI) display view. As described herein, embodiments provide methods for data processing, data fusion, and display data and user interaction. In other embodiments, the ranging sensor system 127 can be implemented as a radar system that is implemented and / or configured to operate similarly to the embodiments described in U.S. patent application Ser. No. 16 / 0007,908, filed on June 13, 2018, entitled “VEHICLE BASED RADAR UPSAMPLING” (now U.S. Patent No. 10,928,512), the entire contents of which are incorporated herein by reference.

[0071] In various embodiments, the sensor bracket 128 can be implemented as a latch mechanism that can be permanently mounted to the platform 110 to provide a mounting position and / or orientation of the environmental sensor 160 relative to the center of gravity of the platform 110, relative to the propulsion system 124, and / or relative to other elements of the platform 110. In addition, the sensor bracket 128 can be configured to provide power, support wired communications, and / or otherwise facilitate the operation of the environmental sensor 160, as described herein. Thus, the sensor bracket 128 can be configured to provide power, telemetry, and / or other sensor data interfaces between the platform 110 and the environmental sensor 160. In some embodiments, the gimbal system 122 can be implemented similarly to the sensor bracket 128, and vice versa.

[0072] For example, the sensor bracket 128 can be implemented as an actuated gimbaled mount that can be controlled by the controller 112 to stabilize the environmental sensor 160 relative to a target or aim the environmental sensor 160 according to a desired direction and / or relative position. Thus, the sensor bracket 128 can be configured to provide the relative orientation of the environmental sensor 160 (e.g., relative to the orientation of the platform 110) to the controller 112 and / or the communication module 120 (e.g., the sensor bracket 128 can include its own orientation sensor 114). In other embodiments, the sensor bracket 128 can be implemented as a gravity-driven mount (e.g., non-actuated). In other embodiments, the sensor bracket 128 can be configured to couple to a laser pointer, a rangefinder, and / or other device, for example, to support, stabilize, power, and / or aim multiple devices (e.g., the environmental sensor 160 and one or more other devices) substantially simultaneously.

[0073] The user interface 132 of the base station 130 may be implemented as one or more of a display, a touch screen, a keyboard, a mouse, a joystick, a knob, a steering wheel, a yoke, and / or any other device capable of accepting user input and / or providing feedback to a user. In various embodiments, the user interface 132 may be adapted to provide user input (e.g., as a signal and / or sensor information transmitted by a communication module 134 of the base station 130) to other devices of the system 100 (e.g., the controller 112). The user interface 132 may also be implemented with one or more logic devices (e.g., similar to the controller 112), which may be adapted to store and / or execute instructions, such as software instructions, to implement any of the various processes and / or methods described herein. For example, the user interface 132 may be adapted, for example, to form a communication link, transmit and / or receive communications (e.g., infrared images and / or other sensor signals, control signals, sensor information, user input, and / or other information), or perform various other processes and / or methods described herein.

[0074] In one embodiment, user interface 132 may be adapted to display time series of various sensor information and / or other parameters as part of or superimposed on a graph or map, which may be referenced to the location and / or orientation of platform 110 and / or other elements of system 100. For example, user interface 132 may be adapted to display time series of the location, heading, and / or orientation of platform 110 and / or other elements of system 100 superimposed on a geographic map, which may include one or more graphs indicating corresponding time series of actuator control signals, sensor information, and / or other sensor and / or control signals.

[0075] In some embodiments, user interface 132 may be adapted to accept user input, including, for example, a user-defined target destination, heading, waypoint, route, and / or orientation of elements of system 100, and generate control signals to move platform 110 in accordance with the target destination, heading, route, and / or orientation, or to correspondingly aim sensor payload 140 or environmental sensor 160. In other embodiments, for example, user interface 132 may be adapted to accept user input to modify control loop parameters of controller 112.

[0076] In other embodiments, user interface 132 may be adapted to accept user input, for example, including a user-defined target pose, orientation, and / or position for an actuation or articulation device (e.g., sensor payload 140 or environmental sensor 160) associated with platform 110, and generate control signals for adjusting the orientation and / or position of the actuation device according to the target pose, orientation, and / or position. Such control signals may be transmitted to controller 112 (e.g., using communication modules 134 and 120), which may then control platform 110 accordingly.

[0077] The communication module 134 may be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of the system 100. For example, the communication module 134 may be configured to transmit flight control signals from the user interface 132 to the communication module 120 or 144. In other embodiments, the communication module 134 may be configured to receive sensor data (e.g., visible spectrum, infrared and / or ultraviolet static images or video images, or other sensor data) from the sensor payload 140. In some embodiments, the communication module 134 may be configured to support, for example, spread spectrum transmission and / or multiple simultaneous communication channels between elements of the system 100. In various embodiments, as described herein, the communication module 134 may be configured to monitor the status of the communication link established between the base station 130, the sensor payload 140, and / or the platform 110 (e.g., including, for example, packet loss for transmitting and receiving data between elements of the system 100 using a digital communication link). As described herein, such status information may be provided to the user interface 132, for example, or transmitted to other elements of the system 100 for monitoring, storage, or further processing.

[0078] For example, other modules 136 of base station 130 may include other and / or additional sensors, actuators, communication modules / nodes, and / or user interface devices for providing additional environmental information associated with base station 130. In some embodiments, for example, other modules 136 may include humidity sensors, wind and / or water temperature sensors, barometers, radar systems, visible spectrum cameras, infrared or thermal cameras, GNSS, and / or other environmental sensors that provide measurements and / or other sensor signals that may be displayed to a user and / or used by other devices of system 100 (e.g., controller 112) to provide operational control of platform 110 and / or system 100, or to process sensor data to compensate for environmental conditions, such as water content of the atmosphere at approximately the same altitude and / or located in the same area as platform 110 and / or base station 130. In some embodiments, other modules 136 may include one or more actuation and / or articulation devices (e.g., multi-spectral active illuminators, visible light and / or infrared cameras, radar, sonar, and / or other actuation devices), wherein each actuation device includes one or more actuators adapted to adjust the orientation of the device in response to one or more control signals (e.g., provided via user interface 132).

[0079] In embodiments where the imaging system / sensor payload 140 is implemented as an imaging device, the imaging system / sensor payload 140 may include an imaging module 142 that may be implemented as a cooled and / or uncooled array of detector elements, such as visible spectrum, infrared, and / or ultraviolet sensitive detector elements, including quantum well infrared photodetector elements, bolometer or microbolometer based detector elements, type II superlattice based detector elements, and / or other infrared spectrum detector elements (e.g., as well as other detector elements sensitive to other spectra) that may be arranged in a focal plane array (FPA). In various embodiments, the imaging module 142 may implement a complementary metal oxide semiconductor (CMOS) based FPA with detector elements that are simultaneously sensitive to portions of the visible, near infrared (NIR), and long wave ultraviolet (LWUV) spectra. In various embodiments, the imaging module 142 may include one or more logic devices (e.g., similar to the controller 112) that may be configured to process an image captured by the detector elements of the imaging module 142 before providing the image to the memory 146 or the communication module 144. More generally, imaging module 142 may be configured, at least in part or in conjunction with controller 112 and / or user interface 132 , to perform any of the operations or methods described herein.

[0080] In some embodiments, sensor payload 140 may, for example, be implemented with a second or additional imaging module similar to imaging module 142, which may include detector elements configured to detect other electromagnetic spectra (e.g., visible light, thermal, ultraviolet, and / or other electromagnetic spectra or subsets of such spectra). In various embodiments, such additional imaging modules may be calibrated or registered to imaging module 142 such that images captured by each imaging module occupy a known and at least partially overlapping field of view of the other imaging module, thereby allowing images of different spectra to be geometrically registered to one another (e.g., by scaling and / or positioning). In some embodiments, in addition to or as an alternative to relying on known overlapping fields of view, pattern recognition processing may be used to register images of different spectra to one another.

[0081] The communication module 144 of the sensor payload 140 may be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of the system 100. For example, the communication module 144 may be configured to transmit an image from the imaging module 142 to the communication module 120 or 134. In other embodiments, the communication module 144 may be configured to receive control signals from the controller 112 and / or the user interface 132 (e.g., control signals that direct the acquisition, focusing, selective filtering, and / or other operations of the sensor payload 140). In some embodiments, the communication module 144 may be configured to support, for example, spread spectrum transmission, and / or multiple simultaneous communication channels between elements of the system 100. In various embodiments, as described herein, the communication module 144 may be configured to monitor the status of the communication link established between the sensor payload 140, the base station 130, and / or the platform 110 (e.g., including, for example, packet loss of data transmitted and received between elements of the system 100 using a digital communication link). As described herein, such status information may be provided to imaging module 142, for example, or transmitted to other elements of system 100 for monitoring, storage, or further processing.

[0082] Memory 146 may be implemented as one or more machine-readable media and / or logic devices configured to, for example, store software instructions, sensor signals, control signals, operating parameters, calibration parameters, infrared images, and / or other data that facilitates operation of system 100 and provide it to various elements of system 100. Memory 146 may also be implemented at least in part as a removable memory, such as a secure digital memory card, for example, including an interface for such a memory.

[0083] The orientation sensor 148 of the sensor payload 140 may be implemented similarly to the orientation sensor 114 or the gyroscope / accelerometer 116 and / or any other device capable of measuring the orientation (e.g., the magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations (e.g., gravity and / or magnetic north)) of the sensor payload 140, the imaging module 142, and / or other elements of the sensor payload 140 and providing such measurements as sensor signals that may be transmitted to various devices of the system 100. The gyroscope / accelerometer (e.g., angular motion sensor) 150 of the sensor payload 140 may be implemented as one or more electronic sextants, semiconductor devices, integrated chips, accelerometer sensors, accelerometer sensor systems, or other devices capable of measuring the angular velocity / acceleration (e.g., angular motion) and / or linear acceleration (e.g., direction and magnitude) of the sensor payload 140 and / or various elements of the sensor payload 140 and providing such measurements as sensor signals that may be transmitted to various devices of the system 100. GNSS 149 may be implemented similarly to GNSS 118 and / or any other device capable of measuring the position of sensor payload 140 , imaging module 142 , and / or other elements of sensor payload 140 and providing such measurements as sensor signals that may be transmitted to various devices of system 100 .

[0084] For example, other modules 152 of sensor payload 140 may include other and / or additional sensors, actuators, communication modules / nodes, cooled or uncooled filters, and / or user interface devices for providing additional environmental information associated with sensor payload 140. In some embodiments, other modules 152 may include humidity sensors, wind and / or water temperature sensors, barometers, radar systems, visible spectrum cameras, infrared cameras, GNSS, and / or other environmental sensors that provide measurements and / or other sensor signals that may be displayed to a user and / or used by imaging module 142 or other devices (e.g., controller 112) of system 100 to provide operational control of platform 110 and / or system 100, or to process images to compensate for environmental conditions.

[0085] In various embodiments, environmental sensor / sensor payload 160 may be implemented as an environmental sensor configured to generate environmental sensor data corresponding to the environment surrounding platform 110. Figure 1In the illustrated embodiment, the environmental sensor 160 includes a sensor controller 162, a memory 163, a communication module 164, a sensor assembly 166, an orientation and / or position sensor (OPS) 167, a power source 168, and other modules 170. In various embodiments, the sensor assembly 166 may be implemented with sensor elements configured to detect the presence of and / or generate sensor data corresponding to dangerous analytes, ionizing radiation, emissivity, thermal radiation, radio frequency signals, and / or other environmental conditions in proximity to the platform 110 and / or the environmental sensor 160 or in its field of view.

[0086] For example, the sensor controller 162 may be implemented as one or more any suitable logic devices (e.g., a processing device, a microcontroller, a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a memory storage device, a memory reader, or other device or combination of devices) that may be adapted to execute, store, and / or receive appropriate instructions, such as software instructions that implement control loops for controlling various operations of the environmental sensor 160 and / or other elements of the environmental sensor 160. Such software instructions may also implement methods for processing sensor signals, determining sensor information, providing user feedback (e.g., via the user interface 132), querying the device for operating parameters, selecting operating parameters for the device, or performing any of the various operations described herein.

[0087] In addition, a non-transitory medium may be provided to store machine-readable instructions for loading into and executing by the sensor controller 162. In these and other embodiments, the sensor controller 162 may be implemented with other components, such as volatile memory, non-volatile memory, one or more interfaces, and / or various analog and / or digital components for interfacing with modules of the environmental sensor 160 and / or devices of the system 100, as appropriate. For example, the sensor controller 162 may store, for example, sensor signals, sensor information, parameters for coordinate system transformations, calibration parameters, calibration point sets, and / or other operating parameters over time and provide such stored data to a user using the user interface 132. In some embodiments, for example, the sensor controller 162 may be integrated with one or more other elements of the environmental sensor 160, or distributed as multiple logical devices within the platform 110, the base station 130, and / or the environmental sensor 160.

[0088] In some embodiments, sensor controller 162 may be configured to substantially continuously monitor and / or store the status of one or more elements of sensor assembly 166 of environmental sensor 160 and / or store sensor data provided thereby, such as, for example, the position and / or orientation of platform 110, environmental sensor 160, and / or base station 130, and the status of a communication link established between platform 110, environmental sensor 160, and / or base station 130. Such a communication link may be configured to substantially continuously establish and then transmit data between elements of system 100 throughout operation of system 100, wherein such data may include various types of sensor data, control parameters, and / or other data.

[0089] The memory 163 may be implemented as one or more machine-readable media and / or logic devices, for example, configured to store software instructions, sensor signals, control signals, operating parameters, calibration parameters, sensor data, and / or other data that facilitates the operation of the environmental sensor 160 and / or other elements of the system 100, and provide it to various elements of the system 100. The memory 163 may also be implemented at least in part as a removable memory, such as a secure digital memory card, for example, including an interface for such a memory.

[0090] The communication module 164 of the environmental sensor 160 may be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of the system 100. For example, the communication module 164 may be configured to transmit sensor data from the environmental sensor 160 and / or the sensor assembly 166 to the communication module 120 of the platform 110 (e.g., for further transmission to the base station 130) or directly to the communication module 134 of the base station 130. In other embodiments, the communication module 164 may be configured to receive control signals (e.g., control signals directing the operation of the environmental sensor 160) from the controller 112 and / or the user interface 132. In some embodiments, the communication module 164 may be configured to support, for example, spread spectrum transmission and / or multiple simultaneous communication channels between elements of the system 100.

[0091] The sensor assembly 166 may be implemented with one or more sensor element supports (e.g., a printed circuit board “PCB”), connectors, sensor elements, and / or other modules configured to facilitate operation of the environmental sensor 160. In certain embodiments, the environmental sensor 160 may be implemented as a relatively high-resolution visible spectrum camera (e.g., an HD or 2K or 4K visible spectrum camera), and the sensor assembly 166 may be implemented as a relatively high-resolution FPA of visible spectrum sensitive detector elements configured to generate relatively high-resolution images and / or videos of the scene imaged substantially simultaneously by the multi-spectral imaging system 140.

[0092] The orientation and / or position sensor (OPS) 167 of the environmental sensor 160 can be implemented similarly to the orientation sensor 114, the gyroscope / accelerometer 116, the GNSS 118, and / or any other device capable of measuring the orientation and / or position (e.g., the magnitude and direction of roll, pitch, and / or yaw, and absolute or relative position relative to one or more reference directions such as gravity and / or magnetic north) of the environmental sensor 160, the sensor assembly 166, and / or other elements of the environmental sensor 160 and providing such measurements as sensor signals that can be transmitted to various devices of the system 100.

[0093] Power source 168 may be implemented as any power storage device that is configured to provide sufficient power to each sensor element of sensor assembly 166 to keep all such sensor elements activated and capable of generating sensor data when environmental sensor 160 is disconnected from external power (e.g., provided by platform 110 and / or base station 130). In various embodiments, power source 168 may be implemented by a supercapacitor so as to be relatively lightweight and facilitate flight of platform 110 and / or relatively easy handheld operation of platform 110 (e.g., where platform 110 is implemented as a handheld sensor platform).

[0094] For example, other modules 170 of environmental sensors 160 may include other and / or additional sensors, actuators, communication modules / nodes, and / or user interface devices for providing additional environmental information associated with environmental sensors 160. In some embodiments, as described herein, other modules 170 may include humidity sensors, wind and / or water temperature sensors, barometers, GNSS, and / or other environmental sensors that provide measurements and / or other sensor signals that may be displayed to a user and / or used by sensor controller 162 or other devices (e.g., controller 112) of system 100 to provide operational control of platform 110 and / or system 100, or to process sensor data to compensate for environmental conditions.

[0095] Generally, each element of system 100 may be implemented with any appropriate logic device (e.g., a processing device, a microcontroller, a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a memory storage device, a memory reader, or other device or combination of devices) that may be adapted to execute, store and / or receive appropriate instructions, such as software instructions implementing methods for, for example, providing sensor data and / or images, or for transmitting and / or receiving communications (e.g., sensor signals, sensor information, and / or control signals) between one or more devices of system 100.

[0096] In addition, one or more non-transitory media may be provided for storing machine-readable instructions for loading into and execution by any logic device implemented with one or more devices of system 100. In these and other embodiments, the logic device may be implemented with other components, such as volatile memory, non-volatile memory, and / or one or more interfaces (e.g., an Inter-Integrated Circuit (I2C) interface, a Mobile Industry Processor Interface (MIPI), a Joint Test Action Group (JTAG) interface (e.g., the IEEE 1149.1 Standard Test Access Port and Boundary Scan Architecture), and / or other interfaces, such as interfaces for one or more antennas, or interfaces for particular types of sensors), where appropriate.

[0097] Sensor signals, control signals, and other signals can be transmitted between the elements of the system 100 using various wired and / or wireless communication technologies, including, for example, voltage signaling, Ethernet, WiFi, Bluetooth, Zigbee, Xbee, Micronet, or other media and / or short-range wired and / or wireless networking protocols and / or implementations. In such embodiments, each element of the system 100 can include one or more modules that support wired, wireless, and / or a combination of wired and wireless communication technologies. In some embodiments, the various elements or portions of the elements of the system 100 can be integrated with each other, for example, or can be integrated onto a single printed circuit board (PCB) to reduce system complexity, manufacturing costs, power requirements, coordinate system errors, and / or timing errors between various sensor measurements.

[0098] For example, each element of system 100 may include one or more batteries, capacitors, or other power storage devices, and may include one or more solar cell modules or other power generation devices. In some embodiments, one or more of the devices may be powered by a power source of platform 110 using one or more power lines. Such power lines may also be used to support one or more communication technologies between elements of system 100.

[0099] Figure 2 A diagram of mobile platforms 110A and 110B of a multispectral navigation system 200 including an embodiment of an environmental sensor 160 and an associated sensor bracket 128 is shown in accordance with an embodiment of the present disclosure. Figure 2In the illustrated embodiment, the multispectral navigation system 200 includes a base station 130, an optional sub-pilot station 230, a mobile platform 110A having an articulated imaging system / sensor payload 140, a gimbal system 122, an environmental sensor 160, and a sensor bracket 128, and a mobile platform 110B having an articulated imaging system / sensor payload 140, a gimbal system 122, an environmental sensor 160, and a sensor bracket 128, wherein the base station 130 and / or the optional sub-pilot station 230 can be configured to control the movement, position, orientation, and / or general operation of the platform 110A, the platform 110B, the sensor payload 140, and / or the environmental sensor 160.

[0100] In various embodiments, the sub-pilot station 230 may be implemented similarly with respect to the base station 130, e.g., include similar elements and / or be capable of similar functionality. In some embodiments, the sub-pilot station 230 may include multiple displays to facilitate operation of the environmental sensors 160 and / or various imaging and / or sensor payloads of the mobile platforms 110A-B, typically separate from the pilot mobile platforms 110A-B, and facilitate substantially real-time analysis, visualization, and communication of sensor data and corresponding instructions, e.g., to first responders in contact with a sub-pilot or user of the system 200. For example, the base station 130 and the sub-pilot station 230 may each be configured to present any of the display views described herein.

[0101] As described herein, embodiments of the multi-spectral navigation system 100 may be implemented with a relatively compact, lightweight, low power multi-spectral imaging system (e.g., the multi-spectral imaging system 140) that may be used to assist in operating a mobile platform, such as through assisted navigation, wherein processed images and / or display views are provided to an operator to assist the operator in navigating the mobile platform, or through autopilot navigation, wherein such images are used to automatically navigate the mobile platform according to a desired route, destination, or other operating parameters.

[0102] In some embodiments, a multispectral imaging system may include an imaging module implemented by a CMOS-based FPA that is formed, manufactured, assembled and / or otherwise configured to have sensitivity in the IR, VIS, and UV spectrum / bands. Such an imaging module may include a Bayer filter that is configured to generate a mosaic or pattern of IR, VIS, and UV pixels in the FPA such that each image captured by the multispectral imaging module includes IR, VIS, and UV information about each scene imaged by the multispectral imaging module. In particular embodiments, such an FPA may be sensitive to portions of the NIR, VIS, and LWUV spectra, including at least 400-750nm (VIS), 750-1100nm (NIR), and 330-400nm (LWUV), and the Bayer filter may be configured to selectively pass such bands according to a particular mosaic or pattern.

[0103] In some embodiments, for example, such a Bayer filter may be deposited directly on the FPA, and / or may form a checkerboard mosaic similar to that used for RGB VIS imaging. Depending on the particular mosaic selected for the Bayer filter and / or depending on a particular demosaicing algorithm (e.g., based on one or more of interpolation, spectral correlation, spatial correlation, and / or other demosaicing techniques), any of the VIS, NIR, or LWUV spectra may be emphasized or de-emphasized relative to the other two spectra in the resulting image.

[0104] In some embodiments, a particular demosaicing algorithm may be selected, for example, based on one or more environmental conditions associated with the imaged scene or object, or one or more environmental conditions associated with the operating state, position, or orientation of the imaging module 142 of the mobile platform 110 and / or the multispectral imaging system 140. For example, a particular demosaicing algorithm may be configured to deemphasize the contribution of the horizon-aligned polarized VIS, NIR, and / or LWUV to the captured image based on the time of day (e.g., the position of the sun in the sky), the position of the mobile platform 110, and / or the orientation of the mobile platform 110 and / or the multispectral imaging system 140 to reduce image artifacts (e.g., pixel saturation artifacts) caused by relatively strong reflections of ambient light from horizon-aligned surfaces. In another example, when UV contribution is expected to be relatively low (e.g., when the sun is below the horizon), a particular demosaicing algorithm may be configured to emphasize UV ​​contribution to a captured image based on the time of day (e.g., the level of natural ambient light provided by the sun), the location of mobile platform 110 (e.g., the geographic location and altitude at which mobile platform 140 is placed within a topographic map of the horizon), and / or the orientation of mobile platform 110 and / or multispectral imaging system 140.

[0105] For example, such a Bayer filter may be implemented as a mosaic of single bandpass filters (e.g., each pixel receives only one of the IR, VIS, UV bands passed), or may be implemented as a mosaic of notched broadband transmission filters (e.g., each pixel receives all bands except one of the notched / filtered IR, VIS, UV bands). In embodiments where the Bayer filter is implemented as a mosaic of notched broadband transmission filters, a selected primary frequency band may be synthesized from a linear combination of two or more pixels receiving differentiated spectra (e.g., associated with spectrally differentiated notched broadband transmission filters). In various embodiments, such synthesis may be implemented within / included as part of a demosaicing algorithm, as described herein. Such techniques may provide increased signal-to-noise characteristics relative to filtering implemented by a mosaic of single bandpass filters.

[0106] In various embodiments, the multispectral imaging system 140 can operate at a reasonably high frame rate so that the resulting image streams are simultaneous enough to be useful for navigation of a mobile platform (e.g., where an operator or autopilot frequently needs to make time-critical maneuvering decisions). For example, embodiments can operate (e.g., capture and process images) at a frame rate approaching about 100 frames per second or higher.

[0107] In certain embodiments, the multispectral imaging system 140 is capable of cycling the integration time associated with the FPA of the imaging module 142 over two or more preset values ​​such that the multispectral imaging system 140 can produce high dynamic range (HDR) images in all imaging bands. For example, such an HDR mode can be used to provide midwell exposure values ​​under various lighting conditions, and in some embodiments, the integration time can be determined and / or adjusted by the multispectral imaging system 140 based on ambient light levels (e.g., in one or more spectrums), contrast levels in previously captured multispectral images, and / or other environmental sensor data and / or derived or processed sensor data and / or images. This is particularly important, for example, where the solar spectrum has very different scene brightnesses in the IR, VIS, and UV, and where the scene spectrum changes over the diurnal cycle.

[0108] The mid-well exposure value is related to the FPA of imaging module 142 (eg, Figure 3Integrating capacitors of sensor elements in a CMOS-based embodiment of the FPA 374 of the multispectral imaging system 140 of the imaging module 142 are allowed to charge to approximately half of their capacity for an exposure event before being read out (e.g., discharged) by the module controller 372 across the PCB 375 (both of the imaging module 142). For example, the readout frequency and / or exposure time (e.g., controlled by a mechanical, electromechanical, and / or electronic (e.g., LCD) embodiment of the shutter 349) can be adjusted based at least in part on average scene radiance (e.g., for a particular band or across multiple bands selected by the filter system 376) such that most sensor elements of the FPA 374 associated with one or more bands captured by the FPA 374 operate at approximately their mid-well exposure value. Operating at such a mid-well exposure value results in image data captured within the most linear portion of the dynamic range of the FPA 374 (e.g., providing a substantially linear response to photons intercepted by the sensor elements of the FPA 374), which helps avoid image noise associated with, for example, low well charge levels. By cycling through different exposure times (eg, integration times), embodiments can achieve mid-well operation performance for each wavelength band captured by the multispectral imaging system 140 .

[0109] For example, the VIS radiance in a scene will typically be higher than the NIR or LWUV radiance in the scene. An exposure / integration time of 5 milliseconds may provide a mid-well exposure level for the NIR and LWUV sensor elements (e.g., as selected by the filter system 376), but may overexpose the VIS sensor elements of the FPA 374. The multispectral imaging system 140 may be configured to capture a first image according to a first 5 millisecond exposure time, then capture a second image according to a second, shorter exposure time, and then combine the NIR and LWUV components of the first image with the VIS component of the second image to generate an HDR image (e.g., as long as the first and second images are captured one after the other and / or when the multispectral imaging system 140, or at least the FOV 345, is substantially stationary relative to the scene 302).

[0110] In various embodiments, multi-spectral imaging system 140 may be equipped with a lens system that is achromatic across the spectral bands captured by imaging module 142. Such a lens system may be implemented with a focal length selected to provide a relatively wide field of view (FOV) sufficient for navigation system and UAS imaging FOV requirements (e.g., mission, specification, and / or regulatory requirements).

[0111] In some embodiments, the multispectral imaging system 140 can be configured to process captured images according to a multispectral image analysis and / or algorithm (e.g., onboard or after transmission to other processing elements of the system 100) that is configured to classify scene pixels according to the likelihood that the scene pixels are part of a particular object class. For example, a clear sky has a unique spectrum for it, being darker in the NIR and brighter in the UV. Vehicles (even those painted white) tend to have opposite spectra: bright in the NIR band and dark in the UV band. The two scene elements can therefore be reliably classified based at least in part on their spectral characteristics. In various embodiments, for example, such a multispectral image analysis and / or algorithm can be performed by a convolutional neural network (CNN) implemented within the multispectral imaging system 140 or within one or more controllers associated with the multispectral navigation system 100.

[0112] In certain embodiments, the image data provided by the multispectral imaging system 140 and / or the imaging module 142 can be encoded using two bytes per pixel, where 12 bits encode the image data (e.g., intensity) and the remaining four bits encode information about the classification probability associated with the pixel, such as a 95% probability that the pixel is sky or not sky. Such data can then be used by the multispectral navigation system 100 to make maneuvering (e.g., braking and steering) decisions in substantially real time (e.g., at time instances of 100+ frames / second).

[0113] Figure 3 FIG. 1 is a diagram of a multispectral imaging system 140 for use in the multispectral navigation system 100 and / or 300 according to an embodiment of the present disclosure. Figure 3, the multispectral imaging system 140 includes an imaging module 142 that includes a multispectral FPA 374 that receives light 308 from a scene 302 along an optical axis 344 and according to a FOV 345 through a filter system 376, a lens system 378, and / or an optional shutter 349. In various embodiments, the imaging module 142 may include a printed circuit board (PCB) 375 or similar structure that is configured to support the FPA 374 and couple the FPA 374 and / or other elements of the imaging module 142 to a module controller 372 of the imaging module 142. As described herein, the filter system 376 may be implemented in some embodiments as a Bayer filter having a selected mosaic configured to provide a differentiated spectrum (e.g., portions of the VIS, IR, and UV spectra) to pixels of the FPA 374. As also described herein, the lens system 378 may be achromatic, for example, for the differentiated spectrum provided to the pixels of the FPA 374, and is configured to provide a FOV 345. In some embodiments, lens system 378 may be actuated to adjust FOV 345, the zoom level of multispectral imaging system 140, and / or the focus of light transmitted to FPA 374. In other embodiments, lens system 378 may be a fixed lens system.

[0114] The module controller 372 may be implemented as any suitable processing device (e.g., a microcontroller, a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other logic device) that may be used by the imaging module 142 and / or the multispectral imaging system 140 to execute appropriate instructions, such as software instructions and / or signal processing operations, for example, for capturing a multispectral image of the scene 302 using the FPA 374, the filter system 376, the lens system 378, and / or the shutter 349, demosaicing raw pixel data associated with such a multispectral image, and / or classifying pixels in such an image associated with an object 304 and / or a background 306 within the scene 302 (e.g., using a CNN implemented within the module controller 372). In addition, the module controller 372 may be implemented with various signal processing devices, such as analog-to-digital converters (ADCs), trigger inputs, timing circuits, and other signal or sensor processing devices described herein.

[0115] In various embodiments, FPA 374 may be implemented by a two-dimensional planar array of similarly manufactured / sized pixel structures, each pixel structure being configured to be sensitive across the entire spectral band of imaging module 142. In other embodiments, FPA 374 may be implemented by an array of structurally differentiated sub-arrays of pixel structures, e.g., where each sub-array is sensitive to a differentiated subset of the full spectral band of imaging module 142, and / or may be non-planar (e.g., concave relative to optical axis 344), three-dimensional (e.g., multi-layered), and / or may include pixels of differentiated sizes (e.g., having a larger surface area as the distance from optical axis 344 increases).

[0116] For example, the filter system 376 may be implemented as a static Bayer filter structure deposited or otherwise attached to the active surface of the FPS 374, or may be implemented as an adjustable or controllable Bayer filter structure or other type of filter structure configured to provide differentiated spectral illumination of the pixels of the FPA 374 or portions of the FPA. In a specific example, such a Bayer filter may be implemented with two VIS pixels for each NIR and LWUV pixel (e.g., similar to some Bayer filter patterns for color VIS cameras with 2 pixels for green and 1 pixel each for red and blue). Such filters may be implemented as multilayer dielectric interference type bandpass filters. More generally, the filter system 376 may be configured to provide spatially and spectrally differentiated illumination of the FPA 374 according to two or more, or three or more different spectra, where each spectrum may be fully differentiated or may partially overlap with an adjacent differentiated spectrum. In one embodiment, the characteristics of the filter system 376 may include a NIR band of 780-1000 nm, a LWUV band of 370 nm with a full width at half maximum (FWHM) of 60 nm, and a general VIS band.

[0117] In other embodiments, the filter system 376 may be implemented at least in part by a multivariate optical element filter array (e.g., a multivariate filter or filter array). Such special interference filters are typically configured according to a relatively complex modified transmission curve that is designed to perform optical computational operations, such as operations similar to the dot product between a scaled regression vector and the spectral response of a scene. For example, the regression vector is typically the result of a filter design and can be optimized for a particular spectrum of interest. In one embodiment, such a filter may include an array of 3 or possibly more different multivariate optical elements (MOEs) with a selected pattern. MOE arrays designed to specifically detect scene spectra such as vegetation, clear sky, cloudy sky, road surface, and vehicles may provide advantages over simple 3-band approaches. Compared to conventional bandpass filters, such MOE filters can be composed of relatively few layers, so they are typically physically thinner than bandpass filters, which makes them an attractive choice for filter arrays in situations where FPA pixels can have a size comparable to the thickness of the filter layer stack. Such MOE filters may also tend to have better overall scene flux throughput (e.g., when they are composed of fewer layers than a comparable bandpass filter array).

[0118] For example, lens system 378 may be implemented with one or more lenses, each configured to pass light to substantially all pixels of FPA 374, or may be implemented with an array of lenses (e.g., a microlens array), each configured to pass light to a subset of pixels of FPA 374. In general, in embodiments where the FPA is sensitive to NIR, VIS, and LWUV bands, each lens of lens system 378 may be configured to be color corrected or achromatic from 330-1100 nm, as described herein. In some embodiments, FOV 345 may be asymmetric (e.g., to match a corresponding FPA size) and be approximately 42 by 34 degrees.

[0119] Although Figure 3 The illustrated embodiment shows a relatively compact multispectral imaging system 140 implemented with a single multispectral imaging module 142 capable of providing a single-view multispectral image of a scene 302, but in other embodiments, for example, the multispectral imaging system 140 may be implemented with multiple imaging modules 142, each imaging module being sensitive to a separately differentiated spectrum of light, and / or each imaging module providing a different view of the scene 302, such as according to a different optical axis and / or a different FOV.

[0120] For example, PCB 375 may be a conventional printed circuit board and adapted to provide electrical access (e.g., via various metal traces) to and physical support for FPA 374 and / or other elements of imaging module 142. In some embodiments, shutter 349 may be implemented as a mechanical or removable light shield adapted to selectively block one or more bands of light 308. In various embodiments, shutter 349 may be electronically actuated (e.g., opened and / or closed), for example, by module controller 372 and / or imaging system controller 312. For example, shutter 349 may be coupled to / supported by housing 348, and housing 348 may be adapted, for example, to protect system 300 from environmental conditions associated with space or atmospheric flight and / or other outdoor environmental conditions, such as fixedly or articulatedly mounted on a land vehicle. In other embodiments, housing 348 may be adapted for handheld use.

[0121] like Figure 3 As shown, the multispectral imaging system 140 may be implemented with various other components suitable for facilitating the operation of the multispectral imaging system 140, including capturing a multispectral image of the scene 302, demosaicing the image of the scene 302, detecting characteristics (e.g., presence, extent, range, translucency, visible color, and / or other characteristics) of objects 304 and background 306 of the scene 302, and / or classifying objects 304 and background 306 of the scene 302 (e.g., as sky or non-sky, maneuvering obstacles or non-maneuvering obstacles, moving objects or non-moving objects, vegetation or non-vegetation, road / land or non-road / land, water surface or non-water surface, and / or the likelihood thereof), and / or reporting such sensor data to other elements of the system 100 as described herein. In some embodiments, the system 300 may report sensor data by aggregating sensor data over time (e.g., multiple frames) to provide duration-based reliability of such characteristics and / or classifications determined by the system 300, and then transmitting the sensor data to other elements of the system 100. In other embodiments, system 300 may report sensor data by activating an LED indicator and / or transmitting an alarm or notification signal to a component of system 300 or 100 (eg, an alarm or electrical switch or relay).

[0122] For example, depending on the desired application and / or overall size of the multi-spectral imaging system 140 and / or imaging module 142, each of the imaging sensor controller 312, memory 146, user interface 332, communication module 144, display 333, and other modules 152, if optionally included in the multi-spectral imaging system 140, can be coupled to the PCB 375 or the housing 348. In other embodiments, any one or a group of such components can be implemented, for example, external to the multi-spectral imaging system 140 and / or in a distributed or grouped manner (e.g., multiple imaging system controllers 312 operate the multi-spectral imaging system 140, or multiple multi-spectral imaging systems 140 are operated by a single imaging system controller 312).

[0123] The imaging system controller 312 may be implemented as any suitable processing device (e.g., a microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other logic device) that the system 300 may use to execute appropriate instructions, such as software instructions and / or signal processing operations, for example, to capture a multispectral image of the scene 302 using the imaging module 142, demosaic the raw pixel data associated with such a multispectral image, classify pixels and / or elements of the scene 302 in such an image (e.g., using a CNN implemented within the imaging system controller 312), and / or report such sensor data / information to other elements of the multispectral navigation system 100 or 300. In addition, as described herein, the imaging system controller 312 may be implemented with various signal processing devices, such as analog-to-digital converters (ADCs), trigger inputs, timing circuits, and other signal or sensor processing devices.

[0124] In various embodiments, at least some portion or functionality of imaging system controller 312 may be part of or implemented with other existing controllers or logic devices of a separate system, such as a server, a personal electronic device (e.g., a mobile phone, a smart phone, a tablet device, a laptop computer, a desktop computer), and / or any other device that may be used to process, report, or act on multispectral images captured by system 300. In other embodiments, imaging system controller 312 may be adapted to interface and communicate with various external controllers or logic devices and associated components and / or perform various operations in a distributed manner.

[0125] In general, imaging system controller 312 may be adapted to interface and communicate with other components of system 300 to perform the methods and processes described herein. In one embodiment, imaging system controller 312 may be adapted, for example, to report multispectral images and / or pixel / object classifications using communication module 144 to display 333 and present and / or display such information or alarm notifications, or present and / or display an image of a classification map corresponding to scene 302. In another embodiment, imaging system controller 312 may be adapted to use communication module 144 to establish a wired or wireless communication link with, for example, a remote reporting system, and report such sensor information.

[0126] The memory 146 is generally in communication with at least the imaging system controller 312 and may include one or more memory devices (e.g., memory components) to store information, including image data, calibration data, other types of sensor data, and / or software instructions. Such memory devices may include various types of volatile and non-volatile information storage devices, such as RAM (random access memory), ROM (read only memory), EEPROM (electrically erasable read only memory), flash memory, disk drives, and / or other types of memory. In one embodiment, the memory 146 may include a removable memory device that can be removed from the system 300 and used to transfer stored data to other systems for further processing and inspection.

[0127] The communication module 144 can be configured to facilitate communication and interfacing between various components of the system 300 (e.g., between the imaging system controller 312 and the memory 146 and / or the display 333) and / or between various external devices (e.g., wireless access points, personal electronic devices, servers, and / or other detectors). For example, components such as the user interface 332 and the display 333 can transmit data to and receive data from the imaging system controller 312 via the communication module 144, which can be adapted to manage wired and / or wireless communication links between the various components. Thus, the communication module 144 can support various interfaces, protocols, and standards for local system networks, such as a controller area network (CAN) bus, a local interconnect network (LIN) bus, a media oriented systems transport (MOST) network, or the ISO 11738 (or ISO bus) standard.

[0128] In some embodiments, the imaging system controller 312 can be adapted to communicate with a remote user interface, notification system, or other detection system via the communication module 144, for example, to aggregate reports from multiple systems or sensors and / or implement specific detection and / or notification methods. Thus, the communication module 144 can include a wireless communication component (e.g., based on the IEEE 802.11 WiFi standard, the Bluetooth™ standard, the ZigBee™ standard, or other suitable short-range wireless communication standards), a wireless broadband component (e.g., based on WiMax technology), a mobile cellular component, a wireless satellite component, or other suitable wireless communication components. The communication module 144 can also be configured to connect to a wired network and / or device interface via a wired communication component such as an Ethernet interface.

[0129] If present, the user interface 332 provides user interaction with the multispectral imaging system 140 and may include one or more buttons, indicators (e.g., LEDs), keyboards, trackballs, knobs, joysticks, displays (e.g., liquid crystal displays, touch screen displays), and / or other types of user interfaces suitable for accepting user input and / or providing user feedback. In one embodiment, the user interface 332 may include a power button, a vibration motor, an LED indicating a manipulated obstacle, and / or a speaker (e.g., a visible, tactile, and / or audible indicator) that provides an audible indication of the manipulated obstacle. In various embodiments, as described herein, the user interface 332 may be used to enter various system configuration settings, such as integration time parameters, de-mosaicing algorithm selections, and / or other configuration settings. In some embodiments, the user interface 332 may be used to view one or more reports, charts, and / or other image data captured by the system 300 and / or processed according to the various operations described herein.

[0130] If present, display 333 may be configured to present, indicate, or otherwise communicate alarms, notifications, and / or other reports of image data and / or object or pixel classification (e.g., generated by imaging system controller 312). Display 333 may be implemented with an electronic display screen, such as a liquid crystal display (LCD), a cathode ray tube (CRT), or various other types of commonly known video displays and monitors, including touch-sensitive displays. Display 333 may be suitable for presenting image data, charts, videos, reports, or other information described herein.

[0131] Other modules 152 may include temperature sensors / probes (e.g., thermocouples, infrared thermometers), LEDs or laser diodes, ambient light sensors, voltage regulators and / or filters, variable voltage sources, and / or other types of devices that may be used to facilitate operation of multi-spectral imaging system 140 as described herein. In some embodiments, other modules 152 may include GNSS, accelerometers, compasses, and / or other orientation sensors capable of sensing the position and / or orientation of multi-spectral imaging system 140. Other modules 152 may additionally include power modules implemented as batteries, power adapters, charging circuits, power interfaces, power monitors, and / or other types of power sources that provide mobile power.

[0132] According to the embodiments described herein, multispectral navigation systems can benefit from various multispectral imaging and visualization technologies configured to improve the operational flexibility, reliability, and accuracy of such systems. In particular, embodiments can be configured to provide various display views, for example, including augmented reality views based on images provided by the multispectral imaging system 140 and / or other imagers of the system 100, thereby allowing users to use and monitor such features and capabilities, and can be implemented according to various processes and / or control loops configured to reduce the burden of navigation, protect the operation of the mobile platform of such systems, and qualitatively and quantitatively evaluate potential maneuvering obstacles and avoidance options faster and more reliably than conventional navigation systems.

[0133] In various embodiments, the system 100 can be configured to visualize and characterize maneuvering obstacles by using the multispectral imaging system 140 and other sensors mounted to the mobile platform 110. Typically, the mobile platform 110 can relay sensor data to an onboard operator or a remote operator at the base station 130 and / or the sub-pilot station 230, where the sensor data can be processed or used to manipulate the mobile platform 110. Such sensor data can also be presented on a display to help visualize and characterize maneuvering obstacles, thereby assisting human operators in detecting and avoiding maneuvering obstacles. For example, the elements of the system 100 can autonomously map the range of one or more maneuvering obstacles and overlay the resulting sensor data onto a geospatial chart or image, so that the operator can visualize the full range of the maneuvering obstacles and proceed safely. In embodiments where the system 100 or 300 includes a handheld mobile platform, the elements of the system 100 or 300 can aggregate various data to provide critical and timely warnings and / or safety instructions to the user of the handheld platform.

[0134] Embodiments may overlay 2D or 3D sensor data onto a geospatial map or image as icons or colored highlights or blobs so that a user can visualize the range of maneuvering obstacles. Embodiments may optionally include a second screen / additional base station so that users other than the UAV / UGV pilot can view / process the sensor data.

[0135] In some embodiments, a display view (e.g., presented by user interface 132 and / or display 333) may include a geospatial chart or augmented image surrounded by various selector / indicator groups (e.g., a header, payload controller menus, video feeds, and platform telemetry indicators) configured to visualize and / or quantify maneuvering obstacles and operating mobile platform 110 and / or elements of mobile platform 110. For example, a header may include one or more selectors and / or indicators configured to, for example, receive user selection of a particular selector to enable, disable, or select an activated sensor payload (e.g., multispectral imaging system 140, environmental sensor 160) to display corresponding geo-referenced sensor data in a geospatial chart or augmented image, or to indicate the operational status of mobile platform 110 and / or various elements of mobile platform 110.

[0136] In a related embodiment, the geospatial chart or augmented image includes a mobile platform indicator and a maneuvering obstacle overlay presented on top of a base map or chart. In various embodiments, the system 100 may be configured to determine the shape, range, and / or other characteristics of the maneuvering obstacle overlay within the geospatial chart or augmented image based at least in part on sensor data provided by the multispectral imaging system 140, environmental sensor 160, and orientation and / or position data (e.g., provided by OPS 167 and / or other orientation and / or position or motion sensors of the mobile platform 110 or elements of the mobile platform 110) when the mobile platform 110 is maneuvered within the area displayed in the geospatial chart or augmented image. For example, the system 100 may be configured to determine the range associated with the object 304 from the perspective of the optical axis 344 based on sensor data and / or environmental conditions provided by the mobile platform 110, and present the maneuvering obstacle overlay according to a color map to indicate a relative range or approach speed, such as warm colors (e.g., red) indicating a relatively close or rapidly approaching maneuvering obstacle, while cold colors (e.g., blue) indicate a relatively far or rapidly receding maneuvering obstacle.

[0137] In another embodiment, the system 100 may be configured to, for example, determine that multiple types of maneuvering obstacles exist within a particular survey area or scene, and present each type of maneuvering obstacle according to a different overlay presented in a display view, each overlay being selectively enabled and / or disabled by a user.

[0138] In various embodiments, the mobile platform 110 can be configured to adjust its route based on, for example, sensor data provided by the multispectral imaging system 140 and / or the environmental sensors 160 and / or based on various environmental conditions measured by sensors mounted to the mobile platform 110 or measured by an external system and transmitted to the system 100 (e.g., regional weather data provided by an online database via a wireless network linked to the base station 130 or the sub-pilot station 230). Thus, the mobile platform 110 can be configured to autonomously avoid maneuvering obstacles or hazardous environments (e.g., significant downdrafts or other undesirable environmental conditions and / or maneuvering obstacles within such undesirable environmental conditions). For example, sending a UAV / UGV into a hazardous environment may put the mobile platform 110 at risk of damage. By adding intelligent maneuvering obstacle avoidance based on multispectral imagery and environmental sensors carried on the vehicle, collision risks and / or inefficient maneuvers can be limited through automatic route adjustments, thereby protecting the mobile platform 110 and its associated sensor suite.

[0139] Embodiments described herein may provide autonomous responses to maneuvering obstacles. For example, the controller 112 and / or the controller of the base station 130 or the sub-pilot station 230 may be configured to receive multispectral images, classification data, and / or other sensor data from the mobile platform 110 and / or from sensors mounted to the mobile platform 110 and determine route adjustments to avoid detected maneuvering obstacles and / or environmental conditions. Examples of route adjustments may include stopping, turning, climbing, and / or reverse routes to retreat from or otherwise avoid maneuvering obstacles or hazardous environments. For example, such route adjustments may be relayed to a user of the base station 130, or may be implemented directly / autonomously by the mobile platform 110. Such autonomous responses are intended to maintain the integrity of the mobile platform 110 and facilitate reaching a desired destination.

[0140] Figure 4-6 Display views 400, 500, and 600 are shown including images generated by the multispectral imaging system 140 for the multispectral navigation system 100 or 300 according to an embodiment of the present disclosure. Figure 4 , display view 400 shows a relatively high resolution visible spectrum RGB or color image 402 of a neighborhood and various scene elements (e.g., roads, sidewalks, fences, vegetation, and structures behind the vegetation, all under a cloudy sky). Figure 5 , display view 500 shows a monochrome visible spectrum image 502 , a NIR image 504 , and a LWUV image 506 of the same scene depicted in visible spectrum color image 402 . Figure 6A display view 600 is shown including a multispectral image 602 that includes characteristics of each of a spectrally monochromatic visible spectrum image 502, a NIR image 504, and a LWUV image 506, wherein each differentiated spectrum is mapped to R, G, and B channels that are typically visible to the human eye (e.g., NIR image data is mapped to the R channel, VIS data is mapped to the G channel, and LWUV data is mapped to the B channel).

[0141] from Figure 4-6 It can be seen that the daytime sky spectral features have relatively low spectral features / brightness in the NIR image, medium spectral features / brightness in the VIS image, and relatively very bright spectral features / brightness in the LWUV image. Therefore, objects that are outlined against the sky in the LWUV image are much easier to segment from the sky than when using the VIS image, and multispectral analysis is better able to distinguish between the sky and foreground objects, such as other mobile platforms. For example, it is difficult to imagine Figure 5 , a daytime scene in which the sky and an enclosed object simultaneously have the same apparent radiance in all three bands depicted in . Moreover, such multispectral analysis is particularly useful when objects or maneuvering obstacles are beyond reliable LIDAR range. Thus, embodiments are generally able to use simple image subtraction to segment the sky from other objects in the scene, even when the objects appear similar in the visible spectrum. In addition, an onboard CNN or other machine vision engine (e.g., implemented within the module controller 372 and / or the imaging system controller 312) can quickly perform pixel and / or object classification and send multispectral image data including "tagged" pixels or groups of pixels to elements of the multispectral navigation system 100 and determine various maneuvering adjustments to avoid maneuvering obstacles, as described herein.

[0142] exist Figure 5 In various images of FIG. 506 , it can be seen that natural light shadows are suppressed in the UV band. For example, Rayleigh scattering typically causes the entire sky to glow relatively brightly in scattered UV sunlight. Since the entire sky is illuminated, shadows in the UV band are less intense (e.g., as shown in LWUV image 506 ). In addition, many foreground objects in the LWUV image will tend to appear dark because UV is absorbed by many molecular surfaces. In contrast, since there is less Rayleigh scattering, shadows tend to be enhanced in the NIR band (e.g., as shown in NIR image 504 ). Therefore, subtracting NIR image 504 and LWUV image 506 from each other produces a multispectral image that emphasizes which pixels in the image may be shadows (e.g., thereby classifying such pixels as shadows or non-shadows, optionally with associated probabilities).

[0143] In another example, white clothing against a clean snow background is often not visible or has low contrast in a VIS image, especially in diffuse ambient light with hazy or nonexistent shadows. However, in a LWUV image, white clothing against a clean snow background is often very visible, has relatively high contrast, and can be detected fairly easily using CNN analysis and / or image subtraction of the LWUV image from the VIS image.

[0144] Figure 7 Flowchart 700 shows various operations for providing assisted or automatic pilot navigation using a multi-spectral navigation system according to an embodiment of the present disclosure. In some embodiments, Figure 7 The operation can be implemented as Figure 1-3 More generally, Figure 7 The operations of may be implemented with any combination of software instructions, mechanical elements, and / or electronic hardware (eg, inductors, capacitors, amplifiers, actuators, or other analog and / or digital components).

[0145] It should also be understood that any step, sub-step, sub-process, or block of process 700 may be performed in different Figure 7 For example, in other embodiments, one or more blocks may be omitted from each individual process, or added to each individual process. In addition, block inputs, block outputs, various sensor signals, sensor information, calibration parameters, and / or other operating parameters may be stored in one or more memories before moving to subsequent portions of the corresponding process. Although reference is made to Figure 1-3 Process 700 is described with reference to the systems described in , but process 700 may be performed by other systems that differ from those systems and include different selections of electronic devices, sensors, components, mechanisms, platforms, and / or platform attributes.

[0146] Figure 7 The process 700 may generally correspond to a method for navigating a survey area using the multispectral navigation system 100 .

[0147] At block 702, multispectral image data corresponding to a projected route of a mobile platform is received. For example, controller 112, 312, and / or 372, communication modules 120, 144, and / or 134, user interface 132, and / or other elements of system 100 may be configured to receive multispectral image data from multispectral imaging system 140 and / or imaging module 142 as mobile platform 110 maneuvers along a projected route (e.g., within scene 302).

[0148] In block 704, orientation and position data corresponding to the multispectral image data is received. For example, the system 100 may be configured to receive orientation and / or position data corresponding to the multispectral image data received in block 702 (e.g., from various orientation, position, and / or other motion sensors of the system 100).

[0149] In block 706, maneuvering obstacle information is generated. For example, system 100 may be configured to generate maneuvering obstacle information (e.g., indicating the position, range, and / or other characteristics of object 304 in scene 302) corresponding to the projected path of mobile platform 110 (e.g., within scene 302) based at least in part on the combination of the orientation and / or position data and the multispectral image data received in blocks 702 and 704.

[0150] In block 708, a display view including the manipulated obstacle information is presented. For example, the system 100 may be configured to present a display view including the manipulated obstacle information generated in block 706 in a display of the user interface 132 and / or a display 333 of the multispectral imaging system 140 (e.g., Figure 4-6 of the display view).

[0151] In block 710 , intersection of the projected path with the maneuvering obstacle area is determined. For example, system 100 may be configured to determine that the projected path of mobile platform 110 intersects the location and / or range of at least one object 304 in scene 302 based at least in part on the maneuvering obstacle information generated in block 706 .

[0152] In block 712, the projected route of the mobile platform is adjusted. For example, the system 100 may be configured to adjust the projected route of the mobile platform 110 to avoid one or more maneuvering obstacles (e.g., multiple objects 304 in the scene 302) that intersect the projected route of the mobile platform 110 as determined in block 710. For example, the system 100 may be configured to determine an avoidance route that is configured to avoid all maneuvering obstacles within the scene 302 and generally reach a predetermined destination or traverse the scene 302 according to a predetermined heading or path. In other embodiments, the system 100 may be configured to determine a series of avoidance routes that are configured to avoid individual maneuvering obstacles within the scene 302 as the mobile platform 110 maneuvers through the scene 302.

[0153] By providing such systems and techniques for multispectral navigation, embodiments of the present disclosure greatly improve the operational flexibility and reliability of manned and unmanned mobile platforms, including unmanned sensor platforms. In addition, such systems and techniques can be used to improve the operational safety of users and operators of mobile platforms (including unmanned mobile sensor platforms) beyond that achievable with conventional systems. Thus, embodiments provide multispectral imaging systems and navigation systems with significantly increased operational convenience and performance.

[0154] As mentioned above, another important class of objects that vehicles often encounter in a scene is vegetation. Healthy vegetation strongly reflects NIR radiation, especially in the 800nm ​​band. A camera system with the ability to measure both visible band radiation and NIR radiation can be configured to detect the so-called red edge: the reflectivity associated with the spongy mesophyll tissue in most vegetation rises sharply from 700nm to 800nm.

[0155] One algorithm for identifying foliage is the Normalized Difference Vegetation Index or NDVI. This metric is commonly used for satellite remote sensing. Traditional NDVI is most often defined as the normalized contrast between the NIR band and the visible red band in a multispectral image. For embodiments of the disclosed multispectral imaging system, there is typically no separate visible red band distinct from visible green or blue, so the traditional NVDI can be modified to form mNDVI to define it in terms of the contrast between NIR and the full visible spectrum light:

[0156] mNDVI=(NIR-VIS) / (NIR+VIS)

[0157] Using this definition of mNDVI, a threshold can be identified and selected to classify pixels in a multispectral image as being associated with vegetation in the imaged scene. Typical threshold ranges include an mNDVI of 0.3-0.4.

[0158] Another useful metric is called the Normalized Difference Sky Index, or NDSI. For example, there is often a strong contrast between LWUV and NIR images of the sky because the Rayleigh scattering cross section varies very strongly with wavelength:

[0159] σ 瑞利 ~Wavelength -4

[0160] LWUV light will scatter about 16 times more than NIR light (e.g., the wavelength is twice that of NIR), which makes the sky appear bright in the LWUV band and dark in the NIR band. This NDSI metric can be defined as:

[0161] NDSI = (LWUV-NIR) / (LWUV+NIR)

[0162] Using this definition of NDSI, a threshold can be identified and selected to classify pixels in a multispectral image as being associated with the sky in the imaged scene. Typical thresholds include an NDSI of about 0.2.

[0163] Figure 8-10 Display views 800 , 900 , and 1000 are shown including images generated by the multispectral imaging system 140 for the multispectral navigation system 100 or 300 according to embodiments of the present disclosure. Figure 8-10 Three views of the same scene are shown. Figure 8 , display view 800 shows a relatively high resolution visible spectrum RGB or color image 802 of a major intersection with a highway entrance ramp and various scene elements (e.g., street lights, roads with painted lanes and directional indicators, street signs, sidewalks, fences, vegetation, bridges, and mountains, all under a clear sky). More generally, Figure 8 A full-color visible-light image of a typical ADAS scene is shown.

[0164] exist Fig. 9 , display view 900 shows a multispectral image 902 that includes spectral characteristics of each of the VIS image, the NIR image, and the LWUV image of the same scene depicted in the visible spectrum color image 802, wherein each differentiated spectrum is mapped to R, G, and B channels that are typically visible to the human eye (e.g., NIR image data is mapped to the R channel, VIS data is mapped to the G channel, and LWUV data is mapped to the B channel). Fig.10 A display view 1000 is shown including a processed image or classification map 1002, as identified pixel by pixel using mNDVI and NDSI and appropriate thresholds, which displays vegetation as red, sky as blue, and the rest as black, as described herein. For example, the specific mNDVI and NDSI thresholds used to generate the classification map 1002 are 0.35 and 0.2, respectively.

[0165] Embodiments of the disclosed multispectral imaging system can be configured to distinguish green objects, such as green road signs, from green vegetation. Such a capability makes it easier for an ADAS to recognize and segment out green road signs and merge the information from the imaged text into its general data stream using optical character recognition. In addition, it is easier to see green vehicles against a background of green vegetation using measurements from the NIR and VIS bands. In contrast, if a conventional color camera is used, both green signs and green vehicles on the road or parked on the side of the road are at risk of being lost against a background of green vegetation.

[0166] For example, a LWUV image of a highway may provide minimal contrast between a sign and the vegetation behind it, but in an LWUV image, there is typically a relatively high contrast between the sky and everything else. Therefore, a multispectral image of the same scene will be able to show the sky and vegetation clearly defined from each other. Using the spectral mapping provided herein, the road can be depicted in a yellow-gray color in a multispectral image. Therefore, the road surface (e.g., using a mapped RGB color threshold) can be classified as different from both the vegetation and the sky in the imaged scene because the multispectral appearance of the road surface is very different from the multispectral appearance of the sky and vegetation. Selecting appropriate thresholds, structural morphologies, and / or identifying other classification processing characteristics or techniques may include implementing appropriate CNN training and classification techniques, wherein the CNN is trained to classify various different image features that are important to ADAS.

[0167] Using the techniques described herein, embodiments of the multispectral navigation system described herein are able to: identify vegetation because it is brighter in the NIR band but dimmer in the other two bands; identify clear sky because it is brighter in the LWUV band but dimmer in the other two bands; distinguish between red LED taillights and incandescent taillights with red filters; and define the position of vehicle windows by visible light passing through the vehicle windows. Embodiments are also able to: distinguish between artificial surfaces and natural surfaces; distinguish between green vehicles and vegetation; distinguish between sky-blue vehicles and clear skies; distinguish between white vehicles and cloudy skies; distinguish between icy roads and ice-free roads; and distinguish between wet roads and dry roads.

[0168] In 2019, traffic accidents in the United States killed more than 6,000 pedestrians, the highest annual total on record, and sent more than 100,000 people to hospitals with injuries. As the automotive industry moves toward autonomous vehicles (AVs), the ability to sense, classify, and make split-second maneuvering decisions based on artificial intelligence (AI) while driving becomes increasingly necessary. Advanced driver assistance systems (ADAS) and related AV systems are tasked with quickly becoming smarter and safer. The embodiments described herein provide automakers, suppliers, regulators, automotive testing facilities, commercial vehicle operators, and consumer systems with systems that maximize the safety of drivers, pedestrians, and other vulnerable road users.

[0169] Fig.11 FIG. 1 is a diagram showing a thermal imaging navigation system (eg, Figure 1 FIG. 1100 shows an embodiment of a multi-spectral navigation system 100 in which the functional benefits associated with the navigation system 100 are shown. In particular, Fig.11 1100 illustrates how a thermal imaging-based navigation system can provide relatively reliable feature performance over a relatively large portion of the safety feature phase space identified in the diagram 1100. In addition, Fig.11Diagram 1100 illustrates that increased reliable signature performance can be achieved by combining thermal imaging with visible spectrum imaging and / or other remote sensor systems (e.g., radar), wherein the overlap or fusion of the different signature performances substantially fills the safety signature phase space identified in diagram 1100.

[0170] Fig.12 A diagram of a mobile platform 110 utilizing a thermal imaging navigation system 1200 is shown in accordance with an embodiment of the present disclosure. For example, as described herein, in some embodiments, the multi-spectral navigation system 100 may be implemented as a thermal imaging navigation system, wherein the sensor payload 140 may be implemented as a thermal imaging system 140 including a thermal imaging module 142, the environmental sensor 160 may be implemented as a visible spectrum imaging system 160 including a visible spectrum imaging module / sensor assembly 166, and the thermal imaging navigation system 100 may include a range sensor system 127, which may be implemented as a radar or other type of range sensor system. In such embodiments, each of the thermal imaging system 140, the visible spectrum imaging system 160, and the range sensor system 127 may be mounted to the platform 110 to have overlapping fields of view (e.g., overlapping sensor data of the scene 302).

[0171] In particular, Fig.12 The thermal imaging navigation system 1200 may include one or more of the following: a controller 112, a propulsion system 124 (e.g., an electric motor or an internal combustion engine or a hybrid motor coupled to a transmission system and / or a drive train), a braking system 125 (e.g., one or more electromechanically controlled clamping or motion-blocking devices disposed along the drive train of the propulsion system 124, including at or within the wheels of the platform / passenger vehicle 110), a range-finding sensor system 127a (a grille-mounted radar system), 127b (a front bumper-mounted radar or sonar system), 127c (a pillar-mounted radar system), and / or 127d (a rear bumper- or trunk-mounted radar system). or sonar system), thermal imaging system 1240a (roof-mounted “shark fin” or “hat” or radio antenna integrated thermal imaging system), 1240b (pillar-mounted thermal imaging system), 1240c (rear windshield-mounted thermal imaging system) and / or 1240d (front windshield-mounted thermal imaging system) and / or visible spectrum imaging system 1260a (front windshield-mounted visible spectrum imaging system), 1260b (roof or roof rack 1226 mounted visible spectrum imaging system), 1260c (rear windshield-mounted visible spectrum imaging system) and / or 1260d (trunk-mounted visible spectrum imaging system).

[0172] More generally, Fig.12Each mounting point identified in may be used to mount any one or combination of a thermal imaging system, a visible spectrum imaging system, and / or a remote sensor system as described herein. In various embodiments, all sensor data generated by each mounted system may be used to generate a display view presented by a user interface 1232 of the platform 110 (e.g., a dashboard display for a passenger vehicle 110).

[0173] Fig.13A A data flow diagram 1300 of a mobile platform 110 utilizing a thermal imaging navigation system 1200 according to an embodiment of the present disclosure is shown. In particular, the data flow diagram 1300 shows the thermal imaging system 1260 and / or the visible spectrum imaging system providing thermal and / or visible spectrum images to a maneuvering obstacle detector 1340, which may be configured to provide the received images and / or associated maneuvering obstacle information (e.g., a marker image) generated by the maneuvering obstacle detector 1340 to a range estimator 1342. The range estimator 1342 can be configured to determine and provide to the sensor data fusion 1344 a range estimate associated with each detected maneuvering obstacle represented in the received tagged images, and the sensor data fusion 1344 can be configured to fuse or otherwise combine the tagged images and associated range estimates generated by the maneuvering obstacle detector 1340, and / or the ranging sensor data provided by the ranging sensor system 127, for example using the ranging sensor data provided by the ranging sensor system 127 and / or the orientation, position, motion and / or other registration or calibration data provided by the aligner 1318, as shown.

[0174] The sensor data fusion 1344 may be configured to provide the combined sensor data and / or images to a braking planner 1346 (e.g., an automatic emergency braking planner), which may be configured to evaluate the combined sensor data and / or images, including the projected path of the platform 110 (e.g., provided by the aligner 1318 ) and selectively activate the braking system 125 and / or other elements of the propulsion system 124 to avoid collision with any maneuvering obstacles detected by the maneuvering obstacle detector 1340 .

[0175] In an optional embodiment, the maneuvering obstacle detector 1340 can be configured to generate a marker image based on any one or a combination of thermal images, visible spectrum images, and / or ranging sensor data. In addition, the range estimator 1342 can be configured to determine and provide a range estimate associated with each detected maneuvering obstacle based on the marker image provided by the maneuvering obstacle detector 1340 and / or the ranging sensor data provided by the ranging sensor system 127.

[0176] In various embodiments, each of the maneuvering obstacle detector 1340, the range estimator 1342, the sensor data fusion 1344, and / or the braking planner 1346 may be implemented as a separate software program and / or executed as a separate software program by the controller 112. In a particular embodiment, the maneuvering obstacle detector 1340 may be implemented as one or more CNNs configured to generate labeled thermal images, visible light images, and / or hybrid images, such as by one or more of feature extraction, semantic segmentation, object recognition, classification, and / or other similar CNN-based image and / or sensor data processing.

[0177] In one embodiment, the maneuvering obstacle detector 1340 may be configured to apply a CNN trained with thermal images to detect maneuvering obstacles represented in the thermal images provided by the thermal imaging system 1240, and generate a corresponding labeled thermal image and / or an associated map score (e.g., an accuracy likelihood value) for each maneuvering obstacle detected in the thermal image. In a related embodiment, the maneuvering obstacle detector 1340 may be configured to apply a CNN trained with visible spectrum images to detect maneuvering obstacles represented in the visible spectrum images provided by the visible spectrum imaging system 1260, and generate a corresponding labeled visible spectrum image and / or an associated map score for each maneuvering obstacle detected in the visible spectrum image. In such an embodiment, the maneuvering obstacle detector 1340 may be configured to combine the two sets of labeled images according to a logical function, such as identifying maneuvering obstacles having a map score above a spectrum-specific threshold according to one or the other, and / or any generally detected maneuvering obstacles having a combined map score (from each spectrum) above a combined threshold.

[0178] In another embodiment, the maneuvering obstacle detector 1340 may be configured to blend the thermal image with the visible spectrum image before applying the CNN trained on the blended image to detect maneuvering obstacles represented in the blended image and generate a corresponding labeled blended image and / or an associated map score for each maneuvering obstacle detected in the blended image, wherein only maneuvering objects with a map score above a blended image threshold are forwarded as labeled blended images to the range estimator 1342. In other embodiments, the maneuvering obstacle detector 1340 may be configured to blend the thermal image with the visible spectrum image and combine the result with the ranging sensor data provided by the ranging sensor system 127 before applying the CNN trained on the fused sensor data to detect maneuvering obstacles represented in the fused sensor data and generate a corresponding labeled blended or spectrum-specific image and / or an associated map score for each maneuvering obstacle detected in the fused sensor data, wherein only maneuvering objects with a map score above a fused sensor data threshold are forwarded as labeled images to the range estimator 1342. In various embodiments, any of the CNNs described herein may be trained by synthetic manipulation obstacles, wherein computer-generated animals and / or other manipulation obstacles are added to images to train the CNN to appropriately label the associated images.

[0179] Thermal and visible spectrum image blending may be performed based on various metrics that emphasize one or more characteristics of one or the other spectrum. In some embodiments, color characteristics of the visible spectrum image may be modulated based on the overlapping thermal image pixel values. In other embodiments, pixel values ​​of the visible spectrum image may be converted to grayscale prior to blending with the overlapping thermal image pixel values. In other embodiments, thermal image pixel values ​​may be mapped to a particular color palette prior to blending with the overlapping visible spectrum image pixel values ​​(e.g., grayscale or color pixel values).

[0180] In some embodiments, range estimator 1342 may be configured to generate range and / or relative direction estimates based solely on thermal or visible spectrum images, such as by identifying common object features (e.g., port and starboard stern lights) having known average feature displacements and determining range estimates based on the known average feature displacements, the pixel displacements of the identified object features, and one or more displacement calibration parameters (e.g., typically specific to each imaging system). In other embodiments, range estimator 1342 may be configured to generate range and / or relative direction estimates based on one or any combination of thermal images, visible spectrum images, or ranging sensor data provided by ranging sensor system 127. For example, if each imaging system is known and fixed relative to ranging sensor system 127, such range estimates may be performed without registration data provided by register 1318.

[0181] For example, sensor data fusion 1344 may be configured to fuse substantially synchronized sensor data provided by any sensor of systems 100 and / or 1200, and / or may be configured to fuse temporally differentiated data, such as a temporal sequence of sensor data and / or tagged images, to facilitate accurate maneuvering obstacle tracking, as described herein. Braking planner 1346 may be configured to receive all of the individual sensor data, tagged images, range and / or relative direction estimates, and / or fused sensor data and selectively activate braking system 125 and / or other elements of propulsion system 124 to stop or reduce the speed of platform 110 to avoid a potential collision with a maneuvering obstacle detected by maneuvering obstacle detector 1340, as described herein.

[0182] Fig. 13B FIG. 1 is a block diagram of an update system 1302 for a mobile platform 110 utilizing a thermal imaging navigation system 1200 according to an embodiment of the present disclosure. Fig. 13B As can be seen in FIG. 1 , update system 1302 may include various platforms 110, each of which is configured to receive CNN configurations (weights) from update server 1390 via network 1320. Each platform 110 (e.g., a passenger vehicle) may be configured as described with respect to Figure 1 In various embodiments, the communication network 1320 may be implemented according to one or more wired and / or wireless network interfaces, protocols, topologies, and / or methods as described herein, and in some embodiments may include one or more LAN and / or WAN networks, including cellular networks and / or the Internet.

[0183] In some embodiments, a CNN maneuvering obstacle detection failure may be identified (e.g., by a user override of the thermal imaging navigation system 1200) and / or stored locally at the platform 110. In some embodiments, the update server 1390 may be configured to receive a CNN maneuvering obstacle detection failure and associated sensor data, and generate an updated CNN configuration to compensate for such an edge case / failure. For example, the update server 1390 and / or a connected dedicated CNN trainer and / or annotation feedback loop may be configured to adjust the current CNN configuration based at least in part on the CNN maneuvering obstacle detection failure, and the updated CNN configuration may be stored for distribution to the platform 110.

[0184] Update server 1390 may be implemented as a logic device, tablet computer, laptop computer, desktop computer and / or server computer, which may be configured to implement a CNN configuration database that stores and manages CNN configurations associated with different platforms and provides updated CNN configurations to platform 110, for example, upon user request or upon push by a manufacturer or regulatory agency. Fig. 13B 13. Although shown as one element, in various embodiments, network 1320 may include multiple network infrastructures and / or combinations of infrastructures, where, for example, each platform 110 may be configured to access update server 1390 using a substantially different network infrastructure.

[0185] Fig.14A -B shows display views 1400, 1402 including images generated by the thermal imaging system 1240 for the thermal imaging navigation system 1200 according to an embodiment of the present disclosure. For example, Fig.14A Display view 1400 shows a visible spectrum image 1460, a co-registered thermal image 1440 including image labels (e.g., CNN-based maneuvering obstacle image labels 1480 and radar sensor data labels 1482), and a top-down or bird's-eye view of the labeled image as a fused image 1427, which includes CNN-based maneuvering obstacle image labels 1480, radar sensor data labels 1482, and local area image labels associated with the location of platform 110. Fig. 14B Display view 1402 shows a visible spectrum image 1462 with a manipulated obstacle label 1486 (e.g., a person) based on a visible spectrum image CNN, a thermal image 1442 with a manipulated obstacle label 1484 (e.g., the same person) based on a thermal image CNN, and a hybrid image 1444 with a manipulated obstacle label 1488 based on a hybrid image CNN.

[0186] Fig.15 1500 is a flowchart showing various operations for providing assisted or automatic pilot navigation including automatic emergency braking using a thermal imaging navigation system according to an embodiment of the present disclosure. Fig.15 The operation can be implemented as Figure 1-14B More generally, software instructions executed by one or more logic devices or controllers associated with the corresponding electronic devices, sensors, structures, and / or related images or display views depicted in FIG. Fig.15 The operations of may be implemented with any combination of software instructions, mechanical elements, and / or electronic hardware (eg, inductors, capacitors, amplifiers, actuators, or other analog and / or digital components).

[0187] Any step, sub-step, sub-process, or block of process 1500 may be performed in the same manner as Fig.15 The embodiments shown may be performed in a different order or arrangement. For example, in other embodiments, one or more blocks may be omitted from each individual process or one or more blocks may be added to each individual process. In addition, the block inputs, block outputs, various sensor signals, sensor information, calibration parameters, and / or other operating parameters may be stored in one or more memories before moving to the subsequent portion of the corresponding process. Although reference is made to Figure 1-14B Process 1500 is described with reference to the systems described in , but process 1500 may be performed by other systems that differ from those systems and include different selections of electronic devices, sensors, components, mechanisms, platforms, and / or platform attributes.

[0188] Fig.15 The process 1500 may generally be similar to that of using the thermal imaging navigation system 1200 (e.g., Figure 1 An embodiment of the multi-spectral navigation system 100 corresponds to a method of navigating a road and braking to avoid maneuvering obstacles.

[0189] At block 1502, thermal image data corresponding to the projected route of the mobile platform is received. For example, the controller 112, 312, and / or 372, the communication module 120, 144, and / or 134, the user interface 1232 and / or 132, and / or other elements of the system 100 and / or 1200 may be configured to receive thermal image data from the thermal imaging system 1240 and / or the imaging module 142 as the mobile platform 110 maneuvers along the projected route (e.g., within the scene 302).

[0190] At block 1504, sensor data corresponding to the thermal image data is received. For example, the system 1200 may be configured to receive visible spectrum image data, radar data, lidar data, other ranging sensor data, and / or orientation and / or position data (e.g., from various orientation, position, and / or other motion sensors of the system 100) corresponding to the thermal image data received in block 1502. In some embodiments, the system 1200 may be configured to blend thermal and visible spectrum images, for example, before proceeding to block 1506, or otherwise combine thermal and visible spectrum images with received ranging sensor data before proceeding to block 1506, as described herein. In some embodiments, the system 1200 may be configured to adjust the frame rate of any imaging system of the system 1200 based at least in part on the speed of the platform 110.

[0191] At block 1506, maneuvering obstacle information is generated. For example, the system 1200 may be configured to generate maneuvering obstacle information (e.g., indicating the location, range, and / or other characteristics of the object 304 in the scene 302) corresponding to the expected route of the mobile platform 110 based at least in part on the thermal image data received in block 1502. In other embodiments, the system 1200 may be configured to generate maneuvering obstacle information corresponding to the expected route of the mobile platform 110 based at least in part on a combination of sensor data and thermal image data received in blocks 1502 and 1504 as described herein. In various embodiments, such maneuvering obstacle information may include a thermal image, a visible spectrum image, and / or a hybrid image of a marker, as described herein. In some embodiments, such maneuvering obstacle information may include range and / or relative direction estimates, fused sensor data corresponding to a detected maneuvering obstacle, and / or other sensor data or processed sensor data, as described herein.

[0192] At block 1508, a display view including the maneuvering obstacle information is presented. For example, system 1200 may be configured to present a display view including the maneuvering obstacle information generated in block 1506 in a display of user interface 1232 and / or a display of user interface 132 (e.g., Figures 13A-14B Such display views may include, for example, visible spectrum images, thermal spectrum images, hybrid images and / or fused sensor data and / or one or more types of image markers, as described herein.

[0193] At block 1510, an intersection of the projected path with the maneuvering obstacle area is determined. For example, the system 1200 may be configured to determine an intersection of the projected path of the mobile platform 110 with the position and / or range of at least one object 304 in the scene 302 based at least in part on the maneuvering obstacle information generated in block 1506. Each of such determined intersections may be determined by the braking planner 1346 and / or the controller 112, for example, as described herein.

[0194] At block 1512, a braking system of the mobile platform is activated. For example, the system 1200 may be configured to activate the braking system 125 and / or control other elements of the propulsion system 124 of the mobile platform 110 to stop or reduce the motion of the platform 110 to avoid one or more maneuvering obstacles (e.g., multiple objects 304 in the scene 302) that intersect the expected path of the mobile platform 110 as determined in block 1510. For example, the system 1200 may be configured to determine an avoidance route that is configured to avoid all maneuvering obstacles within the scene 302 while braking the platform 110 without losing steering control of the platform 110 and generally according to a predetermined heading or path. In other embodiments, the system 1200 may be configured to determine a series of avoidance routes that are configured to avoid individual maneuvering obstacles within the scene 302 while braking the mobile platform 110. More simply, the system 1200 may be configured to determine a braking force to be applied to the braking system 125 to stop the platform 110 with the least amount of linear travel possible.

[0195] By providing such systems and techniques for navigation based on thermal images, embodiments of the present disclosure greatly improve the operational flexibility and reliability of manned and unmanned mobile platforms, including passenger vehicles. In addition, such systems and techniques can be used to improve the operational safety of users and operators of mobile platforms (including manned passenger vehicles) beyond what can be achieved with conventional systems. Thus, embodiments provide a thermal imaging navigation system with significantly increased operational convenience and performance.

[0196] As described herein, an embodiment provides an ingenious software solution that combines visible spectrum images and / or videos with thermal images and / or videos to create a hybrid combination of the two. In some embodiments, such a hybrid image uses a portion of the thermal image as well as the visible spectrum image to provide features such as color and the ability to read signs and lane markings, even in low light conditions. Examples of such hybrid images are provided herein. The embodiments described herein can be configured to use such hybrid images as base images for AI-based image processing and object detection and tracking methods. Such methods include providing de-warping and correction of two video streams, and then running an object detector (e.g., a CNN-based object detector) and tracker on the combined video. The result is a single CNN, which leads to better overall efficiency and computational performance. In contrast, existing systems currently process each video stream separately, which doubles the computational requirements for hardware.

[0197] The embodiments described herein provide additional performance over conventional techniques because the CNN and tracker can utilize additional car information (e.g., speed, radar, GPS, direction) as part of the data in the AI-based processing capabilities because the weighted values ​​of the mixed video on each side can be determined within the AI ​​stack. In addition, the CNN can use additional information from the scene temperature data provided by the radiometric thermal camera to check whether the average temperature of the classified object is within an acceptable temperature range to help limit false positive detections. For example, the pixels of an object classified as a person by the CNN all have a temperature value associated with the identified person. The average temperature value can be checked against an acceptable range associated with a specific type of target in a specific environment.

[0198] Nighttime driving can be difficult for drivers; drivers may find it difficult to see critical road obstructions and / or other objects, such as vulnerable road users (VRUs) and animals. Embodiments described herein provide additional information to the driver so that the driver can see VRUs and animals with sufficient warning to react, so that accidents can be avoided at night and in other challenging lighting conditions. For example, a blended video with an alert can be presented to the driver, or information from a camera (e.g., blended, as described herein) and a detection mechanism can be used to stop the vehicle using AEB (automatic emergency braking).

[0199] In various embodiments, key information from a detected VRU can be displayed to the driver. For example, an example display view (e.g., Fig.16A -F) Pedestrians and vehicles are highlighted using a thermal camera, but a visible light camera displays the view from the car from the driver's perspective (e.g., including what the driver's eyes can see). When a CNN is used with hybrid video, an embodiment uses combined information from thermal and visible spectrum data and reduces computational cost compared to two separate processing stacks operating independently on visible spectrum video and thermal video.

[0200] False positive detections may occur using either or both thermal and visible spectrum images. Embodiments described herein use two video channels to more accurately detect VRUs, animals, and other objects (such as other vehicles), thereby more reliably determining the presence and type of objects classified by the CNN. Embodiments also determine and use additional information corresponding to the average radiant heat value of the detected objects to determine whether the classified objects are within the expected radiant heat value to reduce and / or eliminate false positives when objects are present and / or classified.

[0201] Certain calibration objects in the scene may also be used as reference temperatures to help ensure that the radiant heat values ​​of non-calibrated objects are valid / calibrated / accurate for other non-calibrated objects (e.g., VRUs) detected. For example, the calibration object may include a specified portion of the hood of a vehicle to which the thermal imaging module is attached - the specified portion of the hood may be within the field of view of the thermal camera, and a temperature sensor may be coupled to the specified portion of the hood and provide an accurate temperature of the specified portion of the hood, which temperature may be used to calibrate the thermal imaging module. Such calibration objects may be marked with IR and / or visible spectrum visible markings (e.g., registration dots, crosses, and / or other graphics) to help identify pixels associated with the calibration object. In other embodiments, for example, such calibration objects may be implemented by stationary structures (signs, placards) placed beside roads or at parking lot exits, or structures maintained at a standard temperature or configured to transmit their temperature to the system 100 (e.g., by IR or visible spectrum bright light text, or via wireless beacons).

[0202] Various embodiments include a thermal camera (e.g., a thermal imaging module) and a visible spectrum camera (e.g., a visible spectrum imaging module) mounted on or otherwise coupled to a vehicle with similar or at least overlapping fields of view. The two videos can then be scaled, de-warped, and / or corrected through a calibration process using a registration target that can be seen by both cameras. Such a calibration process can be configured to align pixels between the two camera systems (pixel registration). The visible light and thermal lenses / lens systems may have different orientations, and the calibration process can compensate for the radial and tangential orientations of each lens so that the images from the two cameras are substantially aligned and can be superimposed on each other so that the imaged objects overlap in each video.

[0203] In some embodiments, such a registration calibration can be adjusted in real time using objects in the scene with detectable and clearly defined / sharp edges so that the alignment can be maintained or corrected even after the initial calibration process is performed. The aligned image can then be displayed to the driver or used by a vehicle with ADAS functionality, as described herein. For example, a combined image (e.g., a display view and / or combined image data) can include a variable contribution from 100% thermal and 0% visible light to 100% visible light and 0% thermal or any percentage in between. Such a variation can be selected based on ambient lighting, weather, and / or other environmental conditions in which one of the spectra provides more reliable image data for a particular application (e.g., object detection, classification, image registration, and / or other image processing, as described herein).

[0204] In various embodiments, the registered multispectral video (thermal and visible light) can be displayed to a user and / or input into an object detection algorithm, particularly a CNN-based object detector. As described herein, the hybrid video provides superior object detection / perception compared to visible light video only or thermal video only. Such CNN embodiments can also implement a tracker that maintains target lock between image frames and increases the reliability of object detection over time. Compared to conventional image processing for each sensor type, such CNN embodiments are relatively very efficient in terms of computational cost because only one network is used. The result is lower cost due to computational hardware and more reliable detection / classification due to multiple simultaneous spectra (e.g., thermal and visible light).

[0205] Radiometric heat values ​​per pixel (calibrated to absolute temperature) can also be implemented using a thermal imaging module as described herein. When a classified object (e.g., a VRU) is detected, such per-pixel data can be used and the average of the pixels associated with the VRU combined and a relative temperature value obtained. A threshold range can be used to assess the object temperature compared to the expected value for a specific scene / ambient temperature. Such processing adds confidence to the CNN capabilities and helps reduce or eliminate false positives, which is particularly useful for reliable and comfortable AEB applications.

[0206] In various embodiments, a display view including a mixed spectrum image can be generated for the driver. In some embodiments, the mixed video can be configured to emphasize street signs and colors (from the visible light camera), while in other embodiments, the VRU is emphasized with heat. Other display views are primarily visible spectrum images, where detected objects are displayed with heat. In other embodiments, a mixed thermal and visible light video with CNN detection can be displayed to the driver in a configurable manner so that various driver scenarios are considered and optimized. For example, when driving at night in complete darkness, the thermal imaging module can provide most of the video stream. In well-lit situations, the visible spectrum image module can provide most of the transparency. In other embodiments, thermal detection will be superimposed on the visible spectrum video, especially in situations where oncoming car headlights may glare and wash out the visible spectrum image.

[0207] Fig.16A -B shows display views 1600-1605 including images generated by the thermal imaging system 1240 for the thermal imaging navigation system 1200 according to an embodiment of the present disclosure. For example, Fig.16ADisplay view 1600 of FIG. 1 shows a hybrid image 1610 with hybrid image CNN-based labels for maneuvering obstacles 1620 (oncoming traffic), 1622 (parked vehicles along the lane), and 1630 (pedestrians walking through traffic in a crosswalk). Each maneuvering obstacle label in display view 1600 includes a marker box around the thermal image data, indicating the corresponding detected object / maneuvering obstacle. Fig.16A In the illustrated embodiment, hybrid image 1610 is primarily a visible spectrum image combined with thermal images associated with maneuvering obstacle tags 1620 , 1622 , and 1630 .

[0208] Fig. 16B The display view 1601 of FIG. 1 shows a mixed image 1610 with a manipulated obstacle label 1620 (oncoming traffic) based on the mixed image CNN, which is similar to Fig.16A 1600 , but presents a different environment in which visible spectrum glare from oncoming headlights would otherwise obscure oncoming traffic if that portion of the blended image 1610 were not overlaid or otherwise blended with the thermal image associated with the maneuvering obstacle tag 1620 .

[0209] Fig. 16C The display view 1602 of FIG. 1 shows a mixed image 1610 with a manipulated obstacle label 1632 (oncoming cyclist) based on the mixed image CNN, which is similar to Fig.16A 1600 , but presents a different context in which the visible spectrum image would not otherwise display the oncoming bicyclist if that portion of blended image 1610 were not overlaid or otherwise blended with the thermal image associated with maneuvering obstacle label 1632 .

[0210] Fig.16D Display view 1603 of 1603 shows a hybrid image 1612 that is primarily a thermal image combined with a visible spectrum image associated with various lighting objects, such as taillights 1640, intersection signals 1642, and roadside reflectors 1644 (e.g., all shown as red in hybrid image 1612). Fig.16D In the illustrated embodiment, hybrid image 1612 includes detail sufficient to identify and detect pedestrian 1650 and lane marking 1660 , even in relatively low light conditions (eg, where a visible spectrum image may not include similar detail).

[0211] Fig.16EDisplay view 1604 shows a hybrid image 1612 that is primarily a thermal image combined with visible spectrum images associated with various lighting objects, such as crosswalk sign 1646, lane change sign 1648, corner lights 1670 (with associated holiday lighting), and street lights 1672 (e.g., all displayed in visible spectrum colors - white, yellow, orange - in hybrid image 1612). Fig.16E In the illustrated embodiment, hybrid image 1612 includes detail sufficient to identify and detect lane marking 1660 even in relatively low light conditions (eg, where a visible spectrum image may not include similar detail or may be obscured by glare from headlights of oncoming traffic).

[0212] Fig.16F Display view 1605 shows a hybrid image 1612 that is primarily a thermal image combined with visible spectrum images associated with various lighting objects, such as taillights / sidelights 1640 (e.g., displayed in a visible spectrum color - red - in hybrid image 1612). Fig.16F In the illustrated embodiment, the hybrid image 1612 includes hybrid image CNN-based maneuvering obstacle labels 1624 (relatively small traffic moving in the same direction along the road) and 1626 (relatively large traffic moving in the same direction along the lane). Each maneuvering obstacle label in the display view 1600 includes a marker box around the hybrid image data, indicating the corresponding detected object / maneuvering obstacle and its associated lighting characteristics (e.g., taillights / sidelights 1640).

[0213] By providing systems and techniques including CNN-based image processing of combined or multispectral images, embodiments of the present disclosure greatly improve the operational flexibility and reliability of manned and unmanned mobile platforms, including passenger vehicles. In addition, such systems and techniques can be used to improve the operational safety of users and operators of mobile platforms (including manned passenger vehicles) beyond that achievable with conventional systems. Thus, embodiments provide a thermal imaging navigation system with significantly increased operational convenience and performance.

[0214] The air path is a significant contributor to radiation measurement errors. If a user wants to know the exact absolute temperature of an object at a certain distance, they need to know the amount of atmosphere that the infrared radiation travels through, the composition of the atmosphere (water content, etc.), and the air temperature. In ADAS applications, a variety of sensors (LIDAR, Radar, stereo vision) can be used to determine the distance to the object of interest. If the distance is known, the signal lost in the air path between the camera and the object of interest can be accurately estimated. This can significantly improve radiation measurement accuracy.

[0215] LIDAR (laser radar) and RADAR (radar) are time-of-flight ranging sensor systems that can be used to determine depth. Stereo vision camera systems can determine depth by triangulating common objects in two fields of view with a known distance between image planes. Temperature and humidity sensors in the open air can be used to estimate the water content and temperature of the air path. In various embodiments, the temperature and humidity data can be used to estimate the signal from the air path per unit length. One of the above-mentioned ranging sensor systems can be used to determine the distance between the infrared camera and the object of interest being imaged. This distance can be multiplied by the air path per unit length to determine the amount of signal from the air path.

[0216] It is important to understand what the range required for automotive applications is to understand why infrared camera radiometric accuracy at a certain range is needed. Comparing stopping distances in the United States, Germany, and the United Kingdom, a vehicle would need ~160 meters to stop if traveling at 70 mph. In an automotive environment, assuming the vehicle is traveling at a top speed of ~70MPH, targets that are up to ~160 meters away are of greatest concern. Therefore, embodiments should accurately classify people as far as ~160 meters away. Human targets can be identified using the contrast of the target relative to the background, using shape recognition of the target, however, the embodiments herein focus more on using a radiometrically calibrated camera to use the absolute temperature of the target as a means of determining whether the target is human.

[0217] Using an infrared camera to measure the absolute temperature of a target within a certain range is complicated by the air path. The atmosphere acts as a filter, absorbing the energy emitted by the target, and the atmosphere itself also emits light that looks like a signal from the infrared camera's perspective. Fig.17A As shown, the error components in the radiometric image due to atmospheric reflectivity and atmospheric transmission loss can be determined by analysis, as shown in Figure 1700. There are many models to model the effects of atmospheric transmission, and the effects of humidity and ambient air temperature on atmospheric transmission can be determined, such as Fig. 17B 1702. When considering an air ambient temperature range between 20°C and 40°C, and relative humidity between 5% and 100%, the energy reaching the thermal sensor for a target as far as 500 meters is approximately between 0.93 (93%) and 0.53 (53%). Therefore, when considering a stopping distance of ~160 meters, the air transmission can be between 0.95 (95%) and 0.75 (75%).

[0218] Consider a 37°C target (human body temperature w / emissivity of ~1) in high humidity / high ambient conditions, humans can be roughly observed to range from 36.63°C at one meter to as low as 27.75°C at 160 meters - 27.75°C is well below normal human body temperature and cannot be used for human detection. Fig. 17C As shown in the atmospheric transmission relationship diagram 1704, H2O in the atmosphere can absorb a lot of energy and can vary with atmospheric conditions, which is why accurately estimating humidity is important for radiometric accuracy. Water has an absorption band in the 8-13.5 micron band that can cause signal loss in the air path. Relative humidity can be used to estimate the extent to which the presence of water in the atmosphere blocks the signal from the target downrange.

[0219] Embodiments disclosed herein use vehicle-mounted sensors (e.g., Fig.18 Intake air temperature sensor 1810 and humidity sensor 1812, and corresponding measurement data) are used to measure the absolute temperature of targets of interest (e.g., pedestrians) and / or detect and classify such targets at long range (~160 meters) for stopping high-speed vehicles or taking evasive action. Atmospheric humidity and temperature estimation is used to correct radiation measurement errors in infrared cameras due to air path losses. Off-the-shelf sensors are typically already integrated with the vehicle and configured to record ambient relative humidity and air temperature. Air and humidity measured by the ECU for purposes such as fuel mixture optimization can be used for alternative purposes for the conversion of radiation flux to temperature on the infrared camera.

[0220] Additionally, GPS and / or a clock may also be used along with an almanac (priori knowledge of typical weather patterns in the local climate) to roughly estimate expected air humidity and air temperature. In addition to onboard sensors, most vehicles with modern sensors are also connected to the internet. In most locations that support an internet connection, hyperlocal temperature and humidity data are available to vehicles.

[0221] One target depth measurement method used to estimate air path loss to improve radiometric accuracy is LiDAR. LiDAR consists of a laser source and a detector - the "time of flight" of the laser pulse is measured and the distance to the target is inferred from this measurement in time. In a sensor fusion system, the infrared camera and LiDAR have overlapping fields of view - each pixel of infrared coverage will have a distance measured by the LiDAR. Since each infrared pixel in the scene theoretically has a corresponding distance to the target that emitted the infrared energy, the atmospheric transmission can be calculated on a per-pixel basis for the entire scene, allowing accurate temperature measurements to be made for all objects in the scene (including objects over 100 meters away in high humidity / temperature environments).

[0222] Another target depth measurement method used to estimate air path loss to improve radiometric accuracy is RADAR. RADAR consists of a radio wave source and a detector. RADAR is a superset of "time of flight" sensors, including LiDAR. W-band imaging RADAR is common in autonomous vehicle development. In a sensor fusion system, the infrared camera and RADAR will have overlapping fields of view, and each pixel covered by IR will have a distance measured by RADAR. Since each infrared pixel in the scene theoretically has a corresponding distance to the target emitting infrared energy, the atmospheric transmission can be calculated on a per-pixel basis for the entire scene, allowing accurate temperature measurements of all objects in the scene (including objects more than 100 meters away in high humidity / high temperature environments).

[0223] Another target depth measurement method for estimating air path loss to improve radiometric accuracy is to use a stereo camera array. Using well-known methods and two cameras in fixed positions relative to each other, the distance to the target can be estimated using calculations based on equal-area triangles, provided that the image planes have been calibrated so that they are known to be parallel to each other and aligned on the Z axis with a known focal length. In a stereo camera-based depth measurement system, two infrared cameras or two visible light cameras will have overlapping fields of view - each pixel of infrared coverage will have a distance measured by stereo vision. As described above, since theoretically each infrared pixel in the scene has a corresponding distance to a target emitting infrared energy, atmospheric transmission can be calculated on a per-pixel basis wherever there is scene contrast, allowing accurate temperature measurements to be made for most objects in the scene (including objects over 100 meters away in high humidity / high temperature environments).

[0224] Fig.19 Display view 1900 includes a left image 1910 and a right image 1912 of a stereo thermal camera system, and a coarse (noisy) disparity map 1920, where closer objects found in the left and right images are shown as brighter and farther objects have darker points on the disparity map. Such a disparity map can then be used to determine the range of various objects in the imaged scene (e.g., the average range if averaged over all pixels of the detected / classified objects).

[0225] Embodiments described herein may be configured to create a thermal depth map using a single thermal camera and reflections from a car hood, using reflectivity in LWIR from the car hood as a source for a second image plane that may be used as a virtual camera.

[0226] In automotive ADAS applications such as AEB, it is important to know the distance to objects in front of the vehicle. LWIR (e.g., thermal IR) is useful for identifying targets in challenging weather conditions. This article discloses a technique for determining the distance to objects identified in a thermal image using a single thermal imaging module sensor by utilizing the reflectivity of the vehicle hood. In roof-mounted applications, such as Fig. 20 In the application presented in system 2000, the car hood and the targets to be identified in front of the vehicle are in the field of view of the thermal camera. By using a single thermal camera and a reflective car hood to determine the distance, the overall system cost can be reduced (no need to install two thermal cameras, or thermal + RADAR, or thermal + visible light).

[0227] In a roof-mounted thermal imaging application, the hood of the vehicle may be within the field of view that includes the overall navigation scene. Depending on its emissivity, the hood itself may present a reflection of the scene in front of the vehicle back to the thermal camera. Since the hood and thermal camera are fixed in space, the image reflected from the hood can be treated as a "virtual camera" and the distance between the "virtual camera" and the real camera can be used to compute a disparity map between the common targets of the two images. For example, the hood image can be used to triangulate distances, just as in a stereo imaging system. The hood is at a known fixed distance from the thermal camera, and it moves in unison with the thermal camera, so the reflections in the hood provide information as if it were a secondary camera in a stereo vision system, which allows one to create a depth map from reflections off the hood of a car.

[0228] exist Fig. 20 In system 2000, a roof-mounted thermal camera is shown as having a vertical field of view. A portion of the field of view of the roof-mounted camera intersects with the vehicle hood. In data collected from a thermal camera mounted in this manner, the hood of the vehicle can act as a mirror in the LWIR spectrum. The mirror has the effect of creating a second image plane at a different point in space, which can be considered a "virtual" thermal camera. Because the reflective vehicle hood and the roof-mounted thermal camera are fixed in space relative to each other, the "virtual" thermal camera / second image plane can also be considered to be fixed, and the real and virtual thermal cameras can be considered a stereo pair for the purpose of depth estimation using disparity maps or other such methods in areas of the real world scene where the real thermal camera and the virtual thermal camera have intersecting fields of view.

[0229] There is a structural dependency on the curvature of the hood, detectability dependency on the emissivity / reflectivity of the hood, and additional structural dependencies with respect to the mounting height of the hood and the pose of the thermal camera on the roof. The main benefit of this approach is that a single camera can be treated as two cameras, with simpler calibration than a traditional stereo pair, and no synchronization requirement since both views of the same scene are captured using a single sensor.

[0230] exist Fig.21 In Figure 2100 of , calculations are provided showing how depth in a stereo pair system is estimated. Objects common to both views will have a vertical offset between the real image plane and the virtual image plane that depends on the distance between the two cameras. In the example figure, due to the hood angle, the mirror axis between the real thermal camera and the virtual thermal camera is approximately halfway between the roof and the hood of the vehicle. It can then be determined that the distance between the real image plane and the virtual image plane is approximately equal to the distance between the roof and the hood, or ~1 meter. In various embodiments, this distance will be precisely measured and known. In the case of a wide angle lens used on the thermal camera, the focal length may be approximately 6.2mm, and due to the use of a mirror, it can be assumed that the virtual thermal camera has the same focal length of 6.2mm.

[0231] To compute the disparity, features of an object of interest (such as a pedestrian limb or head) need to be identified in both image planes using a CNN or some other image processing technique, as described in this paper. Once this is done in both the real and virtual image planes, the difference between the centroids of features identified as common in the vertical direction can be saved as the disparity of the features as seen from the two viewpoints. This disparity, along with the known focal length and distance between the real and virtual cameras, can be used to estimate depth by treating the problem as one of equal-area triangles. Fig.21 In the example shown, an object at 160 meters is imaged by a mirror system including a hood reflection and a 12um camera with a focal length of 6.2mm and an image plane difference of 1mm, and will see a vertical difference of 3.23 pixels between the two image planes. An object at 50 meters will appear as a vertical difference of 10.3 pixels between the two image planes.

[0232] Fig. 22 A flowchart 2200 is shown of various operations for generating a range map using a thermal imaging navigation system in accordance with an embodiment of the present disclosure. More particularly, flowchart 2200 illustrates the process of correcting and estimating target depth. The mirror image of the scene cannot be used as is and will need to be flipped at least once along the mirror axis so that it is oriented in the same orientation as the true image plane. Some warping will occur due to the hood contour and curling towards the hood ends. This warping can be removed and calibration only needs to be done once since the hood shape and the pose of the camera relative to the hood are not expected to change over time.

[0233] After rectifying the images, common features between the real and virtual image planes will be found. The real image plane image can be cropped so that only the stereo region is analyzed before the feature detection step. One approach could be to run a CNN for both the real and virtual images, and the centroid of the bounding box drawn on the detected objects can potentially be used for disparity calculations. Another approach is to use a Harris corner detector. After the common features are mapped, their difference in the vertical direction can be calculated and the depth can be estimated using the equations detailed earlier.

[0234] Fig.23A -C shows a display view including an image generated by a thermal imaging system of a thermal imaging navigation system according to an embodiment of the present disclosure. Fig.23A Display view 2300 shows pedestrians in the corridor, the stereo region of the true image, and the corrected mirror image (no attempt is made here to fix the warping due to the hood contour). Fig. 23B Display view 2302 shows the road signs at the turn, the stereo region of the real image, and the corrected mirror image (no attempt is made here to fix the warping due to the hood contour). Fig.23C Display view 2304 shows the headlights of the car on the opposite side of the road, the stereo region of the real image, and the correct mirror image (no attempt is made to fix the warping due to the hood contour). Each display view may include a mixed thermal and visible spectrum image, such as Fig.23C Visible spectrum colors shown in the lighting object (street lamp).

[0235] In some embodiments, the hood of the vehicle may include a radiometric calibration object, such as a portion of the hood that is marked and / or coupled to a thermal sensor so that the temperature of that portion of the hood can be provided to the system 2000 / 100 and the corresponding thermal camera can be calibrated as described herein.

[0236] Where applicable, the various embodiments provided by the present disclosure may be implemented using hardware, software, or a combination of hardware and software. In addition, where applicable, the various hardware components and / or software components set forth herein may be combined into composite components comprising software, hardware, and / or both, without departing from the spirit of the present disclosure. Where applicable, the various hardware components and / or software components set forth herein may be separated into subcomponents comprising software, hardware, or both, without departing from the spirit of the present disclosure. In addition, where applicable, it is conceivable that software components may be implemented as hardware components, and vice versa.

[0237] Software according to the present disclosure, such as non-transitory instructions, program code and / or data, can be stored on one or more non-transitory machine-readable media. It is also contemplated that the software identified herein can be implemented using one or more general or special-purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the order of the various steps described herein can be changed, combined into composite steps and / or separated into sub-steps to provide the features described herein.

[0238] The above embodiments illustrate but do not limit the present invention. It should also be understood that many modifications and variations are possible based on the principles of the present invention. Therefore, the scope of the present invention is limited only by the appended claims.

Claims

1. A multispectral imaging system, the system comprising: a thermal imaging device configured to be mounted on a vehicle, wherein the thermal imaging device is configured to, when mounted on the vehicle, capture an image of a scene including a portion of a hood of the vehicle, wherein the image comprises a first image portion including at least one object in the scene and a second image portion including at least one object in a reflection of the scene from the portion of the hood of the vehicle, and wherein the thermal imaging device and the portion of the hood of the vehicle are fixed in position relative to each other; and A logical device, the logical device being configured to: communicating with the thermal imaging device; identifying at least one object common to the first image portion and the second image portion; flipping the second image portion along the mirror axis to obtain a mirror image; determining a parallax associated with the at least one object based on an offset between the at least one object in a first image plane associated with the first image portion and the at least one object in a second image plane associated with the mirror image; determining a depth associated with the at least one object based on the parallax associated with the at least one object and a distance between the thermal imaging device and a virtual imaging device associated with the second image plane, wherein a reflectivity associated with a hood of the automobile is a source for the virtual imaging device; determining a path loss associated with electromagnetic radiation emitted by at least one object based on ambient temperature data associated with the scene, ambient humidity data associated with the scene, and a depth associated with the at least one object; and Based on the path loss and the depth, radiometric data associated with a scene is determined, wherein the radiometric data includes a temperature associated with the at least one object.

2. The multispectral imaging system according to claim 1, wherein: The logical device is configured as: At least one object common to the first image portion and the second image portion is identified using a convolutional neural network and / or a corner detector.

3. The multispectral imaging system according to claim 1, wherein: The thermal imaging device is configured to be mounted on the roof of the car, The logic device is further configured to dewarp the second image portion based on the shape of the hood and the pose of the thermal imaging device to obtain a dewarped image, and The logic device is configured to determine the disparity based on the first image portion and the dewarped image.

4. The multispectral imaging system according to claim 1, wherein: The thermal imaging device is further configured to provide thermal image data corresponding to the projected route of the automobile as the automobile is maneuvered along the projected route; and The logic device is further configured to receive the thermal image data corresponding to the projected route.

5. The multispectral imaging system according to claim 4, further comprising: a sensor system coupled to the automobile and configured to provide sensor data associated with the projected route; Wherein the logic device is configured to receive the sensor data corresponding to the thermal image data, wherein the predicted route is based at least in part on a combination of the sensor data and the thermal image data.

6. The multispectral imaging system of claim 5, further comprising a communication device configured to establish a wireless communication link with an update server associated with the automobile, wherein: The logic device is further configured to receive the thermal image data from the thermal imaging device as the automobile maneuvers along the projected route and report information corresponding to the projected route to the update server via the wireless communication link.

7. The multispectral imaging system according to claim 5, wherein: The sensor system includes an orientation sensor, a position sensor, a visible spectrum imaging system and / or a ranging sensor system, and wherein the sensor data includes visible spectrum image data corresponding to the expected route, orientation data associated with the movement of the car, position data associated with the movement of the car and / or ranging sensor data corresponding to the expected route.

8. The multispectral imaging system according to claim 7, wherein: The sensor data includes the visible spectrum image data, and the logic device is further configured to generate a hybrid image based at least in part on the visible spectrum image data and the thermal image data; and / or The sensor data includes the ranging sensor data, and the ranging sensor system includes a grid mounted radar system or a grid mounted lidar system.

9. A multispectral imaging method, the method comprising: capturing, with a thermal imaging device mounted on a vehicle, an image of a scene including a portion of the vehicle, wherein the image comprises a first image portion including at least one object in the scene and a second image portion including at least one object in a reflection of the scene from the portion of a hood of the vehicle, and wherein the thermal imaging device and the portion of the hood of the vehicle are fixed in position relative to each other; identifying at least one object common to the first image portion and the second image portion; flipping the second image portion along the mirror axis to obtain a mirror image; determining a disparity based on an offset between the at least one object in a first image plane associated with the first image portion and the at least one object in a second image plane associated with the mirror image; determining a depth associated with the at least one object based on the parallax associated with the at least one object and a distance between the thermal imaging device and a virtual imaging device associated with the second image plane, wherein a reflectivity associated with a hood of the automobile is a source for the virtual imaging device; determining a path loss associated with electromagnetic radiation emitted by at least one object based on ambient temperature data associated with the scene, ambient humidity data associated with the scene, and a depth associated with the at least one object; and Based on the path loss and the depth, radiometric data associated with a scene is determined, wherein the radiometric data includes a temperature associated with the at least one object.

10. The multispectral imaging method according to claim 9, wherein: The identifying includes using a convolutional neural network and / or a corner detector to identify at least one object common to the first image portion and the second image portion.

11. The multispectral imaging method according to claim 9, wherein: The thermal imaging device is mounted on the roof of the car.

12. The multispectral imaging method of claim 9, further comprising capturing, by the thermal imaging device, thermal image data corresponding to the expected route of the vehicle as the vehicle maneuvers along the expected route.

13. The multispectral imaging method of claim 12, further comprising receiving sensor data corresponding to the thermal image data, wherein: The predicted route is based at least in part on a combination of the sensor data and the thermal image data.

14. The multispectral imaging method according to claim 13, wherein: The sensor data includes visible spectrum image data corresponding to the predicted route, orientation data associated with the movement of the automobile, position data associated with the movement of the automobile, and / or ranging sensor data corresponding to the predicted route.

15. The multispectral imaging method according to claim 12, wherein: The thermal image data is captured while the vehicle is maneuvered along the projected route, wherein the method further comprises: Information corresponding to the predicted route is reported to the update server via the wireless communication link by a communication device configured to establish a wireless communication link with an update server associated with the automobile.

Images