Thermal imaging system and method for navigation

Abstract

This invention provides a thermal imaging and navigation system and related technologies to improve the operation of manned or unmanned mobile platforms, including passenger vehicles. A thermal imaging navigation system includes a thermal imaging system and a logic device configured to communicate with the thermal imaging system. The thermal imaging system includes a thermal imaging module configured to provide thermal image data corresponding to a planned route of the mobile platform. The logic device is configured to receive the thermal image data, receive orientation and / or position data corresponding to the thermal image data, and generate maneuvering obstacle information corresponding to the planned route, at least in part, based on the orientation and / or position data and / or the thermal image data.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 159,444, filed March 10, 2021, entitled “THERMAL IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” the entire contents of which are incorporated herein by reference.

[0003] This application is a continuation-in-part of International Patent Application No. PCT / US2021 / 012554, filed January 7, 2021, entitled “VEHICULAR RADIOMETRIC CALIBRATION SYSTEMS AND METHODS,” which claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 959,602, filed January 10, 2020, entitled “VEHICULAR RADIOMETRIC CALIBRATION SYSTEMS AND METHODS,” the entire contents of both of the foregoing applications are incorporated herein by reference.

[0004] This application is a continuation-in-part of International Patent Application No. PCT / US2020 / 048450, filed August 28, 2020, entitled “MULTISPECTRAL IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” which claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 894,544, filed August 30, 2019, entitled “MULTISPECTRAL IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” the entire contents of both of the foregoing applications are incorporated herein by reference. TECHNICAL FIELD

[0005] The present invention relates generally to multispectral imaging, and more particularly, to systems and methods of multispectral imaging for navigation of mobile platforms. BACKGROUND

[0006] Modern mobile platforms, such as assisted or autonomous piloted manned and unmanned land vehicles and aerial vehicles, including unmanned aerial vehicles (UAVs), remotely operated water vehicles (ROVs), unmanned surface vehicles (USVs), and unmanned ground vehicles (UGVs), any of which can be configured as unmanned sensor platforms, capable of operating over long distances and in all environments (rural, urban, and even underwater). Operation of such systems can include real-time feedback to a pilot and / or wireless transmission between the unmanned platform and a remote base station, which typically includes a display to efficiently communicate telemetry, images, and other sensor data captured by the platform to an operator. The operator can typically monitor the autonomous or assisted navigation of the manned or unmanned mobile platform, and pilot or otherwise control the manned or unmanned mobile platform as necessary, relying only on image feedback or data received from the mobile platform throughout the mission.

[0007] Conventional imaging systems are typically too expensive and bulky, or lack sufficient contrast under relatively common environmental conditions, to be used for reliable and safe autonomous or assisted navigation of vehicles or other mobile platforms. Accordingly, there is a need for compact imaging systems and related technology to provide reliable scene assessment for navigation of mobile platforms. SUMMARY

[0008] The present disclosure provides multispectral navigation systems and related technology to improve operation of manned or unmanned mobile platforms, including assisted or autonomously piloted manned vehicles and unmanned sensor or survey platforms. One or more embodiments of the described multispectral navigation systems can advantageously include a multispectral imaging system including a multispectral imaging module, a communication module configured to establish a wireless communication link with a base station associated with the mobile platform, an orientation and / or position sensor configured to measure an orientation and / or position of the multispectral imaging system and / or coupled mobile platform, a controller for controlling the communication module, the orientation and / or position sensor, and / or operation of the mobile platform, and one or more additional sensors for measuring and providing sensor data corresponding to maneuvering and / or other operation of the mobile platform.

[0009] In various embodiments, such additional sensors can include a remote sensor system configured to capture sensor data of a survey area from which a two-dimensional and / or three-dimensional spatial map of the survey area can be generated. For example, the navigation system can include one or more visible spectrum, infrared, and / or ultraviolet cameras and / or other remote sensor systems coupled to a mobile platform. The mobile platform can generally be a flight platform (e.g., manned aircraft, UAS, and / or other flight platforms), a land platform (e.g., motorized vehicle), or a waterborne platform (e.g., boat or submarine). More generally, the multispectral imaging system for the multispectral navigation system can be implemented as a multispectral autonomous vehicle imaging system (e.g., MAVIS, for various autonomous or automatic piloting mobile platforms or vehicles).

[0010] In one embodiment, a system includes a multispectral imaging system including a multispectral imaging module configured to provide multispectral image data corresponding to a projected route of a mobile platform, and a logic device configured to communicate with the multispectral imaging system. The logic device can be configured to receive the multispectral image data corresponding to the projected route; receive orientation and / or position data corresponding to the multispectral image data; and generate maneuver obstacle information corresponding to the projected route based at least in part on a combination of the orientation and / or position data and the multispectral image data.

[0011] In another embodiment, a method includes receiving multispectral image data from a multispectral imaging system including a multispectral imaging module configured to provide multispectral image data corresponding to a projected route of a mobile platform; receiving orientation and / or position data corresponding to the multispectral image data; and generating maneuver obstacle information corresponding to the projected route based at least in part on a combination of the orientation and / or position data and the multispectral image data.

[0012] In one embodiment, a system includes a thermal imaging system including a thermal imaging module configured to provide thermal image data corresponding to a projected route of a mobile platform, and a logic device configured to communicate with the thermal imaging system. The logic device can be configured to receive the thermal image data corresponding to the projected route; and generate maneuver obstacle information corresponding to the projected route based at least in part on the thermal image data.

[0013] In another embodiment, a method includes receiving thermal image data from a thermal imaging system including a thermal imaging module configured to provide thermal image data corresponding to a projected route of a mobile platform; and generating maneuver obstacle information corresponding to the projected route based at least in part on the thermal image data.

[0014] The scope of the application is defined by the claims, which are incorporated in this section by reference. The embodiments of the application will be more fully understood from the following detailed description, taken in conjunction with the accompanying drawings, in which one or more embodiments are shown by way of illustration. The drawings will be described with respect to the following figures: BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A diagram of a multispectral navigation system is shown in accordance with an embodiment of the present disclosure.

[0016] Figure 2 A diagram of a mobile platform utilizing a multispectral navigation system is shown in accordance with an embodiment of the present disclosure.

[0017] Figure 3 A diagram of a multispectral imaging system for a multispectral navigation system is shown in accordance with an embodiment of the present disclosure.

[0018] Figures 4-6 A display view including images generated by a multispectral imaging system for a multispectral navigation system is shown in accordance with an embodiment of the present disclosure.

[0019] Figure 7 A flowchart of various operations for providing assisted or automated piloted navigation using a multispectral navigation system is shown in accordance with an embodiment of the present disclosure.

[0020] Figures 8-10 A display view including images generated by a multispectral imaging system for a multispectral navigation system is shown in accordance with an embodiment of the present disclosure.

[0021] Figure 11 A diagram showing functional benefits associated with a thermal imaging navigation system is shown in accordance with an embodiment of the present disclosure.

[0022] Figure 12 A diagram of a mobile platform utilizing a thermal imaging navigation system is shown in accordance with an embodiment of the present disclosure.

[0023] Figure 13A A data flow diagram of a mobile platform utilizing a thermal imaging navigation system is shown in accordance with an embodiment of the present disclosure.

[0024] Figure 13B A block diagram of an update system for a mobile platform utilizing a thermal imaging navigation system is shown in accordance with an embodiment of the present disclosure.

[0025] Figure 14A A display view including images generated by a thermal imaging system for a thermal imaging navigation system is shown in accordance with an embodiment of the present disclosure.

[0026] Figure 15A flowchart showing various operations of providing assistance or automatic piloting navigation including automatic emergency braking using a thermal imaging navigation system is shown in accordance with embodiments of the present disclosure.

[0027] Figure 16A -F shows a display view including an image generated by a thermal imaging system for a thermal imaging navigation system in accordance with embodiments of the present disclosure.

[0028] Embodiments of the invention and their advantages are best understood by referring to the following detailed description. It should be understood that like reference numerals used in different drawings refer to similar components. DETAILED DESCRIPTION

[0029] Multispectral navigation systems and related technology are provided to improve operational flexibility and reliability of mobile platforms, including unmanned mobile sensor platforms. Imaging systems for advanced driver assistance systems (ADAS) typically acquire real-time video images of the immediate environment surrounding a vehicle or mobile platform. Such images can help a pilot (e.g., a human or an autopilot) make decisions about mobile platform navigation, such as braking or evasive maneuvers. Contemporary commercial ADAS cameras produce images with limited spectral content or no spectral content: the images are RGB color or monochrome visible waveband. Such limited spectral content means that under many common environmental conditions (e.g., heading and / or time of day), physical objects can have little or no contrast compared to the sky or other distant background.

[0030] A real-world example can be the case where an ADAS is unable to detect a relatively large featureless ground obstruction (e.g., a closed trailer) relative to the sky / horizon. Such a deficiency can be caused by limited visible differences (e.g., color, brightness) in the visible spectrum. Such deficiencies are not uncommon: a white object under overcast skies will tend to have a brightness very similar to the sky due to the diffuse reflection of the sky “dome.” Embodiments of the multispectral navigation systems described herein are much less likely to fail under similar conditions because it is extremely unlikely that a physical object will emit or reflect light in a combination of spectral wavebands that can include the visible spectrum and spectra beyond the visible, such as the infrared and / or ultraviolet spectra, that is similar to the sky.

[0031] Scenes presented to imaging navigation systems often include regions with very different near-infrared (NIR), visible (VIS), and long-wave UV (LWUV) spectral content. Multispectral imaging systems that are sensitive to spectra beyond the visible can more reliably determine the composition of scene content, e.g., including the ability to reliably distinguish the sky from other vehicles, trees, shrubs, buildings, or roads. Thus, multispectral imaging systems can provide much more nuanced data streams for imaging navigation systems.

[0032] For example, simultaneous measurement in selected wavebands provides a coarse reflectance / emittance spectrum of the surface materials in the scene. Vegetation and other materials reflect sunlight with unique spectral signatures, while the sky emits a different spectrum. The behavior of vehicles and road surfaces in such selected wavebands is also different from their behavior in the visible waveband. There are several advantages to using a small number (e.g., 2, 3, 4, or 5) of relatively wide spectral wavebands for a multispectral camera compared to many closely spaced, relatively narrow spectral channels. Wider spectral wavebands (e.g., typically associated with broadbandpass filters) generally mean that more scene flux will reach the detector and result in better exposure in low light conditions. Wider spectral wavebands also allow for shorter integration times, reducing motion blur, a particular problem associated with image edges captured by forward-looking systems mounted in fast-moving vehicles with relatively high angular rates of motion.

[0033] Such multispectral imaging capture and processing techniques can also be used in aerial vehicles, including unmanned aerial systems. For example, a UAV with autonomous operating capabilities can be implemented with an imaging system that can help a human or autopilot make decisions about what to do in different flight situations, including during takeoff, landing, and evasive maneuvers. A multispectral imaging system with image analysis capabilities can provide processed sensor information about the physical environment, helping a pilot to steer around obstacles, such as a white blank billboard, which can have the same visible waveband radiation as a cloudy sky behind the billboard. Multispectral images can also help determine the location of vegetation, which can help a mobile platform avoid landing in a tree. Multispectral images can also help a mobile platform know the location of the sky in the imaged scene, as the sky typically has distinct multispectral features. Multispectral images can also help a UAS reliably and accurately locate other UAS in the sky, which can be useful for aerial maneuvering of, for example, a swarm of UAS.

[0034] In addition to the above, embodiments can be made relatively compact, reducing size, weight, and power requirements (relative to conventional systems), and thus suitable for deployment in a variety of applications, such as relatively small unmanned land and aerial vehicle systems. Modern manned and unmanned mobile platforms, including unmanned sensor platforms, such as unmanned aerial vehicles (UAVs), remotely operated underwater vehicles (ROVs), unmanned (water) surface vehicles (USVs), and unmanned ground vehicles (UGVs), are capable of operating over long distances and in all environments. Such systems typically rely on portable power sources that can limit their range of travel. Embodiments described herein provide relatively lightweight, compact, and distinctive multispectral navigation systems that generally increase the achievable range of such mobile platforms, including unmanned sensor platforms, which can be particularly useful when attempting to navigate relatively quickly and exhaustively within a surveyed area.

[0035] In various embodiments, the multispectral images and / or other sensor data can be transmitted to a base station in real-time or after the survey, which can be configured to combine the sensor data with a map or floor plan of the surveyed area to present the sensor data in a survey map within the spatial extent of the map or floor plan. Such a map or floor plan can be two-dimensional or three-dimensional. The survey map can be stored at the base station and presented to an operator / user in real-time as a graphical overlay map if the base station includes a display. For example, during operation, such a map can provide insight for navigating a mobile platform or positioning a mobile platform for stationary observation, or if operations are to be conducted in the same area at a future time, such a map can provide information for route planning for future operations.

[0036] Figure 1 A block diagram of a multispectral navigation system 100 is shown in accordance with embodiments of the present disclosure. In some embodiments, the system 100 can be configured to fly over a scene, fly through a structure, or approach a target and use a gimbal system 122 to aim a multispectral imaging system / sensor payload 140 at the scene, structure, or target or portions thereof and / or use a sensor carriage 128 to aim an environmental sensor 160 at the scene, structure, or target or portions thereof, image or sense the scene, structure, or target or portions thereof. The resulting images and / or other sensor data can be processed (e.g., by the sensor payload 140, the platform 110, and / or the base station 130) and displayed to a user using a user interface 132 (e.g., one or more displays, such as a multi-function display (MFD), a portable electronic device, such as a tablet computer, laptop computer, or smart phone, or other appropriate interface) and / or stored in memory for later review and / or analysis. In some embodiments, the system 100 can be configured to use such images and / or other sensor data to control operation of the platform 110, the sensor payload 140, and / or the environmental sensor 160, as described herein, such as to control the gimbal system 122 to aim the sensor payload 140 in a particular direction or to control a propulsion system 124 to move and / or orient the platform 110 to a desired position / orientation in the scene or structure or relative to a target.

[0037] In further embodiments, the system 100 can be configured to use the platform 110 and / or the sensor carriage 128 to position and / or orient the environmental sensor 160 at or relative to a scene, structure, or target or portion thereof. The resulting sensor data can be processed (e.g., by the environmental sensor 160, the platform 110, and / or the base station 130) and displayed to a user through the use of a user interface 132 (e.g., one or more displays, such as a multifunction display (MFD), a portable electronic device, such as a tablet computer, laptop computer, or smartphone, or other appropriate interface) and / or stored in memory for later review and / or analysis. In some embodiments, the system 100 can be configured to use such sensor data to control operation of the platform 110 and / or the environmental sensor 160, as described herein, such as to control the propulsion system 124 to move and / or orient the platform 110 to a desired position / orientation in a scene or structure or relative to a target.

[0038] In Figure 1 In the illustrated embodiment, the multispectral navigation system 100 includes a platform 110, an optional base station 130, and at least one multispectral imaging system 140. The platform 110 can be a mobile platform configured to move or fly and position the multispectral imaging system 140 and / or environmental sensor 160 (e.g., relative to a designated or detected target). As Figure 1 As illustrated, the platform 110 can include one or more of a controller 112, an orientation sensor 114, a gyroscope / accelerometer 116, a global navigation satellite system (GNSS) 118, a communication module 120, a gimbal system 122, a propulsion system 124, a sensor carriage 128, and other modules 126. Operation of the platform 110 can be substantially autonomous and / or partially or fully controlled by the optional base station 130, which can include one or more of a user interface 132, a communication module 134, and other modules 136. In other embodiments, the platform 110 can include one or more elements of the base station 130, such as for various types of manned aircraft, land vehicles, and / or surface or underwater watercraft.

[0039] The sensor payload 140 and / or the environmental sensors 160 can be physically coupled to the platform 110 and configured to capture sensor data (e.g., visible spectrum images, infrared images, ultraviolet images, narrow aperture radar data, analyte sensor data, directional radiation data, and / or other sensor data) of a target location, area, and / or object selected and / or framed by operation of the platform 110 and / or the base station 130. In some embodiments, one or more elements of the system 100 can be implemented in a combination housing or structure that can be coupled to the platform 110 or within the platform 110 and / or held or carried by a user of the system 100.

[0040] The controller 112 can be implemented as any suitable logic device (e.g., a processing device, a microcontroller, a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a memory storage device, a memory reader, or other device or combination of devices) that can be adapted to execute, store, and / or receive suitable instructions, such as software instructions implementing control loops for controlling various operations of the platform 110 and / or other elements of the system 100, for example. Such software instructions can also implement methods for processing infrared images and / or other sensor signals, determining sensor information, providing user feedback (e.g., through the user interface 132), querying devices for operational parameters, selecting operational parameters for devices, or performing any of the various operations described herein (e.g., operations performed by logic devices of various devices of the system 100).

[0041] Further, a non-transitory medium can be provided to store machine-readable instructions for loading into and execution by the controller 112. In these and other embodiments, the controller 112 can implement other components as appropriate, such as volatile memory, non-volatile memory, one or more interfaces, and / or various analog and / or digital components for interfacing with devices of the system 100. For example, the controller 112 can be adapted to store sensor signals, sensor information, parameters for coordinate system transformations, calibration parameters, sets of calibration points, and / or other operational parameters over time and provide such stored data to a user using the user interface 132. In some embodiments, the controller 112 can be integrated with one or more other elements of the platform 110, or distributed as multiple logic devices within the platform 110, the base station 130, and / or the sensor payload 140, for example.

[0042] In some embodiments, controller 112 can be configured to monitor and / or store substantially continuously state and / or sensor data provided by one or more elements of platform 110, sensor payload 140, environmental sensors 160, and / or base station 130, such as, for example, position and / or orientation of platform 110, sensor payload 140, and / or base station 130, and state of communication links established between platform 110, sensor payload 140, environmental sensors 160, and / or base station 130. Such communication links can be configured to establish and then transfer data between elements of system 100 substantially continuously throughout operation of system 100, where such data includes various types of sensor data, control parameters, and / or other data.

[0043] Orientation sensor 114 can be implemented as a compass, a float, an accelerometer, and / or one or more of other devices capable of measuring orientation of platform 110 (e.g., magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations (e.g., gravity and / or magnetic north)), gimbal system 122, imaging system / sensor payload 140, and / or other elements of system 100 and providing such measurements as sensor signals and / or data that can be transferred to various devices of system 100. Gyroscopes / accelerometers 116 can be implemented as one or more electronic sextants, semiconductor devices, integrated chips, accelerometer sensors, accelerometer sensor systems, or other devices capable of measuring angular velocity / acceleration and / or linear acceleration (e.g., direction and magnitude) of platform 110 and / or other elements of system 100 and providing measurements, such as sensor signals and / or data, that can be transferred to other devices of system 100 (e.g., user interface 132, controller 112).

[0044] GNSS 118 can be implemented according to any global navigation satellite system, including GPS, GLONASS, and / or Galileo-based receivers and / or capable of determining absolute and / or relative position of platform 110 (e.g., or elements of platform 110) based on wireless signals received from space-borne and / or ground-based sources (e.g., eLoran and / or other at least partially ground-based systems), and capable of providing measurements, such as sensor signals and / or data (e.g., coordinates), that can be transferred to various devices of system 100. In some embodiments, GNSS 118 can include an altimeter, or can be used to provide absolute altitude, for example.

[0045] The communication module 120 can be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of the system 100. For example, the communication module 120 can be configured to receive flight control signals and / or data from the base station 130 and provide them to the controller 112 and / or the propulsion system 124. In other embodiments, the communication module 120 can be configured to receive images and / or other sensor information (e.g., visible spectrum, infrared, and / or ultraviolet still or video images) from the sensor payload 140 and relay the sensor data to the controller 112 and / or the base station 130. In other embodiments, the communication module 120 can be configured to receive sensor data and / or other sensor information from the environmental sensors 160 and relay the sensor data to the controller 112 and / or the base station 130. In some embodiments, for example, the communication module 120 can be configured to support spread spectrum transmissions, and / or multiple simultaneous communication channels between elements of the system 100. The wireless communication links can include one or more analog and / or digital radio communication links, such as WiFi, and the like, as described herein, and may, for example, be direct communication links established between elements of the system 100, or can be relayed through one or more wireless relay stations configured to receive and retransmit the wireless communications.

[0046] In some embodiments, the communication module 120 can be configured to monitor the status of the communication links established between the platform 110, the sensor payload 140, and / or the base station 130. Such status information can be provided to, for example, the controller 112, or transmitted to other elements of the system 100, for monitoring, storage, or further processing, as described herein. The communication links established by the communication module 120 can be configured to transmit data between elements of the system 100 substantially continuously throughout the operation of the system 100, where such data includes various types of sensor data, control parameters, and / or other data, as described herein.

[0047] In some embodiments, gimbal system 122 can be implemented as an actuated gimbal mount that can be controlled, e.g., by controller 112, to stabilize sensor payload 140 relative to a target or to aim sensor payload 140 according to a desired direction and / or relative position. Accordingly, gimbal system 122 can be configured to provide controller 112 and / or communication module 120 (e.g., gimbal system 122 can include its own orientation sensor 114) with a relative orientation of sensor payload 140 (e.g., relative to an orientation of platform 110). In other embodiments, gimbal system 122 can be implemented as a gravity driven mount (e.g., non-actuated). In various embodiments, gimbal system 122 can be configured to provide power, support wired communication, and / or otherwise facilitate operation of articulated sensor / sensor payload 140. In other embodiments, gimbal system 122 can be configured to couple to a laser designator, rangefinder, and / or other device, e.g., to substantially simultaneously support, stabilize, power, and / or aim multiple devices (e.g., sensor payload 140 and one or more other devices). In alternative embodiments, multispectral imaging system / sensor payload 140 can be fixed to mobile platform 110 such that gimbal system 122 is implemented as a fixed view mount system for sensor payload 140.

[0048] Propulsion system 124 can be implemented as one or more propellers, turbines, or other thrust-based propulsion systems, and / or other types of propulsion systems that can be used to provide power and / or lift to platform 110 and / or to steer platform 110. In some embodiments, propulsion system 124 can include multiple propellers (e.g., three, four, six, eight, or other types of "rotors") that can be controlled (e.g., by controller 112) to provide lift and motion for platform 110 and to provide an orientation for platform 110. In other embodiments, propulsion system 124 can be configured to primarily provide thrust, while other structures of platform 110 provide lift, e.g., in fixed wing embodiments (e.g., where wings provide lift) and / or airship embodiments (e.g., balloons, airships, hybrid airships).

[0049] In various embodiments, propulsion system 124 can be implemented with a portable power source, e.g., a battery and / or an internal combustion engine / generator and a fuel source, e.g., that can be coupled to a drive train and / or drive system of propulsion system 124 and / or platform 110. In other embodiments, propulsion system 124 can be implemented with a braking system 125, e.g., that can be used or controlled to inhibit or eliminate motion of platform 110, e.g., an electromechanically controlled disc or drum based braking system for a ground vehicle (including a passenger vehicle).

[0050] For example, other modules 126 can include other and / or additional sensors, actuators, communication modules / nodes, and / or user interfaces / interface devices, and can be used to provide, for example, additional environmental information related to operation of platform 110. In some embodiments, other modules 126 can include a humidity sensor, a wind and / or water temperature sensor, a barometer, an altimeter, an analyte detection system, a radar system, a proximity sensor, a visible spectrum camera or infrared / thermal camera (with additional mounts), an irradiance detector, and / or other environmental sensors that provide measurements and / or other sensor signals that can be displayed to a user and / or used by other devices of system 100 (e.g., controller 112) to provide operational control of platform 110 and / or system 100.

[0051] In some embodiments, other modules 126 can include one or more actuation and / or articulation devices (e.g., multi-spectral active illuminators, visible and / or IR cameras, radar, sonar, and / or other actuation devices) coupled to platform 110, where each actuation device includes one or more actuators adapted to adjust an orientation of the device relative to platform 110 in response to one or more control signals (e.g., provided by controller 112). In particular, other modules 126 can include a stereo vision system configured to provide image data that can be used to calculate or estimate a position of, for example, platform 110 or to calculate or estimate a relative position of a navigational hazard proximate to platform 110. In various embodiments, controller 130 can be configured to use such proximity and / or position information to help safely navigate platform 110 and / or monitor communication link quality, as described herein.

[0052] Ranging sensor system 127 can be implemented as a radar, sonar, lidar, and / or other ranging sensor system fixed relative to platform 110, imaging system 140, and / or environmental sensors 160, and configured to provide two-dimensional and / or three-dimensional ranging sensor data corresponding to a depth map overlapping and / or substantially centered about an optical axis of imaging system 140 and / or environmental sensors 160.

[0053] In embodiments in which the ranging sensor system 127 is implemented as a radar system, the ranging sensor system 127 can be implemented as one or more electrically and / or mechanically coupled controllers, transmitters, receivers, transceivers, signal processing logic devices, various electrical components, various shaped and sized antenna elements, multi-channel antenna / antenna modules, radar assemblies, assembly mounts, and / or various actuators adapted to adjust the orientation of any components of the ranging sensor system 127, as described herein. For example, in various embodiments, the ranging sensor system 127 can be implemented according to various radar system arrangements that can be used to detect features of, for example, a terrestrial surface or a body of water, and objects thereon or above, and / or their relative velocities (e.g., their Doppler velocities).

[0054] More generally, the ranging sensor system 127 can be configured to transmit one, multiple, or a series of radar beams (e.g., remote sensor beams), receive corresponding radar echoes (e.g., remote sensor echoes), and convert the radar echoes into radar data and / or images (e.g., remote sensor image data), such as one or more intensity maps and / or aggregations of intensity maps indicative of structures, weather phenomena, waves, other moving structures, surface boundaries, and / or other maneuvering obstacles and / or relative positions, orientations, and / or other characteristics of objects that reflect the radar beams back to the ranging sensor system 127. The ranging sensor system 127 can be configured to provide such data and / or images, for example, to a user interface of the platform 110 and / or the base station 130 for display to a user, or to the controller 112 for additional processing, as described herein. Moreover, such data can be used to generate one or more plots corresponding to AIS data, ARPA data, MARPA data, and / or one or more other target tracking and / or identification protocols.

[0055] In some embodiments, the ranging sensor system 127 can be implemented using a compact design in which multiple radar antennas, sensors, and / or associated processing devices are located within a single radar assembly housing that is configured to interface with the rest of the system 100 by providing power, as well as communication to and from the ranging sensor system 127, through a single cable. In some embodiments, the ranging sensor system 127 can include orientation and / or position sensors configured to help provide two- or three-dimensional waypoints, improve radar data and / or image quality, and / or provide highly accurate radar image data, as described herein.

[0056] Conventional radar systems can be both expensive and bulky, and often cannot be used to provide relatively accurate and / or distortion-free radar image data. Embodiments of the ranging sensor system 127 include low-cost single-channel, dual-channel, and / or multi-channel (e.g., synthetic aperture) radar systems that can be configured to produce detailed two- and three-dimensional radar data and / or images. In some embodiments, the ranging sensor system 127, for example, can integrate electronics and transducers into a single waterproof package to reduce size and cost, and can be implemented with a single connection to other devices of the system 100 (e.g., via a Power over Ethernet Ethernet cable, an integrated power cable, and / or other communication and / or power transfer conduits integrated into a single interface cable).

[0057] In various embodiments, the ranging sensor system 127 can be implemented with its own dedicated orientation and / or position sensors (e.g., similar to the orientation sensor 114, the gyroscope / accelerometer 116, and / or the GNSS 118) that can be incorporated within the radar assembly housing to provide three-dimensional orientation and / or position of the radar assembly and / or antenna for use in processing or post-processing radar data for display. The sensor information may, for example, be used to correct for movement of the radar assembly during and / or between beam transmissions to provide improved alignment of corresponding radar echoes / samples, and / or to generate images based on measured orientation and / or position of the radar assembly / antenna. In other embodiments, external orientation and / or position sensors can be used alone or in combination with one or more integrated sensors.

[0058] In embodiments in which the ranging sensor system 127 is implemented with one or more position sensors, the ranging sensor system 127 can be configured to provide a variety of radar data and / or image enhancements. For example, the ranging sensor system 127 can be configured to provide accurate positioning and / or orientation of radar data and / or user-defined waypoints away from the platform 110. Similarly, the ranging sensor system 127 can be configured to provide accurate two- and / or three-dimensional aggregation and / or display of a series of radar data; without orientation data or position data to help determine a track or heading, radar systems typically assume a straight track, which can result in image artifacts and / or other inaccuracies in corresponding radar data and / or images. Furthermore, when implemented with position sensors, the ranging sensor system 127 can be configured to generate accurate and detailed intensity maps of maneuvering obstacles without the use of magnetometers.

[0059] In embodiments in which the ranging sensor system 127 is implemented with orientation and / or position sensors, the ranging sensor system 127 can be configured to store such position / location information along with other sensor information available to the system 100 (radar returns, temperature measurements, textual descriptions, altitude, platform velocity, and / or other sensor and / or control information). In some embodiments, the controller 112 can be configured to generate look-up tables so that a user can select a desired configuration of the ranging sensor system 127 or coordinate other some sensor information for a particular location. Alternatively, an automatic adjustment algorithm can be used to select an optimal configuration based on the sensor information.

[0060] In various embodiments, the ranging sensor system 127 can also be implemented with a co-located imaging system (e.g., imaging system 140), which can include one or more different types of imaging modules that can be incorporated within the radar assembly housing to provide image data substantially contemporaneously with the radar data for use in processing or post-processing the radar data for display. The image data can be used to improve an operator’s understanding of the radar data and increase the overall functionality of the system 100. For example, embodiments can include one or more imaging modules such that the imaging modules rotate with the radar antennas of the ranging sensor system 127 to generate a panoramic view corresponding to a radar plan position indicator (PPI) display view. As described herein, embodiments provide methods of data processing, data fusion, and displaying data and user interaction. In other embodiments, the ranging sensor system 127 can be implemented as a radar system that is implemented and / or configured to operate similarly to the embodiments described in U.S. Patent Application No. 16 / 000,907, filed June 13, 2018, entitled “VEHICLE BASED RADAR UPSAMPLING,” now U.S. Patent No. 10,928,512, which is incorporated by reference herein in its entirety.

[0061] In various embodiments, the sensor cradle 128 can be implemented as a latching mechanism that can be permanently mounted to the platform 110 to provide a mounting location and / or orientation of the environmental sensor 160 relative to a center of gravity of the platform 110, relative to the propulsion system 124, and / or relative to other elements of the platform 110. Further, the sensor cradle 128 can be configured to provide power, support wired communication, and / or otherwise facilitate operation of the environmental sensor 160 as described herein. Accordingly, the sensor cradle 128 can be configured to provide a power, telemetry, and / or other sensor data interface between the platform 110 and the environmental sensor 160. In some embodiments, the gimbal system 122 can be implemented similarly to the sensor cradle 128, and vice versa.

[0062] For example, the sensor carriage 128 can be implemented as an actuated gimbal base that can be controlled by the controller 112, for example, to stabilize the environmental sensor 160 relative to a target or to aim the environmental sensor 160 according to a desired direction and / or relative position. Thus, the sensor carriage 128 can be configured to provide the controller 112 and / or the communication module 120 with a relative orientation of the environmental sensor 160 (e.g., relative to an orientation of the platform 110) (e.g., the sensor carriage 128 can include its own orientation sensor 114). In other embodiments, the sensor carriage 128 can be implemented as a gravity driven base (e.g., non-actuated). In other embodiments, the sensor carriage 128 can be configured to be coupled to a laser pointer, rangefinder, and / or other device, for example, to substantially simultaneously support, stabilize, power, and / or aim multiple devices (e.g., the environmental sensor 160 and one or more other devices).

[0063] The user interface 132 of the base station 130 can be implemented as one or more of a display, touch screen, keyboard, mouse, joystick, knob, steering wheel, yoke, and / or any other device capable of accepting user input and / or providing feedback to a user. In various embodiments, the user interface 132 can be adapted to provide user input to other devices of the system 100 (e.g., the controller 112) (e.g., as a signal and / or sensor information transmitted by the communication module 134 of the base station 130). The user interface 132 can also be implemented with one or more logic devices (e.g., similar to the controller 112) that can be adapted to store and / or execute instructions, such as software instructions, to implement any of the various processes and / or methods described herein. For example, the user interface 132 can be adapted to form a communication link, transmit and / or receive communications (e.g., infrared images and / or other sensor signals, control signals, sensor information, user input, and / or other information), or perform various other processes and / or methods described herein, for example.

[0064] In one embodiment, the user interface 132 can be adapted to display time series of various sensor information and / or other parameters as part of or superimposed on a chart or map that can be referenced to a position and / or orientation of the platform 110 and / or other elements of the system 100. For example, the user interface 132 can be adapted to display time series of positions, headings, and / or orientations of the platform 110 and / or other elements of the system 100 superimposed on a geographic map that can include one or more charts indicating corresponding time series of actuator control signals, sensor information, and / or other sensor and / or control signals.

[0065] In some embodiments, user interface 132 can be adapted to accept user input, for example, including user-defined target destinations, headings, waypoints, routes, and / or orientations of elements of system 100, and generate control signals to cause platform 110 to move in accordance with the target destinations, headings, routes, and / or directions, or to cause sensor payload 140 or environmental sensors 160 to aim accordingly. In other embodiments, for example, user interface 132 can be adapted to accept user input modifying control loop parameters of controller 112.

[0066] In other embodiments, user interface 132 can be adapted to accept user input, for example, including user-defined target poses, orientations, and / or positions for actuation or articulation devices associated with platform 110 (e.g., sensor payload 140 or environmental sensors 160), and generate control signals for adjusting the orientations and / or positions of the actuation devices in accordance with the target poses, orientations, and / or positions. Such control signals can be transmitted to controller 112 (e.g., using communication modules 134 and 120), which can then control platform 110 accordingly.

[0067] Communication module 134 can be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of system 100. For example, communication module 134 can be configured to transmit flight control signals from user interface 132 to communication module 120 or 144. In other embodiments, communication module 134 can be configured to receive sensor data (e.g., visible spectrum, infrared, and / or ultraviolet still or video images, or other sensor data) from sensor payload 140. In some embodiments, communication module 134 can be configured to support, for example, spread spectrum transmission and / or multiple simultaneous communication channels between elements of system 100. In various embodiments, communication module 134 can be configured to monitor the status of communication links established between base station 130, sensor payload 140, and / or platform 110 (e.g., including packet loss using digital communication links to transmit and receive data between elements of system 100), as described herein. Such status information can be provided to user interface 132, for example, or communicated to other elements of system 100 for monitoring, storage, or further processing, as described herein.

[0068] For example, other modules 136 of base station 130 can include other and / or additional sensors, actuators, communication modules / nodes, and / or user interface devices for providing additional environmental information associated with base station 130. In some embodiments, for example, other modules 136 can include a humidity sensor, a wind and / or water temperature sensor, a barometer, a radar system, a visible spectrum camera, an infrared or thermal camera, a GNSS, and / or other environmental sensors that provide measurements and / or other sensor signals that can be displayed to a user and / or used by other devices of system 100 (e.g., controller 112) to provide operational control of platform 110 and / or system 100, or to process sensor data to compensate for environmental conditions, such as approximating water content of the atmosphere at the same altitude and / or located within the same region as platform 110 and / or base station 130. In some embodiments, other modules 136 can include one or more actuation and / or articulation devices (e.g., a multi-spectral active illuminator, a visible and / or IR camera, a radar, a sonar, and / or other actuation devices), where each actuation device includes one or more actuators adapted to adjust an orientation of the device in response to one or more control signals (e.g., provided through user interface 132).

[0069] In embodiments in which imaging system / sensor payload 140 is implemented as an imaging device, imaging system / sensor payload 140 can include an imaging module 142, which can be implemented as a cooled and / or uncooled array of detector elements, such as visible spectrum, infrared, and / or ultraviolet sensitive detector elements, including quantum well infrared photodetector elements, bolometer or microbolometer based detector elements, type-II superlattice based detector elements, and / or other infrared spectrum detector elements (e.g., as well as other detector elements sensitive to other spectrums) that can be arranged in a focal plane array (FPA). In various embodiments, imaging module 142 can be implemented with a complementary metal-oxide-semiconductor (CMOS) based FPA of detector elements that are simultaneously sensitive to some portions of the visible, near-infrared (NIR), and long-wave ultraviolet (LWUV) spectrums. In various embodiments, imaging module 142 can include one or more logic devices (e.g., similar to controller 112) that can be configured to process an image captured by the detector elements of imaging module 142 prior to providing the image to memory 146 or communication module 144. More generally, imaging module 142 can be configured to perform any of the operations or methods described herein, at least in part or in conjunction with controller 112 and / or user interface 132.

[0070] In some embodiments, the sensor payload 140 can for example implement a second or additional imaging module similar to the imaging module 142, which can include detector elements configured to detect other electromagnetic spectra (e.g. visible light, heat, ultraviolet, and / or other electromagnetic spectra or subsets of such spectra). In various embodiments, such an additional imaging module can be calibrated or registered to the imaging module 142 such that the images captured by each imaging module occupy a known and at least partially overlapping field of view of the other imaging module, thereby allowing the images of different spectra to be geometrically registered to one another (e.g. by scaling and / or positioning). In some embodiments, in addition to or as an alternative to relying on a known overlapping field of view, pattern recognition processing can be used to register the images of different spectra to one another.

[0071] The communication module 144 of the sensor payload 140 can be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of the system 100. For example, the communication module 144 can be configured to transmit images from the imaging module 142 to the communication module 120 or 134. In other embodiments, the communication module 144 can be configured to receive control signals from the controller 112 and / or the user interface 132 (e.g. control signals directing the capture, focusing, selective filtering, and / or other operations of the sensor payload 140). In some embodiments, the communication module 144 can be configured to support for example spread spectrum transmission, and / or multiple simultaneous communication channels between elements of the system 100. In various embodiments, the communication module 144 can be configured to monitor the status of a communication link established between the sensor payload 140, the base station 130, and / or the platform 110 (e.g. including packet loss of data transmitted and received between elements of the system 100 using for example a digital communication link), as described herein. Such status information can for example be provided to the imaging module 142, or transmitted to other elements of the system 100 for monitoring, storage, or further processing, as described herein.

[0072] The memory 146 can be implemented as one or more machine-readable media and / or logic devices configured to for example store software instructions, sensor signals, control signals, operational parameters, calibration parameters, infrared images, and / or other data facilitating the operation of the system 100, and provide the same to various elements of the system 100. The memory 146 can also be at least partially implemented as removable memory, for example a secure digital memory card, for example including an interface for such memory.

[0073] Orientation sensors 148 of sensor payload 140 can be implemented similarly to orientation sensors 114 or gyroscopes / accelerometers 116 and / or be capable of measuring an orientation (e.g., a magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations (e.g., gravity and / or magnetic north)) of sensor payload 140, imaging module 142, and / or other elements of sensor payload 140 and providing such measurements as sensor signals that can be communicated to various devices of system 100. Gyroscopes / accelerometers (e.g., angular motion sensors) 150 of sensor payload 140 can be implemented as one or more electronic sextants, semiconductor devices, integrated chips, accelerometer sensors, accelerometer sensor systems, or other devices capable of measuring angular velocity / acceleration (e.g., angular motion) and / or linear acceleration (e.g., direction and magnitude) of sensor payload 140 and / or various elements of sensor payload 140 and providing such measurements as sensor signals that can be communicated to various devices of system 100. GNSS 149 can be implemented similarly to GNSS 118 and / or be capable of measuring a position of sensor payload 140, imaging module 142, and / or other elements of sensor payload 140 and providing such measurements as sensor signals that can be communicated to various devices of system 100.

[0074] For example, other modules 152 of sensor payload 140 can include other and / or additional sensors, actuators, communication modules / nodes, cryogenic or non-cryogenic filters, and / or user interface devices for providing additional environmental information associated with sensor payload 140. In some embodiments, other modules 152 can include a humidity sensor, a wind and / or water temperature sensor, a barometer, a radar system, a visible spectrum camera, an infrared camera, a GNSS, and / or other environmental sensors that provide measurements and / or other sensor signals that can be displayed to a user and / or used by imaging module 142 or other devices (e.g., controller 112) of system 100 to provide operational control of platform 110 and / or system 100, or to process images to compensate for environmental conditions.

[0075] In various embodiments, environmental sensors / sensor payload 160 can be implemented as environmental sensors configured to generate environmental sensor data corresponding to an environment surrounding platform 110. In some embodiments, environmental sensors / sensor payload 160 can be implemented as a camera, a radar system, a lidar system, a sonar system, a GNSS, a barometer, a humidity sensor, a wind and / or water temperature sensor, a visible spectrum camera, an infrared camera, and / or other environmental sensors that provide measurements and / or other sensor signals that can be displayed to a user and / or used by imaging module 142 or other devices (e.g., controller 112) of system 100 to provide operational control of platform 110 and / or system 100, or to process images to compensate for environmental conditions. Figure 1In the illustrated embodiment, the environmental sensor 160 includes a sensor controller 162, a memory 163, a communication module 164, a sensor assembly 166, an orientation and / or position sensor (OPS) 167, a power source 168, and other modules 170. In various embodiments, the sensor assembly 166 can be implemented with sensor elements configured to detect the presence of hazardous analytes, ionizing radiation, emissivity, thermal radiation, radio frequency signals, and / or other environmental conditions proximate to the platform 110 and / or the environmental sensor 160 or in its field of view and / or generate sensor data corresponding to the hazardous analytes, ionizing radiation, emissivity, thermal radiation, radio frequency signals, and / or other environmental conditions proximate to the platform 110 and / or the environmental sensor 160 or in its field of view.

[0076] For example, the sensor controller 162 can be implemented as one or more any suitable logic devices (e.g., processing devices, microcontrollers, processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), memory storage devices, memory readers, or other devices or combinations of devices) that can be adapted to execute, store, and / or receive suitable instructions, such as software instructions implementing control loops for controlling various operations of the environmental sensor 160 and / or other elements of the environmental sensor 160. Such software instructions can also implement methods for processing sensor signals, determining sensor information, providing user feedback (e.g., through the user interface 132), querying devices for operational parameters, selecting operational parameters for devices, or performing any of the various operations described herein.

[0077] Further, a non-transitory medium can be provided to store machine-readable instructions for loading into and execution by the sensor controller 162. In these and other embodiments, the sensor controller 162 can be implemented with other components as appropriate, such as volatile memory, non-volatile memory, one or more interfaces, and / or various analog and / or digital components for interfacing with modules of the environmental sensor 160 and / or devices of the system 100. For example, the sensor controller 162 can store, over time, for example, sensor signals, sensor information, parameters for coordinate system transformations, calibration parameters, sets of calibration points, and / or other operational parameters, and provide such stored data to a user using the user interface 132. In some embodiments, the sensor controller 162 can be integrated with one or more other elements of the environmental sensor 160, or distributed as multiple logic devices within the platform 110, the base station 130, and / or the environmental sensor 160, for example.

[0078] In some embodiments, the sensor controller 162 can be configured to monitor and / or store the status of one or more elements of the sensor assembly 166 of the environmental sensor 160 and / or store sensor data provided thereby substantially continuously, such as, for example, the location and / or orientation of the platform 110, the environmental sensor 160, and / or the base station 130, as well as the status of a communication link established between the platform 110, the environmental sensor 160, and / or the base station 130. Such a communication link can be configured to establish and then transmit data between elements of the system 100 substantially continuously throughout the operation of the system 100, where such data includes various types of sensor data, control parameters, and / or other data.

[0079] The memory 163 can be implemented as one or more machine-readable media and / or logic devices configured to store software instructions, sensor signals, control signals, operational parameters, calibration parameters, sensor data, and / or other data that facilitates the operation of the environmental sensor 160 and / or other elements of the system 100, and provide the same to various elements of the system 100. The memory 163 can also be implemented at least in part as removable memory, such as a secure digital memory card, for example, including an interface for such memory.

[0080] The communication module 164 of the environmental sensor 160 can be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of the system 100. For example, the communication module 164 can be configured to transmit sensor data from the environmental sensor 160 and / or the sensor assembly 166 to the communication module 120 of the platform 110 (e.g., for further transmission to the base station 130) or directly to the communication module 134 of the base station 130. In other embodiments, the communication module 164 can be configured to receive control signals (e.g., control signals directing the operation of the environmental sensor 160) from the controller 112 and / or the user interface 132. In some embodiments, the communication module 164 can be configured to support spread spectrum transmission and / or multiple simultaneous communication channels between elements of the system 100, for example.

[0081] The sensor assembly 166 can be implemented with one or more sensor element supports (e.g., printed circuit boards “PCBs”), connectors, sensor elements, and / or other modules configured to facilitate the operation of the environmental sensor 160. In particular embodiments, the environmental sensor 160 can be implemented as a relatively high-resolution visible spectrum camera (e.g., a HD or 2K or 4K visible spectrum camera), and the sensor assembly 166 can be implemented as a relatively high-resolution FPA of visible spectrum sensitive detector elements configured to generate relatively high-resolution images and / or video of a scene substantially simultaneously imaged by the multi-spectral imaging system 140.

[0082] An orientation and / or position sensor (OPS) 167 of the environmental sensor 160 can be implemented similarly to the orientation sensor 114, the gyroscope / accelerometer 116, the GNSS 118, and / or any other device capable of measuring an orientation and / or position (e.g., a magnitude and direction of roll, pitch, and / or yaw relative to one or more reference directions, such as gravity and / or magnetic north, as well as an absolute or relative position) of the environmental sensor 160, the sensor assembly 166, and / or other elements of the environmental sensor 160 and providing such measurements as sensor signals that can be communicated to various devices of the system 100.

[0083] The power source 168 can be implemented as any power storage device configured to provide sufficient power to each sensor element of the sensor assembly 166 to keep all such sensor elements active and capable of generating sensor data when the environmental sensor 160 is disconnected from an external power source (e.g., provided by the platform 110 and / or the base station 130). In various embodiments, the power source 168 can be implemented by a supercapacitor in order to be relatively lightweight and to facilitate flight of the platform 110 and / or relatively easy handheld operation of the platform 110 (e.g., in the case where the platform 110 is implemented as a handheld sensor platform).

[0084] For example, other modules 170 of the environmental sensor 160 can include other and / or additional sensors, actuators, communication modules / nodes, and / or user interface devices for providing additional environmental information associated with the environmental sensor 160. In some embodiments, as described herein, the other modules 170 can include a humidity sensor, a wind and / or water temperature sensor, a barometer, a GNSS, and / or other environmental sensors that provide measurements and / or other sensor signals that can be displayed to a user and / or used by the sensor controller 162 or other device (e.g., the controller 112) of the system 100 to provide control of the operation of the platform 110 and / or the system 100, or to process sensor data to compensate for environmental conditions.

[0085] In general, each element of the system 100 can be implemented with any appropriate logic device (e.g., a processing device, microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), memory storage device, memory reader, or other device or combination of devices) that can be adapted to execute, store, and / or receive appropriate instructions, such as software instructions implementing a method for, for example, providing sensor data and / or images, or for transmitting and / or receiving communications (e.g., sensor signals, sensor information, and / or control signals) between one or more devices of the system 100.

[0086] Further, one or more non-transitory media can be provided for storing machine-readable instructions for loading into and execution by any logic device(s) implemented with one or more devices of system 100. In these and other embodiments, the logic device(s) can be implemented with other components as appropriate, such as volatile memory, non-volatile memory, and / or one or more interfaces (e.g., an Inter-Integrated Circuit (I2C) interface, a Mobile Industry Processor Interface (MIPI), a Joint Test Action Group (JTAG) interface (e.g., IEEE 1149.1 Standard Test Access Port and Boundary Scan Architecture), and / or other interfaces, such as interfaces for one or more antennas, or interfaces for particular types of sensors).

[0087] Sensor signals, control signals, and other signals can be communicated between elements of system 100 using various wired and / or wireless communication technologies, including, for example, voltage signaling, Ethernet, WiFi, Bluetooth, Zigbee, Xbee, Micronet, or other media and / or short-range wired and / or wireless networking protocols and / or implementations. In such embodiments, each element of system 100 can include one or more modules supporting wired, wireless, and / or a combination of wired and wireless communication technologies. In some embodiments, various elements or portions of elements of system 100 can be integrated with one another, for example, or can be integrated onto a single printed circuit board (PCB) to reduce system complexity, manufacturing costs, power requirements, coordinate frame errors, and / or timing errors between various sensor measurements.

[0088] For example, each element of system 100 can include one or more batteries, capacitors, or other power storage devices, and can include one or more solar cell modules or other power generation devices. In some embodiments, one or more of the devices can be powered using one or more power lines from a power source of platform 110. Such power lines can also be used to support one or more communication technologies between elements of system 100.

[0089] Figure 2 FIGS. 1 1A and 1 IB show a mobile platform 110A and 110B of a multi-spectral navigation system 200 including embodiments of environmental sensors 160 and associated sensor cradles 128, in accordance with embodiments of the present disclosure. In the illustrated embodiment, mobile platform 110A includes a sensor cradle 128A and a sensor cradle 128B, each of which includes a sensor 160A and a sensor 160B, respectively. In the illustrated embodiment, mobile platform 110B includes a sensor cradle 128C and a sensor cradle 128D, each of which includes a sensor 160C and a sensor 160D, respectively. Figure 2In the illustrated embodiment, the multispectral navigation system 200 includes a base station 130, an optional secondary pilot station 230, a mobile platform 110A having an articulated imaging system / sensor payload 140, a gimbal system 122, an environmental sensor 160, and a sensor carriage 128, and a mobile platform 110B having an articulated imaging system / sensor payload 140, a gimbal system 122, an environmental sensor 160, and a sensor carriage 128, wherein the base station 130 and / or the optional secondary pilot station 230 can be configured to control the motion, position, orientation, and / or general operation of the platforms 110A, 110B, the sensor payload 140, and / or the environmental sensor 160.

[0090] In various embodiments, the secondary pilot station 230 can be similarly implemented with respect to the base station 130, e.g., including similar elements and / or being capable of similar functionality. In some embodiments, the secondary pilot station 230 can include multiple displays to facilitate operation of the various imaging and / or sensor payloads of the environmental sensor 160 and / or the mobile platforms 110A-B, generally separate from the piloting mobile platforms 110A-B, and to facilitate substantially real-time analysis, visualization, and communication of sensor data and corresponding instructions, e.g., to first responders in contact with a secondary pilot or user of the system 200. For example, both the base station 130 and the secondary pilot station 230 can be configured to present any of the display views described herein.

[0091] As described herein, embodiments of the multispectral navigation system 100 can be implemented with relatively compact, lightweight, low-power multispectral imaging systems (e.g., multispectral imaging system 140) that can be used to assist in operating a mobile platform, e.g., by aiding in navigation, where processed images and / or display views are provided to an operator to assist the operator in piloting the mobile platform, or by automated piloting navigation, where such images are used to automatically pilot the mobile platform according to a desired route, destination, or other operational parameters.

[0092] In some embodiments, a multispectral imaging system can include an imaging module implemented by a CMOS-based FPA that is formed, fabricated, assembled, and / or otherwise configured to have sensitivity in the IR, VIS, and UV spectra / bands. Such an imaging module can include a Bayer filter configured to generate a mosaic or pattern of IR, VIS, and UV pixels in the FPA, such that each image captured by the multispectral imaging module includes IR, VIS, and UV information about each scene imaged by the multispectral imaging module. In particular embodiments, such an FPA can be sensitive to some portions of the NIR, VIS, and LWUV spectra, including at least 400-750 nm (VIS), 750-1100 nm (NIR), and 330-400 nm (LWUV), and the Bayer filter can be configured to selectively pass such bands according to a particular mosaic or pattern.

[0093] In some embodiments, for example, such a Bayer filter can be deposited directly on the FPA, and / or can form a checkerboard-like mosaic similar to that used for RGB VIS imaging. Depending on the particular mosaic selected for the Bayer filter and / or depending on a particular demosaicing algorithm (e.g., one or more of interpolation-based, spectral-dependent, spatial-dependent, and / or other demosaicing techniques), any one of the VIS, NIR, or LWUV spectra can be emphasized or de-emphasized relative to the other two spectra in the resulting image.

[0094] In some embodiments, a particular demosaicing algorithm can be selected, for example, based on one or more environmental conditions associated with an imaged scene or object, or one or more environmental conditions associated with an operating state, position, or orientation of the mobile platform 110 and / or imaging module 142 of the multispectral imaging system 140. For example, a particular demosaicing algorithm can be configured to de-emphasize the contribution of VIS, NIR, and / or LWUV pairs of captured images to polarization aligned with the horizon based on the time of day (e.g., the position of the sun in the sky), the position of the mobile platform 110, and / or the orientation of the mobile platform 110 and / or multispectral imaging system 140, to reduce image artifacts (e.g., pixel saturation artifacts) caused by relatively strong reflections of ambient light from horizon-aligned surfaces. In another example, a particular demosaicing algorithm can be configured to emphasize the contribution of UV to captured images based on the time of day (e.g., the level of natural ambient light provided by the sun), the position of the mobile platform 110 (e.g., the geographic position and altitude of the mobile platform 140 placed within a topographic map of the horizon), and / or the orientation of the mobile platform 110 and / or multispectral imaging system 140, when the UV contribution is expected to be relatively low (e.g., when the sun is below the horizon).

[0095] For example, such a Bayer filter can be implemented as a mosaic of single bandpass filters (e.g., each pixel receives only one of the IR, VIS, UV bands that pass through), or can be implemented as a mosaic of notch wideband transmission filters (e.g., each pixel receives all bands except one of the IR, VIS, UV bands that is notched / fi ltered). In embodiments where the Bayer filter is implemented as a mosaic of notch wideband transmission filters, a selected primary band can be synthesized from a linear combination of two or more pixels that receive a differentiated spectrum (e.g., associated with the spectrally differentiated notch wideband transmission filters). In various embodiments, such synthesis can be implemented within / included as part of a demosaicing algorithm, as described herein. Such techniques can provide increased signal-to-noise characteristics relative to filtering implemented by a mosaic of single bandpass filters.

[0096] In various embodiments, the multispectral imaging system 140 is capable of operating at reasonably high frame rates such that the resulting image stream is sufficient for simultaneous use in navigation of a mobile platform (e.g., where an operator or autopilot frequently needs to make time-critical maneuver decisions). For example, embodiments are capable of operating (e.g., capturing and processing images) at frame rates approaching about 100 frames / second or higher.

[0097] In particular embodiments, the multispectral imaging system 140 is capable of cycling the integration time associated with the FPAs of the imaging modules 142 over two or more preset values such that the multispectral imaging system 140 can produce high dynamic range (HDR) images in all imaging bands. For example, such a HDR mode can be used to provide a midwell exposure value under a variety of lighting conditions, and in some embodiments, the integration time can be determined and / or adjusted by the multispectral imaging system 140 based on ambient light levels (e.g., in one or more spectral bands), contrast levels in previously captured multispectral images, and / or other environmental sensor data and / or derived or processed sensor data and / or images. This is particularly important, for example, where the solar spectrum has very different scene luminances in the IR, VIS, and UV, and the scene spectrum varies with the diurnal cycle.

[0098] A midwell exposure value involves an exposure value that is sufficient to capture a scene in the IR, VIS, and UV bands, but not so high that it saturates the FPA (e.g., the FPA of the imaging module 142) in the IR, VIS, and UV bands. Figure 3The integration capacitors of the sensor elements in the CMOS-based embodiment of the FPA 374 of the multispectral imaging system 140 are allowed to charge to about half their capacity before being read out (e.g., discharged) across the PCB 375 (both of which are of the imaging module 142) by the module controller 372 for an exposure event. For example, the readout frequency and / or the exposure time (e.g., controlled by mechanical, electromechanical, and / or electronic (e.g., LCD) embodiments of the shutter 349) can be adjusted based at least in part on the average scene radiance (e.g., of a particular waveband or across multiple wavebands selected by the filter system 376) so that most of the sensor elements of the FPA 374 associated with one or more wavebands captured by the FPA 374 operate at approximately their mid-well exposure values. Operating at such mid-well exposure values results in image data captured within the most linear portion of the dynamic range of the FPA 374 (e.g., providing a substantially linear response to the photons intercepted by the sensor elements of the FPA 374), which helps to avoid image noise associated with, for example, low well charge levels. By cycling through different exposure times (e.g., integration times), embodiments are able to achieve mid-well operating performance for each waveband captured by the multispectral imaging system 140.

[0099] For example, the VIS radiance in a scene will typically be higher than the NIR or LWUV radiance in the scene. An exposure / integration time of 5 milliseconds can provide a mid-well exposure level for the NIR and LWUV sensor elements (e.g., selected by the filter system 376), but will overexpose the VIS sensor elements of the FPA 374. The multispectral imaging system 140 can be configured to capture a first image according to a first 5-millisecond exposure time, then capture a second image according to a second, shorter exposure time, and then combine the NIR and LWUV components of the first image with the VIS components of the second image to generate an HDR image (e.g., so long as the first and second images are captured one after the other and / or when the multispectral imaging system 140 or at least the FOV 345 is substantially stationary relative to the scene 302).

[0100] In various embodiments, the multispectral imaging system 140 can be equipped with a lens system that is achromatic over the spectral bands captured by the imaging module 142. Such a lens system can be implemented with a focal length selected to provide a relatively wide field of view (FOV) sufficient for the navigation system and UAS imaging FOV requirements (e.g., mission, specification, and / or regulatory requirements).

[0101] In some embodiments, the multispectral imaging system 140 can be configured to process the captured images according to multispectral image analysis and / or algorithms (e.g., on-board or after transmission to other processing elements of the system 100) configured to classify scene pixels according to the likelihood that the scene pixels are part of a particular object class. For example, a clear sky has a unique spectrum that is darker in NIR and brighter in UV. Vehicles, even those painted white, tend to have the opposite spectrum: bright in the NIR band and dark in the UV band. Both scene elements can therefore be reliably classified based at least in part on their spectral characteristics. In various embodiments, such multispectral image analysis and / or algorithms can be performed, for example, by a convolutional neural network (CNN) implemented within the multispectral imaging system 140 or within one or more controllers associated with the multispectral navigation system 100.

[0102] In particular embodiments, the image data provided by the multispectral imaging system 140 and / or imaging module 142 can be encoded using two bytes per pixel, with 12 bits encoding the image data (e.g., intensity) and the remaining four bits encoding information about the classification probability associated with the pixel, such as a 95% probability that the pixel is or is not sky. Such data can then be used by the multispectral navigation system 100 to make steering (e.g., braking and turning) decisions in substantially real-time (e.g., at time instances of 100+ frames / second).

[0103] Figure 3 A diagram of a multispectral imaging system 140 for a multispectral navigation system 100 and / or 300 according to embodiments of the present disclosure is shown. In Figure 3In particular embodiments, the multispectral imaging system 140 includes an imaging module 142 that includes a multispectral FPA 374 that receives light 308 from the scene 302 along the optical axis 344 and according to the FOV 345 through a filter system 376, a lens system 378, and / or an optional shutter 349. In various embodiments, the imaging module 142 can include a printed circuit board (PCB) 375 or similar structure configured to support the FPA 374 and couple the FPA 374 and / or other elements of the imaging module 142 to a module controller 372 of the imaging module 142. As described herein, the filter system 376 can be implemented as a Bayer filter having a mosaic configured to provide select differentiated spectra (e.g., some portions of VIS, IR, and UV spectra) to pixels of the FPA 374 in some embodiments. Also as described herein, the lens system 378 can be achromatic, e.g., with respect to the differentiated spectra provided to the pixels of the FPA 374, and configured to provide the FOV 345. In some embodiments, the lens system 378 can be actuated to adjust the FOV 345, a zoom level of the multispectral imaging system 140, and / or a focus of light delivered to the FPA 374. In other embodiments, the lens system 378 can be a fixed lens system.

[0104] The module controller 372 can be implemented as any appropriate processing device (e.g., a microcontroller, a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other logic device) that can be used by the imaging module 142 and / or the multispectral imaging system 140 to execute appropriate instructions, e.g., software instructions and / or signal processing operations, e.g., for capturing multispectral images of the scene 302 using the FPA 374, the filter system 376, the lens system 378, and / or the shutter 349, demosaicking raw pixel data associated with such multispectral images, and / or classifying pixels in such images associated with the object 304 and / or the background 306 within the scene 302 (e.g., using a CNN implemented within the module controller 372). In addition, the module controller 372 can be implemented with various signal processing devices, e.g., analog-to-digital converters (ADCs), trigger inputs, timing circuitry, and other signal or sensor processing devices described herein.

[0105] In various embodiments, FPA 374 can be implemented by a two-dimensional planar array of similarly fabricated / dimensioned pixel structures, each configured to be sensitive across the full spectral band of imaging module 142. In other embodiments, FPA 374 can be implemented by an array of structurally differentiated sub-arrays of pixel structures, e.g., where each sub-array is sensitive to a differentiated subset of the full spectral band of imaging module 142, and / or can be non-planar (e.g., concave with respect to optical axis 344), three-dimensional (e.g., multi-layered), and / or can include dimensionally differentiated pixels (e.g., having greater surface area as distance to optical axis 344 increases).

[0106] For example, filter system 376 can be implemented as a static Bayer filter structure deposited or otherwise attached to the active surface of FPA 374, or can be implemented as an adjustable or controllable Bayer filter structure or other type of filter structure configured to provide spatially and spectrally differentiated illumination of the pixels or FPA portions of FPA 374. In a particular example, such a Bayer filter can be implemented with two VIS pixels for each NIR and LWUV pixel (e.g., similar to some Bayer filter patterns used for color VIS cameras having green 2 pixels and red and blue 1 pixel each). Such a filter can be implemented as a multi-layer dielectric interference bandpass filter. More generally, filter system 376 can be configured to provide spatially and spectrally differentiated illumination of FPA 374 according to two or more, or three or more, different spectra, where each spectrum can be fully differentiated or can partially overlap with an adjacent differentiated spectrum. In one embodiment, the characteristics of filter system 376 can include a NIR band of 780-1000 nm, a LWUV band of 370 nm with a 60 nm full width at half maximum (FWHM), and a general VIS band.

[0107] In other embodiments, the filter system 376 can be implemented at least in part by a multi-element optical element filter array (e.g., a multi-element filter or filter array). Such special interference filters are typically configured according to a relatively complex modified transmission curve that is designed to perform an optical computing operation, such as an operation similar to a dot product between a scaled regression vector and a spectral response of a scene. For example, the regression vector is typically a result of the filter design and can be optimized for a particular spectrum of interest. In one embodiment, such a filter can include an array of 3 or possibly more different multi-element optical elements (MOEs) with a selected pattern. An array of MOEs designed to specifically detect scene spectra of vegetation, clear sky, overcast, road surface, and vehicle, etc. can provide advantages over a simple 3-band approach. Such MOE filters can be constructed from relatively few layers compared to conventional bandpass filters, so they are typically physically thinner than bandpass filters, which makes them an attractive choice for filter arrays in cases where the FPA pixels can have dimensions comparable to the thickness of the filter layer stack. Such MOE filters can also tend to have better overall scene throughput (e.g., when they are constructed from fewer layers than comparable bandpass filter arrays).

[0108] For example, the lens system 378 can be implemented with one or more lenses each configured to pass light to substantially all of the pixels of the FPA 374, or can be implemented with an array of lenses (e.g., a micro-lens array) each configured to pass light to a sub-group of the pixels of the FPA 374. In general, in embodiments where the FPA is sensitive to NIR, VIS, and LWUV bands, each lens of the lens system 378 can be configured to be color-corrected or achromatic over from 330-1100 nm, as described herein. In some embodiments, the FOV 345 can be non-symmetrical (e.g., to match a corresponding FPA size) and be about 42 by 34 degrees.

[0109] While Figure 3 The illustrated embodiment shows a relatively compact multi-spectral imaging system 140 implemented with a single multi-spectral imaging module 142 capable of providing a single view multi-spectral image of the scene 302, but in other embodiments, for example, the multi-spectral imaging system 140 can be implemented with multiple imaging modules 142 each sensitive to a separate differentiated spectrum and / or each providing a different view of the scene 302, for example according to different optical axes and / or different FOVs.

[0110] For example, PCB 375 may be a conventional printed circuit board and is adapted to provide electrical access (e.g., via various metal traces) to FPA 374 and / or other components of imaging module 142, as well as physical support for FPA 374 and / or other components of imaging module 142. In some embodiments, shutter 349 may be implemented as a mechanical or removable light shield adapted to selectively block one or more wavelengths of light 308. In various embodiments, shutter 349 may be electronically actuated (e.g., opened and / or closed), for example, via module controller 372 and / or imaging system controller 312. For example, shutter 349 may be coupled to / supported by housing 348, and housing 348 may be adapted, for example, to protect system 300 from environmental conditions associated with space or atmospheric flight and / or other outdoor environmental conditions, such as fixed or articulated mounting on a land vehicle. In other embodiments, housing 348 may be adapted for handheld use.

[0111] like Figure 3 As shown, the multispectral imaging system 140 may be implemented with various other components suitable for facilitating the operation of the multispectral imaging system 140, including capturing multispectral images of scene 302, de-mosaicing images of scene 302, detecting characteristics of objects 304 and background 306 of scene 302 (e.g., presence, degree, extent, translucency, visible color, and / or other characteristics) and / or classifying objects 304 and background 306 of scene 302 (e.g., classifying them as sky or non-sky, manipulated or non-manipulated obstacles, moving or non-moving targets, vegetation or non-vegetation, road / land or non-road / land, water or non-water, and / or their probabilities) and / or reporting such sensor data to other elements of system 100 as described herein. In some embodiments, system 300 may report sensor data by aggregating sensor data over time (e.g., multiple frames) to provide duration-based reliability of such characteristics and / or classifications determined by system 300, and then transmitting the sensor data to other elements of system 100. In other embodiments, system 300 may report sensor data by activating LED indicators and / or transmitting alarm or notification signals to components of system 300 or 100 (e.g., alarms, electrical switches, or relays).

[0112] For example, depending on the desired application and / or overall size of the multispectral imaging system 140 and / or imaging module 142, each of the imaging sensor controller 312, memory 146, user interface 332, communication module 144, display 333, and other modules 152, if optionally included in the multispectral imaging system 140, can be coupled to the PCB 375 or housing 348. In other embodiments, any one or group of such components can be external to, for example, the multispectral imaging system 140 and / or implemented in a distributed or grouped manner (e.g., multiple imaging system controllers 312 operate the multispectral imaging system 140, or multiple multispectral imaging systems 140 are operated by a single imaging system controller 312).

[0113] The imaging system controller 312 can be implemented as any appropriate processing device (e.g., a microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other logic device) that the system 300 can use to execute appropriate instructions, such as software instructions and / or signal processing operations, for, for example, capturing multispectral images of the scene 302 using the imaging module 142, demosaicking raw pixel data associated with such multispectral images, classifying pixels and / or elements of the scene 302 in such images (e.g., using a CNN implemented within the imaging system controller 312), and / or reporting such sensor data / information to other elements of the multispectral navigation system 100 or 300. Moreover, as described herein, the imaging system controller 312 can be implemented with various signal processing devices, such as analog-to-digital converters (ADCs), trigger inputs, timing circuitry, and other signal or sensor processing devices.

[0114] In various embodiments, at least some portion or function of the imaging system controller 312 can be part of or implemented with other existing controllers or logic devices of separate systems, such as servers, personal electronic devices (e.g., mobile phones, smart phones, tablet devices, laptop computers, desktop computers), and / or any other devices that can be used to process, report, or act on multispectral images captured by the system 300. In other embodiments, the imaging system controller 312 can be adapted to interface and communicate with various external controllers or logic devices and associated components and / or perform various operations in a distributed manner.

[0115] Generally, the imaging system controller 312 can be adapted to interface and communicate with other components of the system 300 to perform the methods and processes described herein. In one embodiment, the imaging system controller 312 can be adapted to report the multispectral images and / or pixel / object classifications, for example, using the communication module 144, to display 333 and present and / or display such information or alert notifications, or to present and / or display images of a classification map corresponding to the scene 302. In another embodiment, the imaging system controller 312 can be adapted to establish a wired or wireless communication link with, for example, a remote reporting system, and report such sensor information using the communication module 144.

[0116] The memory 146 is generally in communication with at least the imaging system controller 312 and can include one or more memory devices (e.g., memory components) to store information, including image data, calibration data, other types of sensor data, and / or software instructions. Such memory devices can include various types of volatile and non-volatile information storage devices, such as RAM (random access memory), ROM (read only memory), EEPROM (electrically erasable programmable read only memory), flash memory, disk drives, and / or other types of memory. In one embodiment, the memory 146 can include a removable memory device that can be removed from the system 300 and used to transfer stored data to other systems for further processing and review.

[0117] The communication module 144 can be configured to facilitate communication and interface between various components of the system 300 (e.g., between the imaging system controller 312 and the memory 146 and / or the display 333) and / or various external devices (e.g., wireless access points, personal electronic devices, servers, and / or other detectors). For example, components such as the user interface 332 and the display 333 can transmit data to and receive data from the imaging system controller 312 through the communication module 144, which can be adapted to manage wired and / or wireless communication links between various components. Thus, the communication module 144 can support various interfaces, protocols, and standards for local system networks, such as Controller Area Network (CAN) bus, Local Interconnect Network (LIN) bus, Media Oriented Systems Transport (MOST) network, or ISO 11738 (or ISO-Bus) standards.

[0118] In some embodiments, the imaging system controller 312 can be adapted to communicate with remote user interfaces, notification systems, or other detection systems through the communication module 144 to, for example, aggregate reports from multiple systems or sensors and / or implement specific detection and / or notification methods. Thus, the communication module 144 can include wireless communication components (e.g., based on IEEE 802.11 WiFi standards, Bluetooth standards, ZigBee standards, Z-Wave standards, or other wireless communication standards) and / or wired communication components (e.g., based on Ethernet standards, USB standards, RS-232 standards, or other wired communication standards) to support such communication.TM Standard, ZigBee TM Communication module 144 may be configured to connect to a wired network and / or device interface via a wired communication component such as an Ethernet interface. This can be a standard or other suitable short-range wireless communication standard, a wireless broadband component (e.g., based on WiMax technology), a mobile cellular component, a wireless satellite component, or other suitable wireless communication component.

[0119] If present, user interface 332 provides user interaction with multispectral imaging system 140 and may include one or more buttons, indicators (e.g., LEDs), a keyboard, trackball, knobs, joysticks, displays (e.g., liquid crystal displays, touchscreen displays), and / or other types of user interfaces adapted to accept user input and / or provide user feedback. In one embodiment, user interface 332 may include a power button, a vibration motor, LEDs indicating the manipulation of obstacles, and / or speakers providing audible indications of obstacle manipulation (e.g., visible, tactile, and / or audible indicators). In various embodiments, as described herein, user interface 332 may be used to input various system configuration settings, such as integration time parameters, demosaic algorithm selection, and / or other configuration settings. In some embodiments, user interface 332 may be used to view one or more reports, charts, and / or other image data captured by system 300 and / or processed according to the various operations described herein.

[0120] If present, display 333 can be configured to present, indicate, or otherwise convey alarms, notifications, and / or image data and / or other reports on object or pixel classification (e.g., generated by imaging system controller 312). Display 333 can be implemented with an electronic display screen, such as a liquid crystal display (LCD), a cathode ray tube (CRT), or various other types of commonly known video displays and monitors, including touch-sensitive displays. Display 333 can be adapted to present image data, charts, videos, reports, or other information as described herein.

[0121] Other modules 152 may include temperature sensors / probes (e.g., thermocouples, infrared thermometers), LEDs or laser diodes, ambient light sensors, voltage regulators and / or filters, variable voltage sources, and / or other types of devices as described herein that may be used to facilitate the operation of the multispectral imaging system 140. In some embodiments, other modules 152 may include GNSS, accelerometers, compasses, and / or other orientation sensors capable of sensing the position and / or orientation of the multispectral imaging system 140. Other modules 152 may additionally include power modules implemented as batteries, power adapters, charging circuitry, power interfaces, power monitors, and / or other types of power sources that provide mobile power.

[0122] According to embodiments described herein, multi-spectral navigation systems can benefit from various multi-spectral imaging and visualization techniques configured to improve the operational flexibility, reliability, and accuracy of such systems. In particular, embodiments can be configured to provide various display views, e.g., including augmented reality views based on images provided by multi-spectral imaging system 140 and / or other imagers of system 100, allowing users to use and monitor such features and capabilities, and can be implemented according to various processes and / or control loops configured to mitigate the piloting burden, protect the operation of the mobile platform of such systems, and assess potential maneuvering obstacles and evasion options qualitatively and quantitatively faster and more reliably than conventional navigation systems.

[0123] In various embodiments, system 100 can be configured to visualize and characterize maneuvering obstacles by using multi-spectral imaging system 140 and other sensors mounted to mobile platform 110. In general, mobile platform 110 can relay sensor data to an onboard operator or a remote operator at base station 130 and / or secondary piloting station 230, where the sensor data can be processed or used to maneuver mobile platform 110. Such sensor data can also be presented on a display to help visualize and characterize maneuvering obstacles, assisting a human operator to detect and evade maneuvering obstacles. For example, elements of system 100 can autonomously map the extent of one or more maneuvering obstacles and overlay the resulting sensor data onto a geospatial chart or image, so that an operator can visualize the full extent of the maneuvering obstacle and proceed safely. In embodiments where system 100 or 300 includes a handheld mobile platform, elements of system 100 or 300 can aggregate various data to provide critical and timely warnings and / or safety instructions to the user of the handheld platform.

[0124] Embodiments can overlay 2D or 3D sensor data as icons or colored highlights or blobs onto a geospatial map or image, so that a user can visualize the extent of a maneuvering obstacle. Embodiments can optionally include a second screen / additional base station, so that users other than the UAV / UGV pilot can view / process sensor data.

[0125] In some embodiments, the display view (e.g., presented by user interface 132 and / or display 333) can include a geospatial chart or augmented image surrounded by various selector / indicator groups (e.g., a header, a payload controller menu, a video feed, and a platform telemetry indicator) configured to visualize and / or quantify the maneuvering obstacles and operate the mobile platform 110 and / or elements of the mobile platform 110. For example, the header can include one or more selectors and / or indicators configured to, for example, receive user selection of a particular selector to enable, disable, or select an active sensor payload (e.g., multispectral imaging system 140, environmental sensors 160) to display corresponding georeferenced sensor data in the geospatial chart or augmented image, or to indicate an operational state of the mobile platform 110 and / or various elements of the mobile platform 110.

[0126] In related embodiments, the geospatial chart or augmented image includes a mobile platform indicator and a maneuvering obstacle overlay presented above a base map or chart. In various embodiments, the system 100 can be configured to determine a shape, extent, and / or other characteristics of the maneuvering obstacle overlay within the geospatial chart or augmented image based at least in part on sensor data provided by the multispectral imaging system 140, environmental sensors 160, and orientation and / or position data (e.g., provided by the OPS 167 and / or other orientation and / or position or motion sensors of the mobile platform 110 or elements of the mobile platform 110) when the mobile platform 110 is maneuvering within a region displayed in the geospatial chart or augmented image. For example, the system 100 can be configured to determine an extent associated with the object 304 from a perspective of the optical axis 344 based on sensor data and / or environmental conditions provided by the mobile platform 110 and present the maneuvering obstacle overlay according to a color mapping to indicate a relative extent or approach speed, e.g., warm colors (e.g., red) indicating a relatively close or fast-approaching maneuvering obstacle, while cool colors (e.g., blue) indicating a relatively more distant or fast-retreating maneuvering obstacle.

[0127] In another embodiment, the system 100 can be configured to, for example, determine that multiple types of maneuvering obstacles exist within a particular survey region or scene and present each type of maneuvering obstacle according to a different overlay layer presented in the display view, each overlay layer can be selectively enabled and / or disabled by a user.

[0128] In various embodiments, the mobile platform 110 can be configured to adjust its route based on sensor data provided, for example, by the multispectral imaging system 140 and / or environmental sensors 160 and / or based on various environmental conditions measured by sensors mounted to the mobile platform 110 or measured by external systems and communicated to the system 100 (e.g., regional weather data provided by an online database through a wireless network linked to the base station 130 or the secondary lead station 230). Thus, the mobile platform 110 can be configured to autonomously avoid maneuvering obstacles or hazardous environments (e.g., significant downdrafts or other undesirable environmental conditions and / or maneuvering obstacles within such undesirable environmental conditions). For example, sending a UAV / UGV into a hazardous environment can put the mobile platform 110 at risk of damage. By adding intelligent maneuvering obstacle avoidance based on multispectral images and environmental sensors carried on board the vehicle, the risk of collision and / or inefficient maneuvering can be limited through automatic route adjustments, thereby protecting the mobile platform 110 and its associated sensor suite.

[0129] Embodiments described herein can provide for autonomous reaction to maneuvering obstacles. For example, the controller 112 and / or the controller of the base station 130 or the secondary lead station 230 can be configured to receive multispectral images, classification data, and / or other sensor data from the mobile platform 110 and / or from sensors mounted to the mobile platform 110 and determine a route adjustment to avoid a detected maneuvering obstacle and / or environmental condition. Examples of route adjustments can include stopping, turning, climbing, and / or reversing course to retreat from or otherwise avoid a maneuvering obstacle or a hazardous environment. For example, such route adjustments can be relayed to a user of the base station 130 or can be implemented directly / autonomously by the mobile platform 110. Such autonomous responses are intended to preserve the integrity of the mobile platform 110 and facilitate reaching a desired destination.

[0130] Figures 4-6 Display views 400, 500, and 600 are shown that include images generated by the multispectral imaging system 140 for the multispectral navigation system 100 or 300, in accordance with embodiments of the present disclosure. In Figure 4 In particular, the display view 400 shows a relatively high resolution visible spectrum RGB or color image 402 of a neighborhood and various scene elements (e.g., a road, a sidewalk, a fence, vegetation, and a structure behind the vegetation, all under a cloudy sky). In Figure 5 In particular, the display view 500 shows a monochrome visible spectrum image 502, an NIR image 504, and an LWUV image 506 of the same scene depicted in the visible spectrum color image 402. Figure 6A display view 600 is shown that includes a multispectral image 602 that includes characteristics of each of the spectral monochromatic visible spectral image 502, the NIR image 504, and the LWUV image 506, where each differentiated spectrum is mapped to the R, G, and B channels that are typically visible to the human eye (e.g., the NIR image data is mapped to the R channel, the VIS data is mapped to the G channel, and the LWUV data is mapped to the B channel).

[0131] From Figures 4-6 It can be seen that the daytime sky spectral characteristics have a relatively low spectral signature / brightness in the NIR image, a moderate spectral signature / brightness in the VIS image, and a relatively very bright spectral signature / brightness in the LWUV image. Thus, objects that are silhouetted against the sky are much easier to segment from the sky in the LWUV image than when using the VIS image, and multispectral analysis can better distinguish the sky from foreground objects, such as other mobile platforms. For example, it is difficult to imagine a daytime scene where the sky and some enclosed object have the same apparent radiance in all three bands depicted in Figure 5 Furthermore, such multispectral analysis is particularly useful when objects or maneuvering obstacles are beyond the reliable LIDAR range. Thus, embodiments are generally able to use simple image subtraction to segment the sky from other objects in a scene, even when the objects appear similar in the visible spectrum. Moreover, onboard CNNs or other machine vision engines (e.g., implemented within the module controller 372 and / or the imaging system controller 312) can quickly perform pixel and / or object classification and send multispectral image data including“labeled” pixels or groups of pixels to elements of the multispectral navigation system 100 and determine various maneuvering adjustments to avoid maneuvering obstacles, as described herein.

[0132] In various images of Figure 5 It can be seen that natural light shadows are suppressed in the UV band. For example, Rayleigh scattering would typically cause the entire sky to glow relatively brightly with scattered UV sunlight. Since the entire sky dome is illuminated, shadows are less intense in the UV band (e.g., as shown in the LWUV image 506). Moreover, many foreground objects in the LWUV image will tend to appear dark because UV is absorbed by many molecular surfaces. In contrast, shadows tend to be enhanced in the NIR band due to less Rayleigh scattering (e.g., as shown in the NIR image 504). Thus, subtracting the NIR image 504 and the LWUV image 506 from each other yields a multispectral image that highlights which pixels in the image are likely to be shadows (e.g., thereby classifying such pixels as shadow or non-shadow, optionally with an associated likelihood).

[0133] In another example, white clothing against a clean snow background is often invisible or has low contrast in VIS images, especially in diffused ambient light with hazy or absent shadows. However, in LWUV images, white clothing against a clean snow background is usually very noticeable with relatively high contrast and can be detected fairly easily using CNN analysis and / or image subtraction from the VIS image to the LWUV image.

[0134] Figure 7 A flowchart 700 illustrates various operations of using a multispectral navigation system to provide assisted or automated navigation according to embodiments of the present disclosure. In some embodiments, Figure 7 The operation can be implemented by and Figures 1-3 The software instructions described herein are executed by one or more logical devices or controllers associated with corresponding electronic devices, sensors, and / or structures. More generally, Figure 7 The operation can be achieved using any combination of software instructions, mechanical components, and / or electronic hardware (e.g., inductors, capacitors, amplifiers, actuators, or other analog and / or digital components).

[0135] It should also be understood that any step, substep, subprocess, or block of process 700 may be different from... Figure 7 The illustrated embodiments are performed in the order or arrangement shown. For example, in other embodiments, one or more blocks may be omitted from each individual process or added to each individual process. Furthermore, block inputs, block outputs, various sensor signals, sensor information, calibration parameters, and / or other operating parameters may be stored in one or more memories before moving to a subsequent part of the corresponding process. Although references... Figures 1-3 The system described herein describes process 700, but process 700 may be performed by other systems that are different from those systems and include different choices of electronic devices, sensors, components, mechanisms, platforms and / or platform attributes.

[0136] Figure 7 The process 700 can generally correspond to the method used to navigate a survey area using a multispectral navigation system 100.

[0137] At box 702, multispectral image data corresponding to the expected route of the mobile platform is received. For example, controllers 112, 312 and / or 372, communication modules 120, 144 and / or 134, user interface 132 and / or other elements of system 100 may be configured to receive multispectral image data from multispectral imaging system 140 and / or imaging module 142 as the mobile platform 110 maneuvers along the expected route (e.g., within scene 302).

[0138] In block 704, orientation and position data corresponding to the multispectral image data is received. For example, the system 100 can be configured to receive orientation and / or position data corresponding to the multispectral image data received in block 702 (e.g., from various orientation, position, and / or other motion sensors of the system 100).

[0139] In block 706, maneuver obstacle information is generated. For example, the system 100 can be configured to generate maneuver obstacle information corresponding to the projected route of the mobile platform 110 (e.g., within the scene 302) based at least in part on a combination of the orientation and / or position data and the multispectral image data received in blocks 702 and 704 (e.g., indicating the location, extent, and / or other characteristics of the objects 304 in the scene 302).

[0140] In block 708, a display view including the maneuver obstacle information is presented. For example, the system 100 can be configured to present a display view including the maneuver obstacle information generated in block 706 in a display of the user interface 132 and / or the display 333 of the multispectral imaging system 140 (e.g., a display view of the user interface 132). Figures 4-6

[0141] In block 710, an intersection of the projected route and a maneuver obstacle region is determined. For example, the system 100 can be configured to determine that the projected route of the mobile platform 110 intersects the location and / or extent of at least one object 304 in the scene 302 based at least in part on the maneuver obstacle information generated in block 706.

[0142] In block 712, the projected route of the mobile platform is adjusted. For example, the system 100 can be configured to adjust the projected route of the mobile platform 110 to avoid one or more maneuver obstacles (e.g., the plurality of objects 304 in the scene 302) determined to intersect the projected route of the mobile platform 110 in block 710. For example, the system 100 can be configured to determine an avoidance route configured to avoid all maneuver obstacles within the scene 302 and generally reach a predetermined destination or traverse the scene 302 according to a predetermined heading or path. In other embodiments, the system 100 can be configured to determine a series of avoidance routes configured to avoid individual maneuver obstacles within the scene 302 as the mobile platform 110 maneuvers through the scene 302.

[0143] ​By providing such systems and techniques for multispectral navigation, embodiments of the present disclosure greatly improve the operational flexibility and reliability of manned and unmanned mobile platforms, including unmanned sensor platforms. Moreover, such systems and techniques can be used to improve the operational safety of users and operators of mobile platforms, including unmanned mobile sensor platforms, beyond that which is possible with conventional systems. Thus, embodiments provide multispectral imaging systems and navigation systems with significantly increased operational convenience and performance.

[0144] As noted above, another important class of objects that vehicles frequently encounter in a scene are vegetation. Healthy vegetation strongly reflects NIR radiation, especially in the 800 nm band. Camera systems with the ability to measure both visible band radiation and NIR radiation can be configured to detect the so-called red edge: a sharp rise in reflectivity from 700 nm to 800 nm associated with the sponge-like mesophyll tissue in most vegetation.

[0145] One algorithm to identify foliage is the normalized difference vegetation index, or NDVI. This metric is commonly used in satellite remote sensing. The traditional NDVI is most often defined as the normalized contrast between the NIR band and the visible red band in a multispectral image. For embodiments of the disclosed multispectral imaging system, there is typically no separate visible red band distinct from the visible green or blue bands, so the traditional NVDI can be modified to form the mNDVI, defined in terms of the contrast between the NIR and the full visible spectral light:

[0146] mNDVI = (NIR - VIS) / (NIR + VIS)

[0147] With this definition of mNDVI, a threshold can be identified and selected to classify pixels in a multispectral image as being associated with vegetation in the imaged scene. Typical threshold values range from 0.3-0.4 mNDVI.

[0148] Another useful metric can be referred to as the normalized difference sky index, or NDSI. For example, there is typically a strong contrast between the LWUV and NIR images of the sky, because the Rayleigh scattering cross section varies very strongly with wavelength:

[0149] σ 瑞利 ~ wavelength -4

[0150] LWUV light will be scattered about 16 times more than NIR light (e.g., twice the wavelength of NIR), which makes the sky appear bright in the LWUV band and dark in the NIR band. This NDSI metric can be defined as:

[0151] NDSI = (LWUV - NIR) / (LWUV + NIR)

[0152] Using this definition of NDSI, thresholds can be identified and selected to classify pixels in a multispectral image as being associated with the sky in the imaging scene. Typical thresholds include an NDSI of approximately 0.2.

[0153] Figures 8-10 Display views 800, 900, and 1000 according to embodiments of the present disclosure are shown, and the display views 800, 900, and 1000 include images generated by a multispectral imaging system 140 for a multispectral navigation system 100 or 300. Figures 8-10 Three views of the same scene are shown. Figure 8 In the image, view 800 shows a relatively high-resolution visible-spectrum RGB or color image 802 of the main intersection, featuring highway entrance ramps and various scene elements (e.g., streetlights, roads with painted lanes and directional indicators, street signs, sidewalks, fences, vegetation, bridges, and mountains, all under a clear sky). More generally, Figure 8 A full-color visible light image of a typical ADAS scenario is shown.

[0154] exist Figure 9 In the middle, the display view 900 shows a multispectral image 902, which includes the spectral characteristics of each of the VIS image, NIR image and LWUV image depicting the same scene in the visible spectral color image 802, wherein each differential spectrum is mapped to the R, G and B channels that are normally visible to the human eye (e.g., NIR image data is mapped to the R channel, VIS data is mapped to the G channel and LWUV data is mapped to the B channel). Figure 10 A display view 1000 is shown, including a processed image or classification map 1002, which is identified pixel-by-pixel using mNDVI and NDSI and appropriate thresholds, displaying vegetation as red, the sky as blue, and the rest as black, as described herein. For example, specific mNDVI and NDSI thresholds used to generate classification map 1002 are 0.35 and 0.2, respectively.

[0155] Embodiments of the disclosed multispectral imaging system can be configured to distinguish green objects, such as green road signs, from green vegetation. This capability makes it easier for ADAS to identify and segment green road signs and to incorporate information from the imaged text into its general data stream using optical character recognition. Furthermore, measurements from both the NIR and VIS bands make it easier to see green vehicles against a background of green vegetation. In contrast, with conventional color cameras, green signs and green vehicles on the road or parked on the side of the road are at risk of being lost against a background of green vegetation.

[0156] For example, a LWUV image of a highway can provide minimal contrast between a sign and the vegetation behind it, but there is typically a relatively high contrast between the sky and everything else in the LWUV image. Thus, a multispectral image of the same scene will be able to show the sky and the vegetation clearly defined from each other. Using the spectral mapping provided herein, the road can be depicted in the multispectral image in a yellowish color. Thus, the road surface (e.g., using the mapped RGB color thresholds) can be classified as being different from both the vegetation and the sky in the imaged scene because the multispectral appearance of the road surface is very different from the multispectral appearance of the sky and the vegetation. Selecting appropriate thresholds, structural morphologies, and / or identifying other classification processing characteristics or techniques can include implementing appropriate CNN training and classification techniques, where the CNN is trained to classify various different image features that are important to ADAS.

[0157] Using the techniques described herein, embodiments of the multispectral navigation system described herein are able to: identify vegetation because it is bright in the NIR band but dark in the other two bands; identify a clear sky because it is bright in the LWUV band but dark in the other two bands; distinguish between a red LED taillight and an incandescent taillight with a red filter; and define the location of a vehicle window by the visible light passing through the vehicle window. Embodiments are also able to: distinguish between artificial surfaces and natural surfaces; distinguish between green vehicles and vegetation; distinguish between vehicles with a sky blue color and a clear sky; distinguish between white vehicles and overcast skies; distinguish between icy roads and non-icy roads; and distinguish between wet roads and dry roads.

[0158] In 2019, traffic accidents in the United States killed over 6,000 pedestrians, the highest annual total on record, and sent over 100,000 people to the hospital. As the automotive industry moves toward autonomous vehicles (AVs), the ability to sense, classify, and make instantaneous maneuver decisions based on artificial intelligence (AI) while driving becomes increasingly necessary. It is the task of advanced driver assistance systems (ADAS) and related AV systems to become increasingly intelligent and safe quickly. Embodiments described herein provide systems for car manufacturers, suppliers, regulators, automotive testing agencies, commercial vehicle operators, and consumers systems to maximize the safety of drivers, pedestrians, and other vulnerable road users.

[0159] Figure 11 is a diagram 1100 illustrating functional benefits associated with embodiments of a multispectral navigation system 100 in a thermal imaging navigation system (e.g., Figure 1 Particularly, Figure 11 The diagram 1100 of FIG. 11 illustrates how a thermal imaging-based navigation system can provide relatively reliable feature performance over a relatively large portion of the safety feature phase space identified in the diagram 1100. Moreover, Figure 11The plot 1100 of FIG. 11 shows that increased reliable feature performance can be achieved by combining thermal imaging with visible spectrum imaging and / or other remote sensor systems (e.g., radar), where the overlap or fusion of different feature performance essentially fills the security feature phase space identified in plot 1100.

[0160] Figure 12 A diagram showing a mobile platform 110 utilizing a thermal imaging navigation system 1200 according to embodiments of the present disclosure. For example, as described herein, in some embodiments, the multispectral navigation system 100 can be implemented as a thermal imaging navigation system, where the sensor payload 140 can be implemented as a thermal imaging system 140 including a thermal imaging module 142, the environmental sensor 160 can be implemented as a visible spectrum imaging system 160 including a visible spectrum imaging module / sensor assembly 166, and the thermal imaging navigation system 100 can include a ranging sensor system 127, which can be implemented as a radar or other type of ranging sensor system. In such embodiments, each of the thermal imaging system 140, the visible spectrum imaging system 160, and the ranging sensor system 127 can be mounted to the platform 110 to have overlapping fields of view (e.g., overlapping sensor data of the scene 302).

[0161] In particular, Figure 12 The thermal imaging navigation system 1200 of FIG. 12 can include one or more of the following: a controller 112, a propulsion system 124 (e.g., an electric motor or internal combustion engine or hybrid motor coupled to a drivetrain and / or drive train), a braking system 125 (e.g., one or more electromechanically controlled clamping or motion arrestment devices disposed along the drive train of the propulsion system 124, including at the wheels or within the wheels of the platform / passenger vehicle 110), a ranging sensor system 127a (a grid-mounted radar system), 127b (a front bumper-mounted radar or sonar system), 127c (a pillar-mounted radar system), and / or 127d (a rear bumper or trunk-mounted radar or sonar system), a thermal imaging system 1240a (a roof-mounted “shark fin” or “hat” or radio antenna-integrated thermal imaging system), 1240b (a pillar-mounted thermal imaging system), 1240c (a rear windshield-mounted thermal imaging system), and / or 1240d (a front windshield-mounted thermal imaging system), and / or a visible spectrum imaging system 1260a (a front windshield-mounted visible spectrum imaging system), 1260b (a roof or roof rack 1226-mounted visible spectrum imaging system), 1260c (a rear windshield-mounted visible spectrum imaging system), and / or 1260d (a trunk-mounted visible spectrum imaging system).

[0162] More generally, Figure 12Each of the identified mounting points can be used to mount any one or combination of the thermal imaging system, the visible spectrum imaging system, and / or the remote sensor system as described herein. In various embodiments, all of the sensor data generated by each mounted system can be used to generate display views presented by the user interface 1232 of the platform 110 (e.g., a dashboard display for the passenger vehicle 110).

[0163] Figure 13A A data flow diagram 1300 of a mobile platform 110 utilizing a thermal imaging navigation system 1200 is shown in accordance with embodiments of the present disclosure. In particular, the data flow diagram 1300 shows a thermal imaging system 1260 and / or a visible spectrum imaging system providing thermal and / or visible spectrum images to a maneuver obstacle detector 1340, which can be configured to provide received images and / or associated maneuver obstacle information (e.g., a labeled image) generated by the maneuver obstacle detector 1340 to a range estimator 1342. The range estimator 1342 can be configured to determine and provide a range estimate associated with each detected maneuver obstacle represented in the received labeled image to a sensor data fusioner 1344, and the sensor data fusioner 1344 can be configured to fuse or otherwise combine the labeled image and associated range estimate generated by the maneuver obstacle detector 1340, and / or ranging sensor data provided by a ranging sensor system 127, e.g., using orientation, position, motion, and / or other registration or calibration data provided by the registrator 1318, as shown.

[0164] The sensor data fusioner 1344 can be configured to provide the combined sensor data and / or images to a braking planner 1346 (e.g., an automatic emergency braking planner), which can be configured to evaluate the combined sensor data and / or images, including a projected route of the platform 110 (e.g., provided by the registrator 1318) and selectively activate braking system 125 and / or other elements of the propulsion system 124 to avoid a collision with any maneuver obstacle detected by the maneuver obstacle detector 1340.

[0165] In optional embodiments, the maneuver obstacle detector 1340 can be configured to generate a labeled image based on any one or combination of thermal images, visible spectrum images, and / or ranging sensor data. Further, the range estimator 1342 can be configured to determine and provide a range estimate associated with each detected maneuver obstacle based on the labeled image provided by the maneuver obstacle detector 1340 and / or ranging sensor data provided by the ranging sensor system 127.

[0166] In various embodiments, each of the maneuver obstacle detector 1340, the range estimator 1342, the sensor data fuser 1344, and / or the braking planner 1346 can be implemented as a separate software program and / or executed as a separate software program by the controller 112. In particular embodiments, the maneuver obstacle detector 1340 can be implemented as one or more CNNs configured to generate labeled thermal images, visible light images, and / or hybrid images, e.g., by one or more of feature extraction, semantic segmentation, object recognition, classification, and / or other similar CNN-based image and / or sensor data processing.

[0167] In one embodiment, the maneuver obstacle detector 1340 can be configured to apply a thermal image-trained CNN to detect maneuver obstacles represented in thermal images provided by the thermal imaging system 1240 and generate corresponding labeled thermal images and / or associated map scores (e.g., accuracy likelihood values) for each maneuver obstacle detected in the thermal images. In related embodiments, the maneuver obstacle detector 1340 can be configured to apply a visible spectrum image-trained CNN to detect maneuver obstacles represented in visible spectrum images provided by the visible spectrum imaging system 1260 and generate corresponding labeled visible spectrum images and / or associated map scores for each maneuver obstacle detected in the visible spectrum images. In such embodiments, the maneuver obstacle detector 1340 can be configured to combine the two sets of labeled images according to a logical function, e.g., according to one or the other to identify maneuver obstacles having map scores above a spectrum-specific threshold, and / or any generally detected maneuver obstacles having combined map scores (from each spectrum) above a combined threshold.

[0168] In another embodiment, the maneuvering obstacle detector 1340 can be configured to blend the thermal image with the visible spectrum image prior to applying the CNN trained on blended images to detect maneuvering obstacles represented in the blended images and generate corresponding labeled blended images and / or associated map scores for each maneuvering obstacle detected in the blended images, where only maneuvering objects with map scores above a blended image threshold are forwarded to the range estimator 1342 as labeled blended images. In other embodiments, the maneuvering obstacle detector 1340 can be configured to blend the thermal image with the visible spectrum image and combine the result with ranging sensor data provided by the ranging sensor system 127 prior to applying the CNN trained on fused sensor data to detect maneuvering obstacles represented in the fused sensor data and generate corresponding labeled blended or spectrum-specific images and / or associated map scores for each maneuvering obstacle detected in the fused sensor data, where only maneuvering objects with map scores above a fused sensor data threshold are forwarded to the range estimator 1342 as labeled images. In various embodiments, any of the CNNs described herein can be trained by synthesizing maneuvering obstacles, where computer-generated animals and / or other maneuvering obstacles are added to images to train the CNN to appropriately label associated images.

[0169] Thermal and visible spectrum image blending can be performed according to various metrics that emphasize one or other spectral characteristics. In some embodiments, color characteristics of the visible spectrum image can be modulated according to overlapping thermal image pixel values. In other embodiments, pixel values of the visible spectrum image can be converted to grayscale prior to blending with overlapping thermal image pixel values. In other embodiments, thermal image pixel values can be mapped to a particular color palette prior to blending with overlapping visible spectrum image pixel values (e.g., grayscale or color pixel values).

[0170] In some embodiments, the range estimator 1342 can be configured to generate range and / or relative direction estimates based on thermal or visible spectrum images only, e.g., by identifying common object features with known average feature displacement (e.g., port and starboard tail lights) and determining a range estimate based on the known average feature displacement, pixel displacement of the identified object features, and one or more displacement calibration parameters (e.g., typically specific to each imaging system). In other embodiments, the range estimator 1342 can be configured to generate range and / or relative direction estimates based on one or any combination of thermal images, visible spectrum images, or ranging sensor data provided by the ranging sensor system 127. For example, if each imaging system is known and fixed relative to the ranging sensor system 127, such range estimates can be performed without registration data provided by the registrator 1318.

[0171] For example, the sensor data fuser 1344 can be configured to fuse substantially synchronized sensor data provided by any sensors of the system 100 and / or 1200, and / or can be configured to fuse temporally differentiated data, such as time series of sensor data and / or labeled images, to facilitate accurate maneuver obstacle tracking, as described herein. The braking planner 1346 can be configured to receive all of the individual sensor data, labeled images, range and / or relative direction estimates and / or fuse the sensor data and selectively activate braking systems 125 and / or other elements of the propulsion system 124 to stop or reduce the speed of the platform 110 to avoid potential collisions with maneuver obstacles detected by the maneuver obstacle detector 1340, as described herein.

[0172] Figure 13B A block diagram of an update system 1302 for a mobile platform 110 utilizing a thermal imaging navigation system 1200 is shown, in accordance with an embodiment of the present disclosure. As Figure 13B As can be seen in FIG. 13, the update system 1302 can include various platforms 110, each configured to receive CNN configurations (weights) from an update server 1390 over a network 1320. Each platform 110 (e.g., a passenger vehicle) can be implemented as described with respect to the platforms 110 of the Figure 1 and / or 12. In various embodiments, the communication network 1320 can be implemented in accordance with one or more wired and / or wireless network interfaces, protocols, topologies, and / or methods as described herein, and in some embodiments can include one or more LAN and / or WAN networks, including cellular networks and / or the Internet.

[0173] In some embodiments, CNN maneuver obstacle detection failures can be identified (e.g., by a user override of the thermal imaging navigation system 1200) and / or stored locally at the platform 110. In some embodiments, the update server 1390 can be configured to receive CNN maneuver obstacle detection failures and associated sensor data, and generate updated CNN configurations to compensate for such edge cases / failures. For example, the update server 1390 and / or connected specialized CNN trainers and / or annotation feedback loops can be configured to adjust a current CNN configuration based at least in part on a CNN maneuver obstacle detection failure, and can store the updated CNN configuration for distribution to platforms 110.

[0174] The update server 1390 can be implemented as a logic device, tablet computer, laptop computer, desktop computer, and / or server computer, which can be configured to implement a CNN configuration database that stores and manages CNN configurations associated with different platforms and provide updated CNN configurations to platforms 110 upon user request or upon push by a manufacturer or regulatory agency, for example. Although the network 1320 is shown as a single network in FIG. 13, it will be appreciated that the network 1320 can include one or more LAN and / or WAN networks, including cellular networks and / or the Internet, in accordance with one or more wired and / or wireless network interfaces, protocols, topologies, and / or methods as described herein.Figure 13B While shown as a single element, in various embodiments, network 1320 may include multiple network infrastructures and / or combinations of infrastructures, wherein, for example, each platform 110 may be configured to use substantially different network infrastructures to access update server 1390.

[0175] Figure 14A -B illustrates display views 1400, 1402 including images generated by a thermal imaging system 1240 for a thermal imaging navigation system 1200, according to embodiments of the present disclosure. For example, Figure 14A The display view 1400 shows a visible spectrum image 1460, a co-registered thermal image 1440 including image tags (e.g., CNN-based maneuvering obstacle image tag 1480 and radar sensor data tag 1482), and a top-down or bird's-eye view of the labeled image as a fused image 1427, which includes the CNN-based maneuvering obstacle image tag 1480, the radar sensor data tag 1482, and local region image tags associated with the position of the platform 110. Figure 14B The display view 1402 shows a visible spectrum image 1462 with a manipulating obstacle label 1486 (e.g., a person) based on a visible spectrum image CNN, a thermal image 1442 with a manipulating obstacle label 1484 (e.g., the same person) based on a thermal image CNN, and a mixed image 1444 with a manipulating obstacle label 1488 based on a mixed image CNN.

[0176] Figure 15 A flowchart 1500 illustrates various operations using a thermal imaging navigation system to provide assisted or automated navigation, including automatic emergency braking, according to embodiments of the present disclosure. In some embodiments, Figure 15 The operation can be implemented by and Figures 1-14B The software instructions executed by one or more logical devices or controllers associated with the corresponding electronic devices, sensors, structures, and / or related images or display views described herein. More generally, Figure 15 The operation can be achieved using any combination of software instructions, mechanical components, and / or electronic hardware (e.g., inductors, capacitors, amplifiers, actuators, or other analog and / or digital components).

[0177] Any step, substep, subprocess, or box in process 1500 can be associated with... Figure 15 The illustrated embodiments are performed in different orders or arrangements. For example, in other embodiments, one or more boxes may be omitted from each individual process or one or more boxes may be added to each individual process. Furthermore, box inputs, box outputs, various sensor signals, sensor information, calibration parameters, and / or other operating parameters may be stored in one or more memories before moving to a subsequent part of the corresponding process. Although references...Figures 1-14B The process 1500 is described with respect to the systems described above, but the process 1500 can be performed by other systems different from those systems and including different selections of electronic devices, sensors, components, mechanisms, platforms, and / or platform attributes.

[0178] Figure 15 The process 1500 can generally correspond overall to a method of navigating a roadway and braking to avoid a maneuvering obstacle using the multispectral navigation system 100 of Figure 1

[0179] At block 1502, thermal image data corresponding to a projected route of a mobile platform is received. For example, the controller 112, 312 and / or 372, the communication module 120, 144 and / or 134, the user interface 1232 and / or 132, and / or other elements of the system 100 and / or 1200 can be configured to receive thermal image data from the thermal imaging system 1240 and / or the imaging module 142 as the mobile platform 110 maneuvers along the projected route (e.g., within the scene 302).

[0180] At block 1504, sensor data corresponding to the thermal image data is received. For example, the system 1200 can be configured to receive visible spectrum image data, radar data, lidar data, other ranging sensor data, and / or orientation and / or position data (e.g., from various orientation, position, and / or other motion sensors of the system 100) corresponding to the thermal image data received in block 1502. In some embodiments, the system 1200 can be configured to blend the thermal and visible spectrum images, or otherwise combine the thermal and visible spectrum images with received ranging sensor data, prior to proceeding to block 1506, e.g., as described herein. In some embodiments, the system 1200 can be configured to adjust a frame rate of any imaging system of the system 1200 based at least in part on a speed of the platform 110.

[0181] ​At block 1506, maneuver obstacle information is generated. For example, the system 1200 can be configured to generate maneuver obstacle information (e.g., indicating locations, extents, and / or other characteristics of the objects 304 in the scene 302) corresponding to the projected route of the mobile platform 110 (e.g., within the scene 302) based at least in part on the thermal image data received in block 1502. In other embodiments, the system 1200 can be configured to generate maneuver obstacle information corresponding to the projected route of the mobile platform 110 based at least in part on a combination of the sensor data and the thermal image data received in blocks 1502 and 1504 as described herein. In various embodiments, such maneuver obstacle information can include labeled thermal images, visible spectrum images, and / or hybrid images as described herein. In some embodiments, such maneuver obstacle information can include extent and / or relative direction estimates, fused sensor data and / or other sensor data or processed sensor data corresponding to detected maneuver obstacles as described herein.

[0182] At block 1508, a display view including the maneuver obstacle information is presented. For example, the system 1200 can be configured to present a display view (e.g., a display view of the user interface 1232 and / or the user interface 132) including the maneuver obstacle information generated in block 1506 in a display of the user interface 1232 and / or a display of the user interface 132. Such a display view can include, for example, visible spectrum images, thermal spectrum images, hybrid images, and / or fused sensor data and / or one or more types of image labels as described herein. Figures 13A-14B

[0183] At block 1510, intersections of the projected route with maneuver obstacle regions are determined. For example, the system 1200 can be configured to determine that the projected route of the mobile platform 110 intersects a location and / or an extent of at least one object 304 in the scene 302 based at least in part on the maneuver obstacle information generated in block 1506. For example, each of such determined intersections can be determined by the braking planner 1346 and / or the controller 112 as described herein.

[0184] ​At block 1512, the braking system of the mobile platform is activated. For example, the system 1200 can be configured to activate the braking system 125 and / or other elements that control the propulsion system 124 of the mobile platform 110 to stop or reduce the motion of the platform 110 to avoid the one or more maneuver obstacles (e.g., the plurality of objects 304 in the scene 302) that intersect the projected course of the mobile platform 110 as determined in block 1510. For example, the system 1200 can be configured to determine an avoidance course that is configured to avoid all of the maneuver obstacles within the scene 302 while braking the platform 110 without losing steering control of the platform 110 and generally according to the predetermined heading or path. In other embodiments, the system 1200 can be configured to determine a series of avoidance courses that are configured to avoid individual maneuver obstacles within the scene 302 while the mobile platform 110 is braked. More simply, the system 1200 can be configured to determine a braking force to be applied to the braking system 125 to stop the platform 110 with the least amount of linear travel possible.

[0185] By providing such systems and techniques for thermal image-based navigation, embodiments of the present disclosure greatly improve the operational flexibility and reliability of manned and unmanned mobile platforms, including passenger vehicles. Moreover, such systems and techniques can be used to improve the operational safety of users and operators of mobile platforms, including manned passenger vehicles, beyond that which is achievable with conventional systems. Thus, embodiments provide thermal imaging navigation systems with significantly increased operational convenience and performance.

[0186] As described herein, embodiments provide a clever software solution that combines visible spectrum images and / or video with thermal images and / or video to create a hybrid combination of both. In some embodiments, such hybrid images use portions of thermal images as well as visible spectrum images to provide features such as color and the ability to read signs and lane markings, even in low light conditions. Examples of such hybrid images are provided herein. Embodiments described herein can be configured to use such hybrid images as the base images for AI-based image processing and object detection and tracking methods. Such methods include providing de-warping and correction of both video streams, then running an object detector (e.g., a CNN-based object detector) and tracker on the combined video. The result is a single CNN, which results in better overall efficiency and computational performance. In contrast, existing systems currently process each video stream separately, which doubles the computational requirements on hardware.

[0187] The embodiments described herein provide additional capabilities over conventional techniques because the CNN and tracker can utilize additional car information (e.g., speed, radar, GPS, direction) as part of the data in the AI-based processing capabilities because the weighted values of the mixed video on each side can be determined within the AI stack. Additionally, the CNN can use additional information from scene temperature data provided by a radiometric camera to check whether the average temperature of a classified object is within an acceptable temperature range to help limit false positive detections. For example, the pixels of an object classified by the CNN as a person all have a temperature value associated with the identified person. The average temperature value can be checked against an acceptable range associated with a particular type of target in a particular environment.

[0188] Nighttime driving can be difficult for drivers; drivers can find it difficult to see critical road obstructions and / or other objects, such as vulnerable road users (VRUs) and animals. The embodiments described herein provide additional information to the driver so that the driver can see VRUs and animals with enough warning to react so that accidents can be avoided at night and other challenging lighting conditions. For example, a mixed video with an alert can be presented to the driver, or information from the cameras (e.g., mixed as described herein) and detection mechanisms can be used to stop the vehicle using AEB (automatic emergency braking).

[0189] In various embodiments, critical information from detecting a VRU can be displayed to the driver. For example, an example display view (e.g., Figure 16A - F) uses a thermal camera to highlight pedestrians and vehicles, but a visible light camera displays a view from the car from the driver’s perspective (e.g., including what the driver’s eyes can see). When using a CNN with the mixed video, the embodiments use combined information from the thermal and visible spectrum data and reduce computational cost compared to two separate processing stacks operating independently on the visible spectrum video and the thermal video.

[0190] False positive detections can occur using either or both of the thermal and visible spectrum images. The embodiments described herein use both video channels to more accurately detect VRUs, animals, and other objects (such as other vehicles) to more reliably determine the presence and type of objects classified by the CNN. The embodiments also determine and use additional information corresponding to the average radiometric heat value of a detected object to determine whether the classified object is within a desired radiometric heat value to reduce and / or eliminate false positives in the presence and / or classification of objects.

[0191] Certain calibration objects in the scene can also be used as reference temperatures to help ensure that the radiometric values of non-calibration objects (e.g., VRUs) are valid / calibrated / accurate for other non-calibration objects detected. For example, a calibration object can include a specified portion of an engine hood of a vehicle to which the thermal imaging module is attached - that specified portion of the engine hood can be within the field of view of the thermal camera, and a temperature sensor can be coupled to the specified portion of the engine hood and provide an accurate temperature of the specified portion of the engine hood that can be used to calibrate the thermal imaging module. Such calibration objects can be marked with IR and / or visible spectrum visible markers (e.g., registration points, crosses, and / or other graphics) to help identify the pixels associated with the calibration object. In other embodiments, such calibration objects can be implemented by stationary structures (signs, placards) placed alongside the road or at the exit of a parking lot, or structures that are maintained at a standard temperature or configured to communicate their temperature to the system 100 (e.g., through IR or visible spectrum bright light text, or via wireless beacons).

[0192] Various embodiments include thermal cameras (e.g., thermal imaging modules) and visible spectrum cameras (e.g., visible spectrum imaging modules) mounted on or otherwise coupled to a vehicle that have similar or at least overlapping fields of view. The two videos can then be scaled, de-warped, and / or corrected through a calibration process that uses a registration target that both cameras can see. Such a calibration process can be configured to align the pixels between the two camera systems (pixel registration). The visible and thermal lens / lens systems can have different orientations, and the calibration process can compensate for the radial and tangential orientation of each lens so that the images from the two cameras are substantially aligned and can be overlaid on each other, thereby causing the imaged objects to overlap in each video.

[0193] In some embodiments, such registration calibration can be adjusted in real-time using objects in the scene that have detectable and well-defined / definitive edges, thereby maintaining or correcting the alignment even after the initial calibration process is performed. The aligned images can then be displayed to the driver or used by the vehicle with ADAS functionality, as described herein. For example, the combined images (e.g., display views and / or combined image data) can include variable contributions from 100% thermal and 0% visible to 100% visible and 0% thermal, or any percentage in between. Such variations can be selected based on the environmental lighting, weather, and / or other environmental conditions in which one of the spectrums provides more reliable image data for a particular application (e.g., object detection, classification, image registration, and / or other image processing, as described herein).

[0194] In various embodiments, the registered multispectral video (thermal and visible) can be displayed to the user and / or input into object detection algorithms, particularly CNN-based object detectors. As described herein, the hybrid video provides superior object detection / awareness compared to visible light video only or thermal video only. Embodiments of such CNNs can also implement a tracker that keeps a target locked between image frames and increases the reliability of object detection over time. Embodiments of such CNNs are relatively very efficient in terms of computational cost compared to conventional image processing for each sensor type, as only one network is used. The result is a lower cost due to computational hardware and more reliable detection / classification due to multiple simultaneous spectrums (e.g., thermal and visible).

[0195] Radiant heat value per pixel (calibrated absolute temperature) can also be implemented using the thermal imaging module, as described herein. Such per-pixel data can be used when a classified object (e.g., VRU) is detected, and the average value of the pixels associated with the VRU is combined and a relative temperature value is obtained. Threshold ranges can be used to evaluate the object temperature compared to expected values for a particular scene / environment temperature. Such processing adds confidence to the CNN capability and helps to reduce or eliminate false positives, which is particularly useful for reliable and comfortable AEB applications.

[0196] In various embodiments, a display view can be generated for the driver that includes hybrid spectral images. In some embodiments, the hybrid video can be configured to emphasize street signs and colors (from the visible light camera), while in other embodiments, to emphasize VRUs in thermal. Other display views are primarily visible spectral images, where detected objects are displayed in thermal. In other embodiments, the driver can be displayed with a configurable way of thermal and visible light hybrid video with CNN detection, such that various driver scenarios are considered and optimized. For example, during completely dark night driving, the thermal image module can provide most of the video stream. In well-lit situations, the visible spectral image module can provide most of the transparency. In other embodiments, thermal detection will be overlaid on the visible spectral video, especially in situations where oncoming car headlights can be blinding and wash out the visible spectral image.

[0197] Figure 16A -B shows display views 1600-1605 including images generated by the thermal imaging system 1240 for the thermal imaging navigation system 1200, in accordance with embodiments of the present disclosure. For example, Figure 16AThe display view 1600 of FIG. 16A shows a hybrid image 1610 with manipulation obstacle labels 1620 (oncoming traffic), 1622 (vehicles parked along the lane), and 1630 (pedestrian crossing the traffic at a crosswalk) based on the hybrid image CNN. Each manipulation obstacle label in the display view 1600 includes a bounding box around the heat map data, representing the corresponding detected object / manipulation obstacle. In Figure 16A In the illustrated embodiment, the hybrid image 1610 is primarily a visible spectrum image combined with heat map images associated with the manipulation obstacle labels 1620, 1622, and 1630.

[0198] Figure 16B The display view 1601 of FIG. 16B shows a hybrid image 1610 with manipulation obstacle label 1620 (oncoming traffic) based on the hybrid image CNN, which is similar to the display view 1600 of FIG. 16A. Figure 16A The display view 1600 of FIG. 16A, but presents a different environment in which the visible spectrum glare of oncoming headlights would otherwise obscure the oncoming traffic if that portion of the hybrid image 1610 was not overlaid or otherwise hybridized with the heat map image associated with the manipulation obstacle label 1620.

[0199] Figure 16C The display view 1602 of FIG. 16C shows a hybrid image 1610 with manipulation obstacle label 1632 (oncoming bicyclist) based on the hybrid image CNN, which is similar to the display view 1601 of FIG. 16B. Figure 16A The display view 1600 of FIG. 16A, but presents a different environment in which the visible spectrum image would not display the oncoming bicyclist if that portion of the hybrid image 1610 was not overlaid or otherwise hybridized with the heat map image associated with the manipulation obstacle label 1632.

[0200] Figure 16D The display view 1603 of FIG. 16D shows a hybrid image 1612 that is primarily a heat map image combined with visible spectrum images associated with various illuminated objects, such as taillights 1640, intersection signal lights 1642, and a roadside reflector 1644 (all shown as red in the hybrid image 1612, for example). In Figure 16D In the illustrated embodiment, the hybrid image 1612 includes sufficient detail to identify and detect a pedestrian 1650 and lane markings 1660, even in relatively low light conditions (e.g., where the visible spectrum image can not include similar detail).

[0201] Figure 16EThe display view 1604 of FIG. 16B shows a hybrid image 1612 that is primarily a thermal image combined with a visible spectrum image associated with various illuminated objects, such as a crosswalk sign 1646, a lane change sign 1648, a corner light 1670 (with associated holiday illumination), and a street light 1672 (e.g., all displayed in visible spectrum colors - white, yellow, orange - in the hybrid image 1612). In the illustrated embodiment, the hybrid image 1612 includes sufficient detail to identify and detect the lane marker 1660, even in relatively low light conditions (e.g., where the visible spectrum image can not include similar detail or be obscured by glare from oncoming vehicle headlights). Figure 16E In the illustrated embodiment, the hybrid image 1612 includes maneuver obstacle labels 1624 (relatively small vehicle traffic traveling in the same direction along the road) and 1626 (relatively large vehicle traffic traveling in the same direction along the lane) based on the hybrid image CNN. Each maneuver obstacle label in the display view 1600 includes a bounding box around the hybrid image data, representing the corresponding detected object / maneuver obstacle and its associated illuminated feature (e.g., the taillight / running light 1640).

[0202] Figure 16F The display view 1605 of FIG. 16B shows a hybrid image 1612 that is primarily a thermal image combined with a visible spectrum image associated with various illuminated objects, such as the taillight / running light 1640 (e.g., displayed in a visible spectrum color - red - in the hybrid image 1612). In the illustrated embodiment, the hybrid image 1612 includes sufficient detail to identify and detect the taillight / running light 1640, even in relatively low light conditions (e.g., where the visible spectrum image can not include similar detail or be obscured by glare from oncoming vehicle headlights). Figure 16F In the illustrated embodiment, the hybrid image 1612 includes maneuver obstacle labels 1624 (relatively small vehicle traffic traveling in the same direction along the road) and 1626 (relatively large vehicle traffic traveling in the same direction along the lane) based on the hybrid image CNN. Each maneuver obstacle label in the display view 1600 includes a bounding box around the hybrid image data, representing the corresponding detected object / maneuver obstacle and its associated illuminated feature (e.g., the taillight / running light 1640).

[0203] By providing systems and techniques that include CNN-based image processing of combined or multispectral images, embodiments of the present disclosure greatly improve the operational flexibility and reliability of manned and unmanned mobile platforms, including passenger vehicles. Moreover, such systems and techniques can be used to improve the operational safety of users and operators of mobile platforms, including manned passenger vehicles, beyond that which is achievable with conventional systems. Thus, embodiments provide thermal imaging navigation systems with significantly increased operational convenience and performance.

[0204] Where applicable, the various embodiments provided by the present disclosure can be implemented using hardware, software, or combinations of hardware and software. Also, where applicable, the various hardware components and / or software components set forth herein can be combined into composite components comprising software, hardware, and / or both, without departing from the spirit of the present disclosure. Where applicable, the various hardware components and / or software components set forth herein can be separated into sub-components comprising software, hardware, and / or both without departing from the spirit of the present disclosure. Furthermore, where applicable, the various hardware components and / or software components set forth herein can be implemented as a "virtual component" comprising software, hardware, and / or both, stored on one or more computer-readable storage media and executed by one or more hardware processors that are not physically a part of the virtual component, but that virtualize the functionality of the virtual component.

[0205] Software according to the present disclosure, such as non-transitory instructions, program code, and / or data, can be stored on one or more non-transitory machine-readable media. It is also contemplated that software identified herein can be implemented using one or more general purpose or special purpose computing systems and / or computer systems. As appropriate, the order of the various steps described herein can be changed, combined into composite steps, and / or separated into sub-steps to provide features described herein.

[0206] The foregoing examples illustrate but do not limit the present application. It should also be understood that numerous modifications and variations are possible in light of the above teachings. Thus, it is intended that the scope of the present application be limited only by the appended claims.

Claims

1. A system for navigation, the system comprising: a thermal imaging system, wherein the thermal imaging system comprises a thermal imaging module configured to provide thermal image data corresponding to a projected route of a mobile platform; a visible spectrum imaging system coupled to the mobile platform and configured to provide visible spectrum image data corresponding to the projected route; a ranging sensor system coupled to the mobile platform and configured to provide ranging sensor data corresponding to the projected route; and a logic device configured to communicate with the thermal imaging system, the visible spectrum imaging system, and the ranging sensor system, wherein the logic device is configured to: receive the thermal image data corresponding to the projected route; receive visible spectrum image data corresponding to the thermal image data; receive ranging sensor data corresponding to the thermal image data; generate hybrid image data based at least in part on the received visible spectrum image data and the received thermal image data; and generate maneuver obstacle information corresponding to the projected route based at least in part on the hybrid image data and the ranging sensor data, wherein the logic device is configured to generate the maneuver obstacle information at least by applying a convolutional neural network trained on fused sensor data to a combination of ranging sensor data and hybrid image data to detect one or more maneuver objects to obtain the maneuver obstacle information; and wherein the logic device is further configured to generate the maneuver obstacle information by associating each detected maneuver object with a map score and determining a range estimate for each detected maneuver object associated with a map score above a threshold value.

2. The system of claim 1, further comprising: an orientation and / or position sensor coupled to the mobile platform and configured to provide orientation and / or position data associated with motion of the mobile structure; wherein the logic device is further configured to: receive orientation and / or position data corresponding to the thermal image data; generate the maneuver obstacle information corresponding to the projected route based at least in part further on the orientation and / or position data; and selectively activate a braking system of the platform based at least in part on the generated maneuver obstacle information corresponding to the projected route.

3. The system of claim 1, further comprising: an orientation and / or position sensor coupled to the mobile platform and configured to provide orientation and / or position data associated with motion of the mobile structure; wherein the logic device is configured to: receive orientation and / or position data corresponding to the thermal image data; and generate the maneuver obstacle information corresponding to the projected route based at least in part further on the orientation and / or position data.

4. The system of claim 1, wherein, the logic device is configured to: generate the hybrid image data based on pixel values of the received visible spectrum image data and pixel values of the received thermal image data to obtain the hybrid image data.

5. The system of claim 1, wherein: the ranging sensor system comprises a grid-mounted radar system and / or a grid- mounted lidar system.

6. The system of claim 1, further comprising a communication module configured to establish a wireless communication link with an update server associated with the mobile platform, wherein: the thermal imaging system is configured to be coupled to the mobile platform; and the logic device is configured to receive the thermal image data from the thermal imaging system as the mobile platform maneuvers along a projected route and report generated maneuver obstacle information corresponding to the projected route to the update server over the wireless communication link.

7. The system of claim 1, wherein, the logic device is configured to determine a range estimate for each detected maneuver object associated with a map score above the threshold based on the ranging sensor data and the blended image data.

8. The system of claim 1, wherein, the logic device is further configured to adjust a frame rate associated with the thermal imaging system and / or the visible spectrum imaging system based at least in part on a speed of the mobile platform.

9. The system of claim 4, wherein, the logic device is further configured to generate the blended image data by modulating pixel values of received visible spectrum image data based on overlapping pixel values of received thermal image data to obtain blended image data.

10. A method for navigation, the method comprising: receiving thermal image data from a thermal imaging system comprising a thermal imaging module configured to provide thermal image data corresponding to a projected route of a mobile platform; receiving visible spectrum image data corresponding to the thermal image data from a visible spectrum imaging system coupled to the mobile platform and configured to provide visible spectrum image data corresponding to the projected route; receiving ranging sensor data corresponding to the thermal image data from a ranging sensor system coupled to the mobile platform and configured to provide ranging sensor data corresponding to the projected route; generating blended image data based at least in part on the received visible spectrum image data and the received thermal image data; and generating maneuver obstacle information corresponding to the projected route based at least in part on the blended image data and the ranging sensor data, wherein generating maneuver obstacle information comprises applying a convolutional neural network trained on fused sensor data to a combination of the ranging sensor data and the blended image data to detect one or more maneuver objects, and wherein generating maneuver obstacle information further comprises associating each detected maneuver object with a map score and determining a range estimate for each detected maneuver object associated with a map score above a threshold.

11. The method of claim 10, further comprising: receiving orientation and / or position data corresponding to the thermal image data from an orientation and / or position sensor coupled to the mobile platform; generating the maneuver obstacle information corresponding to the projected route based at least also in part on the orientation and / or position data; and selectively activate a braking system of the platform based at least in part on the generated maneuver obstacle information corresponding to the projected route.

12. The method of claim 10, further comprising: receiving orientation and / or position data corresponding to the thermal image data from an orientation and / or position sensor coupled to the mobile platform; generating the maneuver obstacle information corresponding to the projected route based at least in part on the orientation and / or position data.

13. The method of claim 10, wherein the blended image data is based on pixel values of the received visible spectrum image data and pixel values of the received thermal image data.

14. The method of claim 10, wherein: the ranging sensor system comprises a radar system and / or a lidar system.

15. The method of claim 10, further comprising: receiving thermal image data from the thermal imaging system as the mobile platform maneuvers along the projected route; and reporting the generated maneuver obstacle information corresponding to the projected route to an update server associated with the mobile platform via a communication module configured to establish a wireless communication link with the update server over a wireless communication link.

Images