Systems and methods for multispectral object detection using thermal imaging

Abstract

This document describes techniques, apparatus, and systems for implementing multispectral object detection with thermal imaging. Thermal sensors, multispectral stereo camera systems, and illumination systems are combined to provide a multispectral object detection system for safety or driving systems of autonomous and semi-autonomous vehicles. The illumination system can provide illumination for the stereo camera system and act as a rangefinder that can determine distances between the stereo camera system and objects in the field of view of the system. The stereo camera system can include one or more of various camera or sensor technologies, including infrared or near-infrared, thermal imaging, or visible light. The system can be used to detect objects in the field of view, determine distances to the objects, and estimate the size and relative motion of the objects. The system can reduce cost and resource usage while enabling accurate and high-quality object detection for safety and autonomous or semi-autonomous control.

Description

Background Technology

[0001] Autonomous and semi-autonomous driving and safety systems can use perception systems to detect objects in their environment. Perception systems can include cameras and other sensors, such as light detection and ranging sensors (LIDAR), to identify pedestrians, vehicles, and other objects that may pose a collision risk. While LIDAR systems can be useful, they involve laser light sources, more sensitive sensors, and fast-response sensor capabilities, which can increase the cost of perception systems or related equipment. Furthermore, detecting and tracking objects using LIDAR systems is a complex data processing task, especially for driving and safety systems supporting vehicles, impacting storage space and processor resources, thus increasing costs. Some perception systems may be too expensive for certain categories of vehicles, potentially forcing consumers to make unnecessarily difficult choices between driving safety and cost savings. Summary of the Invention

[0002] This document describes techniques and systems for implementing multispectral object detection using thermal imaging. It also describes methods performed by the techniques and systems summarized herein, as well as apparatus for performing these methods. For clarity, in this document, the term "AV" will be used to refer to autonomous vehicles, semi-autonomous vehicles, or manually operated vehicles including automated safety or warning functions. As used herein, AV includes ground and / or air vehicles (e.g., drones). For ease of explanation, this disclosure is primarily described in the context of ground vehicles (e.g., trucks, cars, automobiles) configured to drive on and off roads, including passengerless vehicles (e.g., robotic transport vehicles) and driverless vehicles configured to drive based on little or no passenger input (other than voice or computer input to indicate a desired address, route, or final destination).

[0003] By implementing multispectral object detection using thermal imaging, an illumination system and a binary stereo camera system can be combined to provide input to a driving system, aiding in safe and autonomous or semi-autonomous control. The illumination system can both provide illumination for the stereo camera system and act as a rangefinder, determining the distance between the stereo camera system and objects within its field of view (FOV). The stereo camera system can include one or more of a variety of camera technologies, including infrared (e.g., near-infrared radiation (NIR)), thermal imaging, or visible light.

[0004] For example, an AV (Field of View) can include a multispectral camera system with a thermal sensor and two active NIR (Near-Infrared) cameras. In this configuration, the camera system can be used to detect objects within its field of view (FOV), determine the distance to the object, and estimate the object's size and relative motion. To detect objects (or distinguish between objects and background), an emitter on the NIR camera illuminates the FOV with NIR radiation, and the NIR camera captures an NIR image of the FOV. Similarly, a (passive) thermal sensor captures a thermal image of the FOV. The NIR and thermal images can be used to detect objects within the FOV for the AV (e.g., using pattern matching algorithms) to avoid objects such as pedestrians. Furthermore, the thermal sensor can detect emissions from objects within the FOV and from the background surrounding the objects. Since there is typically a temperature difference between the background and the object, the thermal sensor can collect data over time and determine sequential temperature differences across the object's edges, allowing for accurate object detection. Furthermore, since multispectral camera systems use the ranging capabilities of NIR transmitters to map the field of view (FOV) (e.g., scan for potential objects), the thermal imaging incorporated into the multispectral camera system can verify whether the AV is avoiding objects while driving in the environment.

[0005] The distance to the object being detected can be determined in several ways. For example, thermal images from a thermal sensor, along with NIR images from an NIR camera, can be used in conjunction with stereo camera technology to calculate the distance. Furthermore, the NIR transmitter / detector system can determine the distance to the object using time-of-flight or stereo ranging techniques (e.g., without thermal images). In this way, the described systems and techniques can use less processor and memory, which reduces costs while still ensuring accurate and high-quality object detection suitable for safe, autonomous, or semi-autonomous control.

[0006] This invention provides a simplified concept related to indirect verification of speed limits based on contextual information from autonomous and semi-autonomous driving systems, which is further described in the detailed description and accompanying drawings. This invention is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Attached Figure Description

[0007] This document describes in detail one or more aspects of a multispectral object detection system utilizing thermal imaging, with reference to the following figures.

[0008] Figure 1 An example environment is shown in which a multispectral object detection system utilizing thermal imaging can be implemented for vehicles;

[0009] Figure 2An example configuration of the system is shown, which can realize a multispectral object detection system using thermal imaging for vehicles;

[0010] Figure 3 This demonstrates the capability of autonomous vehicles (AVs) (such as...) Figure 1 and Figure 2 A schematic diagram of an example object detection system implemented in a means of transportation;

[0011] Figure 4 It shows the use of Figure 1 , Figure 2 or Figure 3 Example object detection scenario of the object detection system;

[0012] Figure 5 This demonstrates the use of differential time delay and temperature gradient to utilize... Figure 1 , Figure 2 or Figure 3 A flowchart illustrating an example process of an object detection system detecting objects;

[0013] Figure 6 It shows the use of Figure 1 , Figure 2 or Figure 3 Another example object detection scenario for an object detection system; and

[0014] Figure 7-1 and Figure 7-2 An example method for implementing a multispectral object detection system using thermal imaging is shown.

[0015] The same numbers are often used throughout the accompanying drawings to refer to similar features and parts. Detailed Implementation

[0016] Overview

[0017] Autonomous and semi-autonomous vehicles (AVs) use sensor systems to detect objects in the environment surrounding the AV. These systems can include different combinations of sensors, such as radar, cameras (e.g., visible light cameras), or light detection and ranging systems (LIDAR systems), to identify and locate objects that may pose a collision risk, such as pedestrians, vehicles, or other objects within or near the AV's path. Data from sensors and LIDAR may be combined and correlated with other data (e.g., data from navigation or positioning systems) to identify objects and estimate their position, size, shape, and velocity.

[0018] Visible light camera systems are often susceptible to false positives and noise. These "vision-based" cameras detect any object in each camera's two-dimensional image, making it difficult to determine which objects are genuine obstacles to avoid. Shadows and retroreflections also introduce noise into the data, increasing the ambiguity in distinguishing real obstacles from false positives. Furthermore, while LiDAR systems can be effective, they include laser light sources, more sensitive sensors, and fast-response sensor capabilities, which increases the cost associated with LiDAR equipment. Additionally, using LiDAR systems or visible light-based cameras to detect and track objects typically involves very significant data processing requirements, leading to heat generation issues, increased memory usage, higher processor resource consumption, and increased costs.

[0019] In contrast, the described techniques and systems can be used to achieve multispectral object detection using thermal imaging, providing reliable and rapid object detection and ranging calculations while reducing or not increasing data processing and memory consumption. Consider an example where the AV operates on a road with pedestrian crossings.

[0020] In this example, trees and streetlights line the road, and a distracted jogger is about to step onto the road in the AV's path some distance ahead. As the AV travels along the street, for navigation and collision avoidance purposes, the AV uses a sensor system to scan the area around the AV. This sensor system includes, as in this example, a multispectral object detection system utilizing thermal imaging. Assume that in this example, this object detection system includes at least an active near-infrared (NIR) rangefinder, a thermal sensor, and an active NIR camera. The rangefinder scans the environment around the AV and can detect objects on or near the road (e.g., the jogger or trees). The thermal sensor detects heat emissions from the jogger and from the background around the jogger (e.g., stationary objects such as buildings, trees, and streetlights). Based on the thermal data determined from the detected heat emissions, the thermal sensor can verify that the object detected by the rangefinder is the jogger, and the object detection system can communicate with the AV's navigation and collision avoidance systems to control the AV to take evasive action or stop to avoid the jogger.

[0021] In another example, the rangefinder detects another object on the other side of the road. Let's assume this time the object is a ladder on that side of the road, left behind after some landscaping work. A thermal sensor and an NIR camera can act as a stereo camera system, collecting image data including this object (this active NIR rangefinder can also act as an illumination source for the active NIR camera). Using pattern matching techniques and stereo image processing, the perception system determines: the distance of this object to the AV, that this object is not a pedestrian, and that this object will not enter the AV's path. This object detection system can communicate with the AV's navigation and collision avoidance systems to control the AV; this time it might slow down, but without unnecessary avoidance maneuvers, whereas if the ladder were perceived as a pedestrian or other object, avoidance maneuvers might be performed to carefully avoid it. In this way, the system and techniques used for multispectral object detection systems utilizing thermal imaging can perform accurate object detection and ranging, providing a safe and comfortable ride for the object AV and its occupants, while reducing processor and memory usage, which can lower costs and improve the efficiency and sustainability of the AV. Compared to other perception systems, perception systems that perform multispectral object detection using thermal imaging may be less expensive, allowing consumers to choose from a range of luxury and economy vehicle types without having to choose between driving safety and cost.

[0022] These examples are just a small selection of the techniques and systems used for multispectral object detection using thermal imaging. This document describes other examples and implementations.

[0023] Systems and devices

[0024] Figure 1 An example environment 100 is shown, in which a vehicle 102 (which may include an autonomous vehicle (AV), a semi-autonomous vehicle, or a manually operated vehicle including automated safety or warning functions) can implement a multispectral object detection system 104 (also referred to as object detection system 104) utilizing thermal imaging. Environment 100 may include roads, shoulders, sidewalks, parking lots, or structures, or airspace (e.g., for aircraft). Although shown as a car, AV 102 may represent other types of vehicles (e.g., motorcycles, buses, tractor-trailers, semi-trailer trucks, or construction equipment). AV 102 may also be an aircraft (e.g., a drone, helicopter, or airplane). Typically, object detection system 104 can be used with any mobile platform that can move on or near a road or in the airspace within environment 100.

[0025] AV 102 may include a vision system 106 mounted on AV 102. Vision system 106 may capture video and / or photographic images or videos of the environment 100 surrounding AV 102 (including roads and airspace). In some implementations, vision system 106 may be mounted on or near the front of AV 102. In other implementations, at least a portion of vision system 106 may be mounted in a rearview mirror of AV 102 to provide a field of view (FOV) of the environment 100. In other implementations, vision system 106 may project an FOV from any outer surface of AV 102. For example, at least a portion of vision system 106 may be integrated into a side mirror, bumper, roof, or any other internal or external location where the FOV includes the environment 100 surrounding AV 102 (e.g., roads and / or airspace). Typically, vision system 106 may be positioned to provide an instrument FOV that adequately covers the environment 100 surrounding AV 102.

[0026] AV 102 may also include a sensing system 108, which can provide input data to one or more system processors 110. System processor 110 may be a microprocessor or a system-on-a-chip (SoC) of a computing device. Input data from sensing system 108 can be used to detect objects in the environment 100 surrounding AV 102. For example, input data from sensing system 108 can be used to detect pedestrians, animals, other vehicles, debris, buildings, power lines, traffic signals, signs, or other structures. Sensing system 108 may include a radar system, a positioning system (e.g., a radio navigation satellite system (RNSS) such as the US Global Positioning System (GPS) or the European Galileo system), a lidar system, an inertial measurement unit (IMU) such as a microelectromechanical system (MEMS) IMU, or any combination thereof. A radar system or lidar system can use electromagnetic signals to detect objects in the environment 100 surrounding AV 102. As AV 102 moves within environment 100, the positioning system determines the position of AV 102 by receiving signals acquired by the positioning system. An IMU can generate data that can be used to calculate, for example, three-dimensional position, angular rate, linear velocity, and position relative to a global coordinate system. The data generated by the IMU can also be used to predict future position, orientation, velocity, and other parameters of the AV (Automatic Vehicle).

[0027] AV 102 also includes one or more communication devices 112 and one or more computer-readable storage media (CRM) 114, both operatively coupled to system processor 110. Communication device 112 may be a radio frequency (RF) transceiver for transmitting and receiving RF signals. The transceiver may include one or more transmitters and receivers, which may be integrated together on the same integrated circuit (e.g., a transceiver integrated circuit) or separately on different integrated circuits. Communication device 112 may be used to communicate with remote computing devices (e.g., servers or computing systems providing navigation information or sensing data), nearby structures (e.g., traffic signs or signal lights, pedestrian-operated devices such as mobile phones), or other vehicles.

[0028] For example, AV 102 can use communication device 112 to wirelessly exchange information with nearby vehicles or other devices using vehicle-to-vehicle (V2V) or vehicle-to-everything (V2X) communication. AV 102 can use V2V or V2X communication to obtain the speed, position, and heading of nearby vehicles. Similarly, AV 102 can use communication device 112 to wirelessly receive information from other nearby devices to detect objects and pedestrians in environment 100. Communication device 112 may include sensor interfaces and driving system interfaces. For example, when individual components of the multispectral object detection system 104 are integrated within AV 102, the sensor interfaces and driving system interfaces can transmit data via the communication bus of AV 102.

[0029] CRM 114 provides persistent and non-persistent storage for executable instructions (e.g., firmware, recovery firmware, software, applications, modules, programs, functions) and data (e.g., user data, operational data) to AV 102 to support the execution of the executable instructions. For example, CRM 114 may include instructions that, when executed by system processor 110, execute the multispectral object detection system 104. Examples of CRM 114 include volatile and non-volatile memory, fixed and removable media devices, and any suitable memory device or electronic data storage that maintains the executable instructions and supporting data. CRM 114 may include various implementations of random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM (NVRAM), read-only memory (ROM), flash memory, and other memory types in various memory device configurations. CRM 114 does not include propagation signals. CRM 114 may be a solid-state drive (SSD) or a hard disk drive (HDD).

[0030] System processor 110 executes computer-executable instructions that can be stored within CRM 114. As an example, system processor 110 can execute multispectral object detection system 104. Although in Figure 1 The system is shown as separate from CRM 114, but the multispectral object detection system 104 can be stored on and / or executed from CRM 114.

[0031] System processor 110 may receive data from multispectral object detection system 104. As an example, system processor 110 may receive image data or other data (e.g., thermal data or ranging data) from multispectral object detection system 104, vision system 106, or sensing system 108. Similarly, system processor 110 may send configuration data or requests to multispectral object detection system 104, vision system 106, or sensing system 108. System processor 110 may also execute multispectral object detection system 104 to perceive objects in environment 100 and provide data as input to one or more driving systems 116.

[0032] The multispectral object detection system 104 enables an AV 102 to detect objects within the FOV 118 of the object detection system 104 and avoid detected objects if necessary. For example, the multispectral object detection system 104 enables an AV 102 to detect a streetlight 120 on a sidewalk and pedestrians 122 who may be in or near the AV's path. In this way, the AV 102 can control its operating parameters (e.g., speed and / or direction) based on input from the object detection system 104 to avoid pedestrians. The operation and function of the multispectral object detection system 104 will be described in more detail with reference to the following accompanying drawings.

[0033] Driving system 116 may use data from object detection system 104, vision system 106, and / or sensing system 108 to control AV 102. Driving system 116 may include one or both of an autonomous or semi-autonomous driving subsystem that partially relies on data from multispectral object detection system 104 to control the operation of AV 102 (e.g., detecting objects in the driving path of AV 102). Controlling the operation of AV 102 may include controlling AV 102 by direction (e.g., steering, forward, reverse), speed (e.g., acceleration, braking, gear shifting), and other driving functions (e.g., flashing lights, activating the horn, activating hazard or turn signals, or other means to enable safe driving of AV 102).

[0034] For example, a semi-autonomous driving system can provide an alert based on data from the multispectral object detection system 104: a pedestrian (e.g., pedestrian 122) is present and approaching an intersection that AV 102 will enter. Similarly, an autonomous driving system can navigate AV 102 to a specific destination while avoiding pedestrians, vehicles, and other objects that may be in the driving path of AV 102. For example, even if the light in front of AV may be green (e.g., the intersection traffic light is malfunctioning, or pedestrian 122 is distracted and enters the crosswalk when the intersection light is red), driving system 116 can provide braking and / or deceleration based on indications from object detection system 104 that pedestrian 122 has entered the crosswalk in the path of AV 102. In this way, driving system 116 can use information provided by multispectral object detection system 104 (or other sources) to enable AV 102 to navigate safely in environment 100. By identifying multispectral objects in environment 100, the multispectral object detection system 104 enables the AV 102 to drive safely in environment 100, just like other vehicles equipped with lidar or other expensive sensor components, which may be more expensive and complex to implement than multispectral object detection using thermal imaging according to the technology described.

[0035] Figure 2 An example configuration 200 of the system is shown, which can implement a multispectral object detection system 104 using thermal imaging for AV 102. See reference... Figure 1 As described, AV 102 may include an object detection system 104, a vision system 106, a sensing system 108, a system processor 110, a communication device 112, a CRM 114, and a driving system 116.

[0036] The object detection system 104 includes a thermal sensor 202, a non-thermal sensor 204, and an object detection component 206, or related thereto. In this example, the CRM 114 stores instructions for implementing the object detection component 206; however, the object detection component 206 may be a standalone hardware component (e.g., an object detection processor) or a mixture of hardware, software, and / or firmware configured to perform the functions of the object detection component 206, as described herein.

[0037] Object detection component 206 includes an interface with thermal sensor 202, which can be any of various sensors capable of detecting thermal radiation emitted from objects and their surrounding environment and providing thermal data based on the detected thermal radiation. An interface with non-thermal sensor 204 is included in object detection component 206; non-thermal sensor 204 can be any of a variety of sensors capable of detecting non-thermal radiation associated with objects around the sensor and providing non-thermal data based on the detected non-thermal radiation. For example, non-thermal sensor 204 can be an imaging system or camera system (e.g., a stereo camera system) that can use different illumination techniques (e.g., a near-infrared radiation (NIR) camera, a visible light camera, or both visible light and NIR cameras). In some cases, either thermal sensor 202 or non-thermal sensor 204, or either one, can detect thermal radiation or non-thermal radiation respectively over a time period and provide corresponding data for that time period. Furthermore, in some implementations, object detection component 206 may include interfaces with multiple thermal sensors 202 and multiple non-thermal sensors 204.

[0038] In some implementations, the object detection system 104 also includes one or more rangefinders 208 that interface with the object detection component 206. The rangefinder 208 can use any of a variety of ranging techniques.

[0039] Rangefinder 208 may include a transmitter 210 capable of emitting electromagnetic (EM) radiation, and a detector 212 capable of detecting a portion of the EM radiation reflected from an object and providing range data based on the detected EM radiation. For example, rangefinder 208 may be an active NIR rangefinder, wherein transmitter 210 emits NIR radiation, and detector 212 detects a portion of the NIR radiation reflected back from an object within the field of view (FOV) of rangefinder 208 over a period of time. Rangefinder 208 can provide dynamic range data based on the detected portion of the NIR radiation reflected back from the object over the time period. Rangefinder 208 can thus detect objects or potential objects within its FOV.

[0040] The transmitter 210 may also serve as an illumination source for the non-thermal sensor 204. For example, the rangefinder 208 may be an active NIR rangefinder, and the non-thermal sensor 204 may be an active NIR camera (or an active NIR stereo camera system) that includes a narrowband filter 214 (e.g., a narrowband filter that allows NIR radiation to pass through). A portion of the NIR radiation emitted by the active NIR rangefinder is reflected from an object in the rangefinder's field of view (FOV) back to the object detection system 104. The narrowband filter 214 allows this emitted NIR radiation to pass through to reach the active NIR camera, which can provide non-thermal data (e.g., NIR data or NIR image data).

[0041] Because the narrowband filter 214 virtually prevents other radiation (e.g., visible light) from passing through, the non-thermal sensor 204, when configured as an NIR camera, can provide information about objects within the field of view (FOV) with reduced ambiguity. For example, as an active NIR camera, the non-thermal sensor 204 does not need to process large amounts of information about other visible features that might otherwise be included in images based on a broader spectrum (e.g., including visible light), such as shadows or background objects. In this way, the object detection system 104 can have reduced image processing requirements compared to some typical stereo camera systems, which can improve object detection accuracy without overburdening the vehicle's computational resources.

[0042] Typically, object detection component 206 can detect objects in the field of view (FOV) of both the thermal sensor and another sensor (e.g., FOV 118) based on one or more of thermal data, non-thermal data, or distance data. Furthermore, object detection component 206 can determine the distance between the other sensor and the object based on one or more of thermal data, non-thermal data, or distance data.

[0043] Figure 3A schematic diagram of an example object detection system 302, generally shown at 300, which can be implemented in example AV 304, is illustrated. The example object detection system 302 includes a thermal sensor 306 and two non-thermal sensors 308-1 and 308-2, each including narrowband filters 310-1 and 310-2 (or associated with them). The example object detection system 302 also includes an object detection component 312 and three rangefinders 314-1, 314-2, and 314-3. The various components of the example object detection system 302 are combined to implement a field of view (FOV) 316, which represents the area from which the example object detection system 302 can receive reflected radiation to detect objects (e.g., streetlight 120 or pedestrian 122). For example, rangefinders 314-1, 314-2, and 314-3 may be NIR rangefinders. The non-thermal sensors 308-1 and 308-2 can be an NIR camera, a visible light camera, or one NIR camera and one visible light camera. In an implementation with at least one NIR camera, the rangefinders 314-1, 314-2, and 314-3 can serve as illumination for the NIR cameras (multiple).

[0044] refer to Figure 2 Object detection component 206 can receive data over time (e.g., thermal data from thermal sensor 202, non-thermal data from non-thermal sensor 204, or other data from another sensor) and detect objects in the field of view (FOV) based on this data. For example, based on one or both of the thermal and non-thermal data, object detection component 206 can detect objects in the FOV 118 of object detection system 104 and distinguish between background and object to determine whether the detected object is one that needs to be avoided in the driving path of AV 102. Object detection component 206 can also communicate with AV 102 to enable AV 102 to avoid the detected objects (e.g., communicating with a navigation or driving system, as described herein). Figure 2 (and Figure 1 As shown in the diagram, object detection component 206 is integrated with AV 102. In other implementations, object detection component 206 may be executed remotely from AV 102 and at least partially on a remote processor (e.g., cloud-based or server-based) or on another processor of another AV.

[0045] The thermal data received over time can be sequential temperature difference data, indicating the difference between the amount of thermal radiation emitted from the object and the amount of thermal radiation emitted from the background surrounding the object over a time period. Object detection component 206 can determine the temperature gradient between the object and the background over that time period, at least based on the sequential temperature difference data. The temperature gradient can be further considered as the gradient of the object's edge. Furthermore, object detection component 206 can detect objects in the FOV (e.g., FOV 118) of both thermal sensor 202 and non-thermal sensor 204, at least based on dynamic NIR distance data and the temperature gradient. For example, based on dynamic NIR distance data, object detection component 206 can determine the distance of an object (or potential object) to AV 102. Using the thermal data from thermal sensor 202, object detection component 206 can use this temperature gradient to detect objects in FOV 118 or verify that potential objects (e.g., those detected by rangefinder 208) are real objects that need to be avoided by AV 102.

[0046] In some implementations, the temperature gradient can be determined using at least dynamic NIR distance data, and this temperature gradient can be determined by determining the average rate of change of thermal radiation emitted from the object and from the background surrounding the object over a time period. The average rate of change of thermal radiation emitted from the object and from the background surrounding the object over the time period can then be compared. The object detection component 206 can verify the presence of this object (and that it is an object that should be avoided) based on the comparison indicating that the average rate of change of thermal radiation emitted from the object over the time period has decreased relative to the average rate of change of thermal radiation emitted from the background.

[0047] In some cases, the object detection component 206 can determine a temperature gradient when the difference between the temperature of the object and the temperature of the background surrounding the object is a few degrees Celsius or a fraction of a degree Celsius. For example, the temperature difference could be between approximately 0.25 degrees Celsius and approximately 6 degrees Celsius.

[0048] consider Figure 4An example object detection scenario 400 using object detection system 104 is shown. In example scenario 400, vehicle 402 is within the field of view (FOV) 404 of object detection system 104. When there is relative movement between AV 102 and vehicle 402 (e.g., AV 102 moves toward vehicle 402 or vehicle 402 moves across FOV 404), thermal radiation emitted from different parts of vehicle 402 will be sensed over time, depending on the spot size 406 of thermal sensor 202. If spot size 406 is relatively larger than vehicle 402 (e.g., as shown by dashed dot 408), the average rate of change in the amount of thermal radiation emitted from the object over a time period will include a relatively large portion of the thermal radiation emitted from the background. However, when spot size 406 is correctly calibrated (as shown), thermal sensor 202 detects more thermal radiation from vehicle 402. The effective spot size depends on several factors, including the sampling rate and resolution of thermal sensor 202, and the relative speed of the object.

[0049] Furthermore, as the sensor detection area (e.g., spot size 406) moves across the object, thermal radiation from the background is detected when this sensor detection area moves across the edge of the vehicle 402. In this way, the thermal sensor 202 can act as an edge detector. For example, the thermal sensor 202 detects all or most of the background thermal radiation until it detects the vehicle 402. The thermal sensor 202 then detects thermal radiation from both the vehicle 402 and the background, and subsequently detects most of the thermal radiation from the vehicle 402, which can define the edge of the vehicle 402. Combined with dynamic distance data from the rangefinder 208 (which can be used to scan the FOV 404 to obtain potential objects), the average rate of change of the amount of thermal radiation emitted from the background and the vehicle 402 can provide a robust object detection algorithm.

[0050] Therefore, as shown by the multiple dashed circles 410, as the signals from thermal sensor 202 (e.g., signals based on detected thermal energy from vehicle 402) are sequentially combined on vehicle 402, the reflected portion decreases. The rate of change of the thermal signal also tends to stabilize compared to the absence of an object. For example, when an object is present and thermal sensor 202 detects thermal radiation emitted from that object, the rate of change of the amount of thermal radiation is stable because the amount of thermal radiation emitted from that object varies less compared to the amount of thermal radiation from the background, which may contain multiple objects emitting different amounts of thermal radiation. The stability of the signal rate of change thus indicates the presence of an object (with vertical properties), such as vehicle 402. Using a moving average of the sequential rate of change signals, along with low-pass filtering, false alarms about objects (e.g., indicating an object when it is not actually present) will be minimized. The ratio of the rate of change (or derivative) of the thermal signal to the signal from non-thermal sensor 204 (e.g., NIR signal, visible light signal, etc.) can produce additional algorithmic stability.

[0051] In some implementations, differential delay can be used to compare sequential data from thermal sensor 202. Consider... Figure 5 , Figure 5 This is a flowchart illustrating an example process 500 of using object detection system 104 to detect objects using differential time delay and temperature gradient.

[0052] In example process 500, instead of using thermal sensor 202 solely for thermal imaging, the described technique uses the temperature difference between sequential (time-based) thermal measurements of the potential object and similarly sequential measurements of background thermal radiation to aid object detection. For this example, it is assumed that each time-sequential thermal measurement can be considered a thermal "snapshot," and then as measurements are taken from one snapshot to another, a thermal difference exists at each adjacent pixel. The difference between pixels at the next snapshot describes the difference from the previous snapshot and thus describes the temperature gradient. By calculating the root mean square (RMS) of this gradient, the object and its potential movement relative to the background reference can be determined.

[0053] At 502, object detection component 206 reads thermal data from thermal sensor 202 and non-thermal data from non-thermal sensor 204 (e.g., one or more non-thermal sensors, such as NIR sensors or visible light sensors). At 504, a low-pass filter is applied to the data. At 506, a time delay is applied to the thermal data. This time delay is based on a differential calculation at 508, which is based on the speed of AV 102 and the distance between the coverage areas of thermal sensor 202 and the non-thermal sensor(s) 204. At 510, object detection component 206 calculates the RMS of the difference between a snapshot based on thermal data and the same snapshot based on non-thermal data (e.g., visible light data and / or NIR data). At 512, the RMS value is compared to a threshold. The RMS threshold can be based on the temperature difference over a time interval, for example, 0.1 degrees per second. Therefore, a sample might initially be detected as a difference of 0.01 degrees over a 100-millisecond time interval. This threshold can be compared over multiple time periods to confirm the consistency of the object's speed and relative distance with the background. If this threshold is maintained at several time periods and locations (e.g., 0.1 degrees in 1 second and 1 degree in 10 seconds), the RMS temperature difference threshold can be continuously maintained and indicates the presence of the object relative to its direct or relative background.

[0054] At 514, if the RMS value does not exceed the threshold, the object detection component 206 determines that no object exists, and the process returns to operation 502.

[0055] At 516, if the RMS value exceeds a threshold, the object detection component 206 merges the object location prediction based on thermal data and the object location prediction based on non-thermal data (e.g., using sensor fusion technology). At 518, the object detection component 206 updates the object location (e.g., by providing the merged location data to the driving system 116), and the process returns to operation 502.

[0056] In this scenario, differential measurement highlights a smaller difference between the background thermal measurement and the object's thermal measurement compared to using thermal imaging alone. This provides robust detection capability even when the temperature difference between the object and the background is relatively small. In other implementations, the object detection system 104 can use other techniques to detect objects. For example, the object detection component 206 can use image data from one or both of the thermal sensor 202 and the non-thermal sensor 204 to detect objects.

[0057] consider Figure 6This illustrates an example object detection scenario 600 using the example object detection system 602. In example scenario 600, two objects, namely a street lamp 604 and a pedestrian 606, are within the field of view 608 of the example object detection system 602. The example object detection system 602 includes a thermal sensor 610 and two active NIR cameras 612-1 and 612-2 (non-thermal sensor 612), each including narrowband filters 614-1 and 614-2. The example object detection system 602 also includes an object detection component 616 and three active NIR rangefinders 618-1, 618-2, and 618-3, which can be used as an illumination system for the NIR cameras 612-1 and 612-2, as described above (e.g., using narrowband filters 614-1 and 614-2 to filter radiation reflected from the street lamp 604 and the pedestrian 606). Various components of the example object detection system 602 are combined to implement a field of view (FOV) 608, which represents an area from which the example object detection system 602 can receive reflected radiation to detect objects (e.g., streetlight 604 or pedestrian 606). In other implementations, the example object detection system 602 may include any number of the described or illustrated elements.

[0058] In example scenario 600, active NIR cameras 612-1 and 612-2 can detect a portion (e.g., non-thermal radiation associated with streetlight 604 and pedestrian 606) of the NIR radiation emitted from active NIR rangefinders 618-1, 618-2, and 618-3 and reflected from streetlight 604 and pedestrian 606. Active NIR cameras 612-1 and 612-2 provide non-thermal data based on the NIR radiation reflected from streetlight 604 and pedestrian 606, which can be NIR image data. Similarly, thermal data provided by thermal sensor 610 can be thermal image data. Object detection component 616 can determine NIR images of streetlight 604 and pedestrian 606 based on NIR image data and thermal images of streetlight 604 and pedestrian 606 based on thermal image data. Furthermore, the image detection component 616 can detect streetlights 604 and pedestrians 606 in FOV 608 based on pattern matching technology and using the NIR image and thermal image of the object.

[0059] For example, consider detailed views 600-1 and 600-2, which depict NIR and thermal images of streetlight 604 and pedestrian 606, respectively. Object detection component 616 can use pattern matching techniques on one or both images to determine that one object is streetlight 604 (which may not need to be avoided since it is not on the road) and the other object is pedestrian 606. Furthermore, since pedestrian 606 is present in the driving path of AV 102, object detection component 616 can provide location updates or other information to AV 102 (e.g., to driving system 116) so that AV 102 can navigate safely.

[0060] Back Figure 2 In some implementations, the object detection system 104 (or 302 or 602) can also be used to determine: the distance between the detected object and the object detection system 104, as well as the size and / or relative motion of the detected object.

[0061] For example, in an implementation of object detection system 104, rangefinder 208 is an active NIR rangefinder, transmitter 210 can emit NIR radiation, and detector 212 can detect a portion of the NIR radiation reflected from the object over a time period. Active NIR rangefinder 208 can provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object. Further, transmitter 210 can act as an illumination system for non-thermal sensor 204, and the object-related non-thermal radiation can be a portion of the NIR radiation emitted from rangefinder 208 and reflected from the object. Non-thermal sensor 204 can be one or more active NIR cameras that can provide non-thermal data based on the NIR radiation reflected from the object. In this example, non-thermal data can be NIR image data, while thermal data can be thermal image data.

[0062] Object detection component 206 can determine the distance between the NIR camera (or object detection system 104) and the object based on stereo image processing techniques. For example, object detection component 206 can use corresponding time-ordered images extracted from thermal image data and NIR image data. Consider... Figure 6 Detailed views 600-1 and 600-2 show NIR and thermal images of street lamp 604 and pedestrian 606, respectively. Since the distance between thermal sensor 612-1 and non-thermal sensor 610 is known, and the two images were taken from different angles (shown by the different positions of street lamp 604 and pedestrian 606 in the detailed views), the object detection processor can use stereo image processing techniques on the images to determine the distance between the NIR camera and street lamp 604 and pedestrian 606.

[0063] In some cases, because NIR cameras can be configured with a relatively wider / longer baseline (compared to some typical stereo camera systems), NIR cameras may be able to perform ranging over longer distances. NIR camera pairs can therefore also be used, or alternatively, to validate or improve distance measurements performed by the thermal sensor 202.

[0064] In some implementations, object detection component 206 may also determine distance using, for example, time-of-flight techniques based on dynamic NIR distance data provided by rangefinder 208. In other implementations, object detection component 206 may also determine distance based on both image data (e.g., using stereo image processing techniques) and dynamic distance data (e.g., using time-of-flight techniques).

[0065] Additionally or alternatively, object detection component 206 (or another processor) may use distance data (e.g., dynamic NIR distance data) provided by rangefinder 208 to perform real-time calibration of the active NIR stereo camera system (e.g., non-thermal sensor 204) to address slight misalignment of the NIR camera over time. For example, object detection component 206 may compare the distance determined using the distance data with the distance determined using the image data and adjust or calibrate camera parameters as necessary.

[0066] Furthermore, the object detection system 104 (e.g., the example implementation of the object detection system 104 described above, wherein the rangefinder 208 is an active NIR rangefinder, and the non-thermal sensor is one or more active NIR cameras) can be used to determine the size and motion of objects within the FOV of the object detection system 104. In some implementations, the object detection component 206 can use the physical properties of one or more object detection systems 104, along with data from the rangefinder 208 and data from the NIR cameras, to determine the size of detected objects (e.g., pedestrians or other vehicles). For example, the size and motion of an object can be determined using: the geometric configuration of the transmitter 210 and detector 212 of the active NIR rangefinder 208; the relative target distance between one or more active NIR cameras (e.g., the non-thermal sensor 204) and the object; and the amount of frame-to-frame overlap between sequential images extracted from NIR image data; and / or the refresh rate of one or more active NIR cameras. The relative target distance based on the distance data can be determined using dynamic NIR distance data.

[0067] In some implementations, the object detection system 104 (e.g., the example implementation of the object detection system 104 described above, wherein the rangefinder 208 is an active NIR rangefinder and the non-thermal sensor is one or more active NIR cameras) can also use data from the rangefinder 208 and the thermal sensor 202 to determine the motion of the object in the FOV of the object detection system 104. As noted above (e.g., at least referring to...) Figure 2-4 The object detection component 206 can determine the temperature gradient across the object's edge based on thermal data from the thermal sensor 202. Based on the temperature gradient and dynamic NIR distance data, the object detection component 206 can also determine the object's motion.

[0068] For example, stationary objects with thermal differences from their background environment tend to have more blurred edges compared to typical objects based on temperature difference and absolute temperature. In contrast, when relative motion is present (e.g., the object is moving or the sensor is moving toward the object), the object tends to have sharper edges (e.g., temperature gradients). Rangefinder 208 can detect so-called “potential targets” within the FOV, and using temperature gradients allows the system to distinguish with much higher confidence whether an object has a vertical range and is moving relative to the sensor, either getting closer or farther away, or moving across the FOV.

[0069] Example Method

[0070] Figure 7-1 and Figure 7-2 An example method 700 for implementing a multispectral object detection system utilizing thermal imaging is illustrated. Method 700 is shown as a set of operations (or actions) performed, but is not necessarily limited to the order or combination of operations shown herein. Furthermore, any one or more of the operations may be repeated, combined, or recombined to provide other methods. References may be made in the sections discussed below. Figure 1 Example environment 100 and Figures 1 to 6 The entities detailed herein are referenced for illustrative purposes only. The described techniques are not limited to being performed by one or more entities. Furthermore, throughout the description of method 700, reference to actions performed by object detection system 104 may include actions performed by object detection systems 104, 302, or 602, which include corresponding sub-components of object detection systems 104, 302, or 602 (e.g., thermal sensors 202, 306, or 610; non-thermal sensors 204, 308, or 612; object detection components 206, 312, or 616; or rangefinders 208, 314, or 618). Moreover, reference to actions performed by an example sub-component (e.g., thermal sensor 202) may include actions performed by another example sub-component (e.g., thermal processor 610).

[0071] At 702, the object detection processor of the object detection system receives thermal data from a thermal sensor. The thermal data can be based on thermal radiation detected by the thermal sensor and emitted from an object within the thermal sensor's field of view (FOV), from the background surrounding the object, or from both the object and the background. For example, object detection components 206, 312, or 616 can receive thermal data from thermal sensors 202, 306, or 610 based on thermal radiation detected from an object (e.g., streetlight 120 (or 604) or pedestrian 122 (or 606)). The thermal data can be any of a variety of data types, including, for example, thermal image data or sequential differential thermal data. Sequential differential thermal data can indicate changes in thermal radiation emitted from the object and / or from the background surrounding the object over a time period.

[0072] At 704, the object detection processor receives non-thermal data from a non-thermal sensor. The non-thermal data can be based on non-thermal radiation associated with an object and detected by the non-thermal sensor. For example, object detection components 206, 312, or 616 can receive non-thermal data from non-thermal sensors 204, 308, or 612. The non-thermal data can be any of a variety of data types, including, for example, non-thermal image data (e.g., NIR image data or visible light images).

[0073] At 706, the object detection processor detects an object based on one or both of thermal data and non-thermal data. For example, object detection components 206, 312, or 616 may detect an object based on one or both of thermal data from thermal sensors 202, 306, or 610 or non-thermal data from non-thermal sensors 204, 308, or 612. In this example, detecting an object may refer to one or more operations performed by the object detection sensor to determine whether received thermal and / or non-thermal data indicates the presence of an object within the sensor's field of view (FOV).

[0074] At 708, the object detection processor determines the distance between the object detection system and the object based on one or both of thermal data and non-thermal data. For example, object detection components 206, 312, or 616 may determine the distance between the object detection system 104 and the object based on one or both of thermal data from thermal sensors 202, 306, or 610 or non-thermal data from non-thermal sensors 204, 308, or 612.

[0075] In some implementations, the non-thermal sensor may be one or more active NIR cameras, including or associated with a transmitter capable of emitting NIR radiation. The NIR camera may provide NIR image data based on the reflection of at least a portion of the NIR radiation emitted from the object and detected by the active NIR camera. Object detection component 206 may determine a thermal image and an NIR image of the object based on the thermal image data and the NIR image data, respectively. Image detection component 206 may detect the object using the NIR image and the thermal image of the object based on pattern matching techniques.

[0076] Object detection component 206 can also determine the distance between the active NIR camera and the object based on stereo image processing techniques using corresponding time-ordered images extracted from thermal data and NIR data, as described above (e.g., at least referring to...). Figure 2 and Figure 6 In some implementations, the object detection processor can receive dynamic distance data from an active near-infrared (NIR) rangefinder, as described above. The active NIR rangefinder may include: a transmitter capable of emitting NIR radiation, and a detector capable of detecting a portion of the NIR radiation reflected from the object over a time period. The active NIR rangefinder can then provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object, and use this dynamic NIR distance data based on time-of-flight techniques to determine the distance between the NIR camera and the object. Additional optional operations of method 700 are shown in… Figure 7-2 In the middle, in such Figure 7-1 The operation shown is at point "A".

[0077] Figure 7-2 Additional optional operations at operation "A" are shown in method 700. At 710, the object detection processor receives dynamic NIR distance data from an active near-infrared (NIR) rangefinder. The active NIR rangefinder may include a transmitter configured to emit NIR radiation. The active NIR rangefinder may also include a detector configured to: detect a portion of the NIR radiation reflected from the object over a time period, and provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object. For example, object detection components 206, 312, or 616 may receive dynamic NIR distance data from rangefinders 208, 314, or 618.

[0078] At 712, the object detection processor receives sequential temperature difference data (e.g., thermal data) from the thermal sensor during this time period. For example, object detection components 206, 312, or 616 may receive sequential temperature difference data from thermal sensors 202, 306, or 610.

[0079] At 714, the object detection processor determines the temperature gradient between the object and the background over that time period. This temperature gradient may be based at least on sequential temperature difference data. For example, object detection components 206, 312, or 616 may determine the temperature gradient based on sequential temperature difference data received from thermal sensors 202, 306, or 610.

[0080] At 716, the object detection processor verifies the presence of an object based at least on dynamic NIR distance data and a temperature gradient. For example, object detection components 206, 312, or 616 can use a temperature gradient to verify that a potential object detected using dynamic NIR distance data is an actual object (e.g., an object that AV 102 should avoid). In some implementations, determining the temperature gradient includes using object detection components 206, 312, or 616 to determine the average rate of change of the amount of thermal radiation emitted from the object over a given time period, and the average rate of change of the amount of thermal radiation emitted from the background surrounding the object over the same time period. Object detection components 206, 312, or 616 then compare the average rates of change of thermal radiation emitted from the object and from the background surrounding the object over the given time period. Object detection components 206, 312, or 616 can then verify the presence of an object based on a comparison indicating that the average rate of change of thermal radiation emitted from the object over the given time period has decreased relative to the average rate of change of thermal radiation emitted from the background.

[0081] Example

[0082] Examples are provided in the following sections.

[0083] Example 1: A system comprising: a thermal sensor configured to detect thermal radiation emitted from an object in the field of view (FOV) of a thermal processor and from a background surrounding the object, and to provide thermal data based on the thermal radiation; a non-thermal sensor configured to detect non-thermal radiation associated with the object and to provide non-thermal data based on the non-thermal radiation; and a processor configured to receive the thermal data and the non-thermal data; and to detect the object based on at least one of the thermal data and the non-thermal data.

[0084] Example 2: The system of Example 1, wherein: the system further includes an active near-infrared (NIR) rangefinder comprising: a transmitter configured to emit NIR radiation; and a detector configured to detect a portion of NIR radiation reflected from a docking point over a time period and to provide dynamic NIR distance data based on the detected portion of NIR radiation reflected from the object; a thermal sensor further configured to provide thermal data over the time period; the thermal data including sequential temperature difference data indicating the difference between thermal radiation emitted from the object and thermal radiation emitted from the background surrounding the object over the time period; and a processor further configured to: detect an object in the field of view (FOV) based on the dynamic NIR distance data; determine a temperature gradient between the object and the background over the time period, the temperature gradient being based at least on the sequential temperature difference data; and verify the presence of an object based at least on the dynamic NIR distance data and the temperature gradient.

[0085] Example 3: A system of any of the foregoing examples, wherein the processor is further configured to verify the presence of an object by: detecting the object based on dynamic NIR distance data and a temperature gradient by: determining the temperature gradient by: determining the average rate of change of the amount of thermal radiation emitted from the object during the time period; determining the average rate of change of the amount of thermal radiation emitted from the background surrounding the object during the time period; comparing the average rates of change of the amount of thermal radiation emitted from the object and from the background surrounding the object during the time period; and verifying the presence of the object based on the result of the comparison, which indicates that the average rate of change of the amount of thermal radiation emitted from the object during the time period has decreased relative to the average rate of change of the amount of thermal radiation emitted from the background.

[0086] Example 4: A system like any of the preceding examples, where the difference between the temperature of the object and the temperature of the background surrounding the object is between approximately 0.25 degrees Celsius and approximately 6 degrees Celsius.

[0087] Example 5: A system of any of the preceding examples, wherein: the system further includes an active near-infrared (NIR) rangefinder comprising: a transmitter configured to emit NIR radiation; non-thermal radiation associated with an object comprising a portion of NIR radiation emitted from the active NIR rangefinder and reflected from the object; a non-thermal sensor comprising one or more active near-infrared (NIR) cameras configured to provide non-thermal data based on NIR radiation reflected from the object; the non-thermal data comprising NIR image data; the thermal data comprising thermal image data; and a processor further configured to: determine an NIR image of an object based on the NIR image data; determine a thermal image of an object based on the thermal image data; and detect an object in the field of view (FOV) of the thermal sensor and another sensor using a pattern matching technique and the NIR image and thermal image of the object.

[0088] Example 6: A system of any of the preceding examples, wherein: the system further includes an active NIR rangefinder, the NIR rangefinder including at least a transmitter configured to emit NIR radiation; non-thermal radiation associated with the object includes a portion of NIR radiation emitted from the active NIR rangefinder and reflected from the object; the non-thermal sensor includes one or more active NIR cameras configured to provide non-thermal data based on NIR radiation reflected from the object; the non-thermal data includes NIR image data; the thermal data includes thermal image data; and the processor is further configured to determine the distance between the system and the object based on stereo image processing techniques, using corresponding time-ordered images extracted from the thermal image data and the NIR image data.

[0089] Example 7: A system of any of the foregoing examples, wherein: the system further includes an active NIR rangefinder comprising: a transmitter configured to emit NIR radiation; and a detector configured to detect a portion of the NIR radiation reflected from an object over a time period and to provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object; and a processor further configured to: determine the distance between the NIR rangefinder and the object using the dynamic NIR distance data based on time-of-flight and technology.

[0090] Example 8: A system of any of the foregoing examples, wherein: the system further includes an active NIR rangefinder comprising: a transmitter configured to emit NIR radiation; a detector configured to detect a portion of the NIR radiation reflected from an object over a time period and to provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object; non-thermal radiation associated with the object including a portion of the NIR radiation emitted from the active NIR rangefinder and reflected from the object; another sensor including one or more active NIR cameras configured to provide non-thermal data based on the NIR radiation reflected from the object; non-thermal data packet NIR image data; thermal data including thermal image data; and a processor further configured to determine the distance between the system and the object based on: stereo image processing techniques using corresponding time-ordered images extracted from the thermal image data and the NIR image data; and time-of-flight techniques using the NIR distance data.

[0091] Example 9: A system of any of the preceding examples, wherein: the system further includes an active NIR rangefinder comprising: a transmitter configured to emit NIR radiation; and a detector configured to detect a portion of the NIR radiation reflected from an object over a time period and to provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object; non-thermal radiation associated with the object includes a portion of the NIR radiation emitted from the active NIR rangefinder and reflected from the object; a non-thermal sensor comprising one or more active NIR cameras configured to provide non-thermal data based on the NIR radiation reflected from the object; the non-thermal data includes NIR image data; and a processor further configured to determine the size of the object based on one or more of the following: the geometric configuration of the transmitter and detector of the active NIR rangefinder; the relative target distance between the system and the object, the relative target distance being based on the dynamic NIR distance data; the amount of frame-to-frame overlap between sequentially extracted images from the NIR image data; or the refresh rate of one or more active NIR cameras.

[0092] Example 10: A system of any of the preceding examples, wherein: the system further includes an active NIR rangefinder comprising: a transmitter configured to emit NIR radiation; and a detector configured to detect a portion of the NIR radiation reflected from an object over a time period and to provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object; non-thermal radiation associated with the object includes a portion of the NIR radiation emitted from the active NIR rangefinder and reflected from the object; a non-thermal sensor comprising one or more active NIR cameras configured to provide non-thermal data based on the NIR radiation reflected from the object; the non-thermal data including NIR image data; and a processor further configured to determine the motion of the object relative to the system in the FOV of the thermal sensor based on one or more of the following: the geometrical configuration of the transmitter and detector of the active NIR rangefinder; the relative target distance between the system and the object, the relative target distance being based on the distance data; the amount of frame-to-frame overlap between sequentially extracted images from the NIR image data; or the refresh rate of one or more active NIR cameras.

[0093] Example 11: A system of any of the preceding examples, wherein: the system further includes an active NIR rangefinder comprising: a transmitter configured to emit NIR radiation; and a detector configured to detect a portion of the NIR radiation reflected from an object over a time period and to provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object; and a processor further configured to: determine a temperature gradient across an object edge based on thermal data; and determine the motion of the object relative to the system in the FOV of the thermal sensor based on one or more of the following: dynamic NIR distance data; or temperature gradient.

[0094] Example 12: A system comprising: a thermal sensor configured to detect thermal radiation emitted from an object and provide thermal data based on the thermal radiation; a transmitter configured to emit electromagnetic (EM) radiation; a detector configured to detect a portion of the EM radiation reflected from the object and provide distance data based on the detected EM radiation; a non-thermal sensor configured to detect non-thermal radiation associated with the object and provide non-thermal data based on the non-thermal radiation; and a processor configured to: detect an object in the field of view (FOV) of the thermal sensor and another sensor based on one or more of the thermal data, non-thermal data, or distance data; and determine the distance between the system and the object based on one or more of the thermal data, non-thermal data, or distance data.

[0095] Example 13: The system of any of the preceding examples, wherein the EM radiation includes near-infrared (NIR) radiation.

[0096] Example 14: A system of any of the preceding examples, wherein the non-thermal sensor includes: one or more active NIR cameras configured to use NIR radiation emitted from the transmitter as an illumination source; one or more visible light cameras; or at least one visible light camera and at least one NIR camera configured to use NIR radiation emitted from the transmitter as an illumination source.

[0097] Example 15: A method comprising: receiving thermal data from a thermal sensor by a processor, the thermal data being based on thermal radiation detected by the thermal sensor and emitted from an object in the field of view (FOV) of the thermal sensor, from a background surrounding the object, or from both the object and the background; receiving non-thermal data from a non-thermal sensor by the processor, the non-thermal data being based on non-thermal radiation associated with the object and detected by the non-thermal sensor; detecting the object by the processor based on one or both of the thermal data and the non-thermal data; and determining a distance between an object detection system and the object by the processor based on at least one of the thermal data or the non-data.

[0098] Example 16: A method of any of the preceding examples, wherein: the thermal data includes sequential temperature difference data indicating changes in thermal radiation emitted from an object and from the background surrounding the object over a time period, and the method further includes: receiving dynamic NIR distance data from an active near-infrared (NIR) rangefinder by a processor, the active NIR rangefinder including: a transmitter, the reflector being configured to emit NIR radiation, and a detector being configured to detect a portion of the NIR radiation reflected from the object over a time period and provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object; receiving sequential temperature difference data over the time period from a thermal sensor; determining a temperature gradient between the object and the background over the time period by the processor, the temperature gradient being based at least on the sequential temperature difference data; and verifying the presence of the object by the processor at least based on the dynamic NIR distance data and the temperature gradient.

[0099] Example 17: A method of any of the foregoing examples, wherein verifying the presence of an object based on dynamic NIR distance data and a temperature gradient further comprises: detecting the object based on the dynamic NIR distance data; determining the temperature gradient by: determining the average rate of change of the amount of thermal radiation emitted from the object during the time period; determining the average rate of change of the amount of thermal radiation emitted from the background surrounding the object during the time period; and comparing the average rate of change of the amount of thermal radiation emitted from the object and the background surrounding the object during the time period; and verifying the presence of the object based on the comparison result indicating that the average rate of change of the amount of thermal radiation emitted from the object during the time period has decreased relative to the average rate of change of the amount of thermal radiation emitted from the background.

[0100] Example 18: A method of any of the preceding examples, wherein: thermal data includes thermal image data; the non-thermal sensor includes one or more active NIR cameras, the one or more active NIR cameras including or associated with an emitter capable of emitting NIR radiation; the non-thermal data includes NIR image data received from the one or more active NIR cameras, the NIR image data being based on the reflection of at least a portion of NIR radiation reflected from an object and detected by the one or more active NIR cameras; and the method further includes: determining an NIR image of an object based on the NIR image data; determining a thermal image of an object based on the thermal image data; and detecting an object using the NIR image of the object and the thermal image of the object based on a pattern matching technique.

[0101] Example 19: A method of any of the preceding examples, wherein: the thermal data includes thermal image data; the non-thermal sensor includes one or more active NIR cameras, the one or more active NIR cameras including or associated with an emitter capable of emitting NIR radiation; the non-thermal data includes NIR image data received from the one or more active NIR cameras, the NIR image data being based on the reflection of at least a portion of NIR radiation reflected from an object and detected by the one or more active NIR cameras over a time period; and the method further includes: determining the distance between the one or more active NIR cameras and the object using corresponding time-ordered images extracted from the thermal data and from the NIR data based on stereo image processing techniques.

[0102] Example 20: The method of any of the foregoing examples further includes: receiving dynamic distance data from an active near-infrared (NIR) rangefinder, the active NIR rangefinder comprising: a transmitter configured to emit NIR radiation, and a detector configured to detect a portion of NIR radiation reflected from an object over a time period and provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object; and using the dynamic NIR distance data based on a time-of-flight technique to determine the distance between the active NIR rangefinder and the object.

[0103] Example 21: A system including means for performing any of the methods in the foregoing examples.

[0104] Example 22: A system of any of the foregoing examples includes at least one processor configured to perform any of the methods of any of the foregoing examples.

[0105] Example 23: A computer-readable storage medium comprising instructions that, when executed, configure a processor to perform any of the methods described in the preceding examples.

[0106] Conclusion

[0107] While various embodiments of the present disclosure have been described in the foregoing description and illustrated in the accompanying drawings, it should be understood that the present disclosure is not limited thereto, but can be practiced in various ways within the scope of the following claims. It will be apparent from the foregoing description that various modifications can be made without departing from the spirit and scope of the present disclosure as defined by the following claims.

Claims

1. A system for object detection, the system comprising: A thermal sensor configured to detect thermal radiation emitted from an object in the field of view (FOV) of the thermal sensor and from the background surrounding the object, and to provide thermal data based on the thermal radiation; A non-thermal sensor, configured to detect non-thermal radiation associated with the object and provide non-thermal data based on the non-thermal radiation; Active near-infrared (NIR) rangefinders, including: A transmitter configured to emit NIR radiation; and A detector configured to: detect a portion of the NIR radiation reflected from the object over a time period, and provide dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object; and Processor, the processor being configured to: Receive the thermal data and the non-thermal data; The object is detected based on at least one of the thermal data or the non-thermal data; Determine the temperature gradient across the edge of the object based on the thermal data; and The size or motion of the object in the FOV of the thermal sensor relative to the system is determined based on one or more of the following: the dynamic NIR distance data or the temperature gradient.

2. The system as described in claim 1, characterized in that: The thermal sensor is further configured to provide the thermal data during the said time period; The thermal data includes sequential temperature difference data, which indicates the difference between thermal radiation emitted from the object and from the background surrounding the object within the time period; and The processor is further configured to: The object in the FOV is detected based on the dynamic NIR distance data; Determine the temperature gradient between the object and the background within the time period, the temperature gradient being based at least on the sequential temperature difference data; as well as The existence of the object is verified based at least on the dynamic NIR distance data and the temperature gradient.

3. The system as described in claim 2, characterized in that, The processor is further configured to verify the presence of the object by at least the dynamic NIR distance data and the temperature gradient: The object is detected based on the dynamic NIR data; The temperature gradient is determined by the following steps: Determine the average rate of change of the amount of thermal radiation emitted from the object during the time period; Determine the average rate of change of the amount of thermal radiation emitted from the background surrounding the object during the time period; as well as The average rate of change of thermal radiation emitted from the object and from the background surrounding the object during the time period is compared. as well as The existence of the object is verified based on the following comparison results: the comparison results indicate that the average rate of change of the amount of thermal radiation emitted from the object during the time period has decreased relative to the average rate of change of the amount of thermal radiation emitted from the background.

4. The system as described in claim 2, characterized in that, The temperature difference between the object and the background temperature surrounding the object is between 0.25 degrees Celsius and 6 degrees Celsius.

5. The system as described in claim 1, characterized in that: The non-thermal sensor includes one or more active near-infrared (NIR) cameras, which are configured to provide the non-thermal data based on the NIR radiation reflected from the object. The non-thermal data includes NIR image data; The thermal data includes thermal image data; and The processor is further configured to: Determine the NIR image of the object based on the NIR image data; Determine the thermal image of the object based on the thermal image data; as well as The object is detected in the FOV of the thermal sensor and the non-thermal sensor using pattern matching technology and the NIR image and thermal image of the object.

6. The system as described in claim 1, characterized in that: The non-thermal sensor includes one or more active NIR cameras configured to provide the non-thermal data based on the NIR radiation reflected from the object; The non-thermal data includes NIR image data; The thermal data includes thermal image data; and The processor is further configured to: determine the distance between the system and the object using corresponding time-ordered images extracted from the thermal image data and the NIR image data, based on stereo image processing technology.

7. The system as described in claim 1, characterized in that: The processor is further configured to: determine the distance between the NIR rangefinder and the object using the dynamic NIR distance data based on time-of-flight technology.

8. The system as described in claim 1, characterized in that: The non-thermal sensor includes one or more active NIR cameras configured to provide the non-thermal data based on the NIR radiation reflected from the object; The non-thermal data includes NIR image data; The thermal data includes thermal image data; and The processor is further configured to determine the distance between the system and the object based on the following: Based on stereo image processing technology, and using corresponding time-sorted images extracted from the thermal image data and the NIR image data; as well as Based on time-of-flight technology and using the dynamic NIR distance data.

9. The system as described in claim 1, characterized in that: The non-thermal sensor includes one or more active NIR cameras configured to provide the non-thermal data based on the NIR radiation reflected from the object; The non-thermal data includes NIR image data; and The processor is further configured to determine the size of the object based on one or more of the following: The geometric configuration of the transmitter and detector of the active NIR rangefinder; The relative target distance between the system and the object, the relative target distance being based on the dynamic NIR distance data; The amount of frame-to-frame overlap between images extracted sequentially from the NIR image data; or The refresh rate of the one or more active NIR cameras.

10. The system as claimed in claim 1, characterized in that: The non-thermal sensor includes one or more active NIR cameras configured to provide the non-thermal data based on the NIR radiation reflected from the object; The non-thermal data includes NIR image data; and The processor is further configured to determine the motion of the object in the FOV of the thermal sensor relative to the system based on one or more of the following: The geometric configuration of the transmitter and detector of the active NIR rangefinder; The relative target distance between the system and the object, the relative target distance being based on the distance data; The amount of frame-to-frame overlap between images extracted sequentially from the NIR image data; or The refresh rate of the one or more active NIR cameras.

11. The system as claimed in claim 1, characterized in that, The non-thermal sensor includes: One or more active NIR cameras, the one or more active NIR cameras being configured to use the NIR radiation emitted from the transmitter as an illumination source; One or more visible light cameras; or At least one NIR camera and at least one visible light camera, wherein the at least one NIR camera is configured to use the NIR radiation emitted from the transmitter as an illumination source.

12. A method for object detection, the method comprising: The processor of the object detection system receives thermal data from a thermal sensor, the thermal data being based on thermal radiation detected by the thermal sensor and emitted from the object in the field of view (FOV) of the thermal sensor and from the background surrounding the object; The processor receives non-thermal data from a non-thermal sensor, the non-thermal data being based on non-thermal radiation associated with the object and detected by the non-thermal sensor; The processor receives dynamic NIR distance data from an active near-infrared (NIR) rangefinder, the active NIR rangefinder comprising: a transmitter configured to emit NIR radiation; and a detector configured to: detect a portion of the NIR radiation reflected from the object over a time period, and provide the dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object. The processor detects the object based on one or both of the hot data and the non-hot data; The processor determines the distance between the object detection system and the object based on at least one of the thermal data or the non-thermal data, wherein the thermal data is used to determine the temperature gradient across the edge of the object; and The processor determines the size or motion of the object in the FOV of the thermal sensor relative to the system based on one or more of the following: the dynamic NIR distance data or the temperature gradient.

13. The method as described in claim 12, characterized in that: The thermal data includes sequential temperature difference data, which indicates changes in thermal radiation emitted from the object and from the background surrounding the object over a period of time, and the method further includes: Receive temperature difference data in the order of the time period from the thermal sensor; The processor determines the temperature gradient between the object and the background within the time period, the temperature gradient being based at least on the sequential temperature difference data; and The processor verifies the existence of the object based at least on the dynamic NIR distance data and the temperature gradient.

14. The method as described in claim 13, characterized in that, Verifying the existence of the object based on the dynamic NIR distance data and the temperature gradient further includes: The object is detected based on the dynamic NIR distance data; The temperature gradient is determined by the following steps: Determine the average rate of change of the amount of thermal radiation emitted from the object during the time period; Determine the average rate of change of the amount of thermal radiation emitted from the background surrounding the object during the time period; and The average rate of change of thermal radiation emitted from the object and from the background surrounding the object during the time period is compared; and The existence of the object is verified based on the following comparison results: the comparison results indicate that the average rate of change of the amount of thermal radiation emitted from the object during the time period has decreased relative to the average rate of change of the amount of thermal radiation emitted from the background.

15. The method as described in claim 12, characterized in that: The thermal data includes thermal image data; The non-thermal sensor includes one or more active NIR cameras, which include or are associated with a transmitter capable of emitting NIR radiation. The non-thermal data includes NIR image data received from the one or more active NIR cameras, the NIR image data being based on the reflection of at least a portion of the NIR radiation reflected from the object and detected by the one or more active NIR cameras; and The method further includes: Determine the NIR image of the object based on the NIR image data; Determine the thermal image of the object based on the thermal image data; and Based on pattern matching technology, the object is detected using the NIR image and the thermal image of the object.

16. The method as described in claim 12, characterized in that: The thermal data includes thermal image data; The non-thermal sensor includes one or more active NIR cameras, which include a transmitter capable of emitting NIR radiation or are associated with a generator capable of emitting NIR radiation. The non-thermal data includes NIR image data received from the one or more active NIR cameras, the NIR image data being based on the reflection of at least a portion of the NIR radiation reflected from the object and detected by the one or more active NIR cameras over a time period; and The method further includes: using stereo image processing technology, using corresponding time-ordered images extracted from the thermal data and the NIR data, to determine the distance between the one or more active NIR cameras and the object.

17. The method as described in claim 12, characterized in that, Further includes: Dynamic distance data is received from an active near-infrared (NIR) rangefinder, the active NIR rangefinder comprising: a transmitter configured to emit NIR radiation; and a detector configured to: detect a portion of the NIR radiation reflected from the object over a time period, and provide the dynamic NIR distance data based on the detected portion of the NIR radiation reflected from the object; and Based on time-of-flight technology, the distance between the active NIR rangefinder and the object is determined using the dynamic NIR distance data.

Images