Early fire detection systems and methods

Abstract

Multispectral imaging and related techniques are provided to detect fire hazards and alerts that are false positive indicators of the fire hazards. The embodiments are directed to systems for identifying pre-fire hazards using thermal images, visual images, and different rules corresponding to temperature configuration levels. The embodiment are also directed to identifying false positive indicators of fire hazards using regions of interest in the thermal and visual images, identifying objects that have hot components which may generate a false positive indicator for a fire hazard, and identifying sun reflections which may generate a false positive indicator for the fire hazard.

Description

[0001] EARLY FIRE DETECTION SYSTEMS AND METHODS

[0002] Niclas Gruseus and Johan Johansson

[0003] CROSS REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority to and the benefit of U. S. Provisional Patent Application No. 63 / 711,665 filed October 24, 2024 and entitled "‘EARLY FIRE DETECTION SYSTEMS AND METHODS," which is incorporated herein by reference in its entirety.

[0005] TECHNICAL FIELD

[0006] The embodiments are directed to early fire detection systems, and more specifically to techniques for analyzing data to detect or prevent fires.

[0007] BACKGROUND

[0008] Early fire detection systems play a key role in increasing safety m many facilities, such as waste facilities, battery recycling, storage facilities and solar panel inverter condition monitoring to mention a few. One way to increase safety is for early fire detection systems to detect fires sooner. However, conventional early fire detection systems suffer from an increasing number of false positive detections when the early fire detection systems attempt to detect fires earlier. Accordingly, it would be desirable to provide techniques for early fire detection systems to analy ze data to detect fires earlier or prevent fires from starting.

[0009] SUMMARY

[0010] The subject technology provides various techniques for determining fire hazards in an early fire detection system that uses thermal and / or visual images and techniques for identifying alerts for the fire hazards that are false positives. Specifically, the subject technology' provides various techniques for identifying pre-fires by applying different temperature threshold levels for generating alerts and identifying long-term and short-term pre fire trends. Additionally, the subject matter technology identifies alerts corresponding to the fires and pre-fires that are false positive alerts by analyzing thermal images at a pixel level, identifying hot objects in the thermal and visual images that are not fires, and identifying reflections from the sun which can be confused with a fire.

[0011] According to an embodiment, the early fire detection system may divide images that include an object into regions of interest (ROIs). For each ROI in the ROIs in the images, the early fire detection system may associate pixels in the ROI with a heat threshold and a hysteresis time, determine, a portion of the pixels m the ROI with heat indicators above the heat threshold and using the hysteresis time, identify areas m the ROI that include the portions of pixels, and generate an alert based on a number of the areas above an area threshold.

[0012] According to an embodiment, the early fire detection system may classify7, using an artificial intelligence model, athermal image as including an object in a scene that includes at least one component that triggers an alert, determine an area in the thermal image that corresponds to the object, suppress temperature indicators corresponding to the area from the thermal image, and determine whether to generate an alert using remaining temperature indicators in the thermal image. The early fire detection system may also classify', using a second artificial intelligence model, a visual image as having the object that includes the at least one component that triggers the alert, determine a second area in the visual image that corresponds to the object, and transform the second area into the thermal image. The early fire detection system may then suppress temperature indicators corresponding to the transformed area from the thermal image.

[0013] According to an embodiment, the early fire detection system may receive thermal images depicting a scene and weather data, determine that the weather data indicates a weather that is sunny or partly sunny at the scene, determine, using the thermal images, a temperature spike at the scene above a temperature threshold, and delay an issuance of an alert by a time period based on the determination that the weather is sunny or partly sunny and the temperature spike. The early fire detection system may further determine, using a visual image, a red, green, blue (RGB) color parameters in the visual image, determine that the RGB color parameters is within an RGB range and further delay the issuance of the alert by the time period based on the determination that the RGB color parameters is within the RGB range.

[0014] According to an embodiment, the early fire detection system may select a configuration associated with temperature threshold levels, wherein temperature threshold levels are associated with actions and hysteresis times, receive a thermal image of a scene, determine that a temperature in the thermal image passed one of the temperature threshold levels and a corresponding hysteresis time, and execute al least one the action corresponding to the one of the temperature threshold levels.

[0015] According to an embodiment, the early fire detection system may receive thermal images of a scene taken at predefined time intervals, divide each thermal image in the thermal images into regions of interest (ROIs). For each ROI in the ROIs in each thermal image, early fire detection system may determine a maximum temperature and an average temperature, determine a difference between the maximum temperature and the average temperature, and generate an alert based on the difference.

[0016] According to an embodiment, the early fire detection system may identify a temperature trend corresponding to a pre-fire based on a temperature increase rate, generate an alert indicative of the pre-fire, determine that an alert is not a false positive alert based on an increase in temperature in the temperature trend and an ambient temperature, or determine that the alert is not the false alert by analyzing regions of interest (ROIs) in a visual image and determine that energy is increasing in at least some ROIs in the visual image.

[0017] The scope of the disclosure is defined by the claims, which are incorporated into this section by reference, A more complete understanding of embodiments of the present disclosure will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by a consideration of the following detailed description of one or more embodiments. Reference will be made to the appended sheets of drawings that will first be described briefly.

[0018] BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 illustrates a diagram of a multispectral imaging system in accordance with an embodiment of the subject technology.

[0019] FIG. 2 illustrates a diagram of fixed-mount and mobile platforms employing a multispectral imaging system in accordance with an embodiment of the subject technology.

[0020] FIG. 3 illustrates a diagram of a multispectral imaging device for a multispectral imaging system in accordance with an embodiment of the subject technology. FIG. 4 illustrates a block diagram of an image capture component in accordance with an embodiment of the disclosure.

[0021] FIG. 5 illustrates a block diagram of an early fire detection system 350 in accordance with an embodi ment of the disclosure,

[0022] FIG. 6 is a diagram illustrating two thermal images in accordance with an embodiment of the disclosure.

[0023] FIG. 7 is a flowchart of a method for identifying a fire hazard, according to some embodiments.

[0024] FIG. 8 is a diagram illustrating an example thermal image and visual image, in accordance with the embodiment,

[0025] FIG. 9 is a flowchart of a method for identifying a fire hazard, according to some embodiments.

[0026] FIG. 10 illustrates an example visual image of a scene and a corresponding temperature graph, according to some embodiments.

[0027] FIG. 11 illustrates images having various RGB values that indicate “sky"’ and fire colors, according to some embodiments.

[0028] FIG. 12 is a flowchart of a method for identifying a fire hazard, according to some embodiments.

[0029] FIG. 13 is a diagram of a graph illustrating two different configurations for threshold levels.

[0030] FIG. 14 is a flowchart of a method for identifying a fire hazard, according to some embodiments.

[0031] FIG, 15 illustrates an example thermal image and temperature trend graph, according to the embodiments of the disclosure.

[0032] FIG. 16 is a flowchart of a method for identifying a fire hazard, according to the embodiments of the disclosure. FIG. I 7 illustrates an example visual image and temperature trend graph that corresponds to a scene, according to the embodiments of the disclosure.

[0033] FIG. 18 is a flowchart of a method for identifying a fire hazard, according to the embodiments of the disclosure.

[0034] FIG. 19 is a simplified diagram illustrating the neural network structure, according to the embodiments of the disclosure.

[0035] Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures.

[0036] DETAILED DESCRIPTION

[0037] Multispeclral imaging systems (e.g., utilizing imaging data in the thermal, visible, ultraviolet, or other spectrums or combinations thereof) include an early fire detection system for detecting fire hazards in early stages. The early fire detection system, for example, may detect fires using thermal images and / or visual images. Additionally, the early fire detection system may be configured using multiple temperature level configurations, that, when reached, may initiate configurable actions for detecting a fire hazard or detecting a false positive of the fire hazard. The early fire detection system further implements techniques for a long-term trend fire detection that detect pre-fires when obj ects begin to smolder, and for short-term trend fire detection when a rapid temperature increase may indicate a fire.

[0038] Additionally, the early fire detection system includes multiple modules for detecting false lire alerts. These modules may detect false alerts that result from moving objects, from hot objects included on a vehicle, and from reflections from the sun instead of actual fires.

[0039] FIG. 1 illustrates a block diagram of multispectral imaging system 100 in accordance with an embodiment of the subject technology. In some embodiments, system 100 may be configured to fly over a scene, through a structure, or approach a target and image or sense the scene, structure, or target, or portions thereof, using gimbal system 122 to aim multispeclral imaging sensor payload 140 and / or sensor cradle 128 to aim environmental sensor 160 at the scene, structure, or target, or portions thereof. Resulting imagery and / or other sensor data may be processed (e.g., by multispeclral imaging sensor payload 140. platform 110, and / or base station 130) and displayed to a user through use of user interface 132 (e.g.. one or more displays such as a multi-function display (MFD), a portable electronic device such as a tablet, laptop, or smart phone, or other appropriate interface) and / or stored in memory for later viewing and / or analysis. In some embodiments, system 100 may be configured to use such imagery and / or other sensor data to control operation of platform 110, multispectral imaging sensor payload 140, and / or environmental sensor 160, as described herein, such as controlling gimbal system 122 to aim multispectral imaging sensor payload 140 towards a particular direction, controlling propulsion system 124 to move and / or orient platform 110 to a desired position / orientation in a scene or structure or relative to a target, or controlling other modules 126 to deploy a retardant agent towards a scene to extinguish any-detected thermal radiation such as open flames.

[0040] In additional embodiments, system 100 may be configured to use platform 110 and / or sensor cradle 128 to position and / or orient environmental sensor 160 at or relative to the scene, structure, or target, or portions thereof. Resulting sensor data may be processed (e.g., by environmental sensor 160. platform 110, and / or base station 130) and displayed to a user through use of user interface 132 (e.g., one or more display s such as MFD, a portable electronic device such as a tablet, laptop, or smart phone, or other appropriate interface) and / or stored in memory' for later viewing and / or analysis. In some embodiments, system 100 may be configured to use such sensor data to control operation of platform 110 and / or environmental sensor 160, as described herein, such as controlling propulsion system 124 to move and / or orient platform 110 to a desired position in a scene or structure or relative to a target.

[0041] In the embodiment shown in FIG. 1, multispectral imaging system 100 includes platform 110, optional base station 130, and at least one multispectral imaging sensor payload 140. Platform 110 may be a mobile platform configured to move or fly and position multispectral imaging sensor payload 140 and / or environmental sensor 160 (e.g,. relative to a designated or detected target). In other embodiments, the platform 110 may be a fixed-mount platform at a high-elevation configuration to scan a large open area with multispectral imaging sensor payload 140. As shown in FIG. 1, platform 110 may include one or more of a controller 112, an orientation sensor 114, a gyroscope / accelerometer 116, a global navigation satellite system (GNSS) 118, a communication module 120. a gimbal system 122, a propulsion system 124, a sensor cradle 128, and other modules 126. Operation of platform b 110 may be substantially autonomous and / or partially or completely controlled by optional base station 130, which may include one or more of a user interface 132, a communication module 134, and other modules 136. In other embodiments, platform 110 may include one or more of the elements of base station 130, such as with various types of manned aircraft, terrestrial vehicles, and / or surface or subsurface watercraft.

[0042] Multispectral imaging sensor payload 140 and / or environmental sensor 160 may be phy sically coupled to platform 110 and be configured to capture sensor data (e.g., infrared images in the longwave infrared band, ultraviolet images in the shortwave ultraviolet band, visible images, narrow aperture radar data, analyte sensor data, directional radiation data, and / or other sensor data) of a target position, area, and / or object(s) as selected and / or framed by operation of platform 1 10 and / or base station 1 0. In some embodiments, one or more of the elements of system 100 may be implemented in a combined housing or structure that can be coupled to or within platform 110 and / or held or carried by a user of system 100.

[0043] Controller 112 may be implemented as any appropriate logic, device (e.g., processing device, microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), memory storage device, memory reader, or other device or combinations of devices) that may be adapted to execute, store, and / or receive appropriate instructions, such as software instructions implementing a control loop for controlling various operations of platform 110 and / or other elements of system 100, for example. Such software instructions may also implement methods for processing infrared images and / or other sensor signals, determining sensor information, providing user feedback (e.g., through user interface 132). querying devices for operational parameters, selecting operational parameters for devices, or performing any of the various operations described herein (e.g.. operations performed by logic devices of various devices of system 100).

[0044] In addition, a non-transitory medium may be provided for storing machine readable instructions for loading into and execution by controller 112. In these and other embodiments, controller 112 may be implemented with other components where appropriate, such as volatile memory, non-volatile memory, one or more interfaces, and / or various analog and / or digital components for interfacing with devices of system 100. For example, controller 112 may be adapted to store sensor signals, sensor information, parameters for coordinate frame transformations, calibration parameters, sets of calibration points, and / or other operational parameters, over time, for example, and provide such stored data to a user using user interface 132. In some embodiments, controller 112 may be integrated with one or more other elements of platform 110. for example, or distributed as multiple logic devices within platform 110. base station 130. and / or multi spectral imaging sensor payload 140.

[0045] In some embodiments, controller 112 may be configured to substantially continuously monitor and / or store the status of and / or sensor data provided by one or more elements of platform 110, multi spectral imaging sensor payload 140, environmental sensor 160, and / or base station 130, such as the position and / or orientation of platform 110, multispectral imaging sensor payload 140. and / or base station 130, for example, and the status of a communication link established between platform 110, multispectral imaging sensor pay load 140, environmental sensor 160, and / or base station 1 0. Such communication links may be configured to be established and then transmit data between elements of system 100 substantially continuously throughout operation of system 100, where such data includes various types of sensor data, control parameters, and / or other data.

[0046] Orientation sensor 114 may be implemented as one or more of a compass, float, accelerometer, and / or other device capable of measuring an orientation of platform 110 (e.g., magnitude and direction of roll, pitch, and / or yaw, relative to one or more reference orientations such as gravity and / or Magnetic North), gimbal system 122, multispectral imaging sensor payload 140, and / or other elements of system 100, and providing such measurements as sensor signals and / or data that may be communicated to various devices of system 100. Gyroscope / accelerometer 116 may be implemented as one or more electronic sextants, semiconductor devices, integrated chips, accelerometer sensors, accelerometer sensor systems, or other devices capable of measuring angular velocities / accel erations and / or linear accelerations (e.g., direction and magnitude) of platfomi 110 and / or other elements of system 100 and providing such measurements as sensor signals and / or data that may be communicated to other devices of system 100 (e.g., user interface 132, controller 112).

[0047] GNSS 118 may be implemented according to any global navigation satellite system, including a GPS, GLONASS, and / or Galileo based receiver and / or other device capable of determining absolute and / or relative position of platform 110 (e.g., or an element of platform 110) based on wireless signals received from space-born and / or terrestrial sources (e.g, eLoran, and / or other at least partially terrestrial systems), for example, and capable of providing such measurements as sensor signals and / or data (e.g., coordinates) that may be communicated to various devices of system 100. In some embodiments, GNSS 118 may include an altimeter, for example, or may be used to provide an absolute altitude.

[0048] Communication module 120 may be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of system 100. For example, communication module 120 may be configured to receive flight control signals and / or data from base station 130 and provide them to controller 112 and / or propulsion system 124. In other embodiments, communication module 120 may be configured to receive images and / or other sensor information (e.g., visible spectrum, infrared, and / or ultraviolet still images or video images) from multispectral imaging sensor payload 140 and relay the sensor data to controller 112 and / or base station 130. In further embodiments, communication module 120 may be configured to receive sensor data and / or other sensor information from environmental sensor 160 and relay the sensor data to controller 112 and / or base station 130. In some embodiments, communication module 120 may be configured to support spread spectrum transmissions, for example, and / or multiple simultaneous communications channels between elements of system 100. Wireless communication links may include one or more analog and / or digital radio communication links, such as WiFi and others, as described herein, and may be direct communication links established between elements of system 100, for example, or may be relayed through one or more wireless relay stations configured to receive and retransmit wireless communications.

[0049] In some embodiments, communication module 120 may be configured to monitor the status of a communication link established between platform 110. multispectral imaging sensor payload 140, and / or base station 130. Such status information may be provided to controller 112, for example, or transmitted to other elements of system 100 for monitoring, storage, or further processing, as described herein. Communication links established by communication module 120 may be configured to transmit data between elements of system 100 substantially continuously throughout operation of system 100, where such data includes various types of sensor data, control parameters, and / or other data, as described herein.

[0050] In some embodiments, gimbal system 122 may be implemented as an actuated gimbal mount, for example, that may be controlled by controller 112 to stabilize multi spectral imaging sensor payload 140 relative to a target or to aim multispectral imaging sensor

[0051] _ g _ pay load 140 according to a desired direction and / or relative position. As such, gimbal system 122 may be configured to provide a relative orientation of multispectral imaging sensor pay load 140 (e.g. relative to an orientation of platform 110) to controller 112 and / or communication module 120 (e.g., gimbal system 122 may include its own orientation sensor 114). In other embodiments, gimbal system 122 may be implemented as a gravity driven mount (e.g., non-actuated). In various embodiments, gimbal system 122 may be configured to provide power, support wired communications, and / or otherwise facilitate operation of articulated sensor / multispectral imaging sensor payload 140. In further embodiments, gimbal system 122 may be configured to couple to a laser pointer, range finder, and / or other device, for example, to support, stabilize, power, and / or aim multiple devices (e.g., multispectral imaging sensor payload 140 and one or more other devices) substantially simultaneously. In alternative embodiments, multispectral imaging sensor payload 140 may be fixed to mobile platform 110 such that gimbal system 122 is implemented as a fixed perspective mounting system for multispectral imaging sensor payload 140.

[0052] Propulsion system 124 may be implemented as one or more propellers, turbines, or other thrust-based propulsion systems, and / or other types of propulsion systems that can be used to provide motive force and / or lift to platform 110 and / or to steer platform 110. In some embodiments, propulsion system 124 may include multiple propellers (e.g,, a tri, quad, hex, oct, or other type “copter”) that can be controlled (e.g., by controller 112) to provide lift and motion for platform 110 and to provide an orientation for platform 110. In other embodiments, propulsion system 124 may be configured primarily to provide thrust while other structures of platform 110 provide lift, such as in a fixed wing embodiment (e.g., where wings provide the lift) and / or an aerostat embodiment (e.g.. balloons, airships, hybrid aerostats). In various embodiments, propulsion system 124 may be implemented with a portable pow er supply, such as a battery and / or a combustion engine / generator and fuel supply.

[0053] Other modules 126 may include other and / or additional sensors, actuators, communication modules / nodes, and / or user interface devices, for example, and may be used to provide additional environmental information related to operation of platform 110, for example. In some embodiments, other modules 126 may include a humidity sensor, a wind and / or water temperature sensor, a barometer, an altimeter, an analyte detection system, a radar system, a proximity sensor, a visible spectrum camera or infrared / thermal camera (with ail additional mount), an irradiance detector, and / or other environmental sensors providing measurements and / or other sensor signals that can be displayed to a user and / or used by other devices of system 100 (e.g,, controller 1 12) to provide operational control of platform 110 and / or system 100.

[0054] In some embodiments, other modules 126 may include one or more actuated and / or articulated devices (e.g., multi-spectrum active illuminators, visible and / or IR cameras, radars, sonars, and / or other actuated devices) coupled to platform 110, where each actuated device includes one or more actuators adapted to adjust an orientation of the device, relative to platform 110, in response to one or more control signals (e.g., provided by controller 112). In particular, other modules 126 may include a stereo vision system configured to provide image data that may be used to calculate or estimate a position of platform 110. for example, or to calculate or estimate a relative position of a thermal or electrical anomaly in proximity to platform 110. In various embodiments, controller 112 may be configured to use such proximity and-'or position information to help safely pilot platform 110 and / or monitor communication link quality', as described herein. In some embodiments, other modules 126 may include a reservoir containing a type of retardant agent for deployment at or near a scene, structure, target, or portions thereof, to extinguish a fire or open flames.

[0055] In various embodiments, sensor cradle 128 may be implemented as a latching mechanism that may' be permanently mounted to platform 110 to provide a mounting position and / or orientation for environmental sensor 160 relative to a center of gravity- of platform 110, relative to propulsion system 124, and / or relative to other elements of platform 110. In addition, sensor cradle 128 may be configured to provide power, support wired communications, and / or otherwise facilitate operation of environmental sensor 160, as described herein. As such, sensor cradle 128 may be configured to provide a power, telemetry, and / or other sensor data interface between platform 110 and environmental sensor 160. Tn some embodiments, gimbal system 122 may be implemented similarly to sensor cradle 128, and vice versa.

[0056] For example, sensor cradle 128 may be implemented as an actuated gimbal mount, for example, that may be controlled by' controller 112 to stabilize environmental sensor 160 relative to a target or to aim environmental sensor 160 according to a desired direction and / or relative position. As such, sensor cradle 128 may be configured to provide a relative orientation of environmental sensor 160 (e.g., relative to an orientation of platform 110) to controller 112 and / or communication module 120 (e.g., sensor cradle 128 may include its own orientation sensor 1 14). In other embodiments, sensor cradle 128 may be implemented as a gravity driven mount (e.g, non-actuated). In further embodiments, sensor cradle 128 may be configured to couple to a laser pointer, range finder, and / or other device, for example, to support, stabilize, power, and / or aim multiple devices (e.g.. environmental sensor 160 and one or more other devices) substantially simultaneously.

[0057] User interface 132 of base station 130 may be implemented as one or more of a display, a touch screen, a keyboard, a mouse, a joystick, a knob, a steering wheel, a yoke, and / or any other device capable of accepting user input and / or providing feedback to a user. In various embodiments, user interface 132 may be adapted to provide user input (e.g., as a ty pe of signal and / or sensor information transmitted by communication module 134 of base station 130) to other devices of system 100, such as controller 112. User interface 132 may also be implemented with one or more logic devices (e.g., similar to controller 112) that may be adapted to store and / or execute instructions, such as software instructions, implementing any of the various processes and / or methods described herein. For example, user interface 132 may be adapted to form communication links, transmit and / or receive communications (e.g., infrared images and / or other sensor signals, control signals, sensor information, user input, and / or other information), for example, or to perform various other processes and / or methods described herein

[0058] In one embodiment, user interface 132 may be adapted to display a time series of various sensor information and / or other parameters as part of or overlaid on a graph or map, which may be referenced to a position and / or orientation of platform 110 and / or other elements of system 100. For example, user interface 132 may be adapted to display a time series of positions, headings, and / or orientations of platform 110 and / or other elements of system 100 overlaid on a geographical map. which may include one or more graphs indicating a corresponding time series of actuator control signals, sensor information, and / or other sensor and / or control signals. In other examples, user interface 132 may be adapted to display an IR video stream having an overlay that indicates that UV radiation has been detected on a portion of a power line and displays a relative strength of the UV radiation signal, to facilitate assessment of the UV anomaly by a UAS operator (or pilot). In some embodiments, user interface 132 may be adapted to accept user input including a user-defined target destination, heading, waypoint, route, and / or orientation for an element of system 100, for example, and to generate control signals to cause platform 110 to move according to the target destination, heading, route, and / or orientation, or to aim multispectral imaging sensor payload 140 or environmental sensor 160 accordingly. In other embodiments, user interface 132 may be adapted to accept user input modifying a control loop parameter of controller 1 1, for example.

[0059] In further embodiments, user interface 132 may be adapted to accept user input including a user-defined target attitude, orientation, and / or position for an actuated or articulated device (e.g., multispectral imaging sensor payload 140 or environmental sensor 160) associated with platform 110, for example, and to generate control signals for adjusting an orientation and / or position of the actuated device according to the target attitude, orientation, and / or position. Such control signals may be transmitted to controller 112 (e.g., using communication modules 134 and 120), which may then control platform 110 accordingly.

[0060] Communication module 134 may be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of system 100. For example, communication module 134 may be configured to transmit flight control signals from user interface 132 to communication module 120 or 144. In other embodiments, communication module 134 may be configured to receive sensor data (e.g., visible spectrum, infrared, and / or ultraviolet still images or video images, or other sensor data) from multispectral imaging sensor payload 140. In some embodiments, communication module 134 may be configured to support spread spectrum transmissions, for example, and / or multiple simultaneous communications channels between elements of system 100. In various embodiments, communication module 134 may be configured to monitor the status of a communication link established between base station 130, multispectral imaging sensor payload 140, and / or platform 110 (e.g., including packet loss of transmitted and received data between elements of system 100, such as with digital communication links), as described herein. Such status information may be provided to user interface 132, for example, or transmitted to other elements of system 100 for monitoring, storage, or further processing, as described herein. Other modules 136 of base station 130 may include other and / or additional sensors, actuators, communication modules / nodes, and / or user interface devices used to provide additional environmental information associated with base station 130, for example. In some embodiments, other modules 136 may include a humidity’ sensor, a wind and / or water temperature sensor, a barometer, a radar system, a visible spectrum camera, an infrared or thermal camera, a GNSS, and / or other environmental sensors providing measurements and / or other sensor signals that can be displayed to a user and / or used by other devices of system 100 (e.g., controller 112) to provide operational control of platform 110 and / or system 100 or to process sensor data to compensate for environmental conditions, such as an water content in the atmosphere approximately at the same altitude and / or within the same area as platform 110 and / or base station 130, for example. In some embodiments, other modules 136 may include one or more actuated and / or articulated devices (e.g., multi-spectrum active illuminators, visible and / or IR cameras, radars, sonars, and / or other actuated devices), where each actuated device includes one or more actuators adapted to adjust an orientation of the device in response to one or more control signals (e.g., provided by user interface 132).

[0061] In embodiments where multispectral imaging sensor payload 140 is implemented as an imaging device, multispectral imaging sensor payload 140 may include imaging module 142, which may be implemented as a cooled and / or uncooled array of detector elements, such as visible spectrum, infrared, and / or ultraviolet sensitive detector elements, including quantum well infrared photodetector elements, bolometer or microbolometer based detector elements, type II superlattice based detector elements, and / or other infrared spectrum detector elements that can be arranged in a focal plane array (FPA) (e.g., along with other detector elements sensitive to other spectrums). In various embodiments, imaging module 142 may be implemented with a complementary’ metal oxide semiconductor (CMOS) based FPA of detector elements. In various embodiments, imaging module 142 may' include one or more logic devices (e.g., similar to controller 112) that can be configured to process imagery-captured by detector elements of imaging module 142 before providing the imagery' to memory 146 or communication module 144. More generally, imaging module 142 may be configured to perform any of the operations or methods described herein, at least in part, or in combination with controller 112 and / or user interface 132.

[0062] In some embodiments, multispectral imaging sensor payload 140 may be implemented with a second or additional imaging modules similar to imaging module 142, for example, that may include detector elements configured to detect other electromagnetic spectrums, such as visible light, thermal, ultraviolet, and / or other electromagnetic spectrums or subsets of such spectrums. In various embodiments, such additional imaging modules may be calibrated or registered to imaging module 142 such that images captured by each imaging module occupy a known and at least partially overlapping field of view of the other imaging modules, thereby allowing different spectrum images to be geometrically registered to each other (e.g.. by scaling and / or positioning). In some embodiments, different spectrum images may be registered to each other using pattern recognition processing in addition or as an alternative to reliance on a known overlapping field of view.

[0063] Communication module 144 of multispectral imaging sensor payload 140 may be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of system 100. For example, communication module 144 may be configured to transmit images from imaging module 142 to communication module 120 or 134. In other embodiments, communication module 144 may be configured to receive control signals (e.g., control signals directing capture, focus, selective filtering, and / or other operation of multispectral imaging sensor payload 140) from controller 112 and / or user interface 132. In some embodiments, communication module 144 may be configured to support spread spectrum transmissions, for example, and / or multiple simultaneous communications channels between elements of system 100. In various embodiments, communication module 144 may be configured to monitor the status of a communication link established between multispectral imaging sensor payload 140, base station 130, and / or platform 110 (e.g., including packet loss of transmitted and received data between elements of system 100, such as with digital communication links), as described herein. Such status information may be provided to imaging module 142, for example, or transmitted to other elements of system 100 for monitoring, storage, or further processing, as described herein.

[0064] Memory 146 may be implemented as one or more machine readable mediums and / or logic devices configured to store software instructions, sensor signals, control signals, operational parameters, calibration parameters, electromagnetic spectrum images, and / or other data facilitating operation of system 100, for example, and provide it to various elements of system 100. Memory 146 may also be implemented, at least in part, as removable memory', such as a secure digital memoty card for example including an interface for such memory.

[0065] Orientation sensor 148 of multispectral imaging sensor payload 140 may be implemented similar to orientation sensor 114 or gyroscope / accelerometer 116, and / or any other device capable of measuring an orientation of multispectral imaging sensor payload 140, imaging module 142. and / or other elements of multispectral imaging sensor payload 140 (e.g., magnitude and direction of roll, pitch, and / or yaw, relative to one or more reference orientations such as gravity and / or Magnetic North) and providing such measurements as sensor signals that may be communicated to various devices of system 100.

[0066] Gyroscope / accelerometer (e.g., angular motion sensor) 150 of multispectral imaging sensor payload 140 may be implemented as one or more electronic sextants, semiconductor devices, integrated chips, accelerometer sensors, accelerometer sensor systems, or other devices capable of measuring angular velocities / accelerations (e.g., angular motion) and / or linear accelerations (e.g., direction and magnitude) of multispectral imaging sensor payload 140 and / or various elements of multispectral imaging sensor payload 140 and providing such measurements as sensor signals that may be communicated to various devices of system 100. GNSS 149 may be implemented similar to GNSS 118 and / or any other device capable of measuring a osition of multispectral imaging sensor payload 140, imaging module 142, and / or other elements of multispectral imaging sensor payload 140 and providing such measurements as sensor signals that may be communicated to various devices of system 100.

[0067] Other modules 152 of multispectral imaging sensor payload 140 may include other and / or additional sensors, actuators, communication modules / nodes, cooled or uncooled optical filters, and / or user interface devices used to provide additional environmental information associated with multispectral imaging sensor payload 140, for example. In some embodiments, other modules 152 may include a humidity' sensor, a wind and / or water temperature sensor, a barometer, a radar system, a visible spectrum camera, an infrared camera, a GNSS, and / or other environmental sensors providing measurements and / or other sensor signals that can be displayed to a user and / or used by imaging module 142 or other devices of system 100 (e.g., controller 112) to provide operational control of platform 110 and / or system 100 or to process imagery- to compensate for environmental conditions. In various embodiments, environmental sensor / sensor pay load 160 may be implemented as an environmental sensor configured to generate environmental sensor data corresponding to the environment surrounding platform 110, In the embodiment shown in FIG. 1, environmental sensor 160 includes sensor controller 162, memory 163, communication module 164, sensor assembly 166, orientation and / or position sensor (OPS) 167, power supply 168, and other modules 170. In various embodiments, sensor assembly 166 may be implemented with sensor elements configured to detect the presence of and / or generate sensor data corresponding to hazardous analytes, ionizing radiation, emissivities, thermal radiation, radio frequency signals, and / or other environmental conditions proximate to or in view of platform 110 and / or environmental sensor 160.

[0068] Sensor controller 1 2 may be implemented as one or more of any appropriate logic device (e.g., processing device, microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), memory storage device, memory reader, or other device or combinations of devices) that may be adapted to execute, store. and / or receive appropriate instructions, such as software instructions implementing a control loop for controlling various operations of environmental sensor 160 and / or other elements of environmental sensor 160, for example. Such software instructions may also implement methods for processing sensor signals, determining sensor information, providing user feedback (e.g., through user interface 132), querying devices for operational parameters. selecting operational parameters for devices, or performing any of the various operations described herein.

[0069] In addition, non-transitory medium may be provided for storing machine readable instructions for loading into and execution by sensor controller 162 In these and other embodiments, sensor controller 162 may be implemented with other components where appropriate, such as volatile memory, non-volatile memory, one or more interfaces, and / or various analog and / or digital components for interfacing with modules of environmental sensor 160 and / or devices of system 100. For example, sensor controller 162 may be adapted to store sensor signals, sensor information, parameters for coordinate frame transformations, calibration parameters, sets of calibration points, and / or other operational parameters, over time, for example, and provide such stored data to a user using user interface 132. In some embodiments, sensor controller 162 may be integrated with one or more other elements of environmental sensor 160, for example, or distributed as multiple logic devices within platform 110, base station 130, and / or environmental sensor 160.

[0070] In some embodiments, sensor controller 162 may be configured to substantially continuously monitor and / or store the status of and / or store sensor data provided by one or more elements of sensor assembly 166 of environmental sensor 160, such as the position and / or orientation of platform 110, environmental sensor 160, and / or base station 130, for example, and the status of a communication link established between platform 110, environmental sensor 160, and / or base station 130. Such communication links may be configured to be established and then transmit data between elements of system 100 substantially continuously throughout operation of system 100. where such data includes various types of sensor data, control parameters, and / or other data.

[0071] Memory 163 may be implemented as one or more machine readable mediums and / or logic devices configured to store software instructions, sensor signals, control signals, operational parameters, calibration parameters, sensor data, and / or other data facilitating operation of environmental sensor 160 and / or other elements of system 100, for example, and provide it to various elements of system 100. Memory 163 may also be implemented, at least in part, as removable memory, such as a secure digital memory’ card for example including an interface for such memory.

[0072] Communication module 164 of environmental sensor 160 may be implemented as any wired and / or wireless communication module configured to transmit and receive analog and / or digital signals between elements of system 100. For example, communication module 164 may be configured to transmit sensor data from environmental sensor 160 and / or sensor assembly 166 to communication module 120 of platform 110 (e.g., for further transmission to base station 130) or directly to communication modul e 134 of base station 130. In other embodiments, communication module 164 may be configured to receive control signals (e.g., control signals directing operation of environmental sensor 160) from controller 112 and / or user interface 132. In some embodiments, communication module 164 may be configured to support spread spectrum transmissions, for example, and / or multiple simultaneous communications channels between elements of system 100. Sensor assembly 166 may be implemented with one or more sensor element supports (e.g., printed circuit boards ‘ PCBs?’), connectors, sensor elements, and / or other modules configured to facilitate operation of environmental sensor 160. Tn a particular embodiment, environmental sensor 160 may be implemented as a relatively high resolution visible spectrum camera (e.g., an HD or 2K or 4K visible spectrum camera) and sensor assembly 166 may be implemented as a relatively high resolution FPA of visible spectrum sensitive detector elements configured to generate relatively high-resolution imagery and / or video of a scene imaged substantially simultaneously by multispectral imaging sensor payload 140.

[0073] Orientation and / or position sensor (OPS) 167 of environmental sensor 160 may be implemented similar to orientation sensor 114, gyroscope / accelerometer 116. GNSS 118, and / or any other device capable of measuring an orientation and / or position of environmental sensor 160, sensor assembly 166, and / or other elements of environmental sensor 160 (e.g., magnitude and direction of roll, pitch, and / or yaw, relative to one or more reference orientations such as gravity and / or Magnetic North, along with an absolute or relative position) and providing such measurements as sensor signals that may be communicated to various devices of system 100.

[0074] Power supply 168 may be implemented as any power storage device configured to provide enough power to each sensor element of sensor assembly 166 to keep all such sensor elements active and able to generate sensor data while environmental sensor 160 is otherwise disconnected from external power (e.g., provided by platform 110 and / or base station 130). In various embodiments, power supply 168 may be implemented by a supercapacitor so as to be relatively lightweight and facilitate flight of platform 110 and / or relatively easy handheld operation of platform 110 (e.g., where platform 110 is implemented as a handheld sensor platform).

[0075] Other modules 170 of environmental sensor 160 may include other and / or additional sensors, actuators, communication modules / nodes. and / or user interface devices used to provide additional environmental information associated with environmental sensor 160, for example. In some embodiments, other modules 170 may include a humidity' sensor, a wind and / or water temperature sensor, a barometer, a GNSS, and / or other environmental sensors providing measurements and / or other sensor signals that can be displayed to a user and / or used by sensor controller 162 or other devices of system 100 (e.g., controller 112) to provide operational control of platform 110 and / or system 100 or to process sensor data to compensate for environmental conditions, as described herein.

[0076] In general, each of the elements of system 100 may be implemented with any appropriate logic device (e.g., processing device, microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), memory storage device, memory reader, or other device or combinations of devices) that may be adapted to execute, store, and / or receive appropriate instructions, such as software instructions implementing a method for providing sensor data and / or imagery, for example, or for transmitting and / or receiving communications, such as sensor signals, sensor information, and / or control signals, between one or more devices of system 100.

[0077] In addition, one or more non-transitory mediums may be provided for storing machine readable instructions for loading into and execution by any logic device implemented with one or more of the devices of system 100. In these and other embodiments, the logic devices may be implemented with other components where appropriate, such as volatile memory, non-volatile memory, and / or one or more interfaces (e.g., inter-integrated circuit (I2C) interfaces, mobile industry' processor interfaces (MIPI), joint test action group (JTAG) interfaces (e.g.. IEEE 1149.1 standard test access port and boundary-scan architecture), and / or other interfaces, such as an interface for one or more antennas, or an interface for a particular type of sensor).

[0078] Sensor signals, control signals, and other signals may be communicated among elements of system 100 using a variety' of wired and / or wireless communication techniques, including voltage signaling, Ethernet, WiFi, Bluetooth, Zigbee, Xbee, Micronet, or other medium and / or short range wired and / or wireless networking protocols and / or implementations, for example. In such embodiments, each element of system 100 may include one or more modules supporting wired, wireless, and / or a combination of wired and wireless communication techniques. Tn some embodiments, various elements or portions of elements of system 100 may be integrated with each other, for example, or may be integrated onto a single printed circuit board (PCB) to reduce system complexity, manufacturing costs, power requirements, coordinate frame errors, and / or timing errors between the various sensor measurements. Each element of system 100 may include one or more batteries, capacitors, or other electrical power storage devices, for example, and may include one or more solar cell modules or other electrical power generating devices. In some embodiments, one or more of the devices may be powered by a power source for platform 110, using one or more power leads. Such power leads may also be used to support one or more communication techniques between elements of system 100.

[0079] FIG. 2 illustrates a diagram of mobile platforms 110A and HOB of multi spectral imaging system 200 including embodiments of environmental sensor 160 and associated sensor cradle 128 in accordance with an embodiment of the subject technology. In the embodiment shown in FIG. 2, multispectral imaging system 200 includes base station 130. optional co-pilot station 230, mobile platform 110A with articulated imaging system / multispectral imaging sensor payload 140, gimbal system 122, environmental sensor 160, and sensor cradle 128, and mobile platform 110B with articulated imaging system / multispectral imaging sensor payload 140. gimbal system 122, environmental sensor 160, and sensor cradle 128, where base station 130 and / or optional co-pilot station 230 may be configured to control motion, position, orientation, and / or general operation of platform 110A, platform 110B, sensor payloads 140, and / or environmental sensors 160.

[0080] In various embodiments, co-pilot station 230 may be implemented similarly relative to base station 130, such as including similar elements and / or being capable of similar functionality. In some embodiments, co-pilot station 230 may include a number of displays so as to facilitate operation of environmental sensor 160 and / or various imaging and / or sensor payloads of mobile platforms 110A-B, generally separate from piloting mobile platforms 110A-B, and to facilitate substantially real time analysis, visualization, and communication of sensor data and corresponding directives, such as to first responders in contact with a co-pilot or user of system 200. For example, base station 130 and co-pilot station 230 may each be configured to render any of the display views described herein.

[0081] As described herein, embodiments of multispectral imaging system 100 may be implemented with a relatively compact, low weight, and low- power multispectral imaging system (e.g., multispectral imaging sensor payload 140) that can be used to detect LWIR thermal energy in concert with UV radiation, such as through composite imaging with overlaid information about the localized detections, where processed imagery and / or display views are provided to an operator to facilitate assessment of the UV radiation and to facilitate any actions through the mobile platform, such as deploying a retardant agent for extinguishing any open flames confirmed as UV radiation, for example.

[0082] In some embodiments, the multispectral imaging system may include an imaging module implemented by a CMOS based FPA formed, fabricated, assembled, and / or otherwise configured to have sensitivity in the IR and UV spectrums / bands. Such imaging module may include a UV bandpass filter configured to filter out undesirable levels of background noise from out-of-band radiation in the FPA such that each image captured by the multispectral imaging module includes IR and UV information about each scene imaged by the multispectral imaging module.

[0083] In various embodiments, multispectral imaging sensor payload 140 may be equipped with a lens system that is achromatic across the spectral bands captured by imaging module 142. Such lens system may be implemented with a focal length chosen to provide a relatively wide field of view7(FOV) that is sufficient with respect to UAS imaging FOV performance requirements (e.g., directing a narrow field of view in undesirable environmental conditions).

[0084] FIG, 3 illustrates a diagram of multispectral imaging sensor payload 140 for system 100 and / or 300 in accordance with an embodiment of the subject technology7In FIG. 3, imaging module 142 includes UV camera module 370 including multispectral FPA 374 receiving electromagnetic radiation 308 from scene 302 through filter system 376, lens system 378, and / or optional shutter 349 along optical axis 344 and according to FOV 345. In various embodiments, imaging module 142 may include a printed circuit board (PCB) 375 or similar structure configured to support FPA 374 and couple FPA 374 and / or other elements of imaging module 142 to module controller 372 of imaging module 142. As described herein, filter system 376 may in some embodiments be implemented as a UV bandpass filter configured to provide differentiated spectrums (e.g., portions of UV spectrums) to pixels of FPA 374. In various embodiments, filter system 376 is adapted to block light having wavelengths longer than about 260 nanometers and to pass light having wavelengths shorter than about 260 nanometers. As is also described herein, lens system 378 may be achromatic with respect to the differentiated spectrums provided to pixels of FPA 374, for example, and be configured to provide FOV 345. In other embodiments, the lens system 378 may not be achromatic based at least on dimensions of the detector area of the FPA 374 (e.g., the detector area may be on the order of millimeters across). In some embodiments, lens system 378 may be actuated so as to adjust FOV 345. a zoom level of multispectral imaging sensor payload 140, a rotation angle of multispectral imaging sensor payload 140. and / or a focus of light conveyed to FPA 374. In other embodiments, lens system 378 may be a fixed lens system. The imaging module 142 also includes IR camera module 380 receiving the electromagnetic radiation 308 from scene 302 through optional shutter 389 along optical axis 384 and according to FOV 385. In various embodiments, the UV camera module 370 is boresighted to the optical axis 384 of IR camera module 380.

[0085] Module controller 372 may be implemented as any appropriate processing device (e.g.. microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other logic device) that may be used by imaging module 142 and / or multispectral imaging sensor pay load 140 to execute appropriate instructions, such as software instructions and / or signal processing operations for, for example, capturing multispectral images of scene 302 using IR camera module 380 and / or UV camera module 370 including FPA 374, filter system 376. lens system 378, and / or optional shutter 349, generating a composite image from such multispectral images, and / or classifying pixels in such images associated with object 304 (e.g., depicted as an open flame, or a corona discharge) and / or background 306 within scene 302 (e.g.. using a CNN implemented within module controller 372). Further, module controller 372 may be implemented with various signal processing devices, such as analog to digital converters (ADCs), trigger inputs, timing circuitry', and other signal or sensor processing devices as described herein.

[0086] FPA 374 may be implemented by a solar-blind shortwave ultraviolet light imaging sensor. In various embodiments, the solar-blind shortwave ultraviolet light imaging sensor may be implemented as a gas-filled tube having a cathode-anode internal assembly, where the anode is comprised of a two-di ensional planar grid with an active area that allows UV radiation to pass through and strike the photocathode. In some embodiments, FPA 374 may be implemented by a two-dimensional planar array of similarly fabricated / sized pixel structures each configured to be sensitive across the full spectral band of imaging module 142. In other embodiments, FPA 374 may be implemented by an array of structurally differentiated pixel structure subarrays, where each subarray is sensitive to a differentiated subset of the full spectral band of imaging module 142, for example, and / or may be non- planar (e.g., concave with respect to optical axis 344). three dimensional (e.g., multilayered), and / or may include size differentiated pixels (e.g.. with larger surface areas as the distance to optical axis 344 increases).

[0087] In other embodiments, FPA 374 may be implemented with wide bandgap detectors. For example, the wide bandgap (WBG) sensors may include staring arrays of solid-state photon detectors that are inherently insensitive to wavelengths longer than the 290nm ozone cutoff. The WBG sensors may be formed of aluminum gallium nitride alloys, silicon carbide, diamond, or the like. In one or more embodiments, PCB 375 may be implemented with a CMOS based Readout Integrated Circuit (ROIC) that can support high voltages needed to reverse-bias the WBG sensors (e.g., at about 5eV). In some aspects, the WBG sensors may have cost and performance advantages over silicon based sensors For example, a WBG sensor can be integrated into a multispectral imaging system with a significantly lower cost bandpass filter relative to the cost of the bandpass filter implemented with a silicon-based sensor. In some embodiments, the FPA 374 including the WBG sensors may be implemented without a bandpass filter and adapted to operate the multispectral imaging system independent of any bandpass filter provided that the WBG sensor has a bandgap energy that exceeds a predetermined bandgap energy threshold.

[0088] In various embodiments, filter system 376 may be implemented as a static bandpass filter structure deposited or otherwise attached to an active surface of FPA 374, for example, or may be implemented as an adjustable or controllable bandpass filter structure or other type of filter structure configured to provide pixel- or FPA-portion-differentiated spectral illumination of FPA 374. More generally, filter system 376 may be configured to provide spatially and spectrally differentiated illumination of FPA 374 according to one or more different spectrums, each of which may be full differentiated or may partially overlap an adjacent differentiated spectrum.

[0089] In various embodiments, lens system 378 may be implemented as an ultraviolet light transmissive lens that is arranged laterally along an optical axis from the UV camera module 370 implemented as a solar-blmd shortw ave ultraviolet light imaging sensor by a predetermined focal length distance. Lens system 378 may be implemented with one or more lenses each configured to pass light to substantially all pixels of FPA 374, for example, or may be implemented with an array of lenses (e.g., a microlens array) each configured to pass light to a subset of pixels of FPA 374. In general, in embodiments where FPA is sensitive to the NIR, VIS, and LWUV bands, as described herein, each lens of lens system 378 may be configured to be color corrected or achromatic from 330-1 lOOnm. In some embodiments, FOV 345 may be asymmetrical (e.g., to match a corresponding FPA dimension).

[0090] While the embodiment depicted by FIG. 3 shows a relatively compact multispectral imaging sensor payload 140 implemented with a single multispectral imaging module 142 capable of providing single perspective multispectral imagery of scene 302, in other embodiments, multispectral imaging sensor payload 140 may be implemented with multiple imaging modules 142 each sensitive to individually differentiated spectrums, for example, and / or each providing different perspectives of scene 302, such as according to different optical axes and / or different FOVs

[0091] PCB 375 may be a conventional printed circuit board, for example, and be adapted to provide electrical access to FPA 374 and / or other elements of imaging module 142 (e.g.. through various metal traces) as well as physical support for FPA 374 and / or other elements of imaging module 142. In various embodiments, PCB 375 is implemented with a dri ver circuit for controlling FPA 374 and for serving as an output interface to one or more of imaging system controller 312, user interface 332. communication module 144, memory 146, display 333, or other modules 152.

[0092] In some embodiments, each of shutters 349 and 389 may be implemented as a mechanical or removable light shield adapted to selectively block one or more bands of electromagnetic radiation 308. In various embodiments, each of shutters 349 and 389 may be actuated (e.g., opened and / or closed) electronically by module controller 372 and / or imaging system controller 312, for example. Shutters 349 and 389 may be coupled to / supported by housing 348, for example, and housing 348 may be adapted to protect system 300 from environmental conditions associated with space or atmospheric flight, and / or other outdoor environmental conditions, such as fixed or articulated mounting on a terrestrial vehicle, for example. In other embodiments, housing 348 may be adapted for handheld use.

[0093] In various embodiments. UV camera module 370 is implemented as aUV sensorthat acts in concert with IR camera module 380 implemented as athermal LWIR camera. In other embodiments, IR camera module 380 may be implemented as a SWIR camera, a NIR camera

[0094] 2.5 or visible camera. Having the camera modules 370 and 380 acting in concert to image a scene can be efficacious at excluding false alarms, because athermal anomaly such as a flame can generate the correct proportions of radiation in the various spectral bands. The radiation intensity ratio in the two spectral bands (e.g., LWIR band. UV band) can be characterized in advance (or offline) through radiometric experiments on different flames with known fuels and oxidizers. These radiometric experiments can also be performed for the same flame composition at different sizes to determine different ranges of the radiant intensity ratio for different flame types. These radiometric experiments can also be performed at different times in the diurnal cycle and in different weather conditions. In some aspects, the ability to characterize flames by a ratio of radiant intensity in the LWIR band to the UV band may not be feasible for some types of flames. For example, the radiant intensity ratio may be affected by environmental conditions such as wind, or by the presence of gases such as oxygen or the composition of combustible materials.

[0095] In various embodiments, multispectral imaging sensor payload 140, with module controller 372, can be adapted to trigger an alert (or some type of notification) when two preset conditions are met. For example, the first condition may include an occurrence of a hot spot detected in the field of view of the LWIR camera (e.g., IR camera module 380) that has a temperature associated with an intensity value of a pixel of the LWIR image, the composite image, or a combination thereof, which exceeds a preset threshold (e.g., 300°F). The second condition may include an occurrence of UV radiation detected within the field of view of the LV camera (e.g., UV camera module 370) that has a temperature associated with an intensity value of a pixel of the UV image, the composite image, or a combination thereof, which exceeds a preset radiant intensity’ threshold. In some examples, PCB 375 feeds an output trigger signal w ith the driver circuit to a digital input interface of module controller 372 (or imaging system controller 312) with an audible alarm condition set on the input including a pulsed alarm on a Readout Interface (ROI) with a temperature threshold setting. When a flame is presented to the two sensors (e.g., LWIR sensor, UV sensor), the image rendered for display on display 333 (e.g., IR monitor) includes visual and audible indicators (e.g., flashes and beeps).

[0096] In some examples, the output of FPA 374 implemented by the solar-blind shortwave ultraviolet light imaging sensor may not be inherently radiometric in the sense that a pulse height of the output is constant with in-band irradiance, however, the pulse rate of the output can be proportional to the incident irradiance up to a maximum pulse rate of 20Hz by the output of PCB 375 with FPA 374, In one or more embodiments, PCB 375 may be implemented with a metal-filled heatsink board and a spectrally pure 5mW 260nm light-emitting diode (LED) mounted on the metal-filled heatsink board, in which PCB 375 with FPA 374 can analyze the relationship betw een the pulse height and pulse rate of the output with visual indications to a user by the LED. The LED can be driven by a laser diode driver implemented within PCB 375. The LED irradiance at a predetermined distance can be measured using a calibrated UV-enhanced silicon detector and an optical power meter coupled to, or implemented within, multispectral imaging sensor payload 140. In one or more embodiments, the LED has a power pattern that is substantially uniform to simultaneously irradiate the active area of FPA 374 and a co-located silicon detector with a corresponding irradiance. In an embodiment, the pulse rate of FPA 374 through PCB 375 can be measured along with the LED irradiance in terms of watts per square centimeters. PCB 375 coupled with FPA 374 can output a maximum pulse rate of about 20 Hz in some embodiments, or a wider range of pulse rates beyond 20 Hz in other embodiments.

[0097] In some embodiments, module controller 372 may be implemented with machinelearning algorithms (e.g., supervised or unsupervised neural networks, convolutional neural networks, or the like) that evaluate the temporal and spatial signature of a hot spot (or flame), for example, as detected by IR camera module 380 to advantageously provide additional gating that further reduces detections of false alarms. In some examples, thermal events such as fires have an associated temporal signature with a low-frequency oscillatory pattern (e.g., flickering). This distinct oscillatory pattern occurs as oxygen in the immediate vicinity of the fire is consumed, leading to a momentary reduction in flame intensity7. This reduction in flame intensity can cause an inrush of oxygen that causes an increase in the flame intensity. and then the cycle is repeated. However, certain environmental conditions such as windy conditions can lead to a disparate temporal signature of a flame by continuously replenishing the oxygen and sweeping away the non-flammable combustion products. In some embodiments, the machine-learning algorithms can be trained to detect a flame with a more stable radiance time profile (e.g.. the oscillatory pattern of the flame is significantly reduced or non-existent). In one or more embodiments, the machine-learning algorithms can be trained to evaluate the infrared signature of a fire w'ith an increasing temperature and size of the fire. In some examples, the exhaust stack on heavy machinery can exhibit an increasing temperature, particularly when the diesel engine transitions into a regeneration mode. In the regeneration mode, the temperature of the exhaust system is increased by way of additional fuel injection to bum off any carbon soot that accumulates in the diesel particulate filter. As such, the machine-learning algorithms can be trained to distinguish this type of thermal anomaly from an actual growing fire.

[0098] As shown in FIG. 3, multispectral imaging sensor payload 140 may be implemented with a variety of other components adapted to facilitate operation of multispectral imaging sensor payload 140, including capturing multispectral images of scene 302, generating composite images of scene 302, classifying object 304 (e.g., as open flame or not open flame, corona discharge or not corona discharge, and / or likelihood thereof), and / or reporting such sensor data to other elements of system 100 as described herein. In some embodiments, system 300 may report sensor data by aggregating sensor data over time (e.g., multiple frames) to provide a time-duration-based reliability of such classifications determined by system 300, and then transmitting the sensor data to other elements of system 100. In other embodiments, system 300 may report sensor data by energizing an LED indicator and / or transmitting an alert or notification signal to a component (e.g., an alarm, or an electrical switch or relay ) of systems 300 or 100.

[0099] Each of imaging sensor controller 312, memory 146, user interface 332, communication module 144, display 333, and other modules 152, if optionally included in multispectral imaging sensor payload 140, may be coupled to PCB 375 or to housing 348, for example, depending on a desired application and / or overall size of multispectral imaging sensor payload 140 and / or imaging module 142 In other embodiments, any one or group of such components may be implemented externally to multispectral imaging sensor payload 140, for example, and / or in a distributed or grouped manner (e.g.. multiple imaging system controllers 312 operating multispectral imaging sensor pay load 140, or multiple multispectral imaging systems 140 operated by a single imaging system controller 312),

[0100] Imaging system controller 312 may be implemented as any appropriate processing device (e.g., microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA). or other logic device) that may be used by system 300 to execute appropriate instructions, such as software instructions and / or signal processing operations for, for example, capturing multispectral images of scene 302 using imaging module 142, generating composite pixel data associated with such multispectral images, classifying pixels and / or elements of scene 302 in such images (e.g., using a CNN implemented within imaging system controller 312), and / or reporting such sensor data / information to other elements of multispectral imaging system 100 or 300. Further, imaging system controller 312 may be implemented with various signal processing devices, such as analog to digital converters (ADCs), trigger inputs, timing circuitry, and other signal or sensor processing devices as described herein.

[0101] In various embodiments, at least some portion or some functionality' of imaging system controller 312 may be part of or implemented with other existing controllers or logic devices of separate systems, such as a server, a personal electronic device (e.g., a mobile phone, smartphone, tablet device, laptop computer, desktop computer), and / or any other device that may7be used to process, report, or act on multispectral images captured by system 300. In other embodiments, imaging system controller 312 may be adapted to interface and communicate with various external controllers or logic devices and associated components and / or perform various operations in a distributed manner.

[0102] In general, imaging system controller 312 may be adapted to interface and communicate with other components of system 300 to perform the methods and processes described herein. In one embodiment, imaging system controller 312 may be adapted to use communication module 144 to report multispectral imagery and / or pixel / object classifications to display' 333 and render and / or display such information or an alert notification, for example, or render and / or display an image of a classification map corresponding to scene 302. In another embodiment, imaging system controller 312 may be adapted to use communication module 144 to establish a wired or wireless communication link with a remote reporting system, for example, and report such sensor information.

[0103] Memory' 146 is typically in communication with at least imaging system controller 312 and may include one or more memory devices (e.g., memory components) to store information, including image data, calibration data, other types of sensor data, and / or software instructions. Such memory devices may include various ty pes of volatile and nonvolatile information storage devices, such as RAM (Random Access Memory). ROM (Read-Only Memory), EEPROM (Electrically-Erasable Read-Only Memory), flash memory, a disk drive, and / or other ty pes of memory. In one embodiment, memory 146 may include a portable memory device that can be removed from system 300 and used to convey stored data to other systems for further processing and inspection.

[0104] Communication module 144 may be configured to facilitate communication and interfacing between various components of system 300 (e.g., between imaging system controller 312 and memory' 146 and / or display 333) and / or various external devices, such as a wireless access point, a personal electronic device, a server, and / or other detectors. For example, components such as user interface 332 and display 333 may transmit and receive data to and from imaging system controller 312 through communication module 144, which may be adapted to manage w ired and / or wireless communication links between the various components. As such, communication module 144 may support various interfaces, protocols, and standards for local system networking, such as the controller area network (CAN) bus, the local interconnect netw ork (LIN) bus, the media-oriented systems transport (MOST) netw ork, or the ISO 11738 (or ISO bus) standard.

[0105] In some embodiments, imaging system controller 312 may be adapted to communicate, via communication module 144, with a remote user interface, a notification system, or other detection systems to, for example, aggregate reports from multiple systems or sensors and / or implement a particular detection and / or notification method. As such, communication module 144 may include a wireless communication component (e.g., based on the IEEE 802.11 WiFi standards, the Bluetooth™ standard, the ZigBee™ standard, or other appropriate short range wireless communication standards), a wireless broadband component (e.g., based on WiMax technologies), a mobile cellular component, a wireless satellite component, or other appropriate wireless communication components.

[0106] Communication module 144 may also be configured to interface with a wired network and / or device via a wired communication component, such as an Ethernet interface.

[0107] If present, user interface 332 provides for user interaction with multispectral imaging sensor payload 140 and may include one or more buttons, indicators (e g., LEDs), keyboards, trackballs, knobs, joysticks, displays (e.g., a liquid crystal display, a touch-screen display), and / or other type of user interface adapted to accept user input and / or provide user feedback. In one embodiment, user interface 332 may include a power button, a vibration motor, an LED to indicate a maneuvering obstruction, and'or a speaker to provide an audible indication of a maneuvering obstruction (e.g., visible, tactile, and / or audible indicators). In various embodiments, user interface 332 may be used to input a variety of system configuration settings, such as integration time parameters, machine-learning algorithm selections, and / or other configuration settings, as described herein. In some embodiments, user interface 332 may be used to view one or more reports, graphs, and / or other image data captured by system 300 and / or processed according to the various operations described herein.

[0108] If present, display 333 may be configured to present, indicate, or otherwise convey alerts, notifications, and / or other reports of image data and / or object or pixel classifications (e.g., generated by imaging system controller 12). Display 333 may be implemented with an electronic display screen, such as a liquid cry stal display (LCD), a cathode ray tube (CRT), or various other types of generally known video displays and monitors, including touch-sensitive displays. Display 333 may be suitable for presenting image data, graphs, video, reports, or other information as described herein.

[0109] Other modules 152 may include a fiber-coupled SWIR spectrometer, a temperature sensor / probe (e.g., a thermocouple, an infrared thermometer), an LED or laser diode, an ambient light sensor, a voltage regulator and / or filter, a variable voltage source, and / or other types of devices that can be used to facilitate operation of multispectral imaging sensor payload 140, as described herein. In some embodiments, other modules 152 may include a GNSS, accelerometer, compass, and / or other orientation sensor capable of sensing a position and / or orientation of multispectral imaging sensor payload 140. Other modules 152 may additionally include a power module implemented as a battery, a power adapter, a charging circuit, a power interface, a power monitor, and / or other ty pe of power supply providing a mobile power source.

[0110] In various embodiments, system 100 may be configured to determine fire hazards, pre-fire hazards, and generated alerts based on the determination. Moreover, system 100 may be configured to minimize false positive alerts through use of multispectral imaging sensor payload 140 and other sensors mounted to mobile platform 110. In general, mobile platform 110 can relay sensor data to an onboard operator or remote operators at base station 130 and / or co-pilot station 230 where the sensor data can be processed or used to maneuver mobile platform 110. Such sensor data may also be rendered on a display to help visualize and characterize the detected thermal or electrical anomaly at a scene to assist a human operator with assessing the detection radiation and determining any next courses of action through the mobile platform 110.

[0111] Multispectral imaging sensor payload 140 may include an image capture component 330 and early fire detection system 350,

[0112] Fig 4 illustrates a block diagram of image capture component 330 in accordance with an embodiment of the disclosure. In this illustrated embodiment, image capture component 330 is a thermal imager implemented as a focal plane array (FPA) including an array of unit cells 432 (e.g., sensors) and a read out integrated circuit (ROIC) 402. Each unit cell 432 may be provided with an infrared detector (e.g., a microbolometer, indium antimonide (InSb) sensor, multilayer sensor, or other appropriate cooled or uncooled sensor) and associated circuitry to provide image data for a pixel of a captured thermal image frame. In this regard, time-multiplexed electrical signals may be provided by the unit cells 432 to ROIC 402 For example, in some embodiments, such sensors may be cooled sensors, high operating temperature (HOT) cooled sensors (e.g, operating at or near 120 degrees K). or uncooled sensors. In some embodiments, anomalous pixels may be more likely in HOT cooled sensors or uncooled sensors than conventional cooled sensors (e.g., InSb sensors). Accordingly, the various embodiments disclosed herein are particularly advantageous in implementations employing HOT cooled sensors or uncooled sensors.

[0113] ROIC 402 includes bias generation and timing control circuitry 404, column amplifiers 405, a column multiplexer 406, a row multiplexer 408, and an output amplifier 410. Image frames captured by infrared sensors of the unit cells 232 may be provided by output amplifier 410 to image system controller 312, early fire detection system 350 and / or any other appropriate components to perform various processing techniques described herein. Although an 8 by 8 array is shown in Fig. 4, any desired array configuration may be used in other embodiments. Further descriptions of ROICs and infrared sensors (e.g., microbolometer circuits) may be found in U. S. Patent No. 6,028,309 issued February 22, 2000 which is incorporated by reference herein in its entirety.

[0114] FIG. 5 illustrates a block diagram of an early fire detection system 350 in accordance with an embodiment of the disclosure. Fire detection system 350 may have various modules for detecting fire hazards from visual and thermal images, generating alerts in response to the fire hazards, and identifying false positive alerts that may indicate a fire but may be caused by a sun reflection, hot objects, such as exhaust pipes, and the like. In this illustrated embodiment, fire detection system 350 includes a false alert reduction module (FARM) 502. FARM 502 comprises multiple modules that execute separately or in combination to reduce false positives in fire identification in early fire detection system 350. FARM 502 may include a moving object FARM 504, a vehicle FARM 506, and a reflection FARM 508. Moving object FARM 504, vehicle FARM 506, and reflection FARM 508 may identify false alerts individually or as a combination and suppress these alerts. Further. FARM 502 may access one or more configurable rules that may indicate which one or a combination of moving object FARM 504, vehicle FARM 506, and reflection FARM 508 to invoke to reduce false alerts.

[0115] Moving object FARM 504 identifies false positive alerts for moving objects, such as moving vehicles. Moving object FARM 504 may receive a thermal image and divide the thermal image into multiple regions of interest (ROIs). Each ROI may be a non-overlapping square or rectangular region in the thermal image. Moving object FARM 504 may associate each pixel within each ROI with a heat threshold and a hysteresis time. For example, a 640 x 480 thermal image may be associated with twenty -five ROIs and 307,200 pixels, where each pixel may be associated with a heat threshold and a hysteresis time. The hysteresis time maybe a parameter that affects sensitivity and reaction time in early fire detection system 350. A shorter hysteresis time may trigger alerts quicker and with a higher sensitivity, but may create more false alerts because there is less time for objects to move. Typically, the hysteresis time may be configurable based on the image capture frequency and requirements for reaction time. Next, moving object FARM 504 may determine pixels with heat indicators above the heat threshold within each ROI while tracking hysteresis time. Moving object FARM 504 may then aggregate the pixels and determine a number of areas with pixels above the heat threshold. When alerts are detected in a more than a threshold number of areas in an ROI, moving object FARM 504 may generate an alert.

[0116] In some instances, moving objects FARM 504 may also determine whether the number of areas within the ROI have decreased by more than a predefined threshold. In this case, moving objects FARM 504 may suppress the alert. For example, suppose moving objects FARM 504 receives two thermal images that are taken at a first time and a second time. The first time and the second time may be a few seconds, minutes, etc., apart. Moving objects FARM 504 receives the two thermal images and determines the number of areas with pixels above the heat threshold in an ROI in the first thermal image and the number of areas with pixels above the heat threshold in an ROI in the second thermal image, where the ROI in the first thermal image and the ROI in the second thermal image cover the same area of the two thermal images. Moving objects FARM 504 may determine to suppress the alert if the number of areas in the second thermal image decreased by more than a predefined threshold. The predefined threshold may be a percentage decrease between the number of areas in the ROI in the first thermal image and the ROI in the second thermal image.

[0117] FIG. 6 is a diagram 600 illustrating two thermal images in accordance with an embodiment of the disclosure. FIG. 6 illustrates images 602A and 602B. Images 602A and 602B depict a moving object, which is an exhaust pipe 606 of a moving vehicle. Image 602 A is taken first and has a first timestamp and image 602B is taken second and has a second timestamp.

[0118] Image 602A is divided into twenty-five ROls 604A and image 602B is divided into twenty-five ROIs 604B. Each ROI 604A in image 602A has a corresponding ROI 604B in image 602B. This way, ROIs 604A and 604B cover the same area in images 602A and 602B.

[0119] As discussed above, each pixel within each ROI 604A and 604B may be associated with a heat threshold and a hysteresis time. For example, moving object FARM 504 may examine pixels in ROI 604A_1 and ROI 604B_1, both of -which depict the exhaust pipe 606, and determine a number of pixels that are above the heat threshold based on their heat indicators. Next, moving object FARM 504 may combine the pixels with heat indicators above the heat threshold into areas and determine a number of areas within ROI 604A__1 and 604B_1 with pixels above the heat threshold. Moving object FARM 504 may generate an alert if a number of areas in ROI 604A__l or ROI 604B_1 are above the threshold number of areas.

[0120] Moving object FARM 504 may also determine if the number of areas decreased by a predefined percentage between ROI 604A_1 and ROI 604B_1. In this case, moving object FARM 504 may suppress the alert. In this way, moving object FARM 504 may determine that the exhaust pipe 606 is not a fire hazard even if the exhaust pipe 606 has moved slightly between image 602 A and 602B. FIG. 7 is a flowchart of a method 700 for identifying a fire hazard, according to some embodiments. One or more of the operations 702-710 of method 700 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the operations 702-710. Prior to operation 702, thermal images of a moving object are generated.

[0121] At operation 702, thermal images are divided into regions of interest. For example. Moving object FARM 504 divides thermal images 602 of a moving object into ROIs 604.

[0122] At operation 704, pixels in each ROI are associated with a heat threshold and a hysteresis time. For example, moving object FARM 504 associates pixels in each ROI 604 with a heat threshold and hysteresis time.

[0123] At operation 706, pixels with heat indicators above the heat threshold are determined. For example, moving object FARM 504 determines a portion of pixels with heat indicators above a heat threshold in each ROI 604 while tracking the hysteresis time.

[0124] At operation 708, areas with pixels above the heat threshold are determined. For example, moving object FARM 504 determines areas within each ROI 604 with pixels above the heat threshold.

[0125] At operation 710, a determination is made as to whether the number of areas in each ROI is above an area threshold. If so, method 700 proceeds to operation 712, where an alert indicating a potential fire risk is generated. If not, 700 proceeds to operation 714, where no alert is generated.

[0126] Going back to Fig. 5, vehicle FARM 506 may reduce false alarms that are associated with vehicles in images. The hot parts in the vehicles and other machines, such as exhaust pipes, may trigger fire alarms. As a result, conventional early fire detection systems would remove alerts when a vehicle is in the image. However, removing these alerts may cause false negatives and prevent the conventional early fire detection system from issuing an alert when the vehicle is on fire or when the fire is near the vehicle. Further, conventional early fire detection systems may not detect a vehicle in the image. In this case, the conventional early detection system may detect an exhaust pipe and generate an alert that is a false positive. To reduce the false alarms due to images that include vehicles, vehicle FARM 506 may receive and process visual and thermal images using Al models 510, such as AI models 510T and 510V. AI models 510T and 510V may be neural network structures discussed in detail in Fig. 19. AI model 510T may be trained on a training dataset that includes historical thermal images to identify vehicles in thermal images. Al model 510V may be trained on a training dataset that includes historical visual images to identify vehicles in visual images. Once trained. Al models 510T and 510V may execute within vehicle FARM 506, in early fire detection system 350, or be stored and execute separately and be communicatively coupled to vehicle FARM 506. The trained Al model 510T may receive thermal images and identify an object(s) in the thermal images that are vehicle(s). The trained Al model 510V may receive visual images and identify an object(s) in the visual images that are vehiclefs).

[0127] Vehicle FARM 506 may receive a thermal image and a visual image of the same scene. FIG. 8 is a diagram illustrating an example thermal image and visual image, in accordance with the embodiment. In FIG. 8. a thermal image 802T and a visual image 802V depict an image of a vehicle that is an excavator Vehicle FARM 506 may pass thermal image 802T through Al model 510T, which may classify thermal image 802T as including or not including an object that is a vehicle. Al model 510T may also identify an area 804T of the object in thermal image 802T. A classification may be a probability that an object is a vehicle. If the probability is above a predefined probability threshold, such as above 90 percent, the vehicle FARM 506 may remove the temperature indicators from the pixels, ROI, etc., that correspond to area 804T. This may suppress items in area 804T from being identified as a fire and may prevent the early fire detection system 350 from triggering an alert.

[0128] In some embodiments, vehicle FARM 506 may also pass visual image 802V through Al model 510V, which may classify visual image 802V as including or not including an object that is a vehicle. Al model 510V may also identify an area 804V that corresponds to the object in visual image 802V. A classification may be a probability that an object is a vehicle. If the probability is above a predefined probability threshold, such as above 90 percent, the vehicle FARM 506 may transform the area 804V to thermal image 802T. The transformation of area 804V may be to the same location in thermal image 802T as area 804V was in visual image 802V. Vehicle FARM 506 may then suppress the temperature indicators at the location of thermal image 802T to which area 804V was transformed. As discussed above, this may suppress the portion of thermal image 802T from being identified as a fire and may prevent the early fire detection system 350 from triggering a false positive alert, such as an alert due to an exhaust pipe of a vehicle.

[0129] By analyzing both thermal image 802T and visual image 802V of the same scene, vehicle FARM 506 increases the likelihood of identifying a vehicle in the scene. Further, by using areas 804T and 804V to suppress the alerts, the vehicle FARM 506 allows the early fire detection system 350 to trigger alerts based on heat temperature in other areas and / or thermal indicators in those other areas in thermal image 802T.

[0130] In an alternative embodiment, AI model 510T and AI model 510V may be trained to use semantic segmentation to identify pixels in corresponding thermal image 802T and visual image 802V with various categories or objects, Similarly to the embodiments above, vehicle FARM 506 may use Al model 510T and Al model 510V, but trained on semantic segmentation to identify vehicles in thermal image 802T and visual image 802V. and suppress alerts indicative of a fire from areas where both thermal image 802T and visual image 802V indicate a presence of a vehicle. Similarly, vehicle FARM 506 may also issue alerts or perform further analytics as discussed below in segments of thermal image 802T and visual image 802V that Al model 510T and Al model 510V have classified to include a fire.

[0131] in some embodiments, vehicle FARM 506 may have a heat threshold. This heat threshold may be configured to be high, such as 280°C, such that an alert would be triggered in area 804T or in the portion of thermal image 802T to which area 804V was transformed. The high heat threshold may reduce the likelihood of failing to generate an alert when the fire is on a vehicle or in an area surrounding the vehicle. For example, suppose an exhaust pipe on a vehicle that is excavator is approximately 204°C, Vehicle FARM 506 may suppress the alert triggered by the exhaust pipe because it is below the heat threshold. However, in case of a real fire in the vicinity of the vehicle, the temperature of the fire would exceed the heat threshold since the temperature of the fire is above 280°C, thus triggering the alert.

[0132] Vehicle FARM 506 may also determine whether an area temperature in area 804T or in the portion of thermal image 802T to which area 804V was transformed is growing between the medium threshold passed and the end. When the temperature is fixed, vehicle FARM 506 may suppress the alert. This is because a fixed temperature indicates a standing vehicle and not fire as the source of the heat. For example, when the fire starts, the maximum temperature increases by time to a certain level that is above the medium threshold.

[0133] Additionally, the area that has the temperature above the medium threshold also increases. By using a combination of a maximum temperature within the area, and a sum of temperature pixels that are above the medium threshold, vehicle FARM 506 may determine whether the temperature and / or area with the temperature above the medium threshold is increasing. If neither the sum of the temperature is increasing, nor the maximum temperature is increasing, then vehicle FARM 506 may determine that thermal image 802T and visual image 802V do not display a fire in area 804T and 804V, and suppresses the alert.

[0134] FIG. 9 is a flowchart of a method 900 for identifying a fire hazard, according to some embodiments One or more of the operations 902-916 of method 900 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the operations 902-916. Prior to operation 902, thermal image 802T and visual image 802V of the same scene are generated, and Al models 510T and 510V were trained on historical thermal images and visual images, respectively, to identify’ objects, such as vehicles.

[0135] At operation 902, a thermal image and a visual image of the same scene are received. For example, vehicle FARM 506 receives thermal image 802T and visual image 802V.

[0136] At operation 904, an object and its area in the thermal image is identified. For example, vehicle FARM 506 may use Al model 510T to determine whether an object, e.g., a vehicle, exists in thermal image 802T with a probability above a first predefined probability threshold. Al model 510T may also identify area 804T of the object in thermal image 802T.

[0137] At operation 906, temperature indicators from an area in the thermal image that corresponds to the location of the object are suppressed. For example, vehicle FARM 506 may suppress temperature indicators in an area in thermal image802T that corresponds to area 804T.

[0138] At operation 908, a determination whether a fire exists based on the non-suppressed temperature indicators in the thermal image is made. For example, early fire detection system 350 may determine existence of a fire in the thermal image 802T using temperature indicators that were not suppressed and ignoring the suppressed temperature indicators. In this way, the thermal indicators that may correspond to the vehicle with a hot exhaust and may generate a false positive fire identification are suppressed. If the existence of the fire is determined, early fire detection system 350 may generate an alert (not shown).

[0139] At operation 910. an object and its area in the visual image are identified. For example, vehicle FARM 506 may use Al model 510V to determine whether an object, e.g., a vehicle, exists in visual image 802V with a probability above a second predefined probability threshold. The second predefined probability threshold may be the same or different from the first predefined probability threshold. Al model 510V may also identify area 804V of the object in visual image 802V.

[0140] At operation 912, an area corresponding to the location of the object in the visual image is transferred into the thermal image. For example, vehicle FARM 506 may transfer area 804V from visual image 802V into the thermal image 802T. Area 804V may be placed into thermal image 802T at the same location as area 804V was in visual image 802V.

[0141] At operation 914, temperature indicators corresponding to the transferred area are suppressed For example, vehicle FARM 506 may suppress temperature indicators in the area of the thermal image 802T that corresponds to the transferred area 804V.

[0142] At operation 916, a determination of whether a fire exists based on the nonsuppressed temperature indicators in the thermal image is made. For example, early fire detection system 350 may determine the existence of a fire in the thermal image 802T with the suppressed temperature indicators. In this way, the thermal indicators that may correspond to the vehicle and may generate a false positive fire identification are suppressed. If the existence of the fire is determined, early fire detection system 350 may generate an alert (not shown).

[0143] Going back to Fig. 5, reflection FARM 508 may reduce false positive alarms due to sun reflections. Sun reflections may indicate a temperature that is several hundred degrees higher than the temperature in other areas of the scene. Further, the reflection may take on any shape, which makes it difficult for Al models 510T and 510V to detect an object next to or behind the sun reflection. Notably, due to the Earth’s rotation, the sun reflection may disappear. For example, because the Earth rotates approximately one degree every 240 seconds, the sun reflection may change or disappear in approximately four minutes.

[0144] Additionally, majority of sun reflections may have a "sky" color because the sun reflections reflect the heat from the sun via the sky. The sky color may have a certain RGB (“R” - red. “G” - green, “B” - blue) range, which depend on a location of the scene on Earth, the time of the year, and / or a type of a visual camera.

[0145] In some embodiments, reflection FARM 508 may include rules that may predict whether a detected temperature spike is due to sun reflection rather than a fire hazard, thus reduce the number of false alarms generated due to the sun reflections. The rules may be based on a combination of one or more of an object’s (e.g., Earth’s) rotation, weather data, a spike in a temperature at a location that may be identified from the thermal image, and a reflection color (e.g., “sky” color) that may be identified from a visual image. First, reflection FARM 508 may receive a thermal image and a visual image of a scene from IR camera module 380 (or UV camera module 370, if applicable). Next, reflection FARM 508 may receive weather data, and use the weather data to determine whether the weather is sunny or partly sunny at the location of the scene. In some instances, reflection FARM 508 may receive data from the weather service using a communication module 144. Reflection FARM 508 may receive thermal images of the scene from IR camera module 380 over a time period, e.g., 10 minutes, and determine the average temperature over the time period. Reflection FARM 508 may also determine a temperature increase that is more than a predefined percentage, e.g., 200% from the average temperature at the scene that may be due to a sun reflection or a fire.

[0146] FIG. 10 illustrates an example visual image of a scene and a corresponding temperature graph, according to some embodiments, FIG. 10 illustrates a visual image 1002 of a scene with a sun reflection and a temperature graph 1004. Temperature graph 1004 includes a temperature 1006 at the scene at various time intervals, such as every ten minutes. and an average temperature 1008 from the weather data. Temperature graph 1004 also illustrates a spike in the temperature from 44°C at 19:03:37 pm to 552°C at 19:06:43 pm and then back to 47°C at 19:07:42 pm. A spike from 44°C at 19:03:37 pm to 552°C at 19:06:43 pm is more than a 200% spike in temperature, when the average temperature 1008 is around 45°C, and would generate an alert as a potential fire hazard.

[0147] Next, reflection FARM 508 may determine whether the visual image 1002 has an RGB that is a “sky” color. The “sky” color may have "R ”, “G”. and “B” values configured to be within predefined ranges based on the location, time of year, type of camera, etc. Reflection FARM 508 may7analyze visual image 1002 to determine whether the “R”, “G”, and “B” values are within the ranges of the configured “R”. “G”, and “B” values. If the reflection FARM 508 determines that the weather is sunny (or partly sunny), that there is a temperature increase above a predefined temperature threshold based on the thermal image, and the RGB values in the visual image 1002 fall within the configured RGB ranges, then reflection FARM 508 may extend the time to trigger the alert by a predefined number of seconds. The predefined number of seconds may be greater than the time it takes for the Earth to rotate by one degree, such as 300 seconds. Accordingly, if the color in the visual image 1002 and the temperature in the thermal image are due to the reflection of the sun, then the color in the visual image 1002 and the temperature in the thermal image will drop after the Earth rotates one degree. In this case, the reflection FARM 508 would suppress the alert. However, if the color in the visual image 1002 and the temperature in the thermal image are due to a fire, then the color will not change after the Earth rotates by7one degree, and the alert would be valid After the time to trigger the alert expires, reflection FARM 508 may reevaluate the conditions using new thermal and visual images to determine whether the fire hazard persists.

[0148] FIG. 11 illustrates images having various RGB values that indicate “‘sky” and fire colors, according to some embodiments. As discussed above, reflection FARM 508 may detect a “sky ” color to differentiate between sun reflection and a fire hazard. In some instances, the RGB ranges for the “sky'” color may have the following ranges: “R” > 248, 210 < ■ ■( / ’ < 240. and “B” > 248. The RGB ranges may further vary based on the type of a camera, location, and season.

[0149] As illustrated in FIG 11, visual images 1102-1110 show sun reflection and the corresponding RGB values. For example, in visual image 1102. R=255, G=218, and B=255. The RGB values for images 1102-1110 are within the above "sky sun reflection” color RGB range. Image 1112 is a portion of image 1104 that illustrates a sun reflection. The temperature colorization of image 1112 is changed to highlight an area 1113 in image 1112 that is hotter than the temperature threshold. Visual images 1114-1120 include images with fires and the corresponding RGB values due to fires. The RGB values for images 1114-1120 are outside of the above “sky ” color RGB range. In some embodiments, reflection FARM 508 may be configured with a reflection width threshold and a percentage area threshold. Reflection FARM 508 may check the RGB values of the corresponding area in each one of visual images 1 102-1120 extended with a frame around the area for "sky sun reflection’" color. In particular, reflection FARM 508 may check the RGB values for each one of visual images 1102-1120 where the number of pixels is above the reflection width threshold. If the area of the "sky sun reflection” color in one of visual images 1102-1120 is above a percentage area threshold, then reflection FARM 508 extends the hysteresis time by a predefined number of second, such as 300 seconds.

[0150] As discussed above, the reflection width threshold and a percentage area threshold are configurable. The higher the reflection width threshold, the larger the area checked to compensate for possible misalignment between the corresponding thermal and visual images. This is done to compensate for instances when the visual sun reflection has a different spread than the heat that is seen in the thermal image. Hie percentage area threshold indicates sensitivity. For example, if the percentage area threshold is set to 25%, then the percentage area threshold is sufficient if 25% of the pixels in the area have the ‘‘sky sun reflection” color to extend the hysteresis time to 300 seconds.

[0151] Notably, although the embodiments in FIGs. 10-11 are discussed with reference to RGB color scheme, the embodiments are not limited to the RGB color scheme, and the "sky sun reflection” color may be detected using different color schemes, such as YCbCr, HSV, YUV, etc. Moreover, colors in one color schemes may be converted to colors in a different color scheme according to methods known to one or ordinary skill m the art.

[0152] FIG. 12 is a flowchart of a method 1200 for identifying a fire hazard, according to some embodiments. One or more of the operations 1202-1210 of method 1200 may be implemented, at least in part, in the form of executable code stored on a non-transilory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the operations 1202-121. Prior to operation 1202, a thermal image and visual image 1002 of the same scene are generated. Further, one or more rules in operations 1204-1208 may be implemented. At operation 1202, a thermal image and a visual image of the same scene are received. For example, reflection FARM 508 receives thermal images and visual images of a scene.

[0153] At operation 1204, weather conditions are determined to be conducive to generating a sun reflection. For example, reflection FARM 508 may receive weather data that may indicate that the weather is sunny or partly sunny at the scene and is conducive to generating a sun reflection.

[0154] At operation 1206, a temperature increase above a temperature threshold is determined. For example, reflection F RM 508 may use thermal images collected over a time period to identify an average temperature at the scene and identify7a spike in the temperature to be abo ve a predefined temperature threshold. An example temperature threshold may be more than a 200% temperature spike from an average temperature.

[0155] At operation 1208, an RGB color in a visual image is determined to be within an RGB range. For example, reflection FARM 508 may determine that the RGB color in the visual image 1002 is within the configurable RGB range that indicates a “sky" color.

[0156] At operation 1210, an issuance of an alert is delayed. For example, reflection FARM 508 may cause the early fire detection system 350 to delay issuing an alert for a time period, provided the rules in one or more operations 1204-1208 have been met. The time period may exceed the time period it takes the Earth to rotate one degree. In some cases, the time period may be set to 300 seconds After the time period expires, reflection FARM 508 may receive new thermal and visual images of the scene and determine if the fire conditions persist (not shown) using method 1200. Because the Earth would have rotated during the time period, the fire conditions due to a sun reflection would not persist.

[0157] Going back to FIG. 5, in some embodiments, early lire detection system 350, may also include a fire detection module 512. Fire detection module 512 may detect a fire based on thermal images and visual images discussed above. Fire detection module 512 may include a multi-level fire detection module 514, a long-term trend pre-fire detection module 516, and a short-term trend pre-fire detection module 518. Multi-level fire detection module 514 may generate an alert based on multiple thresholds, [degree, std., ] Each threshold may be associated with a temperature threshold level and a corresponding hysteresis time. The hysteresis time may be a time between a threshold level being passed and an action being taken. The temperature and hysteresis time may vary’ for each threshold level. Although the embodiments below will be discussed with reference to three temperature-based threshold levels, the embodiments may also apply to any number of threshold levels.

[0158] In some embodiments, the threshold levels and hysteresis times may be configurable For example, multi-level fire detection module 514 may access a configuration file or a memory’ storage that stores different threshold levels and corresponding hysteresis time configurations. The configurations may vary- based on different context, such as the location of the scene, whether the location is a facility’ or an open space, whether the facility' is open or closed, the time of the year, and the like. Multi-level fire detection module 514 may also access and select various configurations based on external input and / or settings. Moreover. multi-level fire detection module 514 may apply multiple configurations to the same location and / or scene based on the time of day, which may correspond to the time the facility’ is open or closed, based on the time sunset occurs, based on a month, etc.

[0159] Each threshold level may also be associated with actions. When multi-level lire detection module 514 determines that the temperature passes each threshold level, multi-level fire detection module 514 may be configured to perform certain actions. The actions may’ also be configurable and associated with different threshold level configurations. For example, when the first threshold level (e.g., the threshold level associated with the lowest temperature) is passed, multi-level fire detection module 514 may perform analytics. The analytics may use a thermal image to determine energy' and isotherms at a location Multi-level fire detection module 514 may determine energy by calculating the area and the temperature at the location from the thermal image. Multi-level fire detection module 514 may' determine the isotherms by analyzing the thermal image to determine an area in the thermal image with temperatures above the first threshold level.

[0160] In another example, when the second threshold level (e.g., the threshold level associated with the medium temperature) is passed, multi-level fire detection module 14 may generate an alert. In some instances, multi-level fire detection module 514 may' detect that the second threshold level is passed for a hysteresis time associated with the second threshold level before generating an alert. In yet a further embodiment, prior to generating an alert, multi-level fire detection module 514 may initiate FARM 502 module that includes various modules 504-508 that determine whether the alert may be suppressed or deferred due to the alert being a false positive.

[0161] In another example, when the third threshold level (e.g.. the threshold level associated with the high temperature) is passed, multi-level fire detection module 514 may generate an alert. This alert may be generated regardless of the false positive alert suppression that may be initialed by FARM 502.

[0162] FIG. 13 is a diagram 1300 of a graph illustrating two different configurations for threshold levels. A graph 1302 illustrates a first configuration 1304 and a second configuration 1306. The first configuration 1304 is set to three temperature threshold levels that are at 80°C, 100°C, and 210°C. The second configuration 1306 is set to three temperature threshold levels that are at 60°C, 80°C, and 210°C. Moreover, multi-level fire detection module 514 may be configured to switch between the first configuration 1304 and the second configuration 1306 at a time 1308.

[0163] Notably, although the embodiments above discuss temperature threshold levels in Celsius, the implementation is not limited to this embodiment, as temperature units may also be in Fahrenheit, Kelvin, and the like. Additionally, the temperature thresholds may be statistical measurements, such as those based on the standard deviation from a configurable “‘normal”, average, etc., temperature measurement.

[0164] FIG. 14 is a flowchart of a method 1400 for identifying a fire hazard, according to some embodiments. One or more of the operations 1402-1412 of method 1400 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the operations 1402-1412. Moreover, method 1400 may repeat for thermal images generated over multiple days, months, etc. Prior to operation 1402, a thermal image of the scene is generated.

[0165] At operation 1402, a configuration is selected. For example, multi-level fire detection module 514 may select a configuration that is associated with multiple temperature threshold levels and corresponding hysteresis times. Additionally, each temperature threshold level may be configured with one or more actions that multi-level fire detection module 514 may implement once a temperature threshold is crossed and corresponding hysteresis times.

[0166] At operation 1404, a thermal image is received. For example, multi-level fire detection module 14 may receive a thermal image from IR camera module 380.

[0167] At operation 1406, a determination is made as to whether a temperature in the thermal image passed one of the temperature thresholds for corresponding hysteresis time. If the temperature passed a first level threshold (e.g., low level), the method 1400 proceeds to operation 1408. If the temperature passed a second level threshold (e g., medium level), the method 1400 proceeds to operation 1410. If the temperature passed a third level threshold (e.g., high level), the method 1400 proceeds to operation 1412. If none of the temperature threshold levels are passed, method 1400 terminates and reverts to operation 1404 (not shown).

[0168] At operation 1408, actions corresponding to the first threshold level are executed. For example, multi-level fire detection module 514 may execute actions corresponding to the first threshold level that calculate energy at the scene and isotherms from the thermal image. As discussed above, energy may be based on the area of the scene and corresponding temperature, and isotherms may be based on a portion of the area of the scene above first level threshold.

[0169] At operation 1410, actions corresponding to the second threshold level are executed. For example, multi-level fire detection module 514 may execute actions corresponding to the second threshold level that generate an alert subject to false positive logic that may be included in the FARM 502.

[0170] At operation 1412, actions corresponding to the third threshold level are executed. For example, multi-level fire detection module 514 may execute actions corresponding to the third threshold level that generate an alert that is not subject to false positive logic that may be included in the FARM 502.

[0171] Going back to FIG. 5, a long-term trend pre-fire detection module 516 may be configured to perform pre-fire detection from a thermal image. A pre-fire may occur, for example, when a pile of wood, paper, etc., begins to smolder and the heat builds up to the level where there is a risk of fire. Accordingly, detecting a pre-fire during this stage may be particularly beneficial to prevent a fire

[0172] Long-term trend pre-fire detection module 516 may use thermal images to perform pre-fire detection. In some instances, thermal images of a scene may be collected each night or when it is dark to avoid heat caused by the sun, etc. Thermal images may also be collected at the same time every night. Once long-term trend pre-fire detection module 516 receives a thermal image, long-term trend pre-fire detection module 516 may divide the image into multiple samples. Each sample may be an ROI. Each ROI may cover the same or different amount of area as other ROls in the thermal image. Long-term trend pre-fire detection module 516 may determine a maximum, average, and median thermal temperature of each ROI.

[0173] Long-term trend pre-fire detection module 516 may also receive weather data from an external source, e.g., a weather sendee. The weather data may include outdoor temperature, humidity, and weather conditions at the scene in the thermal image.

[0174] Long-term trend pre-fire detection module 516 may determine a temperature trend based on the delta between the maximum temperature in each ROI sample and an average, median, and outdoor temperature over multiple days and multiple images. If the trend passes a temperature trend threshold within a certain number of days, long-term trend pre-fire detection module 516 may trigger an alert

[0175] In some instances, FARM 502 may suppress the alert based on factors that may reduce fire risk. These factors may include weather data indicating rain at the scene, such as rain over a configurable number of centimeters a day, or rain over a configurable number of days. The factors may be configurable and accessible to FARM 502 and / or long-term trend pre-fire detection module 516

[0176] FIG. 15 illustrates an example thermal image and temperature trend graph, according to the embodiments of the disclosure. A thermal image 1502 in FIG. 15 is divided into multiple ROIs 1504. Thermal image 1502 also illustrates a maximum temperature 1506 at each ROI 1504. A temperature trend graph 1508 illustrates a trend based on multiple thermal images submitted over a time period 1510. There may be multiple temperature trend graphs, one trend graph for each ROI in some embodiments. The temperature trend graph 1508 displays a maximum temperature 1512 at each ROI 1504, the median temperature 1514 and the ambient temperature 1516.

[0177] The temperature trend graph 1508 illustrates via arrow 1518 that a maximum temperature of one of ROTs 1504 on April 12thwas 11°C, and via arrow 1520 that the median temperature was 1°C and the ambient temperature was 7°C. Thus, on April 12th, the difference between the median temperature and the maximum temperature was 10°C, and the difference between the ambient temperature and the maximum temperature was 7°C. Longterm trend pre-fire detection module 516 may determine that the difference of 10°C and 7°C does not cross the temperature trend threshold.

[0178] The temperature trend graph 1508 illustrates via arrow 1522 that a maximum temperature of the same ROI 1504 on April 19thwas 29°C, and via arrow 1524 that the median temperature was 3°C and the ambient temperature was 3°C. Thus, the difference between the median temperature and the maximum temperature was 26°C, and the difference between the ambient temperature and the maximum temperature was 26°C. Long-term trend pre-fire detection module 516 may determine that the difference of 26°C is above the temperature trend threshold and may generate an alert.

[0179] FIG. 16 is a flowchart of a method 1600 for identifying a fire hazard, according to the embodiments of the disclosure. One or more of the operations 1602-1612 of method 1600 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable media that when run by one or more processors maycause the one or more processors to perform one or more of the operations 1602-1612.

[0180] Moreover, method 1600 may repeat for thermal images generated over multiple days, months, etc. Prior to operation 1602, thermal images of the scene are generated.

[0181] At operation 1602, thermal images are received. For example, long-term trend pre-fire detection module 516 may receive thermal images from IR camera module 380 taken over a predefined time period, such as over several days or months. Further, in some embodiments, each thermal image may have been taken at the same time every- night. At operation 1604, thermal images are divided into ROIs. For example, long-term trend pre-fire detection module 516 may divide each thermal image into ROIs 1504, where ROIs 1504 from multiple thermal images cover the same areas.

[0182] At operation 1606, for each ROI in each thermal image, a maximum, average, and median temperature are determined. For example, long-term trend pre-fire detection module 516 may determine a maximum, average, and median temperature for each ROI in each thermal image.

[0183] At operation 1608, for each ROI in each thermal image, a difference between a maximum temperature and an average temperature, and a difference between a maximum temperature and median temperature are determined. For example, long-term trend pre-fire detection module 516 may determine differences between the maximum temperature and the average temperature, and the differences between maximum temperature and the median temperature for each ROI in each thermal image.

[0184] At operation 1610, an alert is generated when the differences between the maximum temperature and the average temperature, and / or the differences between maximum temperature and the median temperature for one or more ROIs across multiple thermal images are above the temperature trend threshold.

[0185] At operation 1612, the alerts are suppressed based on weather data. For example, FARM 502 and / or long-term trend pre-fire detection module 516 may receive weather data, and suppress the alert based on the weather data, such as an indication of ram.

[0186] Going back to FIG. 5, short-term trend pre-fire detection module 518 may be configured to perform pre-fire detection. When a pre-fire occurs, a temperature increase trend may occur where a temperature increases at a defined rate. An example temperature defined rate may be 4°C-10°C degrees per minute. Short-term trend pre-fire detection module 518 may detect whether the temperature increase occurs at a defined rate and whether the temperature passes a predefined temperature threshold within a certain time. Once the temperature passes the predefined temperature threshold, short-term trend pre-fire detection module 518 may generate an alert. To reduce the likelihood of false positive alerts, short-term trend pre-fire detection module 518 may include additional logic (or access the logic in FARM 502) that may suppress the false positive alerts For example, short-term trend pre-fire detection module 518 may receive visual images and divide the visual images into ROIs. For each ROI, short-term trend pre-fire detection module 518 may summarize the temperature per area at a lowest possible level. Using multiple visual images collected every few minutes, short-term trend pre-fire detection module 518 may identify temperature increase in summarized temperature from multiple ROIs. If the ROIs show that the energy increases, then the alert should not be suppressed. On the other hand, if the ROIs show that the energy does not increase, the alert should be suppressed.

[0187] In another example, short-term trend pre-fire detection module 518 may also receive weather data and determine whether the temperature increases faster than the average temperature in combination with the weather data to avoid scenarios where the false positive alerts may occur due to a sun heating a dark surface or a sun reflection.

[0188] FIG. 17 illustrates an example visual image and temperature trend graph that corresponds to a scene, according to some embodiments. FIG. 17 includes a visual image 1702 and a temperature trend graph 1708. Visual image 1702 in FIG. 17 is divided into multiple ROIs 1704. Visual image 1702 also illustrates a summarized temperature 1706 at each ROI 1704.

[0189] A temperature trend graph 1708 ill ustrates a detected temperature trend 1710 and an ambient temperature 1712 from the weather data over a time period 1714. The temperature trend graph 1708 illustrates via arrows 1716 and 1718 a temperate increase at the scene from 0°C at time 00:38 to 107°C at time 00:57, which is a 107°C increase over a nineteen-minute period. Such increase may be indicative of a pre-fire, particularly compared to ambient temperature 1712 that remains below70°C.

[0190] From the summarized temperatures 1706 for each ROI 1704, short-term trend pre-fire detection module 518 may determine whether the alert is generated due to the detected temperature trend 1710 or is a false positive alert. For example, from the three ROIs 1704 having a summarized temperature of 130°C that cover the same area, short-term trend pre-fire detection module 518 may determine that the pre-fire has started because energy is increasing, and the alert is not a false positive alert.

[0191] FIG. 18 is a flowchart of a method 1800 for identifying a fire hazard, according to the embodiments of the disclosure. One or more of the operations 1802-1808 of method 1800 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the operations 1802-1808.

[0192] At operation 1802, a temperature trend corresponding to a pre-fire is identified. For example, short-term trend pre-fire detection module 518 may identify a temperature trend from the temperature data. The temperature trend may indicate that the temperature increases at a defined rate, which is indicative of a pre-fire.

[0193] At operation 1804, an alert is generated. For example, once a pre-fire is identified, short-term trend pre-fire detection module 518 may generate an alert.

[0194] At operations 1806 and 1808, a determination is made as to whether the alert generated in operation 1802 is a false positive alert. In some embodiments, operation 1806, operation 1808, or both may be performed. In operation 1806, a comparison between the increase in temperature and the ambient temperature is made. For example, short-term trend pre-fire detection module 518 may receive weather data and compare the ambient temperature in the weather data to the increasing temperature. A temperature that is increasing faster than the ambient temperature may indicate that the alert is not a false positive alert.

[0195] In operation 1808, a determination whether the energy is spreading in the scene is made. For example, short-term trend pre-fire detection module 518 may receive a visual image, divide the image into ROI 1704, and for each ROI 1704 summarize the temperature in the region at a lowest possible level. Short-term trend pre-fire detection module 518 may determine whether a temperature corresponding to the increased temperate is spreading through different ROIs 1704 which may be indicative over a pre-fire, and that an alert is not a false positive alert. FIG. 19 is a simplified diagram 1900 illustrating the neural network structure, according to the embodiments of the disclosure. The neural network structure may be included in Al model 510 and may be accessed by FARM 502. The neural network structure may be a perceptron neural network, a feed-forward neural network, a multilayer perceptron network, a convolutional neural network, a radial basis function neural network, a recurrent neural network, an LSTM (Long Short-Term Memory ) network, a deep neural network, and the like, or be included as a component thereof.

[0196] The neural network structure may comprise a neural network architecture. The example neural network architecture may comprise an input layer 1902, one or more hidden layers 1904 and an output layer 1906. The neural network structure may be built as a collection of connected units or nodes, referred to as neurons 1908. Each layer 1902, 1904, or 1906 may comprise the same or a different number of neurons or nodes 1908, with neurons between layers being interconnected according to a specific topology’. Each neuron 1908 may be associated with an adjustable weight. The neurons 1908 may be aggregated into layers 1902, 1904, 1906 such that different layers may perform different transformations on the respective input to generate a transformed output, which is an input for the subsequent layer. Further, different layers in neural network structure may be combined into their own neural network models, such that an output layer of one neural network model is an input into the next neural network model, until a final output layer 1906 is reached.

[0197] Input layer 1902 receives input data, such as thermal images or visual images. The number of nodes (neurons) in the input layer 1902 may be determined by the dimensionality of the input data (e.g., the length of a vector of a given example of the input). Each node 1908 in the input layer 1902 may represent a feature or attribute of the input. In some embodiments, input layer 1902 may receive an image that is represented by a matrix of pixel values. For a color image, the matrix may be a three-dimensional matrix that includes the width and height of the image and channels. The channels may be the RGB color values.

[0198] The hidden layers 1904 are intermediate layers located between the input and output layers 1902, 1906 of the neural network structure. Although three hidden layers 1904 are shown, there may be any number of hidden layers in the neural network structure. Hidden layers 1904 may extract and transform the input data through a series of weighted computations and activation functions associated with individual neurons. For example, the neural network structure may receive prompts and data at input layer 1902 and generate images as output of output layer 1906. To perform the transformation, each neuron 1908 receives input signals (which may be input to the neural network structure or output of the preceding layer), performs a weighted sum of the inputs according to weights assigned to each connection, and then applies an activation function associated with the respective neuron 1908 to the result. The output of the neuron is passed to the next layer of neurons or serves as the final output of the network. The activation function may be the same or different across different layers 1902, 1904, 1906, and may be different at neurons 1908 within each layer. Example activation functions include Sigmoid, hyperbolic tangent, Rectified Linear Unit (ReLU), Leaky ReLU, and Softmax, among others. In this way, input data received at the input layer 1902 is transformed by hidden layers 1904 into different values indicative of data characteristics corresponding to a task that the neural network structure has been trained to perform.

[0199] In some embodiments, hidden layers 1904 may further be combined into layers. For example, hidden layers 1904 may be combined into one or more convolutional layers, activation layers (ReLu), pooling layers, and fully connected layers. The convolution layer may apply filters to the output of the input layer 1902 and extract features, such as edges, textures, and patterns from the image. In some instances, the ReLu layer may be interspersed within the convolution layer and uses activation functions to replace negative pixel values with zeros. The pooling layer may reduce the spatial dimensions of the output of the convolution layer, while preserving important features. The fully connected layer may be a dense neural network where every neuron is connected to every other neuron in the previous layer. The fully connected layer may receive the output of the preceding layers, e.g., convolutional layer, the ReLus, and / or pooling layers, and generate a classification for the image.

[0200] The output layer 1906 is the final layer of the neural network structure. It produces the network’s output or prediction based on the computations performed in the preceding layers (e.g., 1902, 1904). The number of nodes in the output layer depends on the nature of the task being addressed. For example, in a binary classification problem, the output layer may consist of a single node representing the probability of belonging to one class. In a multi-class classification problem, the output layer may have multiple nodes, each representing the probability of belonging to a specific class. The neural network structure may also be implemented by hardware, software, and / or a combination thereof. For example, the neural network structure may comprise a specific neural network structure implemented and run on various hardware platforms, such as CPUs (central processing units), GPUs (graphics processing units), FPGAs (field-programmable gate arrays), Application-Specific Integrated Circuits (ASICs), dedicated Al accelerators like TPUs (tensor processing units), and specialized hardware accelerators designed specifically for the neural network computations described herein. Example specific hardware for neural network structures may include, but is not limited to, Google Edge TPU, Deep Learning Accelerator (DLA), NVIDIA Al-focused GPUs, and / or the like. The hardware used to implement the neural network structure is specifically configured based on factors such as the complexity of the neural network, the scale of the tasks (e.g., training time, input data scale, size of training dataset, etc.), and the desired performance.

[0201] The neural network structure may be trained by iteratively updating the underlying weights of the neurons 1908, etc., bias parameters and / or coefficients in the activation functions associated with neurons 1908. The weights may be updated based on a loss function, such as a mean squared estimation error (MSEE), cross-entropy loss, log-loss, and the like. For example, during training, the training data are fed into neural network structure over thousands of iterations. The training data flows through the network's layers 1902, 1904, 1906, with each layer performing computations based on its weights, biases, and activation functions until the output layer 1906 produces the output.

[0202] The training data may be labeled with an expected output (e.g., a "ground-truth" label). The output generated by the output layer 1906 is compared to the expected output from the training data to compute a loss function that measures the discrepancy between the predicted output and the expected output. In some embodiments, the negative gradient of the loss function may be computed with respect to the weights of each layer individually. This negative gradient is computed one layer at a time, iteratively backward from the last layer 1906 to the input layer 1902. These gradients quantify the sensitivity of the network's output to changes in the parameters. The chain rule of calculus is applied to efficiently calculate these gradients by propagating them backward (in a backpropagation network) from the output layer 1906 to the input layer 1902, The parameters of the neural network are updated backwardly from the last layer to the input layer (backpropagating) based on the computed negative gradient using an optimization algorithm to minimize the loss. The backpropagation from the last layer 1906 to the input layer 1902 may be conducted for a number of training samples in a number of iterative training epochs. In this way, parameters of the neural network structure may be gradually updated in a direction that results in a lesser or minimized loss, indicating that the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching a maximum number of epochs or achieving satisfactory performance on validation data. In a multiple neural network embodiment, the neural network models may be trained separately and then combined and trained as a single neural network structure.

[0203] The neural network parameters may be trained over multiple stages. For example, initial training (e.g., pre-training) may be performed on one set of training data, and then an additional training stage (e.g., fine-tuning) may be performed using a different set of training data, such as machine-readable code in one or more programming languages. In some embodiments, all, or a portion of, the parameters of one or more neural-network models being used together may be frozen, such that the "frozen" parameters are not updated during that training phase. This may allow, for example, a smaller subset of the parameters to be trained without the computing cost of updating all the parameters. In some instances, neural network structures associated with different Al models 510 may be trained differently to perform different tasks.

[0204] Therefore, the training process transforms the neural network into an “‘updated” trained neural network with updated parameters such as weights, activation functions, and biases. The trained neural network thus improves neural network technology for classifying objects in thermal and visual images.

[0205] Once training is complete, the neural network structure may enter an inference stage where the neural network structure may be used to make predictions on new, unseen data, such as identifying objects in thermal and visual images. Where applicable, various embodiments provided by the present disclosure can be implemented using hardware, software, or combinations of hardware and software. Also, where applicable, the various hardware components and / or software components set forth herein can be combined into composite components comprising software, hardware, and / or both without departing from the spirit of the present disclosure. Where applicable, the various hardware components and / or software components set forth herein can be separated into subcomponents comprising software, hardware, or both without departing from the spirit of the present disclosure. In addition, where applicable, it is contemplated that software components can be implemented as hardw are components, and vice-versa.

[0206] Software in accordance with the present disclosure, such as non-transitory instructions, program code, and / or data, can be stored on one or more non-transitory machine-readable mediums. It is also contemplated that software identified herein can be implemented using one or more general purpose or specific purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the ordering of various steps described herein can be changed, combined into composite steps, and / or separated into sub-steps to provide features described herein.

[0207] Embodiments described above illustrate but do not limit the disclosure. It should also be understood that numerous modifications and variations are possible in accordance with the principles of the present disclosure. Accordingly, the scope of the disclosure is defined only by the following claims.

Claims

CLAIMS1. A system comprising:a multispectral imaging system, wherein the multispectral imaging system comprises an early fire detection system; anda logic device configured to communicate with the multispectral imaging system, wherein the logic device is configured to:divide images that include an object into regions of interest (ROIs); and for an ROI in the ROIs in the images:associate pixels in the ROI with a heat threshold and a hysteresis time; determine, a portion of the pixels in the ROI with heat indicators above the heat threshold and using the hysteresis time; andidentify areas in the ROI that include the portions of pixels; and generate an alert based on a number of the areas above an area threshold.

2. The system of claim 1, wherein the logic device is further configured to: determine a first number of areas in a first ROI associated with a first image; determine a second number of areas in a second ROI associated with a second image, wherein the second image is taken after the first image and the first ROI and the second ROI are in a same location in the first image and the second image;determine a difference between the first number of areas and the second number of areas; andsuppress the alert based on the difference.

3. The system of claim 1, wherein the object in the images is a moving object.

4. The system of claim 1, wherein the images are thermal images.

5. The system of claim 1, wherein to divide the images into the ROls the logic device is further configured to:divide each image in the images into square and'or rectangle areas, wherein ROIs in each image cover the same area as ROIs in other images in the images,6. A system comprising:a multi spectral imaging system, wherein the multispectral imaging system comprises an early fire detection sy stem; anda logic device configured to communicate with the multispectral imaging system, wherein the logic device is configured to:classify, using an artificial intelligence model, a thermal image as including an object in a scene that includes at least one component that triggers an alert;determine an area in the thermal image that corresponds to the object; suppress temperature indicators corresponding to the area from the thermal image; anddetermine whether to generate an alert using remaining temperature indicators in the thermal image.

7. The system of claim 6, wherein the logic device is further configured to:classify, using a second artificial intelligence model, a visual image as having the object that includes the at least one component that triggers the alert;determine a second area in the visual image that corresponds to the object; transform the second area into the thermal image; andsuppress temperature indicators corresponding to the transformed area from the thermal image.

8. The system of claim 7, wherein the logic device is further configured to:generate the alert when at least one temperature indicator in the temperature indicators corresponding to the area in the thermal image and / or the transformed area is above a temperature threshold.

9. The system of claim 6, wherein the object is a vehicle.

10. A system comprising:a multispectral imaging system, wherein the multispectral imaging system comprises an early fire detection system; anda logic device configured to communicate with the multispectral imaging system, wherein the logic device is configured to:receive thermal images depicting a scene and weather data;determine that the weather data indicates a weather that is sunny or partly sunny at the scene;determine, using the thermal images, a temperature spike at the scene above a temperature threshold; anddelay an issuance of an alert by a time period based on the determination that the weather is sunny or partly sunny and the temperature spike.

11. The system of claim 10, wherein the logic device is further configured to: determine, using a visual image, a red, green, blue (RGB) color parameters in the visual image;determine that the RGB color parameters is within an RGB range; andfurther delay the issuance of the alert by the time period based on the determination that the RGB color parameters is within the RGB range.

12. The system of claim 10, wherein the time period is greater than a rotation of the Earth by one degree.

13. The system of claim 10, wherein the logic device is further configured to generate the alert after the time period expires.

14. A system comprising:a multispectral imaging system, wherein the multispectral imaging system comprises an early fire detection system; anda logic device configured to communicate with the multispectral imaging system, wherein the logic device is configured to:select a configuration associated with temperature threshold levels, wherein temperature threshold levels are associated with actions and hysteresis times;receive a thermal image of a scene;determine that a temperature in the thermal image passed one of the temperature threshold levels and a corresponding hysteresis time; andexecute at least one the action corresponding to the one of the temperature threshold levels.

15. The system of claim 14, wherein the at least one action corresponding to the one temperature threshold level comprises determining energy and isotherms from the thermal image of the scene.

16. The system of claim 14, wherein the at least one action corresponding to the one temperature threshold level comprises generating an alert subject to a false alert detection.

17. The system of claim 14, wherein the at least one action corresponding to the one temperature threshold level comprises generating an alert.

18. The system of claim 14, wherein the logic device is further configured to select a second configuration with different temperature threshold levels at a predefined time during a day.

19. A system comprising:a multispectral imaging system, wherein the multispectral imaging system comprises an early fire detection system; anda logic device configured to communicate with the multispectral imaging system, wherein the logic device is configured to:receive thermal images of a scene taken at predefined time intervals; divide each thermal image in the thermal images into regions of interest (ROIs); andfor each ROI in the ROIs in each thermal image:determine a maximum temperature and an average and / or median temperature;determine a difference between the maximum temperature and the average and / or median temperature; andgenerate an alert based on the difference.

20. The system of claim 19, wherein the logic device is further configured to: receive weather data: andsuppress the alert based on the weather data.

21. A system comprising:a multispectral imaging system, wherein the multispectral imaging system comprises an early fire detection system; anda logic device configured to communicate with the multispectral imaging system, wherein the logic device is configured to:identify a temperature trend corresponding to a pre-fire based on a temperature increase rate;generate an alert indicative of the pre-fire:determine that an alert is not a false positive alert based on an increase in temperature in the temperature trend and an ambient temperature, ordetermine that the alert is not the false alert by analyzing regions of interest (ROIs) in a visual image and determine that energy is increasing in at least some ROIs in the visual image.

Images