An enhanced fire source detection method and system based on multi-sensor fusion

Abstract

The application discloses a kind of based on multi-sensor fusion's enhanced fire source detection method and system, it is related to intelligent security technology field, the present method includes the following steps: initialization detection is carried out to sensor, and environment benchmark data is collected;Audio data and visual data are collected, audio features and visual features are extracted respectively, and audio confidence and visual confidence are calculated according to the audio features and visual features;When the audio confidence or visual confidence meets preset wake-up condition, start thermal imaging sensor to collect thermal imaging data, and calculate thermal imaging confidence;Based on the audio confidence, visual confidence and thermal imaging confidence are weightedly distributed based on the environment benchmark, and each confidence is time series fusion to obtain fusion determination result;Based on audio data, visual data and thermal imaging data, corresponding candidate coordinates are calculated respectively, and the candidate coordinates are weightedly fused to output the coordinate information of fire source.

Description

Technical Field

[0001] This invention relates to the field of intelligent security technology, specifically an enhanced fire source detection method and system based on multi-sensor fusion. Background Technology

[0002] As the level of intelligence in industrial parks increases, inspection robots have become core equipment for achieving 24 / 7 safety monitoring, replacing manual labor. Fire source detection, as a key function, directly impacts the efficiency of park safety protection. Traditional fire source detection technology has evolved from early single smoke and temperature sensors to multi-sensor solutions integrating audio, vision, and thermal imaging, with the core objective of improving detection coverage and response speed. Among these, detection technologies suitable for inspection robots need to balance accuracy, real-time performance, and power consumption to adapt to the battery life limitations of mobile operations and the dynamic needs of complex park scenarios, becoming a core direction for industry technology research and development.

[0003] Current multi-sensor fire detection solutions applied to park inspection robots still have key shortcomings: most adopt fixed weight fusion and fixed threshold judgment, which cannot adapt to dynamic scenarios such as day and night light fluctuations, seasonal temperature changes, and environmental noise interference in parks, resulting in a high false alarm rate; some solutions rely on complex algorithms to improve accuracy, but are difficult to run in real time on resource-constrained robot embedded platforms, and the response delay affects fire response; at the same time, the continuous operation of thermal imaging sensors leads to excessive power consumption, shortening the robot's battery life; in addition, the single-modal positioning deviation is large, which cannot provide the inspection robot with accurate fire location guidance, seriously restricting its practical application effect in complex parks. Summary of the Invention

[0004] The purpose of this invention is to provide an enhanced fire source detection method and system based on multi-sensor fusion to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an enhanced fire source detection method based on multi-sensor fusion, the method comprising the following steps:

[0006] S1. Initialize the detection of the audio sensor, vision sensor and thermal imaging sensor, and collect environmental noise, ambient light and ambient temperature data as environmental reference.

[0007] To establish an environmental reference baseline for subsequent testing, this eliminates the interference of inherent differences in different scenarios on the test results and ensures consistency in cross-scenario testing. Specifically, environmental parameter acquisition must be performed under conditions of no significant sensor interference and stable equipment operation.

[0008] S2. Collect audio data and visual data, extract audio features and visual features respectively, and calculate audio confidence and visual confidence based on the audio features and visual features respectively;

[0009] By capturing the inherent audio and visual attributes of the fire source through multi-dimensional features, misjudgment caused by interference from a single feature is avoided, providing a reliable basis for subsequent wake-up decisions;

[0010] S3. When the audio confidence or visual confidence meets the preset wake-up condition, the thermal imaging sensor is activated to collect thermal imaging data, and the thermal imaging confidence is calculated based on the thermal imaging data.

[0011] The strategy of "audio / vision first, thermal imaging confirmation" aims to reduce the invalid working time of thermal imaging sensors while ensuring detection accuracy, thereby reducing the overall power consumption of the system and adapting to the battery life requirements of mobile devices such as park inspection robots.

[0012] S4. Based on the environmental benchmark, assign weights to the audio confidence, visual confidence, and thermal imaging confidence, and perform time-series fusion of each confidence to obtain the fusion judgment result;

[0013] By dynamically weighting to adapt to environmental changes and smoothing out instantaneous interference through time series fusion, a stable and reliable fire source determination result is finally output, reducing false alarms and false negatives.

[0014] S5. Calculate the corresponding candidate coordinates based on audio data, visual data and thermal imaging data respectively, and perform weighted fusion on the candidate coordinates to output the coordinate information of the fire source;

[0015] By complementing and fusing multimodal positioning data, the accuracy and stability of fire source positioning are improved, providing accurate location guidance for firefighting operations in the park.

[0016] In S1, during system startup or preset calibration, the following environmental parameters are collected as environmental references:

[0017] Environmental noise standard N base : Reflects the background acoustic characteristics of the current scene;

[0018] Ambient Lighting Standard L base This reflects the brightness level of the environment in which the vision sensor is located;

[0019] Ambient temperature reference T base : Reflects the background temperature state of thermal imaging;

[0020] Environmental benchmarks are used to eliminate absolute differences between different scenarios. For example, the absolute values ​​of audio energy or image brightness can differ greatly between daytime and nighttime, and between indoor and outdoor environments. Directly comparing absolute values ​​can easily lead to misjudgment. For instance, the average brightness of a visual image may reach 200 grayscale values ​​during the day, but only 30 grayscale values ​​at night. If a fixed brightness threshold is used to judge fire sources, real fire sources at night may be missed, and bright objects that are not fire sources during the day may be misjudged.

[0021] In S2, the fire source is often accompanied by popping or continuous burning sounds during combustion, and its energy distribution is significantly different from the background noise; therefore, the extraction of audio features is used to capture this relatively abrupt change characteristic.

[0022] The audio signal is divided into frames, and the audio energy is calculated as: E = 1 / NΣ n=0 N-1 |x[n]| 2 Where, x[n]: discrete audio signal, n=0,1,…,N-1; N is the analysis window length;

[0023] The spectral centroid is calculated after performing a frequency domain transformation on the framed audio signal: SC=(Σ k=0 N / 2 k·|X[k]|) / (Σ k=0 N / 2 |X[k]|); where X[k] is the discrete Fourier transform of the audio signal; k represents the frequency index;

[0024] The Mel frequency cepstral coefficients are calculated based on the Mel filter bank: m i =Σ k=1 K log(M k )·cos[i·(k-1 / 2)·π / K]; where, m i M represents the i-th MCFF coefficient; k This represents the output energy of the Mel filter bank; K represents the number of Mel filters, which is usually set to 40, conforming to the industry standard setting for speech signal processing and fully characterizing the spectral envelope features of audio.

[0025] The audio confidence score is determined based on the audio energy and spectral centroid of the current audio frame, combined with the spectral centroid range and the corresponding audio energy dynamic range in the environmental benchmark, through a normalized calculation relationship, and is used to characterize the current audio state.

[0026] Audio confidence level C a =(EE min ) / (E max -E min )·(SC-SC min ) / (SC max-SC min ); where E max E min Indicates the energy dynamic range under the current environment; SC max SC min Indicates the range of the centroid of the spectrum;

[0027] Flames and smoke are typically visually represented by: specific color distribution; continuous or expanding areas of motion; periodically changing brightness characteristics; single visual features are easily interfered with, so multiple visual features are introduced simultaneously for joint judgment.

[0028] The acquired images are converted to the HSV color space based on visual data; image pixels are filtered based on a preset smoke or flame color threshold range to obtain a color mask; and the color matching degree R is determined according to the ratio of the number of pixels in the color mask that meet the threshold conditions to the total number of pixels in the image. c ;

[0029] The motion region ratio R is obtained by calculating the proportion of the foreground motion region to the total area based on the frame difference method of continuous visual data frames. m Typically, a three-frame consecutive frame difference is used (the current frame is differentially divided with the previous two frames and the intersection is taken), which can effectively suppress noise and static background interference and accurately extract moving smoke or flame areas.

[0030] Analyze the periodic changes in brightness of the flame area within a preset frequency band (2-10Hz) and extract the dominant frequency as the flicker frequency f. f This frequency band represents a typical range of brightness fluctuations during flame combustion. The main frequency can be extracted by analyzing the brightness time series of the flame region using Fast Fourier Transform (FFT).

[0031] The visual confidence score is obtained by combining the color matching degree, the proportion of the motion area, and the flashing frequency according to a preset function calculation relationship, and is used to characterize the current visual state.

[0032] Visual confidence level C v =R m ·R c ·(f f -3) / 12;

[0033] In S3, after calculating the audio confidence and visual confidence, the audio confidence and visual confidence at the current moment are judged; when any confidence meets the preset wake-up condition, the thermal imaging sensor is activated to collect data.

[0034] The wake-up condition is used to determine whether there is a possible fire source abnormality in the current environment. It is set based on environmental benchmarks to avoid false wake-ups caused by environmental noise or light fluctuations.

[0035] Specifically, let the audio wake-up threshold be θ. a The visual arousal threshold is θ v Then the wake-up condition can be expressed as: C a >θ a Or C v >θ v ;θ a and θ v Through extensive experimental calibration, we ensure that while effectively detecting fire sources, the false wake-up rate caused by environmental interference is kept at a low level.

[0036] When the above conditions are met, the thermal imaging sensor is activated to collect thermal imaging data.

[0037] Thermal imaging data processing and confidence calculation:

[0038] The data collected by the thermal imaging sensor is represented as a temperature matrix: T(x,y); where x and y represent the horizontal and vertical coordinates of the pixel in the thermal imaging image, respectively, and T(x,y) represents the temperature value of the corresponding pixel.

[0039] Based on ambient temperature reference T base Set the high temperature threshold T th Threshold segmentation is performed on the temperature matrix to obtain the set of pixels in the high-temperature region: Ω hot ={(x,y)|T(x,y)>T th};

[0040] Further calculation of the proportion of high-temperature regions: P hot =|Ω hot | / |Ω|;where |Ω hot | represents the number of pixels in the high-temperature area; |Ω| represents the total number of pixels in the thermal imaging image;

[0041] To describe the spatial variation characteristics of the high-temperature region, the gradient magnitude is calculated from the temperature matrix:

[0042] ;

[0043] The average temperature gradient is obtained by statistically analyzing the gradient within the high-temperature region.

[0044] G=1 / (∣Ω hot ∣)∑(x,y)∈Ω hot G(x,y);

[0045] Based on the proportion of high-temperature regions P hot The thermal imaging confidence level, C, is obtained using a preset thermal anomaly calculation function, along with the average temperature gradient G. t =H(P hot ,G);

[0046] Where H(·) is a combined calculation function used to characterize the degree of thermal anomaly;

[0047] By making preliminary judgments using audio or vision, the thermal imaging sensor is activated only when necessary, reducing the overall power consumption of the system. At the same time, by combining temperature distribution and spatial variation characteristics to calculate the confidence level of thermal imaging, the system can effectively distinguish between a continuous high-temperature background and a real fire source.

[0048] In S4, the audio confidence C is obtained. a Visual confidence level C v and thermal imaging confidence C t Subsequently, the system assigns weights to each confidence level based on environmental benchmarks and performs time series fusion on the confidence levels to obtain stable fusion judgment results.

[0049] Based on the ambient noise and ambient lighting conditions, determine the weighting coefficients for each mode: w a w v w t Among them, w a For audio confidence weights; w v For visual confidence weights; w t The weights are the confidence scores for thermal imaging; each weight satisfies w a +w v +w t =1; In noisy environments, reduce w a When lighting conditions are poor, reduce w v When thermal imaging is activated, increase w t ;

[0050] The time series fusion calculation specifically includes:

[0051] Let the time window length be T, and perform fusion processing on the confidence scores within this window:

[0052] C fusion (t)=Φ(C a (t),C v (t),C t (t))

[0053] Among them, C fusion (t) represents the fusion judgment result at time t; Φ(·) represents the time series fusion function, which is used to integrate the confidence change trends of the current and historical times;

[0054] By introducing a time-series fusion mechanism, misjudgments caused by transient noise, light flicker, or brief thermal anomalies are avoided, making the detection results more stable and reliable.

[0055] In S5, while determining the state of the fire source, the system calculates candidate coordinates based on audio data, visual data, and thermal imaging data, and performs weighted fusion on the candidate coordinates to output the coordinate information of the fire source.

[0056] The arrival time difference Δt is calculated by receiving the sound of the fire source through multiple audio sensors. ij Based on the speed of sound c, the following relationship is established: ∥s a -p i ∥-∥s a -p j ∥=c·Δt ij ; where s a For audio candidate coordinates; p i ,p j These are the coordinates of the audio sensor's location.

[0057] Extracting a set of flame or smoke feature points p from a visual image k Visual candidate coordinates are calculated using geometric mapping relationships: s v =1 / R∑ k=1 R p k ; where s v R represents the visual candidate coordinates; R is the number of feature points.

[0058] Determine the geometric center of the high-temperature region in the thermal imaging image and estimate its spatial location using system calibration parameters: s t =f(T(x,y),"calibration parameters"); where s t Candidate coordinates for thermal imaging;

[0059] Based on the weights corresponding to the confidence levels of each modality, the candidate coordinates are weighted and fused: s=W a ·s a +W v ·s v +W t ·s t Where s represents the output fire source coordinates; W a W v W t To define the fusion weights for the coordinates corresponding to each mode;

[0060] By fusing multimodal candidate coordinates, the stability and accuracy of fire source localization are improved, avoiding the deviation of a single localization method in complex environments.

[0061] An enhanced fire source detection system based on multi-sensor fusion is provided, which is applied to the aforementioned enhanced fire source detection method based on multi-sensor fusion.

[0062] Compared with existing technologies, the beneficial effects of this invention are as follows: By collecting environmental benchmarks and dynamically adjusting weights and thresholds, this invention can adapt to complex environments such as day-night cycles and seasonal changes in parks; by extracting multimodal core features and introducing a lightweight time-series fusion algorithm, this invention can run efficiently on resource-constrained embedded platforms; through the hierarchical collaborative logic of "audio / visual wake-up first, thermal imaging for precise confirmation," this invention can reduce the ineffective working time of thermal imaging sensors, effectively reduce system power consumption, and extend the battery life of park inspection robots; through complementary extraction of multimodal features and dynamic fusion decision-making, this invention can overcome the limitations of single sensors. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating an enhanced fire source detection method based on multi-sensor fusion according to the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Example: Figure 1 As shown, the present invention provides a technical solution, an enhanced fire source detection method based on multi-sensor fusion, the method comprising the following steps:

[0066] S1. Initialize the detection of the audio sensor, vision sensor and thermal imaging sensor, and collect environmental noise, ambient light and ambient temperature data as environmental reference.

[0067] To establish an environmental reference baseline for subsequent testing, this eliminates the interference of inherent differences in different scenarios on the test results and ensures consistency in cross-scenario testing. Specifically, environmental parameter acquisition must be performed under conditions of no significant sensor interference and stable equipment operation.

[0068] S2. Collect audio data and visual data, extract audio features and visual features respectively, and calculate audio confidence and visual confidence based on the audio features and visual features respectively;

[0069] By capturing the inherent audio and visual attributes of the fire source through multi-dimensional features, misjudgment caused by interference from a single feature is avoided, providing a reliable basis for subsequent wake-up decisions;

[0070] S3. When the audio confidence or visual confidence meets the preset wake-up condition, the thermal imaging sensor is activated to collect thermal imaging data, and the thermal imaging confidence is calculated based on the thermal imaging data.

[0071] The strategy of "audio / vision first, thermal imaging confirmation" aims to reduce the invalid working time of thermal imaging sensors while ensuring detection accuracy, thereby reducing the overall power consumption of the system and adapting to the battery life requirements of mobile devices such as park inspection robots.

[0072] S4. Based on the environmental benchmark, assign weights to the audio confidence, visual confidence, and thermal imaging confidence, and perform time-series fusion of each confidence to obtain the fusion judgment result;

[0073] By dynamically weighting to adapt to environmental changes and smoothing out instantaneous interference through time series fusion, a stable and reliable fire source determination result is finally output, reducing false alarms and false negatives.

[0074] S5. Calculate the corresponding candidate coordinates based on audio data, visual data and thermal imaging data respectively, and perform weighted fusion on the candidate coordinates to output the coordinate information of the fire source;

[0075] By complementing and fusing multimodal positioning data, the accuracy and stability of fire source positioning are improved, providing accurate location guidance for firefighting operations in the park.

[0076] In S1, during system startup or preset calibration, the following environmental parameters are collected as environmental references:

[0077] Environmental noise standard N base : Reflects the background acoustic characteristics of the current scene;

[0078] Ambient Lighting Standard L base This reflects the brightness level of the environment in which the vision sensor is located;

[0079] Ambient temperature reference T base : Reflects the background temperature state of thermal imaging;

[0080] Environmental benchmarks are used to eliminate absolute differences between different scenarios. For example, the absolute values ​​of audio energy or image brightness can differ greatly between daytime and nighttime, and between indoor and outdoor environments. Directly comparing absolute values ​​can easily lead to misjudgment. For instance, the average brightness of a visual image may reach 200 grayscale values ​​during the day, but only 30 grayscale values ​​at night. If a fixed brightness threshold is used to judge fire sources, real fire sources at night may be missed, and bright objects that are not fire sources during the day may be misjudged.

[0081] In S2, the fire source is often accompanied by popping or continuous burning sounds during combustion, and its energy distribution is significantly different from the background noise; therefore, the extraction of audio features is used to capture this relatively abrupt change characteristic.

[0082] The audio signal is divided into frames, and the audio energy is calculated as: E = 1 / NΣ n=0 N-1 |x[n]| 2 Where, x[n]: discrete audio signal, n=0,1,…,N-1; N is the analysis window length;

[0083] The spectral centroid is calculated after performing a frequency domain transformation on the framed audio signal: SC=(Σ k=0 N / 2 k·|X[k]|) / (Σ k=0 N / 2 |X[k]|); where X[k] is the discrete Fourier transform of the audio signal; k represents the frequency index;

[0084] The Mel frequency cepstral coefficients are calculated based on the Mel filter bank: m i =Σ k=1 K log(M k )·cos[i·(k-1 / 2)·π / K]; where, m i M represents the i-th MCFF coefficient; k This represents the output energy of the Mel filter bank; K represents the number of Mel filters, which is usually set to 40, conforming to the industry standard setting for speech signal processing and fully characterizing the spectral envelope features of audio.

[0085] The audio confidence score is determined based on the audio energy and spectral centroid of the current audio frame, combined with the spectral centroid range and the corresponding audio energy dynamic range in the environmental benchmark, through a normalized calculation relationship, and is used to characterize the current audio state.

[0086] Audio confidence level C a =(EE min ) / (E max -E min )·(SC-SC min ) / (SC max -SC min ); where E max E min Indicates the energy dynamic range under the current environment; SC max SC min Indicates the range of the centroid of the spectrum;

[0087] Flames and smoke are typically visually represented by: specific color distribution; continuous or expanding areas of motion; periodically changing brightness characteristics; single visual features are easily interfered with, so multiple visual features are introduced simultaneously for joint judgment.

[0088] The acquired images are converted to the HSV color space based on visual data; image pixels are filtered based on a preset smoke or flame color threshold range to obtain a color mask; and the color matching degree R is determined according to the ratio of the number of pixels in the color mask that meet the threshold conditions to the total number of pixels in the image. c ;

[0089] The motion region ratio R is obtained by calculating the proportion of the foreground motion region to the total area based on the frame difference method of continuous visual data frames. m Typically, a three-frame consecutive frame difference is used (the current frame is differentially divided with the previous two frames and the intersection is taken), which can effectively suppress noise and static background interference and accurately extract moving smoke or flame areas.

[0090] Analyze the periodic changes in brightness of the flame area within a preset frequency band (2-10Hz) and extract the dominant frequency as the flicker frequency f. f This frequency band represents a typical range of brightness fluctuations during flame combustion. The main frequency can be extracted by analyzing the brightness time series of the flame region using Fast Fourier Transform (FFT).

[0091] The visual confidence score is obtained by combining the color matching degree, the proportion of the motion area, and the flashing frequency according to a preset function calculation relationship, and is used to characterize the current visual state.

[0092] Visual confidence level C v =R m ·R c ·(f f -3) / 12;

[0093] In S3, after calculating the audio confidence and visual confidence, the audio confidence and visual confidence at the current moment are judged; when any confidence meets the preset wake-up condition, the thermal imaging sensor is activated to collect data.

[0094] The wake-up condition is used to determine whether there is a possible fire source abnormality in the current environment. It is set based on environmental benchmarks to avoid false wake-ups caused by environmental noise or light fluctuations.

[0095] Specifically, let the audio wake-up threshold be θ. a The visual arousal threshold is θ v Then the wake-up condition can be expressed as: C a >θ a Or C v >θ v ;θ a and θ v Through extensive experimental calibration, we ensure that while effectively detecting fire sources, the false wake-up rate caused by environmental interference is kept at a low level.

[0096] When the above conditions are met, the thermal imaging sensor is activated to collect thermal imaging data.

[0097] Thermal imaging data processing and confidence calculation:

[0098] The data collected by the thermal imaging sensor is represented as a temperature matrix: T(x,y); where x and y represent the horizontal and vertical coordinates of the pixel in the thermal imaging image, respectively, and T(x,y) represents the temperature value of the corresponding pixel.

[0099] Based on ambient temperature reference T base Set the high temperature threshold T th Threshold segmentation is performed on the temperature matrix to obtain the set of pixels in the high-temperature region: Ω hot ={(x,y)|T(x,y)>T th};

[0100] Further calculation of the proportion of high-temperature regions: P hot =|Ω hot | / |Ω|;where |Ω hot | represents the number of pixels in the high-temperature area; |Ω| represents the total number of pixels in the thermal imaging image;

[0101] To describe the spatial variation characteristics of the high-temperature region, the gradient magnitude is calculated from the temperature matrix:

[0102] ;

[0103] The average temperature gradient is obtained by statistically analyzing the gradient within the high-temperature region.

[0104] G=1 / (∣Ω hot ∣)∑(x,y)∈Ω hot G(x,y);

[0105] Based on the proportion of high-temperature regions P hot The thermal imaging confidence level, C, is obtained using a preset thermal anomaly calculation function, along with the average temperature gradient G. t =H(P hot ,G);

[0106] Where H(·) is a combined calculation function used to characterize the degree of thermal anomaly;

[0107] By making preliminary judgments using audio or vision, the thermal imaging sensor is activated only when necessary, reducing the overall power consumption of the system. At the same time, by combining temperature distribution and spatial variation characteristics to calculate the confidence level of thermal imaging, the system can effectively distinguish between a continuous high-temperature background and a real fire source.

[0108] In S4, the audio confidence C is obtained. a Visual confidence level C v and thermal imaging confidence C tSubsequently, the system assigns weights to each confidence level based on environmental benchmarks and performs time series fusion on the confidence levels to obtain stable fusion judgment results.

[0109] Based on the ambient noise and ambient lighting conditions, determine the weighting coefficients for each mode: w a w v w t Among them, w a For audio confidence weights; w v For visual confidence weights; w t The weights are the confidence scores for thermal imaging; each weight satisfies w a +w v +w t =1; In noisy environments, reduce w a When lighting conditions are poor, reduce w v When thermal imaging is activated, increase w t ;

[0110] The time series fusion calculation specifically includes:

[0111] Let the time window length be T, and perform fusion processing on the confidence scores within this window:

[0112] C fusion (t)=Φ(C a (t),C v (t),C t (t))

[0113] Among them, C fusion (t) represents the fusion judgment result at time t; Φ(·) represents the time series fusion function, which is used to integrate the confidence change trends of the current and historical times;

[0114] By introducing a time-series fusion mechanism, misjudgments caused by transient noise, light flicker, or brief thermal anomalies are avoided, making the detection results more stable and reliable.

[0115] In S5, while determining the state of the fire source, the system calculates candidate coordinates based on audio data, visual data, and thermal imaging data, and performs weighted fusion on the candidate coordinates to output the coordinate information of the fire source.

[0116] The arrival time difference Δt is calculated by receiving the sound of the fire source through multiple audio sensors. ij Based on the speed of sound c, the following relationship is established: ∥s a -p i ∥-∥s a -p j ∥=c·Δt ij ; where s a For audio candidate coordinates; pi ,p j These are the coordinates of the audio sensor's location.

[0117] Extracting a set of flame or smoke feature points p from a visual image k Visual candidate coordinates are calculated using geometric mapping relationships: s v =1 / R∑ k=1 R p k ; where s v R represents the visual candidate coordinates; R is the number of feature points.

[0118] Determine the geometric center of the high-temperature region in the thermal imaging image and estimate its spatial location using system calibration parameters: s t =f(T(x,y),"calibration parameters"); where s t Candidate coordinates for thermal imaging;

[0119] Based on the weights corresponding to the confidence levels of each modality, the candidate coordinates are weighted and fused: s=W a ·s a +W v ·s v +W t ·s t Where s represents the output fire source coordinates; W a W v W t To define the fusion weights for the coordinates corresponding to each mode;

[0120] By fusing multimodal candidate coordinates, the stability and accuracy of fire source localization are improved, avoiding the deviation of a single localization method in complex environments.

[0121] An enhanced fire source detection system based on multi-sensor fusion is provided, which is applied to the aforementioned enhanced fire source detection method based on multi-sensor fusion.

[0122] In this embodiment, an industrial park uses an inspection robot to perform nighttime security patrols. The robot is equipped with the enhanced fire source detection system based on multi-sensor fusion of the present invention. The core objective is to accurately and quickly detect and locate fire sources in complex environments such as dim lighting and noise interference from equipment at night, while controlling the robot's power consumption to ensure continuous operation throughout the night.

[0123] After the inspection robot is started, it first performs a sensor self-test and environmental calibration process: the audio sensor collects the background noise of the park at night as a reference noise level, the vision sensor records the ambient light reference under the streetlights, and the thermal imaging sensor captures the normal temperature environment of the park at night as a temperature reference, providing a reference for subsequent dynamic adjustments.

[0124] While the robot is inspecting along a preset route, the audio sensor continuously collects ambient sounds and captures a sound resembling a burst of flames coming from near the outer wall of a workshop. Simultaneously, it extracts the audio energy, spectral centroid, and Mel frequency cepstral coefficient features. The visual sensor simultaneously captures an image of the area, converts it to HSV space, and identifies a light gray, smoke-like color area. The frame difference method detects that the area has a slowly expanding motion feature, and the analysis also shows that the area has a periodic brightness fluctuation of 2-10Hz. The audio confidence and visual confidence are calculated respectively.

[0125] Since both confidence levels met the preset wake-up conditions, the system triggered secondary detection and started the thermal imaging sensor to collect data on the area. The thermal imaging sensor quickly detected a significant high-temperature cluster in the area, and further analysis revealed the proportion of high-temperature areas and significant temperature gradient characteristics, calculating the probability of thermal anomalies.

[0126] Based on nighttime environmental parameters, the system dynamically adjusts the weight allocation: due to insufficient nighttime lighting, the visual confidence weight is reduced; since environmental noise is at the baseline level, the audio confidence weight is maintained; and because the thermal imaging sensor is activated and the data is reliable, the thermal imaging confidence weight is increased. Simultaneously, the detection threshold is dynamically adjusted based on the nighttime temperature baseline. By integrating the various confidence data through a time-series fusion algorithm, the area is ultimately confirmed as the initial fire source.

[0127] Subsequently, the audio localization algorithm used the sound delay difference captured by the microphone array for initial localization, the visual localization algorithm extracted smoke edge feature points for auxiliary localization, and the thermal imaging distance estimation algorithm combined with lens parameters to estimate the distance to the fire source. After weighted fusion of the three localization results, accurate fire source coordinate information was obtained. The robot immediately sent an alarm signal and coordinates to the park monitoring center, providing precise guidance for subsequent firefighting operations.

[0128] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. An enhanced fire source detection method based on multi-sensor fusion, characterized in that: The method includes the following steps: S1. The inspection robot performs initial detection on multiple sensors and collects environmental noise, ambient light and ambient temperature data as environmental benchmarks. S2. Collect audio data and visual data, extract audio features and visual features respectively, and calculate audio confidence and visual confidence based on the audio features and visual features respectively; S3. When the audio confidence or visual confidence meets the preset wake-up condition, the thermal imaging sensor is activated to collect thermal imaging data, and the thermal imaging confidence is calculated based on the thermal imaging data. S4. Based on the environmental benchmark, assign weights to the audio confidence, visual confidence, and thermal imaging confidence, and perform time-series fusion of each confidence to obtain the fusion judgment result; S5. Calculate the corresponding candidate coordinates based on audio data, visual data and thermal imaging data respectively, and perform weighted fusion on the candidate coordinates to output the coordinate information of the fire source.

2. The enhanced fire source detection method based on multi-sensor fusion according to claim 1, characterized in that: In S2, the audio features specifically include: The audio energy is calculated after the audio data is processed into frames; the spectral centroid is calculated after the framed audio signal is transformed in the frequency domain; and the Mel frequency cepstral coefficients are calculated based on the Mel filter bank. The audio confidence score is determined by normalizing the audio energy and spectral centroid of the current audio frame, combined with the spectral centroid range and the corresponding dynamic range of audio energy in the environmental benchmark, and is used to characterize the current audio state.

3. The enhanced fire source detection method based on multi-sensor fusion according to claim 2, characterized in that: In S2, the visual features include: Color matching degree is obtained by color segmentation in the HSV color space based on visual data; motion region ratio is calculated by frame difference method based on continuous visual data frames; and flicker frequency is obtained by frequency domain analysis based on brightness change of target region. The visual confidence score is obtained by combining the color matching degree, the proportion of the motion area, and the flashing frequency according to a preset function calculation relationship, and is used to characterize the current visual state.

4. The enhanced fire source detection method based on multi-sensor fusion according to claim 3, characterized in that: In S3, the wake-up condition specifically includes: the audio confidence level exceeding an audio threshold determined based on environmental noise in an environmental reference, or the visual confidence level exceeding a visual threshold determined based on color matching degree, motion region ratio, and flicker frequency.

5. The enhanced fire source detection method based on multi-sensor fusion according to claim 4, characterized in that: In S3, the calculation of the thermal imaging confidence level includes: Temperature matrix obtained from thermal imaging data; Determine the set of pixels in the high-temperature region from the temperature matrix and calculate the proportion of the high-temperature region; Calculate the temperature gradient of the pixel set in the high-temperature region; The confidence level of thermal imaging is determined based on the calculated relationship between the proportion of high-temperature areas and the temperature gradient.

6. The enhanced fire source detection method based on multi-sensor fusion according to claim 5, characterized in that: In S4, the weight allocation is determined based on the environmental noise and ambient light in the environmental reference, and is used to fuse the audio confidence, visual confidence and thermal imaging confidence.

7. The enhanced fire source detection method based on multi-sensor fusion according to claim 6, characterized in that: In S5, the calculation of the candidate coordinates includes: Calculate candidate audio coordinates based on the time difference of arrival between audio sensors; Calculate visual candidate coordinates based on the position of flame or smoke feature points in the visual sensor; The candidate coordinates for thermal imaging are calculated based on the target size and system calibration parameters in the thermal imaging data.

8. The enhanced fire source detection method based on multi-sensor fusion according to claim 7, characterized in that: S5 also includes: performing a fusion operation on the audio candidate coordinates, visual candidate coordinates, and thermal imaging candidate coordinates according to their corresponding weights to obtain the coordinate information of the fire source.

9. An enhanced fire source detection system based on multi-sensor fusion, characterized in that, The system is applied to an enhanced fire source detection method based on multi-sensor fusion as described in any one of claims 1-8.

Images