Multi-modal sensing fire detection method based on Transform time sequence feature fusion model

Abstract

The invention discloses a multi-mode sensing fire detection method based on a Transform time sequence feature fusion model, and relates to the technical field of fire safety monitoring, the method comprises a flame detector combining video, multi-band infrared and ultraviolet sensing technologies, the detector is mainly composed of an image sensor, four infrared sensors and one ultraviolet sensor, the image sensor is mainly used for sensing wavelength information within the range of 400 nm to 800 nm, the infrared sensor is particularly sensitive to detection of heat radiation, and the ultraviolet sensor can detect ultraviolet light generated by flames. According to the invention, the method can adapt to the condition of missing of actual data and partial sensor data, and the designed multi-modal data fusion module carries out effective feature extraction on the sensor data and image data, so that the generalization ability of the model is improved; according to the method, the correlation between different sensor data and the depth feature information of the specific time sequence mode of the fire can be captured from long-distance context information, and the historical data and the current data are adaptively fused.

Description

Technical Field

[0001] The present invention relates to the technical field of fire safety monitoring, and in particular to a multimodal perception fire detection method based on a Transformer temporal feature fusion model. Background Art

[0002] Fire detection technology is an important part of modern building safety systems. It involves the use of electronic technology, computer technology and sensor technology to detect, alarm and deal with fire hazards. With the advancement of science and technology, the technology of fire detectors is also constantly developing and improving, with the aim of detecting the occurrence of fire more quickly and accurately, thereby buying precious time for personnel evacuation and fire fighting.

[0003] Traditional fire detectors rely on smoke sensors, temperature sensors, or single infrared or ultraviolet sensors to detect fires. Although these detectors can detect fires to a certain extent, they have problems such as high false alarm rate, slow response time, and limited detection range. Summary of the invention

[0004] The present invention provides a multimodal perception fire detection method based on a Transformer temporal feature fusion model to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A multimodal perception fire detection method based on a Transformer temporal feature fusion model includes a flame detector that combines video, multi-band infrared and ultraviolet sensing technologies. The detector is mainly composed of an image sensor, four infrared sensors and an ultraviolet sensor. The image sensor is mainly used to sense wavelength information within the range of 400nm-800nm. The infrared sensor is particularly sensitive to detecting thermal radiation, while the ultraviolet sensor can detect ultraviolet light generated by flames. The multimodal perception fire detection method includes the following steps:

[0007] Step 1: Use the built-in camera of the detector to collect flame and smoke image data, filter out valid frames, and then use the darklabel tool to annotate the data set as positive samples. At the same time, use the fire image and several interference source image data as the test set. The image sensor is mainly used to sense wavelength information in the range of 400nm-800nm.

[0008] Step 2: Use the four infrared sensors and one ultraviolet sensor built into the detector to collect 500 sets of real data in the fire experiment scene. Among them, infrared sensor 1 uses a 3.8μm wavelength, infrared sensor 2 uses a 4.4μm wavelength, infrared sensor 3 uses a 5.0μm wavelength, infrared sensor 4 uses a 2.7μm wavelength, and the ultraviolet sensor uses a 190nm-260nm wavelength;

[0009] Step 3: Solve the problem of missing actual sensor data through data enhancement methods and simulated data missing collection;

[0010] Step 4: The occurrence of fire is accompanied by stage and time series characteristics. The data format and mode of each sensor are different, so data preprocessing is particularly important. The sensor data is defined as: S = {s1, s2, ..., s n}, but the range of each sensor data is inconsistent, so the data needs to be preprocessed and normalized with the maximum and minimum values, that is, The result after sensor data processing is S'={s'1,s'2,…,s' n}, the definition of image data is: P = {p1, p2, ..., p m}, the image data can be regarded as a three-dimensional matrix, namely w, h, b, where w is the width of the image, h is the height of the image, and b is the band of the image. The image data is also normalized and pre-processed by maximum and minimum normalization;

[0011] Step 5: After preprocessing the data collected by various sensors, send them to the multimodal data fusion module to fuse data of multiple data types to obtain feature data. Then, a Transformer-based time series feature fusion module is used to fuse the feature data of each time series. Its multi-head self-attention mechanism can adaptively fuse historical data and current data to form a comprehensive fire detection index. Finally, the desired training model is obtained through iterative training of the objective function and the cosine annealing algorithm is used to adjust the learning rate.

[0012] Step 6: After the training is completed, the test data to be detected is input into the optimal model, and an alarm is issued according to the set fire alarm threshold to achieve the expected fire detection accuracy and performance.

[0013] A further improvement of the technical solution of the present invention is that the step five further includes the following steps:

[0014] A1: Based on the proposed Multi-Modal Data Fusion (MMDF) module, the pre-processed data is used for feature extraction. For image data, good feature data is extracted through the Convolutional Neural Network (CNN). For numerical sensor data, the data is concatenated and then used to extract effective features using a Multi-layer Perceptron consisting of a fully connected layer and an activation function. Finally, the extracted feature data is subjected to a cascade operation in the channel dimension to obtain the feature data of the input data.

[0015] A2: The Transformer-based time series feature fusion module fuses the feature data extracted by MMDF. First, the current time feature data (i.e., time t) and the fusion of the feature data of the previous time (i.e., time t-1) and the historical feature data (i.e., out t-1 The fused feature data is then used for deep feature extraction through a Transformer Encoder-based network structure.

[0016] A3: Iteratively train the model through the objective function, observe the indicators and continuously adjust and optimize the network parameters until the network model reaches the expected accuracy and obtains the optimal model.

[0017] A further improvement of the technical solution of the present invention is that in order to accurately detect fire signals under various ambient light conditions, a strategy of adaptive multi-step switching of device detection sensitivity according to ambient light intensity can be adopted. When the ambient light intensity is high, the system will automatically reduce the sensitivity of the fire detection device to reduce the false alarm rate caused by light interference. When the ambient light intensity is low, the system will increase the sensitivity of the device to ensure that the sensor can still efficiently detect fire signals when there is less light interference.

[0018] A further improvement of the technical solution of the present invention is that the adaptive adjustment of sensitivity can be achieved by the following methods:

[0019] B1: Real-time ambient light monitoring: Integrate ambient light sensors in the detection equipment to monitor the ambient light intensity in real time;

[0020] B2: Multi-step sensitivity adjustment: Based on the feedback from the ambient light sensor, the system uses a multi-step sensitivity adjustment mechanism to automatically adjust the detection sensitivity of the infrared and ultraviolet sensors;

[0021] B3: Intelligent algorithm control: Use intelligent algorithms to analyze ambient light data and dynamically adjust detection sensitivity to ensure optimal detection results under different lighting conditions.

[0022] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art:

[0023] 1. The present invention provides a multimodal perception fire detection method based on the Transformer time series feature fusion model. The present invention adapts to the situation where some sensor data of the actual data is missing through data enhancement method and simulated data missing collection.

[0024] 2. The present invention provides a multimodal perception fire detection method based on the Transformer time series feature fusion model. The multimodal data fusion module designed by the present invention effectively extracts features from sensor data and image data, thereby improving the generalization ability of the model.

[0025] 3. The present invention provides a multimodal perception fire detection method based on the Transformer time series feature fusion model. The present invention designs a Transformer-based time series feature fusion module, which can capture the correlation between different sensor data and the deep feature information of the time series pattern unique to the fire from the long-distance context information, and adaptively fuse historical data and current data.

[0026] 4. The present invention provides a multimodal perception fire detection method based on the Transformer time series feature fusion model. The present invention has a self-learning function and can self-adjust and learn according to the on-site environment after installation, thereby improving the accuracy and efficiency of fire monitoring and fire situation capture, and reducing the probability of false alarms and missed alarms of fires. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart for realizing the present invention;

[0028] Figure 2 It is the overall network model flow chart of the algorithm of the present invention;

[0029] Figure 3 Multi-Modal Data Fusion, MMDF module diagram designed for the algorithm of the present invention;

[0030] Figure 4 Diagram of the temporal feature fusion module designed for the algorithm of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below in conjunction with embodiments:

[0032] Example 1

[0033] like Figure 1-2As shown, the present invention provides a multimodal perception fire detection method based on the Transformer temporal feature fusion model, including a flame detector combining video, multi-band infrared and ultraviolet sensing technology, the detector mainly consists of an image sensor, four infrared sensors and an ultraviolet sensor, the image sensor is mainly used to sense wavelength information within the range of 400nm-800nm, the infrared sensor is particularly sensitive to detecting thermal radiation, and the ultraviolet sensor can detect ultraviolet light generated by the flame, the multimodal perception fire detection method includes the following steps:

[0034] Step 1: Use the built-in camera of the detector to collect flame and smoke image data, filter out valid frames, and then use the darklabel tool to annotate the data set as positive samples. At the same time, use the fire image and several interference source image data as the test set. The image sensor is mainly used to sense wavelength information in the range of 400nm-800nm.

[0035] Step 2: Use the four infrared sensors and one ultraviolet sensor built into the detector to collect 500 sets of real data in the fire experiment scene. Among them, infrared sensor 1 uses a 3.8μm wavelength, infrared sensor 2 uses a 4.4μm wavelength, infrared sensor 3 uses a 5.0μm wavelength, infrared sensor 4 uses a 2.7μm wavelength, and the ultraviolet sensor uses a 190nm-260nm wavelength;

[0036] Step 3: Solve the problem of missing actual sensor data through data enhancement methods and simulated data missing collection;

[0037] Step 4: The occurrence of fire is accompanied by stage and time series characteristics. The data format and mode of each sensor are different, so data preprocessing is particularly important. The sensor data is defined as: S = {s1, s2, ..., s n}, but the range of each sensor data is inconsistent, so the data needs to be preprocessed and normalized with the maximum and minimum values, that is, The result after sensor data processing is S'={s'1,s'2,…,s' n}, the definition of image data is: P = {p1, p2, ..., p m}, the image data can be regarded as a three-dimensional matrix, namely w, h, b, where w is the width of the image, h is the height of the image, and b is the band of the image. The image data is also normalized and pre-processed by maximum and minimum normalization;

[0038] Step 5: After preprocessing the data collected by various sensors, send them to the multimodal data fusion module to fuse data of multiple data types to obtain feature data. Then, a Transformer-based time series feature fusion module is used to fuse the feature data of each time series. Its multi-head self-attention mechanism can adaptively fuse historical data and current data to form a comprehensive fire detection index. Finally, the desired training model is obtained through iterative training of the objective function and the cosine annealing algorithm is used to adjust the learning rate.

[0039] Step 6: After the training is completed, the test data to be detected is input into the optimal model, and an alarm is issued according to the set fire alarm threshold to achieve the expected fire detection accuracy and performance.

[0040] The step five also includes the following steps:

[0041] A1: Based on the proposed Multi-Modal Data Fusion (MMDF) module, the pre-processed data is used for feature extraction. For image data, good feature data is extracted through the Convolutional Neural Network (CNN). For numerical sensor data, the data is concatenated and then used to extract effective features using a Multi-layer Perceptron consisting of a fully connected layer and an activation function. Finally, the extracted feature data is subjected to a cascade operation in the channel dimension to obtain the feature data of the input data.

[0042] A2: The Transformer-based time series feature fusion module fuses the feature data extracted by MMDF. First, the current time feature data (i.e., time t) and the fusion of the feature data of the previous time (i.e., time t-1) and the historical feature data (i.e., out t-1 The fused feature data is then used for deep feature extraction through a Transformer Encoder-based network structure.

[0043] A3: Iteratively train the model through the objective function, observe the indicators and continuously adjust and optimize the network parameters until the network model reaches the expected accuracy and obtains the optimal model.

[0044] In this embodiment, the present invention provides a multimodal perception fire detection method based on the Transformer time series feature fusion model. The method uses a data enhancement method and simulates data missing collection, and then pre-processes the data set through maximum and minimum normalization, which solves the problem of sensor data missing and ensures the stability and reliability of the monitoring system. A multimodal data fusion module based on CNN is proposed to adaptively fuse data from different types of sensors, thereby improving the comprehensiveness and accuracy of the data. A time series feature fusion module based on Transformer is proposed, and its multi-head self-attention mechanism can adaptively fuse historical data and current data, thereby accurately capturing the development and changes of the fire situation and reducing the probability of false alarms and missed alarms of fire.

[0045] Example 2

[0046] On the basis of Example 1, the present invention provides a technical solution: Preferably, in order to accurately detect fire signals under various ambient light conditions, a strategy of adaptive multi-step switching of device detection sensitivity based on ambient light intensity can be adopted. When the ambient light intensity is high, the system will automatically reduce the sensitivity of the fire detection device to reduce the false alarm rate caused by light interference. When the ambient light intensity is low, the system will increase the sensitivity of the device to ensure that the sensor can still efficiently detect fire signals when there is less light interference.

[0047] Adaptive adjustment of sensitivity can be achieved in the following ways:

[0048] B1: Real-time ambient light monitoring: Integrate ambient light sensors in the detection equipment to monitor the ambient light intensity in real time;

[0049] B2: Multi-step sensitivity adjustment: Based on the feedback from the ambient light sensor, the system uses a multi-step sensitivity adjustment mechanism to automatically adjust the detection sensitivity of the infrared and ultraviolet sensors;

[0050] B3: Intelligent algorithm control: Use intelligent algorithms to analyze ambient light data and dynamically adjust detection sensitivity to ensure optimal detection results under different lighting conditions.

[0051] In this embodiment, the detection method of the present invention not only ensures the high accuracy of fire detection, but also has self-learning ability. It can automatically adjust and learn according to the on-site environment after deployment, and quickly adapt to environmental changes, which significantly improves the accuracy and efficiency of fire monitoring and prevention, realizes intelligent fire monitoring and prediction of fire situation, and provides important support for the security and disaster prevention of future society.

[0052] The above generally describes the present invention in detail, but it is obvious to a person skilled in the art that some modifications or improvements can be made to the present invention. Therefore, modifications or improvements that do not depart from the spirit of the present invention are within the scope of protection of the present invention.

Claims

1. A multimodal perception fire detection method based on Transformer temporal feature fusion model, characterized by: The invention comprises a flame detector combining video, multi-band infrared and ultraviolet sensing technologies. The detector is mainly composed of an image sensor, four infrared sensors and one ultraviolet sensor. The image sensor is mainly used to sense wavelength information within the range of 400nm-800nm. The infrared sensor is particularly sensitive to detecting thermal radiation, while the ultraviolet sensor can detect ultraviolet light generated by flames. The multi-modal perception fire detection method comprises the following steps: Step 1: Use the built-in camera of the detector to collect flame and smoke image data, filter out valid frames, and then use the darklabel tool to annotate the data set as positive samples. At the same time, use the fire image and several interference source image data as the test set. The image sensor is mainly used to sense wavelength information in the range of 400nm-800nm. Step 2: Use the four infrared sensors and one ultraviolet sensor built into the detector to collect 500 sets of real data in the fire experiment scene. Among them, infrared sensor 1 uses a 3.8μm wavelength, infrared sensor 2 uses a 4.4μm wavelength, infrared sensor 3 uses a 5.0μm wavelength, infrared sensor 4 uses a 2.7μm wavelength, and the ultraviolet sensor uses a 190nm-260nm wavelength; Step 3: Solve the problem of missing actual sensor data through data enhancement methods and simulated data missing collection; Step 4: The occurrence of fire is accompanied by stage and time series characteristics. The data format and mode of each sensor are different, so data preprocessing is particularly important. The sensor data is defined as: S = {s1, s2, ..., s n }, but the range of each sensor data is inconsistent, so the data needs to be preprocessed and normalized with the maximum and minimum values, that is, The result after sensor data processing is S'={s'1,s'2,…,s' n }, the definition of image data is: P = {p1, p2, ..., p m }, the image data can be regarded as a three-dimensional matrix, namely w, h, b, where w is the width of the image, h is the height of the image, and b is the band of the image. The image data is also normalized and pre-processed by maximum and minimum normalization; Step 5: After preprocessing the data collected by various sensors, send them to the multimodal data fusion module to fuse data of multiple data types to obtain feature data. Then, a Transformer-based time series feature fusion module is used to fuse the feature data of each time series. Its multi-head self-attention mechanism can adaptively fuse historical data and current data to form a comprehensive fire detection index. Finally, the desired training model is obtained through iterative training of the objective function and the cosine annealing algorithm is used to adjust the learning rate. Step 6: After the training is completed, the test data to be detected is input into the optimal model, and an alarm is issued according to the set fire alarm threshold to achieve the expected fire detection accuracy and performance.

2. According to claim 1, a multimodal perception fire detection method based on Transformer temporal feature fusion model is characterized by: The step five also includes the following steps: A1: Based on the proposed Multi-Modal Data Fusion (MMDF) module, the pre-processed data is used for feature extraction. For image data, good feature data is extracted through the Convolutional Neural Network (CNN). For numerical sensor data, the data is concatenated and then used to extract effective features using a Multi-layer Perceptron consisting of a fully connected layer and an activation function. Finally, the extracted feature data is subjected to a cascade operation in the channel dimension to obtain the feature data of the input data. A2: The Transformer-based time series feature fusion module fuses the feature data extracted by MMDF. First, the current time feature data (i.e., time t) and the fusion of the feature data of the previous time (i.e., time t-1) and the historical feature data (i.e., out t-1 The fused feature data is then used for deep feature extraction through a Transformer Encoder-based network structure. A3: Iteratively train the model through the objective function, observe the indicators and continuously adjust and optimize the network parameters until the network model reaches the expected accuracy and obtains the optimal model.

3. According to claim 1, a multimodal perception fire detection method based on Transformer temporal feature fusion model is characterized by: In order to accurately detect fire signals under various ambient light conditions, a strategy of adaptive multi-step switching of device detection sensitivity based on ambient light intensity can be adopted. When the ambient light intensity is high, the system will automatically reduce the sensitivity of the fire detection equipment to reduce the false alarm rate caused by light interference. When the ambient light intensity is low, the system will increase the sensitivity of the equipment to ensure that the sensor can still efficiently detect fire signals when there is less light interference.

4. According to claim 3, a multi-modal perception fire detection method based on Transformer temporal feature fusion model is characterized in that: Adaptive adjustment of sensitivity can be achieved in the following ways: B1: Real-time ambient light monitoring: Integrate ambient light sensors in the detection equipment to monitor the ambient light intensity in real time; B2: Multi-step sensitivity adjustment: Based on the feedback from the ambient light sensor, the system uses a multi-step sensitivity adjustment mechanism to automatically adjust the detection sensitivity of the infrared and ultraviolet sensors; B3: Intelligent algorithm control: Use intelligent algorithms to analyze ambient light data and dynamically adjust detection sensitivity to ensure optimal detection results under different lighting conditions.

Images