A multimodal perception fire detection method based on Transformer temporal feature fusion model

Abstract

The present invention discloses a multimodal perception fire detection method based on a Transformer temporal feature fusion model, which relates to the field of fire safety monitoring technology. The method includes a flame detector that combines video, multi-band infrared, and ultraviolet sensing technologies. The detector is mainly composed of an image sensor, four infrared sensors, and one ultraviolet sensor. The image sensor is mainly used to sense wavelength information within the range of 400nm-800nm. The infrared sensor is particularly sensitive to detecting thermal radiation, while the ultraviolet sensor can detect ultraviolet light generated by flames. The present invention can adapt to situations where some sensor data is missing from the actual data. The multimodal data fusion module designed by the present invention effectively extracts features from sensor data and image data, improving the generalization ability of the model. The present invention can capture the correlation between different sensor data and the deep feature information of the time series pattern unique to fire from long-distance context information, and adaptively fuse historical data and current data.

Description

Technical Field

[0001] The present invention relates to the technical field of fire safety monitoring, and in particular to a multimodal perception fire detection method based on a Transformer temporal feature fusion model. Background Art

[0002] Fire detection technology is a crucial component of modern building safety systems. It utilizes electronics, computers, and sensors to detect, alert, and address fire hazards. With technological advancements, fire detector technology is constantly evolving and improving, aiming to more quickly and accurately detect fires, thereby buying valuable time for evacuation and firefighting.

[0003] Traditional fire detectors rely on smoke sensors, temperature sensors, or single infrared or ultraviolet sensors to detect fires. While these detectors can detect fires to a certain extent, they suffer from high false alarm rates, slow response times, and limited detection ranges. Summary of the Invention

[0004] The present invention provides a multimodal perception fire detection method based on a Transformer temporal feature fusion model to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A multimodal fire detection method based on a Transformer temporal feature fusion model includes a flame detector that combines video, multi-band infrared, and ultraviolet sensing technologies. The detector primarily consists of an image sensor, four infrared sensors, and one ultraviolet sensor. The image sensor primarily senses wavelength information within the 400nm-800nm ​​range. The infrared sensor is particularly sensitive to thermal radiation, while the ultraviolet sensor can detect ultraviolet light generated by flames. The multimodal fire detection method includes the following steps:

[0007] Step 1: Use the detector's built-in camera to collect flame and smoke image data, filter out valid frames, and then use the darklabel tool to annotate the dataset as positive samples. At the same time, use fire images and several interference source image data as test sets. The image sensor is mainly used to sense wavelength information in the range of 400nm-800nm.

[0008] Step 2: Use the detector's built-in four infrared sensors and one ultraviolet sensor to collect 500 sets of real data from the fire experiment scene. Among them, infrared sensor 1 uses a 3.8μm wavelength, infrared sensor 2 uses a 4.4μm wavelength, infrared sensor 3 uses a 5.0μm wavelength, infrared sensor 4 uses a 2.7μm wavelength, and the ultraviolet sensor uses a 190nm-260nm wavelength;

[0009] Step 3: Use data augmentation methods and simulated data missing collection to solve the problem of missing sensor data;

[0010] Step 4: Fire occurs in stages and in a sequential manner. The data formats and modes of various sensors are different, so data preprocessing is particularly important. The sensor data is defined as: S = {s1, s2, ..., s n However, the range of each sensor data is inconsistent, so the data needs to be preprocessed and normalized with the maximum and minimum values, that is, The result after sensor data processing is S'={s'1,s'2,…,s' n}, the image data is defined as: P = {p1, p2, ..., p m}, the image data can be regarded as a three-dimensional matrix, namely w, h, b, where w is the width of the image, h is the height of the image, and b is the band of the image. The image data is also normalized and pre-processed by maximum and minimum normalization;

[0011] Step 5: After preprocessing the data collected by various sensors, the data is sent to the multimodal data fusion module to fuse the data of multiple data types to obtain feature data. Then, a Transformer-based time series feature fusion module is used to fuse the feature data of each time series. Its multi-head self-attention mechanism can adaptively fuse historical data and current data to form a comprehensive fire detection indicator. Finally, through iterative training of the objective function and adjusting the learning rate using the cosine annealing algorithm, the desired training model is obtained.

[0012] Step 6: After the training is completed, the test data to be detected is input into the optimal model, and an alarm is issued according to the set fire alarm threshold to achieve the expected fire detection accuracy and performance.

[0013] A further improvement of the technical solution of the present invention is that the step five further includes the following steps:

[0014] A1: Based on the proposed Multi-Modal Data Fusion (MMDF) module, feature extraction is performed on preprocessed data. For image data, good feature data is extracted through a Convolutional Neural Network (CNN). For numerical sensor data, the data is concatenated and then used to extract effective features using a Multi-Layer Perceptron (MLP), which consists of fully connected layers and activation functions. Finally, the extracted feature data is concatenated through a channel-dimensional cascade operation to obtain the feature data of the input data.

[0015] A2: The Transformer-based time series feature fusion module fuses the feature data extracted by MMDF. First, the current time feature data (i.e., time t) and the fusion of the feature data of the previous time (i.e., time t-1) and the historical feature data (i.e., time out) are combined. t-1 The fused feature data is then used to perform deep feature extraction through a Transformer Encoder-based network structure.

[0016] A3: Iteratively train the model using the objective function, observe the indicators, and continuously adjust and optimize the network parameters until the network model reaches the expected accuracy and obtains the optimal model.

[0017] A further improvement to the technical solution of the present invention is that, in order to accurately detect fire signals under various ambient light conditions, a strategy of adaptive multi-step switching of device detection sensitivity based on ambient light intensity can be adopted. When the ambient light intensity is high, the system will automatically reduce the sensitivity of the fire detection device to reduce the false alarm rate caused by light interference. When the ambient light intensity is low, the system will increase the sensitivity of the device to ensure that the sensor can still efficiently detect fire signals when there is less light interference.

[0018] A further improvement of the technical solution of the present invention is that the adaptive adjustment of sensitivity can be achieved by the following methods:

[0019] B1: Real-time ambient light monitoring: Integrate an ambient light sensor into the detection equipment to monitor the ambient light intensity in real time;

[0020] B2: Multi-step sensitivity adjustment: Based on the feedback from the ambient light sensor, the system uses a multi-step sensitivity adjustment mechanism to automatically adjust the detection sensitivity of the infrared and ultraviolet sensors;

[0021] B3: Intelligent algorithm control: Use intelligent algorithms to analyze ambient light data and dynamically adjust detection sensitivity to ensure optimal detection results under different lighting conditions.

[0022] Due to the adoption of the above technical solution, the present invention has the following technical advancements compared to the prior art:

[0023] 1. The present invention provides a multimodal perception fire detection method based on the Transformer temporal feature fusion model. The present invention adapts to the situation where some sensor data is missing in the actual data through data enhancement method and simulated data missing collection.

[0024] 2. The present invention provides a multimodal perception fire detection method based on the Transformer temporal feature fusion model. The multimodal data fusion module designed in the present invention effectively extracts features from sensor data and image data, thereby improving the generalization ability of the model.

[0025] 3. The present invention provides a multimodal perception fire detection method based on the Transformer time series feature fusion model. The present invention designs a Transformer-based time series feature fusion module, which can capture the correlation between different sensor data and the deep feature information of the time series pattern unique to fire from long-distance context information, and adaptively fuse historical data and current data.

[0026] 4. The present invention provides a multimodal perception fire detection method based on the Transformer temporal feature fusion model. The present invention has a self-learning function and can self-adjust and learn according to the on-site environment after installation, thereby improving the accuracy and efficiency of fire monitoring and fire situation capture, and reducing the probability of false alarms and missed alarms of fires. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart for the implementation of the present invention;

[0028] Figure 2 It is the overall network model flow chart of the algorithm of the present invention;

[0029] Figure 3 Multi-Modal Data Fusion (MMDF) module diagram designed for the algorithm of the present invention;

[0030] Figure 4 Diagram of the temporal feature fusion module designed for the algorithm of the present invention. DETAILED DESCRIPTION

[0031] The present invention is described in further detail below in conjunction with the embodiments:

[0032] Example 1

[0033] like Figure 1-2As shown, the present invention provides a multimodal perception fire detection method based on the Transformer temporal feature fusion model, including a flame detector that combines video, multi-band infrared and ultraviolet sensing technologies. The detector is mainly composed of an image sensor, four infrared sensors and one ultraviolet sensor. The image sensor is mainly used to sense wavelength information in the range of 400nm-800nm. The infrared sensor is particularly sensitive to detecting thermal radiation, while the ultraviolet sensor can detect ultraviolet light generated by flames. The multimodal perception fire detection method includes the following steps:

[0034] Step 1: Use the detector's built-in camera to collect flame and smoke image data, filter out valid frames, and then use the darklabel tool to annotate the dataset as positive samples. At the same time, use fire images and several interference source image data as test sets. The image sensor is mainly used to sense wavelength information in the range of 400nm-800nm.

[0035] Step 2: Use the detector's built-in four infrared sensors and one ultraviolet sensor to collect 500 sets of real data from the fire experiment scene. Among them, infrared sensor 1 uses a 3.8μm wavelength, infrared sensor 2 uses a 4.4μm wavelength, infrared sensor 3 uses a 5.0μm wavelength, infrared sensor 4 uses a 2.7μm wavelength, and the ultraviolet sensor uses a 190nm-260nm wavelength;

[0036] Step 3: Use data augmentation methods and simulated data missing collection to solve the problem of missing sensor data;

[0037] Step 4: Fire occurs in stages and in a sequential manner. The data formats and modes of various sensors are different, so data preprocessing is particularly important. The sensor data is defined as: S = {s1, s2, ..., s n However, the range of each sensor data is inconsistent, so the data needs to be preprocessed and normalized with the maximum and minimum values, that is, The result after sensor data processing is S'={s'1,s'2,…,s' n}, the image data is defined as: P = {p1, p2, ..., p m}, the image data can be regarded as a three-dimensional matrix, namely w, h, b, where w is the width of the image, h is the height of the image, and b is the band of the image. The image data is also normalized and pre-processed by maximum and minimum normalization;

[0038] Step 5: After preprocessing the data collected by various sensors, the data is sent to the multimodal data fusion module to fuse the data of multiple data types to obtain feature data. Then, a Transformer-based time series feature fusion module is used to fuse the feature data of each time series. Its multi-head self-attention mechanism can adaptively fuse historical data and current data to form a comprehensive fire detection indicator. Finally, through iterative training of the objective function and adjusting the learning rate using the cosine annealing algorithm, the desired training model is obtained.

[0039] Step 6: After the training is completed, the test data to be detected is input into the optimal model, and an alarm is issued according to the set fire alarm threshold to achieve the expected fire detection accuracy and performance.

[0040] The step five further comprises the following steps:

[0041] A1: Based on the proposed Multi-Modal Data Fusion (MMDF) module, feature extraction is performed on preprocessed data. For image data, good feature data is extracted through a Convolutional Neural Network (CNN). For numerical sensor data, the data is concatenated and then used to extract effective features using a Multi-Layer Perceptron (MLP), which consists of fully connected layers and activation functions. Finally, the extracted feature data is concatenated through a channel-dimensional cascade operation to obtain the feature data of the input data.

[0042] A2: The Transformer-based time series feature fusion module fuses the feature data extracted by MMDF. First, the current time feature data (i.e., time t) and the fusion of the feature data of the previous time (i.e., time t-1) and the historical feature data (i.e., time out) are combined. t-1 The fused feature data is then used to perform deep feature extraction through a Transformer Encoder-based network structure.

[0043] A3: Iteratively train the model using the objective function, observe the indicators, and continuously adjust and optimize the network parameters until the network model reaches the expected accuracy and obtains the optimal model.

[0044] In this embodiment, the present invention provides a multimodal perception fire detection method based on a Transformer temporal feature fusion model. This method uses data augmentation methods and simulated data loss collection, followed by data set preprocessing through maximum and minimum normalization, to address the problem of missing sensor data and ensure the stability and reliability of the monitoring system. A CNN-based multimodal data fusion module is proposed to adaptively fuse data from different sensor types, thereby improving the comprehensiveness and accuracy of the data. A Transformer-based temporal feature fusion module is proposed, whose multi-head self-attention mechanism can adaptively fuse historical and current data, thereby accurately capturing the development and changes of fire situations and reducing the probability of false alarms and missed fire reports.

[0045] Example 2

[0046] Based on Example 1, the present invention provides a technical solution: Preferably, in order to accurately detect fire signals under various ambient light conditions, a strategy of adaptive multi-step switching of device detection sensitivity based on ambient light intensity can be adopted. When the ambient light intensity is high, the system will automatically reduce the sensitivity of the fire detection device to reduce the false alarm rate caused by light interference. When the ambient light intensity is low, the system will increase the sensitivity of the device to ensure that the sensor can still efficiently detect fire signals when there is less light interference.

[0047] Adaptive adjustment of sensitivity can be achieved in the following ways:

[0048] B1: Real-time ambient light monitoring: Integrate an ambient light sensor into the detection equipment to monitor the ambient light intensity in real time;

[0049] B2: Multi-step sensitivity adjustment: Based on the feedback from the ambient light sensor, the system uses a multi-step sensitivity adjustment mechanism to automatically adjust the detection sensitivity of the infrared and ultraviolet sensors;

[0050] B3: Intelligent algorithm control: Use intelligent algorithms to analyze ambient light data and dynamically adjust detection sensitivity to ensure optimal detection results under different lighting conditions.

[0051] In this embodiment, the detection method of the present invention not only ensures high-precision fire detection, but also has self-learning capabilities. It can automatically adjust and learn according to the on-site environment after deployment, and quickly adapt to environmental changes. It significantly improves the accuracy and efficiency of fire monitoring and prevention, realizes intelligent fire monitoring and prediction of fire situations, and provides important support for future social security and disaster prevention.

[0052] The above generally describes the present invention in detail. However, it is obvious to those skilled in the art that modifications or improvements may be made based on the present invention. Therefore, modifications or improvements that do not depart from the spirit of the present invention are within the scope of protection of the present invention.

Claims

1. A multimodal perception fire detection method based on a Transformer temporal feature fusion model, characterized by: The flame detector includes a combination of video, multi-band infrared and ultraviolet sensing technologies. The detector is mainly composed of an image sensor, four infrared sensors and a ultraviolet sensor. The image sensor is mainly used to sense wavelength information in the range of 400nm-800nm. The infrared sensor is particularly sensitive to detecting thermal radiation, while the ultraviolet sensor can detect ultraviolet light generated by flames. The multimodal perception fire detection method includes the following steps: Step 1: Use the detector's built-in camera to collect flame and smoke image data, filter out valid frames, and then use the darklabel tool to annotate the dataset as positive samples. At the same time, use fire images and several interference source image data as test sets. The image sensor is mainly used to sense wavelength information in the range of 400nm-800nm. Step 2: Use the detector's built-in four infrared sensors and one ultraviolet sensor to collect 500 sets of real data from the fire experiment scene. Among them, infrared sensor 1 uses a 3.8μm wavelength, infrared sensor 2 uses a 4.4μm wavelength, infrared sensor 3 uses a 5.0μm wavelength, infrared sensor 4 uses a 2.7μm wavelength, and the ultraviolet sensor uses a 190nm-260nm wavelength; Step 3: Use data augmentation methods and simulated data missing collection to solve the problem of missing sensor data; Step 4: Fire occurs in stages and in a sequential manner. The data formats and modes of various sensors are different, so data preprocessing is particularly important. The sensor data is defined as: S = {s1, s2, ..., s n However, the range of each sensor data is inconsistent, so the data needs to be preprocessed and normalized with the maximum and minimum values, that is, The result after sensor data processing is S'={s'1,s'2,…,s' n }, the image data is defined as: P = {p1, p2, ..., p m }, the image data can be regarded as a three-dimensional matrix, namely w, h, b, where w is the width of the image, h is the height of the image, and b is the band of the image. The image data is also normalized and pre-processed by maximum and minimum normalization; Step 5: After preprocessing the data collected by various sensors, the data is sent to the multimodal data fusion module to fuse the data of multiple data types to obtain feature data. Then, a Transformer-based time series feature fusion module is used to fuse the feature data of each time series. Its multi-head self-attention mechanism can adaptively fuse historical data and current data to form a comprehensive fire detection indicator. Finally, through iterative training of the objective function and adjusting the learning rate using the cosine annealing algorithm, the desired training model is obtained. Step 6: After the training is completed, the test data to be detected is input into the optimal model, and an alarm is issued according to the set fire alarm threshold to achieve the expected fire detection accuracy and performance.

2. The multimodal perception fire detection method based on the Transformer temporal feature fusion model according to claim 1 is characterized by: The step five further comprises the following steps: A1: Based on the proposed Multi-Modal Data Fusion (MMDF) module, feature extraction is performed on preprocessed data. For image data, good feature data is extracted through a Convolutional Neural Network (CNN). For numerical sensor data, the data is concatenated and then used to extract effective features using a Multi-Layer Perceptron (MLP), which consists of fully connected layers and activation functions. Finally, the extracted feature data is concatenated through a channel-dimensional cascade operation to obtain the feature data of the input data. A2: The Transformer-based time series feature fusion module fuses the feature data extracted by MMDF. First, the current time feature data (i.e., time t) and the fusion of the feature data of the previous time (i.e., time t-1) and the historical feature data (i.e., time out) are combined. t-1 The fused feature data is then used to perform deep feature extraction through a Transformer Encoder-based network structure. A3: Iteratively train the model using the objective function, observe the indicators, and continuously adjust and optimize the network parameters until the network model reaches the expected accuracy and obtains the optimal model.

3. The multimodal perception fire detection method based on the Transformer temporal feature fusion model according to claim 1 is characterized by: In order to accurately detect fire signals under various ambient light conditions, a strategy of adaptive multi-step switching of device detection sensitivity based on ambient light intensity can be adopted. When the ambient light intensity is high, the system will automatically reduce the sensitivity of the fire detection equipment to reduce the false alarm rate caused by light interference. When the ambient light intensity is low, the system will increase the sensitivity of the equipment to ensure that the sensor can still efficiently detect fire signals when there is less light interference.

4. The multimodal perception fire detection method based on the Transformer temporal feature fusion model according to claim 3 is characterized by: Adaptive adjustment of sensitivity can be achieved in the following ways: B1: Real-time ambient light monitoring: Integrate an ambient light sensor into the detection equipment to monitor the ambient light intensity in real time; B2: Multi-step sensitivity adjustment: Based on the feedback from the ambient light sensor, the system uses a multi-step sensitivity adjustment mechanism to automatically adjust the detection sensitivity of the infrared and ultraviolet sensors; B3: Intelligent algorithm control: Use intelligent algorithms to analyze ambient light data and dynamically adjust detection sensitivity to ensure optimal detection results under different lighting conditions.

Images