Electrical fire detection method based on multi-modal information fusion
By preprocessing and feature extraction of electrical signals and visual information, and using the Transformer architecture to achieve efficient fusion of multimodal information, the problem of low fusion efficiency between electrical signals and visual information in the prior art is solved, and the fire detection is significantly improved Accuracy and real-time.
Patent Information
- Application Number
- CN202411496949.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-02-14
AI Technical Summary
The existing multimodal fire detection technology has not yet achieved efficient fusion of electrical signals and visual information. In particular, how to extract useful features from multimodal data and maintain their complementarity after fusion is still an urgent problem to be solved.
A variety of sensors are used to obtain raw data, and the time series and visual information are standardized and denoised through data preprocessing tools. Then, the features of the time series and visual sequence are extracted using an encoder based on PatchTST and ViT, and spliced them through a feature connector and input them into a multimodal feature fusion device based on Transformer to achieve efficient fusion of information.
Through the integration of multimodal information, the comprehensive understanding of electrical fault fire scenes and the ability to judge electrical fire information are significantly improved, the missed and false alarms are reduced, and the accuracy and real-time nature of fire detection are improved.
Smart Images
Figure CN119441978A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of electrical fire detection, in particular to an electrical fire detection method based on multi-modal information fusion. Background Art
[0002] With the continuous improvement of the level of social electrification, electrical fires have become one of the important factors causing major disasters. Due to the complexity of electrical equipment and the diversity of the use environment, the occurrence of electrical fires is usually accompanied by complex electrical faults, such as short circuits, overloads, and leakage. In addition, it is often difficult to accurately detect the early stage of a fire through single-mode sensor information (such as current or temperature). Traditional fire warning systems based on single electrical signals or ambient temperature sensors cannot capture early signals of a fire in a timely manner, which can easily lead to missed reports and false alarms. Therefore, how to accurately predict and detect fires in complex and changeable electrical scenarios has become a hot topic and difficulty in current research.
[0003] At present, with the improvement of sensor technology and computing power, the fusion technology of multimodal information has been widely used in intelligent detection of various complex scenarios. Especially in the field of electrical fire detection, combining data from different modalities (such as electrical signals and video information) can effectively make up for the limitations of single modal information and provide more decision-making basis for early fire detection. However, the existing multimodal fire detection technology has not yet achieved efficient fusion of electrical signals and visual information. In particular, how to extract useful features from multimodal data and maintain their complementarity after fusion is still an urgent problem to be solved. In addition, how to achieve automated and standardized processes in the preprocessing and sample construction of multimodal data is also a key technical challenge to improve the efficiency and accuracy of fire detection. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide an electrical fire detection method based on multimodal information fusion to enhance the comprehensive understanding of electrical fault fire scenes and the ability to distinguish electrical fire information.
[0005] To achieve the above object, the present invention adopts the following technical solution: an electrical fire detection method based on multimodal information fusion, comprising the following steps:
[0006] Step 1: Obtain raw data through multiple sensors, namely time series signals and visual sequence information related to electrical faults; the system uses data preprocessing tools to standardize and denoise the raw time series and visual information;
[0007] Step 2: Sample preparation tools are used to build high-quality training and test datasets; the preprocessed time series and visual sequence information will be input into their respective encoders; the time series data will be feature extracted by the PatchTST-based encoder, while the visual data will be feature extracted by the ViT-based encoder;
[0008] Step 3: The system concatenates the feature representations of the time series and the visual sequence through the feature connector, and further inputs them into the Transformer-based multimodal feature fuser; the Transformer-based multimodal feature fuser uses the self-attention mechanism of the Transformer architecture to capture the cross-modal correlation between the time series and the visual sequence, thereby achieving efficient fusion of information;
[0009] Step 4: The fused feature representation will be used for classification tasks or regression tasks. The specific task objectives are set according to the needs of the application scenario. Through the branch task module, the risk of electrical fires is predicted, or the time and intensity of fires are estimated.
[0010] In a preferred embodiment, the data preprocessing in step 1 includes the following steps:
[0011] Step A1: Multimodal data synchronization: For the original multimodal experimental data collected by multi-source sensors, the data preprocessing tool first performs synchronization processing on the multimodal data; multi-video stream playback control and multi-electrical signal experimental visualization methods are used to ensure the time synchronization of different modal data, and the occurrence time of key events is marked by timestamps;
[0012] Step A2: Multimodal data alignment: Based on synchronization, the data alignment and recording module is further used to obtain the precise moment of the video and electrical signal when the electrical fault occurs; the multimodal data alignment tool is used to calibrate the time clues of the electrical signal and the video sequence to ensure the consistency of different modal data on the time axis before data fusion;
[0013] Step A3: Data cropping and optimization: The preprocessing tool performs data cropping and noise reduction processing; the specific operations include cropping video frames and signal data noise reduction, removing irrelevant or interfering data, and retaining the core information related to electrical fires;
[0014] Step A4: Formatting and storage: The data after synchronization, alignment and optimization will be formatted and stored to form pre-processed high-quality data sets; these data sets will be used for subsequent sample construction and model training to ensure the integrity and consistency of the data during the multimodal fusion process.
[0015] In a preferred embodiment, the sample preparation tool in step 2 includes:
[0016] Step B1: Sample data acquisition: The sample preparation tool first extracts the time series and corresponding visual sequence signals from the preprocessed data set; through the sliding window method, the tool automatically generates samples from the continuous time series data according to the set window size and step size; in addition, the visual sequence frames matching the time window are extracted to ensure the synchronization of time and space information; the sample data extraction module ensures the efficient acquisition and pairing of data under different modalities;
[0017] Step B2: Sample label acquisition: While extracting samples, the sample preparation tool automatically assigns labels to each sample according to the requirements of the fire detection task; the label acquisition module assigns corresponding status labels to each sample through timestamp and event trigger mechanism; the label acquisition module automatically annotates samples according to the time related to the fire event to ensure the accurate match between label data and sample data; at the same time, dynamic labels are assigned to reflect the time change information during the fire process;
[0018] Step B3: Sample data standardization: The sample preparation tool standardizes the extracted time series data and visual series data; standardization operations include removing outliers and normalizing to ensure that data from different sources have the same scale and range;
[0019] Step B4: Sample storage and management: The processed sample data includes time series data, image sequence data and their corresponding label information; all sample data will be stored in a unified formatted data set for subsequent model training and verification.
[0020] In a preferred embodiment, the time series processing based on PatchTST in step 2 includes:
[0021] Step C1: Input multidimensional time series: First, receive multidimensional electrical signals as input. The multidimensional time series data contains rich electrical fault information.
[0022] Step C2: Instance normalization and block processing: The input multi-dimensional time series data is first processed by instance normalization; the normalized time series data is divided into multiple time patches, each of which represents the change of the signal within a specific time window;
[0023] Step C3: Projection and position embedding: After the block processing is completed, each time block will be projected into a high-dimensional space, and the time series information will be introduced through the position embedding technology;
[0024] Step C4: Time Series-Transformer Encoder: After the above processing, the block-wise time series data will be input into the time series-Transformer encoder; the Transformer encoder adopts a multi-head attention mechanism;
[0025] Step C5: Flattening and linear output: The encoded high-dimensional feature vector is flattened, and the multi-dimensional features are merged into a linear sequence representation Seq-token; this linear sequence representation contains the temporal and spatial features of all input electrical signals, providing an information basis for subsequent feature fusion and classification tasks.
[0026] In a preferred embodiment, the ViT-based image information processing in step 2 includes:
[0027] Step D1: Input image sequence: First, receive the fire video image sequence collected by the surveillance camera. These sequence data contain important visual information during the fire. The video frames are passed into ViT as the image input sequence for subsequent processing.
[0028] Step D2: Block processing and linear projection: The input image sequence is first divided into multiple fixed-size image patches; each image patch represents a local spatial region in the video frame; each image patch is linearly projected to map the two-dimensional image data into a high-dimensional feature space, generating a series of image embedding vectors Image tokens;
[0029] Step D3: Position Embedding: ViT uses position embedding technology; after embedding the image blocks into high-dimensional space, position embedding adds position information to each image block, thus ensuring that the model can recognize the spatial structure and position dependencies in the image;
[0030] Step D4: Image sequence-Transformer encoder: After the block segmentation and embedding are completed, the image block embedding vector is input into the image sequence-Transformer encoder;
[0031] Step D5: Classification tag with category mark: During the encoding process, ViT introduces a specific category tag, namely the classification tag Cls-token, for each image sequence to represent the category characteristics of the fire; the classification tag is input into the Transformer encoder together with the image embedding, and through interaction with the features of other image blocks, a comprehensive classification feature representation is finally generated to help the model identify the occurrence and category of fire.
[0032] In a preferred embodiment, the Transformer-based multimodal information fusion in step 3 includes:
[0033] Step E1: Feature connection: The first step of multimodal information fusion is to connect the features extracted from different modalities. The time series features output by the PatchTST module and the image features output by the ViT module represent different aspects of electrical fires. Through the feature connection module, the features of these two modalities are connected in series to form a comprehensive feature representation. This comprehensive feature representation contains the spatiotemporal dependency features of electrical signals and visual information, laying the foundation for subsequent deep fusion.
[0034] Step E2: Fused Transformer Encoder: The comprehensive feature representation is input into the fused Transformer encoder for deep fusion; the fused Transformer encoder adopts a similar structure to the unimodal Transformer, including a multi-head attention mechanism and a feed-forward network;
[0035] During the fusion process, the fusion Transformer can simultaneously focus on the long-range dependencies between multi-dimensional time series and image features, and dynamically adjust the weights of different modal features using the self-attention mechanism;
[0036] Step E3: Addition and normalization: In the multi-layer encoding process of the fused Transformer, the system also performs addition and normalization operations; by standardizing the fused feature representation, the scale differences between features are eliminated;
[0037] Step E4: Fused feature representation: Finally, the feature representation of the fused Transformer encoder output is a deep comprehensive representation of multimodal information, including multi-level features of time series and image information.
[0038] In a preferred embodiment, the classification task in step 4 includes:
[0039] (1) Feature input: The fused feature representation generated by the fused Transformer is used as the input of the classifier;
[0040] (2) Fully connected layer: The classifier consists of a multi-layer fully connected network, which is used to map the fused feature representation to different category label spaces;
[0041] (3) Activation function: The output layer of the classifier usually uses a softmax activation function to ensure that the output is in the form of a probability distribution, representing the probability of various types of fire or no fire;
[0042] (4) Output results: Based on the predicted category labels, the classification task ultimately determines whether a fire has occurred and its type.
[0043] In a preferred embodiment, the regression task in step 4 includes:
[0044] 1) Feature input: The fused feature representation is used as the input of the regressor;
[0045] 2) Fully connected layer: The regressor consists of multiple layers of fully connected networks, extracting features layer by layer and outputting a continuous value;
[0046] 3) Activation function: In regression tasks, the output layer usually uses a linear activation function to ensure that the regression result is a continuous value;
[0047] 4) Output result: The output of the regression task is a continuous physical quantity.
[0048] Compared with the prior art, the present invention has the following beneficial effects:
[0049] 1. Automated integrated system for multimodal data preprocessing and sample construction
[0050] The innovation involves proposing an automated integrated system for multimodal data preprocessing and sample construction. The system automatically constructs high-quality training and test sample data sets for different modes of electrical signals and video information through preprocessing steps such as standardization, synchronization, cropping, and alignment. By uniformly processing and managing multimodal data, the system can ensure efficient synchronization and consistency between time series signals and visual information, providing accurate and complete data support for subsequent model training. This automated integrated system significantly improves data processing efficiency and ensures the effectiveness of multimodal information fusion.
[0051] 2. Electrical fire detection method based on complementary characteristics of electrical signals and video information
[0052] This innovation involves the introduction of multimodal fusion technology of electrical signals and video information into the electrical fire detection method, which is particularly suitable for complex scenes of electrical fires. This method effectively enhances the detection capability of electrical faults and fire events by combining the time series data of electrical signals with the visual information of fire monitoring videos. Electrical signals can reflect abnormal current behavior during faults, while video information provides the system with visual evidence of fire events. The complementary characteristics between the two significantly improve the accuracy and real-time performance of fire detection. Compared with traditional single-modal methods, this inventive method provides more comprehensive scene analysis and risk warning capabilities.
[0053] 3. Electrical fire detection model based on Transformer-based multimodal information fusion mechanism
[0054] This innovation involves a multimodal information fusion mechanism based on the Transformer architecture. The model uses the Transformer's self-attention mechanism to effectively integrate multiple data features from time series signals (electrical signals) and visual information (video). Through the feature fusion module, the model can deeply explore the correlation and complementary information between different modalities to form a fused feature representation. This fusion mechanism not only retains the spatiotemporal dependency of single modal features, but also further enhances the detection capability of complex fire scenes through a multi-head attention mechanism, providing a strong model support for the efficient prediction and accurate identification of electrical fires. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a schematic diagram of the overall process of a preferred embodiment of the present invention;
[0056] Figure 2 A schematic diagram of a data processing flow in a preferred embodiment of the present invention;
[0057] Figure 3 A schematic diagram of a sample preparation process according to a preferred embodiment of the present invention;
[0058] Figure 4 A schematic diagram of a component architecture of time series processing based on PatchTST in a preferred embodiment of the present invention;
[0059] Figure 5 A schematic diagram of a component architecture of image information processing based on ViT according to a preferred embodiment of the present invention;
[0060] Figure 6 Schematic diagram of component architecture of Transformer-based multimodal information fusion according to a preferred embodiment of the present invention;
[0061] Figure 7 The figure is a flow chart of training and reasoning of a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0062] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0063] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0064] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.
[0065] refer to Figure 1-7 As shown in the figure, an electrical fire detection method based on multimodal information fusion includes the following key steps:
[0066] 1. Obtain time series signals (such as current signals) and visual sequence information (such as surveillance camera images) related to electrical faults through a variety of sensors. These raw data contain rich information about the early stages of electrical fires, but there are problems such as data dimension differences and information redundancy when used directly for detection. To solve these problems, the system uses data preprocessing tools to standardize and denoise the original time series and visual information to ensure data quality and consistency.
[0067] 2. Sample preparation tools are used to build high-quality training and test datasets. The preprocessed time series and visual sequence information will be input into their respective encoders. The time series data will be feature extracted by the PatchTST-based encoder, while the visual data will be feature extracted by the ViT-based encoder. The design of these two encoders can effectively capture the spatiotemporal features in their respective modalities.
[0068] 3. The system concatenates the feature representations of the time series and visual sequence through feature connectors, and further inputs them into the Transformer-based multimodal feature fuser. The fuser uses the self-attention mechanism of the Transformer architecture to capture the cross-modal correlation between the time series and the visual sequence, thereby achieving efficient fusion of information.
[0069] 4. The fused feature representation will be used for classification or regression tasks, and the specific task objectives are set according to the needs of the application scenario. Through the branch task module, this method can accurately predict the risk of electrical fires, or estimate the time and intensity of fires.
[0070] The entire method is centered on precise synchronization and deep integration of multimodal information, which significantly enhances the ability to understand and judge complex electrical fire scenarios and provides strong technical support for early fire warning.
[0071] 1) [Data acquisition and sample preparation]
[0072] 1.1) Data preprocessing tools
[0073] In the multimodal information fusion process of the electrical fire detection method of the present invention, data preprocessing is a key step to ensure the efficient fusion of time series and visual sequence information. The main function of the data preprocessing tool is to synchronize, align and reduce noise of multimodal experimental data from electrical signals and visual monitoring to improve the accuracy and effectiveness of multimodal data fusion. The specific process of the tool is as follows Figure 2 As shown, it includes the following steps:
[0074] 1. Multimodal data synchronization: For the original multimodal experimental data (including electrical signals and video stream data) collected by multi-source sensors, the data preprocessing tool first performs multimodal data synchronization. The system uses methods such as multi-video stream playback control and multi-electrical signal experiment visualization to ensure the time synchronization of different modal data, and annotate the occurrence time of key events through timestamps.
[0075] 2. Multimodal data alignment: On the basis of synchronization, the system further uses the data alignment and recording module to obtain the precise moment of the video and electrical signal when the electrical fault occurs. The multimodal data alignment tool can calibrate the time clues of the electrical signal and the video sequence to ensure the consistency of different modal data on the time axis before data fusion.
[0076] 3. Data trimming and optimization: In order to reduce data redundancy and enhance key features, the preprocessing tool also performs data trimming and noise reduction. Specific operations include trimming video frames and signal data noise reduction, removing irrelevant or interfering data, and retaining core information related to electrical fires. This step significantly improves the efficiency of subsequent sample processing and model training.
[0077] 4. Formatted storage: After synchronization, alignment and optimization, the data will be formatted and stored to form a pre-processed high-quality data set. These data sets will be used for subsequent sample construction and model training to ensure the integrity and consistency of the data in the multimodal fusion process.
[0078] Through the above steps, the data preprocessing tool of the present invention can effectively process multimodal experimental data, ensure the accurate synchronization and alignment of time series and visual information, and lay a solid foundation for the fusion of multimodal features.
[0079] 1.2) [Sample preparation tools]
[0080] The sample preparation tool of the present invention is designed to provide high-quality training and test samples for electrical fire detection based on multimodal information fusion. The tool extracts time series and visual sequence data from the preprocessed data set through a series of steps, automatically generates annotated samples, and ensures that the sample data has high accuracy and multimodal feature consistency. The specific process is as follows Figure 3 As shown, it includes the following main steps:
[0081] 1. Sample data acquisition: The sample preparation tool first extracts the time series and corresponding visual sequence signals from the preprocessed data set. Using the sliding window method, the tool automatically generates samples from continuous time series data according to the set window size and step size. In addition, the system also extracts visual sequence frames that match the time window to ensure the synchronization of time and space information. The sample data extraction module ensures efficient acquisition and pairing of data under different modalities.
[0082] 2. Sample label acquisition: While extracting samples, the sample preparation tool automatically assigns labels to each sample according to the requirements of the fire detection task. The label acquisition module assigns corresponding status labels to each sample, such as "normal" or "fire", through timestamp and event triggering mechanisms. The module automatically labels samples according to the moments related to the fire event to ensure that the label data accurately matches the sample data. At the same time, the system will assign dynamic labels to reflect the time-varying information during the fire process.
[0083] 3. Sample data standardization: In order to ensure the consistency and availability of training samples, the sample preparation tool standardizes the extracted time series data and visual sequence data. Standardization operations include removing outliers and normalization to ensure that data from different sources have the same scale and range. In addition, the tool also supports batch processing of large amounts of data, greatly improving the efficiency of sample preparation.
[0084] 4. Sample storage and management: The processed sample data includes time series data, image sequence data and their corresponding label information. All sample data will be stored in a unified formatted data set for subsequent model training and verification. Through this process, the sample preparation tool ensures the integrity and efficient management of multimodal data, providing a solid sample foundation for subsequent fire detection models.
[0085] 2) [Model architecture and component structure]
[0086] 2.1) [Time series processing based on PatchTST]
[0087] In the electrical fire detection method of the present invention, the electrical signal processing module adopts a multi-dimensional time series processing module based on PatchTST, which aims to extract effective features from complex multi-dimensional electrical signals and improve the accuracy and real-time performance of fire detection. PatchTST is a model that efficiently processes long time series data. It divides the long sequence into multiple small blocks for encoding processing to cope with the multi-dimensional features and time dependencies in the electrical signal. The specific processing flow is as follows: Figure 4 As shown, it includes the following key steps:
[0088] 1. Input multidimensional time series: The system first receives multidimensional electrical signals as input. These signals usually come from different sensors of electrical equipment and have different physical quantity characteristics. Multidimensional time series data contains rich electrical fault information, but due to its high dimensionality and complexity, it is difficult to directly apply it to traditional fire detection models.
[0089] 2. Instance normalization and block processing: The input multi-dimensional time series data is first processed by instance normalization to ensure that electrical signals of different dimensions are processed at a unified scale. The normalized time series data is divided into multiple time patches, each of which represents the change of the signal within a specific time window. This step not only reduces the computational burden of the model in processing long time series, but also preserves the temporal local characteristics of the signal.
[0090] 3. Projection and position embedding: After the block processing is completed, each time block will be projected into a high-dimensional space, and the time series information of the time series will be introduced through position embedding technology. Position embedding ensures that the model can capture the time dependency characteristics in the time series and helps the model identify the correlation between different time segments.
[0091] 4. Time Series-Transformer Encoder: After the above processing, the block-wise time series data will be input into the time series-Transformer encoder. The Transformer encoder uses a multi-head attention mechanism, which can simultaneously focus on the relationship between multiple time points, thereby effectively capturing the long-range dependency features in multi-dimensional electrical signals. The structure of the encoder is shown on the left side of the block diagram. Through multi-layer stacked feedforward networks and multi-head attention modules, a high-dimensional vector representation representing the characteristics of the electrical signal is further refined.
[0092] 5. Flattening and linear output: The encoded high-dimensional feature vector is flattened to merge the multi-dimensional features into a linear sequence representation (Seq-token). This linear sequence representation contains the temporal and spatial features of all input electrical signals, providing a rich information basis for subsequent feature fusion and classification tasks.
[0093] Through multi-dimensional time series processing of PatchTST, the present invention can effectively deal with high-dimensional complex electrical signals that appear in the process of electrical fire detection. The module can extract representative features from long time series and deeply mine the spatiotemporal dependencies in the signal through the Transformer encoder, providing more accurate and stable input for the fire detection model.
[0094] 2.2) [Image information processing based on ViT]
[0095] In the electrical fire detection method of the present invention, the image information processing module uses a visual Transformer (ViT)-based architecture to process the visual sequence data of the fire monitoring video. ViT divides the video frame sequence into multiple image blocks and introduces a Transformer encoder to extract spatiotemporal features, effectively improving the ability to recognize key information in the fire video. Figure 5 The image processing process based on ViT is shown, which includes the following steps:
[0096] 1. Input image sequence: The system first receives the fire video image sequence collected by the surveillance camera. These sequence data contain important visual information during the fire. The video frames are passed to ViT as image input sequences for subsequent processing. Since video frames contain information in spatial and temporal dimensions, effective processing and feature extraction can significantly improve the accuracy of fire detection.
[0097] 2. Block processing and linear projection: The input image sequence is first divided into multiple fixed-size image blocks (patches). Each image block represents a local spatial area in the video frame. This block division method can reduce computational complexity and retain local information of the image. Next, the system performs linear projection on each image block, maps the two-dimensional image data into a high-dimensional feature space, and generates a series of image embedding vectors (image tokens).
[0098] 3. Position embedding: In order to capture the spatial relationship between image blocks, ViT uses position embedding technology. After embedding the image blocks into a high-dimensional space, position embedding adds position information to each image block, thereby ensuring that the model can recognize the spatial structure and position dependency in the image. This step helps enhance the model's ability to determine the location of the fire.
[0099] 4. Image sequence-Transformer encoder: After the block segmentation and embedding are completed, the image block embedding vector is input into the image sequence-Transformer encoder. The encoder can effectively capture the spatiotemporal dependency features in the video frame sequence through the multi-head attention mechanism and layer normalization module. The multi-head attention mechanism allows the model to focus on the long-range relationship between multiple image blocks, while layer normalization helps to improve the training stability of the model. The encoded feature vectors (Cls-token and Img-token) will be used for the classification and recognition of fire scenes.
[0100] 5. Classification token with category mark: During the encoding process, ViT also introduces a specific category mark, namely the classification token (Cls-token), for each image sequence to represent the category characteristics of the fire. This classification token is input into the Transformer encoder together with the image embedding, and through interaction with the features of other image blocks, a comprehensive classification feature representation is finally generated to help the model identify the occurrence and category of the fire.
[0101] Through the ViT-based image information processing module, the present invention can efficiently extract key visual features related to electrical fires from surveillance videos. This module not only retains the spatiotemporal dependency information in the image sequence, but also enhances the accuracy and real-time performance of fire detection through the powerful learning ability of the Transformer architecture, laying a solid visual foundation for multimodal information fusion.
[0102] 2.3) [Multimodal information fusion based on Transformer]
[0103] The multimodal information fusion module in the present invention is based on the Transformer architecture and is designed to integrate the multidimensional time series features (multidimensional time series tokens) output from the PatchTST module and the image features (image sequence tokens) output from the ViT module. By effectively fusing information from different modalities, the system can improve the comprehensive understanding and discrimination capabilities of electrical fire detection. The specific multimodal fusion process is as follows: Figure 6 As shown, it includes the following steps:
[0104] 1. Feature connection: The first step of multimodal information fusion is to connect the features extracted from different modalities. The time series features output by the PatchTST module and the image features output by the ViT module represent different aspects of electrical fires. Through the feature connection module, the features of these two modalities are connected in series to form a comprehensive feature representation. This comprehensive feature representation contains the spatiotemporal dependency features of electrical signals and visual information, laying the foundation for subsequent deep fusion.
[0105] 2. Fusion Transformer Encoder: The comprehensive feature representation is input into the fusion Transformer encoder for deep fusion. The fusion Transformer encoder adopts a similar structure to the unimodal Transformer, including a multi-head attention mechanism and a feedforward network. This module can capture the cross-modal correlation between time series features and image features, and discover the complementary information between different modal features through the multi-head attention mechanism, thereby achieving deep fusion of features.
[0106] During the fusion process, the fused Transformer can simultaneously focus on the long-range dependencies between multidimensional time series and image features, and dynamically adjust the weights of different modal features using the self-attention mechanism. This approach ensures that the fused feature representation can more comprehensively reflect the multimodal signal characteristics when a fire occurs.
[0107] 3. Addition and normalization: In the multi-layer encoding process of the fused Transformer, the system also performs addition and normalization operations. This process standardizes the fused feature representation to eliminate the scale differences between features, ensuring the consistency of feature representation and the stability of model training. Addition and normalization further improve the robustness of the model when fusing features of different modalities.
[0108] 4. Fusion feature representation: Finally, the feature representation of the fusion Transformer encoder output (fusion feature representation) is a deep comprehensive representation of multimodal information, including multi-level features of time series and image information. This fusion feature representation can provide a more comprehensive input for the fire detection model, thereby improving the accuracy and effectiveness of electrical fire detection.
[0109] Through the Transformer-based multimodal information fusion module, the present invention can make full use of the complementary characteristics of time series and image information to achieve more accurate judgment and early warning in electrical fire detection. The self-attention mechanism of the fusion module ensures information sharing and interaction between different modalities, thereby improving the model's adaptability to complex fire scenes.
[0110] 2.4) [Classification task or regression task]
[0111] In the electrical fire detection method of the present invention, the task module undertakes the final discrimination task based on the fusion feature representation. According to the requirements of different application scenarios, the tasks can be divided into classification tasks and regression tasks. The classification task is mainly used to determine whether an electrical fire has occurred or to identify the type of fire, while the regression task focuses on predicting the specific parameters of the fire development, such as the time of occurrence of the fire, temperature changes, etc. The implementation process of the specific task is as follows:
[0112] 1. Classification task:
[0113] The classification task is mainly used to determine whether an electrical fire has occurred and to identify the type of fire. After completing the multimodal feature fusion, the fused feature representation will be input into the classifier. The classifier uses a fully connected network and can classify electrical fire scenes based on the fused features. The specific classification process includes:
[0114] (1) Feature input: The fused feature representation generated by the fused Transformer is used as the input of the classifier.
[0115] (2) Fully connected layer: The classifier consists of a multi-layer fully connected network, which is used to map the fused feature representation to different category label spaces.
[0116] (3) Activation function: In order to distinguish the type of fire, the output layer of the classifier usually uses a softmax activation function to ensure that the output is in the form of a probability distribution, indicating the possibility of various types of fire or no fire.
[0117] (4) Output results: Based on the predicted category labels, the classification task ultimately determines whether a fire has occurred and its type.
[0118] Through the classification task, the present invention can quickly and accurately identify the occurrence of electrical fires, thereby providing a reliable basis for fire early warning.
[0119] 2. Regression task:
[0120] In addition to classification tasks, the present invention also supports regression tasks, which are used to predict specific parameters of electrical fires, such as the time of fire occurrence, temperature change trend, etc. In the regression task, the fused feature representation is input into the regressor, which also uses a fully connected network, and its output is a continuous value, which is used to predict the time or physical quantity of the fire event. The specific regression process includes:
[0121] (1) Feature input: The fused feature representation is used as the input of the regressor.
[0122] (2) Fully connected layer: The regressor consists of multiple layers of fully connected networks, which extract features layer by layer and output a continuous value.
[0123] (3) Activation function: In regression tasks, the output layer usually uses a linear activation function to ensure that the regression result is a continuous value.
[0124] (4) Output result: The output of the regression task is a continuous physical quantity, such as the time of fire occurrence or temperature change value.
[0125] The design of the regression task can provide refined parameter prediction for the monitoring and early warning of electrical fires, helping relevant personnel to grasp the key data in the development of fires and take effective countermeasures.
[0126] 3) [Training and inference process]
[0127] The electrical fire detection method of the present invention is based on a multimodal information fusion model, and through the model training and reasoning process, it can achieve efficient detection and prediction of electrical fires. The whole process includes model construction, training and reasoning operations after deployment, ensuring that the model has stable fire detection capabilities in complex scenarios. The specific process of this process is as follows Figure 7 As shown in the figure, it is mainly divided into two parts: model building process and deployment model reasoning process.
[0128] 1. Model building process
[0129] The model construction process mainly includes the following key steps:
[0130] (1) Data preprocessing and sample preparation: The raw data from electrical signals and video information is preprocessed for sample construction. Data preprocessing tools synchronize, align, and normalize multimodal data to ensure that the model can learn effective features from high-quality datasets.
[0131] (2) Feature extraction and fusion: The preprocessed sample data is processed through a time series processing module based on PatchTST and an image processing module based on ViT to extract features from the time series and visual series. These extracted multimodal features are concatenated by a feature connection module and finally deeply fused by a multimodal information fusion module based on Transformer to form a fused feature representation.
[0132] (3) Training / validation process: The fused feature representation is input into the decision layer model for classification or regression training. The training process uses labeled sample data sets to optimize the model parameters so that it can accurately identify the occurrence of fire or predict related fire parameters. During the model training process, the performance and generalization ability of the model are further evaluated through cross-validation and other techniques.
[0133] (4) Model performance evaluation: The trained model will undergo a rigorous performance evaluation, including multiple indicators such as accuracy, recall, and F1 value, to ensure that the model has strong fire detection capabilities. If the evaluation results do not meet expectations, it may be necessary to adjust the hyperparameters of the feature extraction module, fusion module, or decision layer to optimize the model again.
[0134] 2. Deployment model reasoning process
[0135] After the model is trained and optimized, it will be deployed to the actual fire detection system for real-time reasoning. The specific steps for deploying model reasoning are as follows:
[0136] (1) On-site real-time data input: The deployed model receives real-time electrical signals and video information from the site. The system preprocesses and simplifies these real-time data to ensure that the data can be quickly and efficiently input into the model for reasoning.
[0137] (2) Data processing and windowing: During the inference process, real-time data is processed in blocks according to the set window size and step size so that the model can analyze the data features in each time period in real time. The electrical signal and the corresponding visual information will be synchronously input into the feature extraction module after the same preprocessing process as the training phase.
[0138] (3) Model reasoning and fire risk prediction: After the real-time data passes through the feature extraction and feature fusion modules, the corresponding fused feature representation is generated. The fused feature representation will be inferred through the decision-making layer model to determine the fire risk in the current scenario. If the model detects a fire, the system will trigger the alarm mechanism; if regression task reasoning is performed, the system will output prediction parameters such as the time and intensity of the fire.
[0139] Through the training and reasoning process, the electrical fire detection system of the present invention can quickly and accurately detect fires in complex and changeable application scenarios and provide real-time warning information. At the same time, the efficient deployment and reasoning capabilities of the model ensure that the system can operate stably in the long term, providing a solid technical guarantee for the early prevention and rapid response of electrical fires.
Claims
1. An electrical fire detection method based on multimodal information fusion, characterized in that: The following steps are involved: Step 1: Obtain raw data through multiple sensors, namely time series signals and visual sequence information related to electrical faults; the system uses data preprocessing tools to standardize and denoise the raw time series and visual information; Step 2: Sample preparation tools are used to construct high-quality training and testing datasets; The preprocessed time series and visual sequence information will be input into their respective encoders; the time series data will be feature extracted by the PatchTST-based encoder, while the visual data will be feature extracted by the ViT-based encoder; Step 3: The system concatenates the feature representations of the time series and the visual sequence through the feature connector, and further inputs them into the Transformer-based multimodal feature fuser; the Transformer-based multimodal feature fuser uses the self-attention mechanism of the Transformer architecture to capture the cross-modal correlation between the time series and the visual sequence, thereby achieving efficient fusion of information; Step 4: The fused feature representation will be used for classification tasks or regression tasks. The specific task objectives are set according to the needs of the application scenario. Through the branch task module, the risk of electrical fires is predicted, or the time and intensity of fires are estimated.
2. The electrical fire detection method based on multimodal information fusion according to claim 1 is characterized in that: The data preprocessing in step 1 includes the following steps: Step A1: Multimodal data synchronization: For the original multimodal experimental data collected by multi-source sensors, the data preprocessing tool first performs synchronization processing on the multimodal data; multi-video stream playback control and multi-electrical signal experimental visualization methods are used to ensure the time synchronization of different modal data, and the occurrence time of key events is marked by timestamps; Step A2: Multimodal data alignment: Based on synchronization, the data alignment and recording module is further used to obtain the precise moment of the video and electrical signal when the electrical fault occurs; the multimodal data alignment tool is used to calibrate the time clues of the electrical signal and the video sequence to ensure the consistency of different modal data on the time axis before data fusion; Step A3: Data cropping and optimization: The preprocessing tool performs data cropping and noise reduction processing; the specific operations include cropping video frames and signal data noise reduction, removing irrelevant or interfering data, and retaining the core information related to electrical fires; Step A4: Formatting and storage: The data after synchronization, alignment and optimization will be formatted and stored to form pre-processed high-quality data sets; these data sets will be used for subsequent sample construction and model training to ensure the integrity and consistency of the data during the multimodal fusion process.
3. The electrical fire detection method based on multimodal information fusion according to claim 1 is characterized in that: Sample preparation tools in step 2 include: Step B1: Sample data acquisition: The sample preparation tool first extracts the time series and corresponding visual sequence signals from the preprocessed data set; through the sliding window method, the tool automatically generates samples from the continuous time series data according to the set window size and step size; in addition, the visual sequence frames matching the time window are extracted to ensure the synchronization of time and space information; the sample data extraction module ensures the efficient acquisition and pairing of data under different modalities; Step B2: Sample label acquisition: While extracting samples, the sample preparation tool automatically assigns labels to each sample according to the requirements of the fire detection task; the label acquisition module assigns corresponding status labels to each sample through timestamp and event trigger mechanism; the label acquisition module automatically annotates samples according to the time related to the fire event to ensure the accurate match between label data and sample data; at the same time, dynamic labels are assigned to reflect the time change information during the fire process; Step B3: Sample data standardization: The sample preparation tool standardizes the extracted time series data and visual series data; standardization operations include removing outliers and normalizing to ensure that data from different sources have the same scale and range; Step B4: Sample storage and management: The processed sample data includes time series data, image sequence data and their corresponding label information; all sample data will be stored in a unified formatted data set for subsequent model training and verification.
4. The electrical fire detection method based on multimodal information fusion according to claim 1 is characterized in that: The time series processing based on PatchTST in step 2 includes: Step C1: Input multidimensional time series: First, receive multidimensional electrical signals as input. The multidimensional time series data contains rich electrical fault information. Step C2: Instance normalization and block processing: The input multi-dimensional time series data is first processed by instance normalization; the normalized time series data is divided into multiple time patches, each of which represents the change of the signal within a specific time window; Step C3: Projection and position embedding: After the block processing is completed, each time block will be projected into a high-dimensional space, and the time series information will be introduced through the position embedding technology; Step C4: Time Series-Transformer Encoder: After the above processing, the block-wise time series data will be input into the time series-Transformer encoder; the Transformer encoder adopts a multi-head attention mechanism; Step C5: Flattening and linear output: The encoded high-dimensional feature vector is flattened, and the multi-dimensional features are merged into a linear sequence representation Seq-token; this linear sequence representation contains the temporal and spatial features of all input electrical signals, providing an information basis for subsequent feature fusion and classification tasks.
5. The electrical fire detection method based on multimodal information fusion according to claim 1 is characterized in that: The ViT-based image information processing in step 2 includes: Step D1: Input image sequence: First, receive the fire video image sequence collected by the surveillance camera. These sequence data contain important visual information during the fire. The video frames are passed into ViT as the image input sequence for subsequent processing. Step D2: Block processing and linear projection: The input image sequence is first divided into multiple fixed-size image patches; each image patch represents a local spatial region in the video frame; each image patch is linearly projected to map the two-dimensional image data into a high-dimensional feature space, generating a series of image embedding vectors Image tokens; Step D3: Position Embedding: ViT uses position embedding technology; after embedding the image blocks into high-dimensional space, position embedding adds position information to each image block, thus ensuring that the model can recognize the spatial structure and position dependencies in the image; Step D4: Image sequence-Transformer encoder: After the block segmentation and embedding are completed, the image block embedding vector is input into the image sequence-Transformer encoder; Step D5: Classification tag with category mark: During the encoding process, ViT introduces a specific category tag, namely the classification tag Cls-token, for each image sequence to represent the category characteristics of the fire; the classification tag is input into the Transformer encoder together with the image embedding, and through interaction with the features of other image blocks, a comprehensive classification feature representation is finally generated to help the model identify the occurrence and category of fire.
6. The electrical fire detection method based on multimodal information fusion according to claim 1, characterized in that: The Transformer-based multimodal information fusion in step 3 includes: Step E1: Feature connection: The first step of multimodal information fusion is to connect the features extracted from different modalities. The time series features output by the PatchTST module and the image features output by the ViT module represent different aspects of electrical fires. Through the feature connection module, the features of these two modalities are connected in series to form a comprehensive feature representation. This comprehensive feature representation contains the spatiotemporal dependency features of electrical signals and visual information, laying the foundation for subsequent deep fusion. Step E2: Fused Transformer Encoder: The comprehensive feature representation is input into the fused Transformer encoder for deep fusion; the fused Transformer encoder adopts a similar structure to the unimodal Transformer, including a multi-head attention mechanism and a feed-forward network; During the fusion process, the fusion Transformer can simultaneously focus on the long-range dependencies between multi-dimensional time series and image features, and dynamically adjust the weights of different modal features using the self-attention mechanism; Step E3: Addition and normalization: In the multi-layer encoding process of the fused Transformer, the system also performs addition and normalization operations; by standardizing the fused feature representation, the scale differences between features are eliminated; Step E4: Fused feature representation: Finally, the feature representation of the fused Transformer encoder output is a deep comprehensive representation of multimodal information, including multi-level features of time series and image information.
7. The electrical fire detection method based on multimodal information fusion according to claim 1, characterized in that: The classification tasks in step 4 include: (1) Feature input: The fused feature representation generated by the fused Transformer is used as the input of the classifier; (2) Fully connected layer: The classifier consists of a multi-layer fully connected network, which is used to map the fused feature representation to different category label spaces; (3) Activation function: The output layer of the classifier usually uses a softmax activation function to ensure that the output is in the form of a probability distribution, representing the probability of various types of fire or no fire; (4) Output results: Based on the predicted category labels, the classification task ultimately determines whether a fire has occurred and its type.
8. The electrical fire detection method based on multimodal information fusion according to claim 1, characterized in that: The regression tasks in step 4 include: 1) Feature input: The fused feature representation is used as the input of the regressor; 2) Fully connected layer: The regressor consists of multiple layers of fully connected networks, extracting features layer by layer and outputting a continuous value; 3) Activation function: In regression tasks, the output layer usually uses a linear activation function to ensure that the regression result is a continuous value; 4) Output result: The output of the regression task is a continuous physical quantity.