A harmful gas leakage detection method and system based on industrial multimodal data fusion

Abstract

The present invention relates to the field of gas leakage detection technology, and specifically to a method and system for detecting harmful gas leakage based on industrial multimodal data fusion, including: obtaining sensor scalar data and infrared image data in an industrial environment collected by a gas sensor and a thermal imager, and preprocessing these data; constructing a dual-branch encoder network and a multimodal fusion network based on a multi-head attention mechanism; using the preprocessed sensor data and infrared image data to train the encoder network and the multimodal fusion network; using the trained encoder network and the multimodal fusion network, using the multi-head attention mechanism to perform feature fusion; based on the fused features, obtaining the identification result of the harmful gas leakage through a classifier. The present invention can significantly improve the detection accuracy of harmful gas leakage, realize automatic and accurate identification and rapid detection of harmful gas leakage, and is conducive to enhancing the safety and reliability of the industrial production process.

Description

Technical Field

[0001] The present invention relates to the field of smart industrial technology, and in particular to a method and system for detecting harmful gas leakage based on industrial multimodal data fusion. Background Art

[0002] In industrial environments, the leakage of hazardous gases is one of the main causes of environmental pollution and major accidents. These harmful gases often leak into the air, water and land through cracks in aging tanks or pipelines. If the leakage of some harmful gases cannot be detected or repaired in time, it will cause huge economic losses, environmental pollution, and even huge casualties. Studies have shown that the risk level of poisoning accidents caused by dangerous gas leaks is unacceptable. In addition, the leakage of harmful gases will affect the normal operation of the factory, thereby reducing the availability and productivity of the factory and causing economic losses. In order to avoid these consequences, the leakage of harmful gases must be detected in time when an accident occurs, which is crucial to formulate early response plans, reveal leakage conditions and prevent escalation.

[0003] Traditional gas leak detection methods mainly include manual inspection, sound monitoring, flow detection, sensor detection, etc. However, these methods rely heavily on the accuracy of data acquisition and the mathematical model used. In addition, these methods not only require a lot of manpower, financial resources and time investment, but may also cause certain damage to pipelines, gas storage and the surrounding environment. With the increase of pipeline distance and the complexity of plant structure, the effectiveness of these methods will be further reduced.

[0004] At present, the gas leak identification method based on neural network still has limitations in some aspects. The sensor-based method cannot accurately detect the presence of a specific gas in a mixed gas environment; and the infrared image-based method, although it can detect the presence of gas, has difficulty locating the leak and determining the gas type. However, in complex environments such as manufacturing plants, the gas composition is complex and the data has multiple modalities, including both sensor data and infrared images. Therefore, there is an urgent need for a harmful gas leak detection method that integrates multi-modal gas data to improve the accuracy of harmful gas leak detection in industrial environments and ensure timely and accurate detection and response when leaks occur.

[0005] The existing Chinese patent document CN202310428798.9 discloses a training method, a detection method, and an electronic device for industrial leakage target gas detection model. The training method includes: in response to receiving a gas image to be trained, extracting features of the gas image to be trained; enhancing the image features of the gas image to be trained based on deformable convolution and channel attention to obtain a deep feature image for the gas image to be trained; predicting features of the deep feature image to obtain multiple output feature maps; screening positive and negative sample prediction frames based on confidence; calculating the target frame regression loss of the screened positive and negative sample prediction frames using a complete loss function; and updating the industrial leakage target gas detection model based on at least multiple output feature maps and target frame regression losses. In this way, industrial leakage gas can be detected in real time and accurately, improving the safety of industrial gas use. However, although the above method can detect gas leakage, it cannot determine the specific type of gas. For example, if it is an ordinary water vapor leak, the existing technology may not be able to distinguish and thus cannot effectively deal with it. In addition, the feature enhancement and fusion methods are limited, and there is still room for improvement in dealing with complex backgrounds.

[0006] The present invention overcomes the above-mentioned shortcomings through a multimodal data fusion method. First, by combining sensor data and infrared image data, the characteristics of industrial gas leakage can be captured more comprehensively and accurately, thereby improving the sensitivity and reliability of detection. This method can not only detect gas leaks, but also determine the specific type of gas, thereby distinguishing between common gases and hazardous gases. Secondly, the multimodal data fusion method can effectively handle complex backgrounds and changeable industrial environments, thereby enhancing the robustness of the model. Finally, by optimizing the feature extraction and fusion algorithms, the adaptability and real-time detection performance of the model in multiple scenarios are improved, thereby greatly improving the accuracy and reliability of industrial gas leakage detection and ensuring the safety of industrial production. Summary of the invention

[0007] The purpose of the present invention is to provide a harmful gas leakage detection method and system based on industrial multimodal data fusion, so as to realize accurate identification and rapid detection of harmful gas leakage through industrial multimodal data.

[0008] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions: The method of the present invention is designed from the two perspectives of feature encoder and fusion strategy. Specifically, first, SwinTransformer is used to extract infrared image features, and the local and global features of the image are effectively captured through the hierarchical structure and window self-attention mechanism; secondly, gMLP is used to extract sensor data features, and the sequence data is efficiently processed through MLP and gating mechanism, and its complex patterns are captured; finally, the multi-head attention mechanism is used to fuse multimodal features, while paying attention to the information of different feature subspaces to improve the feature fusion effect. In the above manner, the method can effectively fuse multimodal gas data and achieve fast and accurate gas leak detection.

[0009] A method for detecting harmful gas leakage based on industrial multimodal data fusion comprises the following steps:

[0010] S1: Obtain sensor data and infrared image data in industrial environments collected by gas sensors and thermal imagers, and pre-process these data;

[0011] S2: Construct a dual-branch encoder network and a multi-modal fusion network based on a multi-head attention mechanism;

[0012] S3: Use the preprocessed sensor data and infrared image data to train the encoder network and multimodal fusion network;

[0013] S4: Feature fusion is performed using the multi-head attention mechanism through the trained encoder network and multimodal fusion network;

[0014] S5: Based on the fused features, the identification results of harmful gas leakage are obtained through the classifier, realizing accurate identification and rapid detection of harmful gas leakage.

[0015] Furthermore, the step S1 specifically includes the following sub-steps:

[0016] S1.1: Clean the acquired and captured sensor data and infrared image data;

[0017] S1.2: Perform image enhancement on the cleaned infrared image data;

[0018] S1.3: Match and annotate the cleaned sensor data and enhanced infrared image data respectively.

[0019] Furthermore, the step S2 specifically includes the following sub-steps:

[0020] S2.1: Building a sensor data encoder network based on gMLP;

[0021] S2.2: Construct infrared image encoder network based on Swin Transformer;

[0022] S2.3: Construct a multimodal feature fusion network based on the multi-head attention mechanism.

[0023] Furthermore, the step S2.1 specifically includes:

[0024] Selecting a gMLP block based on a gating mechanism to construct a sensor data encoder network; repeatedly stacking gMLP blocks to construct a sensor data encoder network;

[0025] Where: Each gMLP block is defined as:

[0026] Z=σ(E s U)

[0027]

[0028] In the formula, E s is defined as the input sensor data features; U and V are defined as linear projections along the channel dimension; σ is defined as the GeLU activation function; s is defined as the linear projection along the spatial dimension; and Y defines the output of each layer of the gMLP block.

[0029] Furthermore, the step S2.2 specifically includes:

[0030] The infrared image encoder network is constructed by selecting the SwinTransformer block based on the sliding window mechanism; the sensor data encoder network is constructed by repeatedly stacking the Swin Transformer blocks and inserting downsampling blocks between each layer;

[0031] in:

[0032] The input and output of the continuous Swin Transformer block can be expressed as follows:

[0033]

[0034] In the formula, represents the output of W-MSA and SW-MSA at layer l; f l represents the output of the MLP in the lth layer; LN represents the LayerNorm layer.

[0035] Furthermore, the step S2.3 specifically includes:

[0036] The encoder part of the Transformer model is selected and the MSA module is optimized. The optimization of the MSA module includes: deleting the three QKV matrices of the original MSA module input; using the extracted sensor data features as the Q matrix, splicing the sensor data features and the infrared image features as the K matrix and the V matrix to replace the originally deleted QKV matrices and input them into the MSA module to calculate attention; stacking the optimized Transformer models to build a multimodal feature fusion network;

[0037] in:

[0038] The process can be described as:

[0039]

[0040] F fus =MLP(LN(F mca ))+F mca

[0041] In the formula, E s and E i Defined as sensor features and infrared image features after pooling and linear layers; Represents three weight matrices; represents the three recalculated matrices; MCA represents the multi-head attention layer; F mca represents the features initially fused by the multi-head attention layer; F fus Represents the output of the multimodal fusion network.

[0042] Furthermore, the step S3 specifically includes the following sub-steps:

[0043] S3.1: Determine the loss function L;

[0044] S3.2: Train the sensor data encoder network based on the preprocessed sensor data;

[0045] S3.3: Train the infrared image encoder network based on the preprocessed infrared image data;

[0046] S3.4: Train a multimodal fusion network based on pre-trained weights.

[0047] Furthermore, the step S3.1 specifically includes:

[0048] Select cross entropy loss as the loss function for encoder network training;

[0049] Among them: The expression of the loss function L is:

[0050]

[0051] In the formula, y represents the true value of the multi-label, p i and i They represent the predicted value and true value of the i-th label respectively, N represents the number of multiple labels, and CE represents the cross entropy loss.

[0052] Furthermore, the step S3.2 specifically includes:

[0053] The preprocessed sensor data is input into the constructed sensor data encoder network, the loss value of the output of the sensor data encoder network under the loss function L is obtained, the encoder network parameters are updated according to the loss value, and the weight parameters of the trained sensor data encoder network are saved.

[0054] Furthermore, the step S3.3 specifically includes:

[0055] Load the intermediate weights of the SwinTransformer network pre-trained on ImageNet21k; input the preprocessed infrared image into the constructed infrared image encoder network; obtain the loss value of the output of the infrared image encoder network under the loss function L; update the encoder network parameters according to the loss value; save the trained infrared image encoder network weight parameters.

[0056] Furthermore, the step S3.4 specifically includes:

[0057] Load the trained infrared sensor data encoder network weight parameters; load the trained infrared image encoder network weight parameters; input sensor data and infrared image data to extract and fuse features; obtain the loss value of the output of the multimodal fusion network under the loss function L; update the multimodal fusion network parameters according to the loss value; save the trained infrared multimodal fusion network weight parameters.

[0058] Furthermore, the step S4 specifically includes the following sub-steps:

[0059] S4.1: Load the saved network weight parameters;

[0060] S4.2: Input the preprocessed sensor data into the constructed sensor data encoder network to extract sensor data features;

[0061] S4.3: Input the preprocessed infrared image into the constructed infrared image encoder network to extract infrared image features;

[0062] S4.4: Input the extracted sensor data features and infrared image features into the constructed multimodal fusion network for feature fusion, and output the fused features.

[0063] Furthermore, the step S5 specifically includes: inputting the fused features into a classifier to obtain an identification result of the harmful gas leakage.

[0064] On the other hand, the present invention proposes a harmful gas leakage detection system based on industrial multimodal data fusion, comprising:

[0065] A data acquisition module configured to acquire gas sensor data and thermal imager data in an industrial plant environment;

[0066] A data processing module, configured to pre-process the gas sensor data and the thermal imager data to obtain sensor data and infrared images that have undergone data cleaning and data enhancement;

[0067] The feature extraction module is configured to extract effective features from the sensor data and the infrared image to obtain a high-dimensional, distinguishable feature representation.

[0068] A network training module is configured to learn the preprocessed sensor data and infrared images, calculate the overall loss value through the loss function, and update the network parameters through the loss value;

[0069] A feature fusion module configured to fuse the sensor data and the high-dimensional feature representation in the infrared image to obtain a unified high-dimensional feature representation after fusion;

[0070] The gas detection module is configured to obtain a detection result of harmful gas leakage according to the unified high-dimensional feature representation after fusion.

[0071] Beneficial effects of the present invention:

[0072] The present invention effectively captures the complex patterns in the quantitative concentration data of the gas sensor through gMLP, and dynamically adjusts the importance of the features through the gating mechanism, thereby enhancing the generalization ability and robustness of the model; effectively captures the local and global features in the infrared image through Swin-Transformer, so that the model can pay attention to the details and overall structural information in the image at the same time, thereby more comprehensively understanding the image content; based on the multi-head attention mechanism, the features of the two modes are fused, which significantly improves the effect of feature fusion. In the above way, the method can effectively fuse multi-modal gas data and improve the robustness and detection accuracy of the system in complex environments.

[0073] The present invention constructs a dual-branch encoder network, and the sensor data and infrared images are subjected to their own feature extraction, and then feature fusion is performed under the multi-head attention mechanism, so that the model can fully understand the relationship between different modalities. This process not only helps to extract richer and more effective features, but also reduces the probability of false positives and false negatives by learning the potential patterns in the data. In a multimodal environment, the detection accuracy of the model can be improved by more than 2.08% compared with a single modality method. This improvement provides a reliable guarantee for industrial safe operation.

[0074] In industrial environments, rapid response to gas leaks is crucial to preventing accidents. The system of the present invention can process data streams from different sensors and thermal imaging devices in real time through a deep learning framework, thereby meeting the need for rapid detection. The trained multimodal fusion network of the present invention can complete the analysis of input data within a few milliseconds through efficient feature extraction and fusion strategies. This rapid response capability is due to the use of an optimized loss function and an efficient parallel computing mechanism, which enables the model to fully utilize GPU acceleration when predicting. Test results show that after multimodal fusion, the response time of the system can be shortened to less than 500 milliseconds, greatly reducing the time interval between leak detection and emergency response, and reducing the risk of safety accidents.

[0075] The risk of misoperation is an important safety hazard in industrial environments, especially in gas leak detection, where incorrect judgments may lead to serious consequences. The present invention effectively reduces the occurrence of misoperation by introducing a gating mechanism in data preprocessing and deep learning models. First, through data cleaning and image enhancement steps, the system eliminates outliers and noise in sensor data and infrared images to ensure the quality of input data. Secondly, gMLP and Swin Transformer based on the gating mechanism are adopted to enable the network to adaptively select important information during the feature extraction process. This selective mechanism reduces the interference of irrelevant information and improves the stability and reliability of the model. Through the above method, the false alarm rate of the system is only 2.29%, thereby reducing emergency response and resource waste caused by incorrect judgments and improving the overall operational safety.

[0076] Modern industrial environments are usually complex and changeable, with many gas components. Traditional single-modal detection methods are easily limited in complex environments. The present invention can effectively meet these challenges through a multimodal data fusion strategy. The multi-head attention mechanism of the present invention allows the system to automatically identify and focus on the feature areas that have the greatest impact on the detection results when processing sensor and infrared imaging data. This strategy not only improves the system's ability to detect specific gases, but also can accurately distinguish and identify potential sources of leakage when multiple gases are mixed. In addition, the modular design of the system enables it to be flexibly adjusted according to different industrial environments and gas types, further enhancing its adaptability. Through comparative tests with other methods, the gas detection accuracy of this method in complex environments reached 98.02%, demonstrating its powerful capabilities in real applications.

[0077] In summary, the present invention comprehensively improves the accuracy and response speed of harmful gas leakage detection, reduces the risk of misoperation, and the ability to adapt to complex environments through multimodal data fusion technology, making it have broad prospects and important practical significance in actual industrial applications.

[0078] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0080] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0081] Figure 2 It is a schematic diagram of the overall network structure of the present invention;

[0082] Figure 3 This is a schematic diagram of the multimodal fusion network structure of the present invention;

[0083] Figure 4 It is a schematic diagram of the system principle of the present invention;

[0084] Figure 5 Confusion matrix diagram of different modal inputs of the present invention;

[0085] Figure 6 This is a graph showing the accuracy comparison experiment of the present invention;

[0086] Figure 7 This is a false alarm rate control experiment diagram of the present invention. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] Example 1

[0089] The harmful gas leakage detection method based on industrial multimodal data fusion described in this embodiment is as follows: Figure 1 As shown, the following steps are included:

[0090] S1: Obtain sensor data and infrared image data in industrial environments collected by gas sensors and thermal imagers, and pre-process these data;

[0091] S2: Construct a dual-branch encoder network and a multi-modal fusion network based on a multi-head attention mechanism;

[0092] S3: Use the preprocessed sensor data and infrared image data to train the encoder network and multimodal fusion network;

[0093] S4: Feature fusion is performed using the multi-head attention mechanism through the trained encoder network and multimodal fusion network;

[0094] S5: Based on the fused features, the identification results of harmful gas leakage are obtained through the classifier, realizing accurate identification and rapid detection of harmful gas leakage.

[0095] It should be understood that in step S1, sensor data and infrared image data in an industrial environment collected by a gas sensor and a thermal imager are obtained, and these data are preprocessed; wherein, the gas sensor works on the principle of chemical reaction between gas and metal oxide surface. When a specific gas is present, it reacts with the metal oxide surface, causing the electrical properties on the sensor to change. By measuring and capturing the changes in these electrical properties, the presence of a gas leak can be detected. However, this gas sensor cannot accurately detect the presence of a specific gas in a mixed gas environment. As a non-contact technology, infrared images can detect and measure the thermal radiation of different gases, even for invisible gases (such as coal gas), through infrared images. However, it is difficult to locate the leak and determine the type of gas with infrared images. Therefore, it is possible to consider combining the two modes to learn from each other so that when a harmful gas leaks, it can be detected more quickly and accurately.

[0096] In this embodiment, step S1 specifically includes the following sub-steps:

[0097] S1.1: Clean the acquired and captured sensor data and infrared image data; data cleaning includes processing missing values ​​and outliers in the data, and ensuring that the time and location of the two data acquisitions are consistent. For sensor data, it is necessary to ensure the consistency of information such as timestamps and units, and eliminate duplicate data or logical errors. For infrared images, it is necessary to ensure the consistency of image format, resolution and timestamp, and unify the size and direction of the image; after processing, the sensor data and infrared image data need to be converted into a unified format and labeled with different categories for easy storage and processing.

[0098] S1.2: Perform image enhancement on the cleaned infrared image data; wherein, image enhancement includes normalizing the image pixel values, normalizing the image pixel values ​​to between 0 and 1, so that the subsequent image processing algorithm is stable and consistent. Use image filtering technology to remove noise in the image, and use Gaussian filtering to smooth the image and reduce high-frequency noise. Use median filtering to replace the current pixel value with the median of the neighboring pixels to remove impulse noise. Use bilateral filtering to smooth while retaining edge information to reduce noise. Use image enhancement algorithms to enhance the image, and enhance the image edges and improve the image clarity by sharpening the image. Adjust the image brightness by performing gamma correction to make the overall visual effect of the image more natural;

[0099] S1.3: Match and annotate the cleaned sensor data and enhanced infrared image data respectively.

[0100] In this embodiment, step S2 specifically includes the following sub-steps:

[0101] S2.1: Select gMLP blocks based on the gating mechanism to construct the sensor data encoder network, and repeatedly stack gMLP blocks to construct the sensor data encoder network. Each gMLP encoder consists of 4 identical layers, and the core of each layer is the spatial gating unit (SGU), in which each feature is divided into two parts along the channel dimension, one part is projected by the spatial projection layer and multiplied with the other part. Spatial projection allows the extracted features to share information globally, while element-by-element multiplication retains the local features extracted by the previous module. Therefore, SGU can adjust the extracted features based on global and local information. In addition, channel projection layers are used at the beginning and end of the module to mix different components of each feature along the channel dimension, which makes the learned features more flexible;

[0102] Each gMLP block is defined as:

[0103] Z=σ(E s U)

[0104]

[0105] In the formula, E s is defined as the input sensor data features; U and V are defined as linear projections along the channel dimension; σ is defined as the GeLU activation function; s is defined as the linear projection along the spatial dimension; and Y defines the output of each layer of the gMLP block.

[0106] S2.2: Select the Swin Transformer block based on the sliding window mechanism to build the infrared image encoder network, repeatedly stack the Swin Transformer blocks and insert downsampling blocks between each layer to build the sensor data encoder network. Compared with CNN, Swin Transformer can generate hierarchical feature representations and model long-range dependencies. In addition, unlike Vision Transformer (ViT) which performs self-attention calculations on all patches, Swin Transformer performs self-attention calculations on patches within the window. This allows Swin Transformer to both generate hierarchical feature representations and model long-range dependencies and improve computational efficiency. Each Swin Transformer block consists of a LayerNorm (LN) layer, a multi-head self-attention (MSA) module, a multi-layer perceptron (MLP) layer, and a residual connection. The multi-head self-attention modules of two consecutive Swin Transformer blocks are composed of a window-based MSA (W-MSA) and a sliding window-based MSA (SW-MSA) module. The former mainly performs attention calculations on patches within the window, while the latter performs attention calculations by moving to the upper left direction through cyclic shifts. The sliding window mechanism can cover multiple non-adjacent sub-windows in the feature map to capture a wider range of contextual information while maintaining the same number of batch windows as conventional partitioning. This approach can both improve computational efficiency and enhance the model's ability to model long-range dependencies;

[0107] The input and output of the continuous Swin Transformer block can be expressed as follows:

[0108]

[0109] In the formula, represents the output of W-MSA and SW-MSA at layer l; f l represents the output of the MLP in the lth layer; LN represents the LayerNorm layer.

[0110] S2.3: Select the encoder part in the Transformer model and optimize the MSA module therein. The Transformer model is an existing model architecture, which calculates attention through the MSA module and can effectively capture long-range dependencies. By deleting the three matrices QKV that were originally input to the MSA module. The extracted sensor data features are used as the Q matrix, and the sensor data features and infrared image features are concatenated as the K matrix and the V matrix to replace the originally deleted three matrices QKV and input into the MSA module to calculate attention. This method is different from the original calculation of attention through three separate matrices. It uses the feature distribution in the current modality and the semantic information in other modalities to calculate attention. This not only improves the model's understanding of global information, but also enhances the model's ability to express complex features. This enables the Transformer model to better integrate information from different sources, making the model more robust and accurate when fusing multimodal data. Finally, the optimized Transformer models are stacked to construct a multimodal feature fusion network. The structure of the multimodal feature fusion network is as follows: Figure 3 As shown;

[0111] The optimization process can be described as:

[0112]

[0113] F fus =MLP(LN(F mca ))+F mca

[0114] In the formula, E s and E i Defined as sensor features and infrared image features after pooling and linear layers; Represents three weight matrices; represents the three recalculated matrices; MCA represents the multi-head attention layer; F mca represents the features initially fused by the multi-head attention layer; F fus Represents the output of the multimodal fusion network.

[0115] In this embodiment, the step S3 specifically includes: using the preprocessed sensor data and infrared image data to train the encoder network and the multimodal fusion network. Among them, the encoder network should be trained using the preprocessed sensor data and infrared image data to ensure that the data input into the two networks are matched during training. For example: if it is necessary to detect whether there is a gas leak in the current environment, it is necessary to input the preprocessed sensor data and thermal imager data in the environment at the same time, and ensure that the labels of the two types of data are consistent. It should be noted that the two encoder networks should be trained separately and the network weights should be saved after training. The multimodal fusion network needs to import the trained encoder network weights before training, and train the multimodal fusion network according to the loss function L.

[0116] The specific steps include the following

[0117] S3.1: Determine the loss function L, wherein the cross entropy loss is selected as the loss function for encoder network training;

[0118] The expression of the loss function L is:

[0119]

[0120] In the formula, y represents the true value of the multi-label, p i and i They represent the predicted value and true value of the i-th label respectively, N represents the number of multiple labels, and CE represents the cross entropy loss.

[0121] S3.2: Train the sensor data encoder network based on the preprocessed sensor data. The training process is as follows: input the preprocessed sensor data into the constructed sensor data encoder network; obtain the loss value of the output of the sensor data encoder network under the loss function L; update the encoder network parameters according to the loss value; save the trained sensor data encoder network weight parameters;

[0122] S3.3: Train the infrared image encoder network based on the preprocessed infrared image data. The training process is as follows: load the intermediate weights of the Swin Transformer network pre-trained on ImageNet21k; input the preprocessed infrared image into the constructed infrared image encoder network; obtain the loss value of the output of the infrared image encoder network under the loss function L; update the encoder network parameters according to the loss value; save the trained infrared image encoder network weight parameters;

[0123] S3.4: Train the multimodal fusion network based on pre-trained weights. The training process is as follows: load the trained infrared sensor data encoder network weight parameters; load the trained infrared image encoder network weight parameters; input sensor data and infrared image data to extract and fuse features; obtain the loss value of the output of the multimodal fusion network under the loss function L; update the multimodal fusion network parameters according to the loss value; save the trained infrared multimodal fusion network weight parameters.

[0124] In this embodiment, step S4 includes: based on the trained network model and the preprocessed sensor data and infrared image data, multimodal feature fusion is performed through a multimodal fusion network using a multi-head attention mechanism. Among them, before inputting the sensor data and infrared image data, it is necessary to import the trained network weights. It should be noted that the imported weights include the network weights of the two encoder networks and the multimodal feature fusion network. By inputting the preprocessed sensor data and infrared image data into the network model, the fused features can be obtained. Specifically including the following sub-steps:

[0125] S4.1: Load the saved network weight parameters;

[0126] S4.2: Input the preprocessed sensor data into the constructed sensor data encoder network to extract sensor data features;

[0127] S4.3: Input the preprocessed infrared image into the constructed infrared image encoder network to extract infrared image features;

[0128] S4.4: Input the extracted sensor data features and infrared image features into the constructed multimodal fusion network for feature fusion, and output the fused features.

[0129] In this embodiment, step S5 specifically includes: based on the fused features, obtaining the identification result of the harmful gas leakage through the classifier, and realizing the accurate identification and rapid detection of the harmful gas leakage. Among them, the classifier is composed of a linear layer, and the fused features refer to the sensor features extracted after the encoder model in the preprocessing network and the infrared image features are input into the multimodal fusion network for feature fusion and output. The identification result of the harmful gas leakage can be obtained by inputting the fused features into the classifier.

[0130] Example 2

[0131] A harmful gas leakage detection system based on industrial multimodal data fusion as described in this embodiment, such as Figure 4 As shown, including:

[0132] A data acquisition module configured to acquire gas sensor data and thermal imager data in an industrial plant environment;

[0133] A data processing module, configured to pre-process the gas sensor data and the thermal imager data to obtain sensor data and infrared images that have undergone data cleaning and data enhancement;

[0134] The feature extraction module is configured to extract effective features from the sensor data and the infrared image to obtain a high-dimensional, distinguishable feature representation.

[0135] A network training module is configured to learn the preprocessed sensor data and infrared images, calculate the overall loss value through the loss function, and update the network parameters through the loss value;

[0136] A feature fusion module configured to fuse the sensor data and the high-dimensional feature representation in the infrared image to obtain a unified high-dimensional feature representation after fusion;

[0137] The gas detection module is configured to obtain a detection result of harmful gas leakage according to the unified high-dimensional feature representation after fusion.

[0138] Example 3

[0139] In order to verify the method proposed in this paper, the public multimodal gas dataset MultimodalGasData is used for experiments. The MultimodalGasData dataset provides two modes for the smoke leak detection task and collects four gas types, namely smoke, mixed gas, perfume and no gas. There are 1600 samples of each type, and each sample includes gas data collected by 7 gas sensors (MQ2, MQ3, MQ5, MQ6, MQ7, MQ8 and MQ135), as well as smoke infrared image data taken by a thermal imager.

[0140] In the experiment, the dataset is randomly divided into a training set and a validation set in a ratio of 7:3, which are used for training and testing this method, respectively.

[0141] In order to verify the effectiveness of multimodal fusion, experiments were conducted on the same dataset using infrared images and sensor data, respectively, under the premise of using only one encoder network. The confusion matrices of the three methods are shown in Figure 2. Figure 5 As shown in the figure. The results show that the accuracy of multimodal fusion is significantly improved compared with single modality data. The calculation of the confusion matrix shows that after fusing sensor data and infrared images, the accuracy is increased by 2.08% and 2.97% respectively compared with using sensor data and infrared images alone. This shows that multimodal fusion can significantly improve the accuracy of recognition.

[0142] In order to verify the effectiveness of the present invention, CNN, ResNet and MobileNet were used as infrared image encoders on the same data set; RNN and LSTM were used as sensor data encoders; after feature extraction, the features were fused in a splicing manner for experiments. The accuracy and false alarm rates of the LSTM+CNN, RNN+CNN, LSTM+ResNet50, RNN+ResNet50, LSTM+Mobv2, RNN+Mobv2, MFT and other models were compared.

[0143] The experimental results are as follows Figure 6 and Figure 7 As shown, the horizontal axis represents different methods, and the vertical axis represents the result percentage. The experimental results show that the accuracy and false alarm rate of the present invention are improved compared with other methods. The overall accuracy rate is increased by 0.97% to 4.4%, and the overall false alarm rate is reduced by 1.25% to 14.13%.

[0144] In summary, the method for detecting harmful gas leakage based on industrial multimodal data fusion provided by the present invention is to learn the preprocessed sensor data and infrared image data based on the encoder model constructed by gMLP and Swin Transformer, and to fuse the features of the two modal data using the multi-head attention mechanism. The fused features are input into the classifier to identify harmful gases, thereby realizing accurate identification and rapid detection of harmful gas leakage, and having high response speed and accuracy. By using the dual encoder model and the multi-head attention fusion strategy, image features and sensor data features are effectively extracted and fused. In order to detect the leakage of hazardous gases in time when an accident occurs, the staff in the industrial plant can formulate early response plans, reveal the leakage situation and prevent the situation from escalating.

[0145] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A harmful gas leakage detection method based on industrial multimodal data fusion, characterized in that: The following steps are involved: S1: Obtain sensor data and infrared image data in industrial environments collected by gas sensors and thermal imagers, and pre-process these data; S2: Construct a dual-branch encoder network and a multi-modal fusion network based on a multi-head attention mechanism; S3: Use the preprocessed sensor data and infrared image data to train the encoder network and multimodal fusion network; S4: Feature fusion is performed using the multi-head attention mechanism through the trained encoder network and multimodal fusion network; S5: Based on the fused features, the identification results of harmful gas leakage are obtained through the classifier, realizing accurate identification and rapid detection of harmful gas leakage; The step S2 specifically includes the following sub-steps: S2.1: Building a sensor data encoder network based on gMLP; Selecting a gMLP block based on a gating mechanism to construct a sensor data encoder network; repeatedly stacking gMLP blocks to construct a sensor data encoder network; Where: Each gMLP block is defined as: Z=σ(E s AT) In the formula, E s Defined as the input sensor data features; U and V are defined as linear projections along the channel dimension; σ is defined as the GeLU activation function; s is defined as the linear projection along the spatial dimension; Y defines the output of each layer of gMLP block; S2.2: Construct infrared image encoder network based on Swin Transformer block; The infrared image encoder network is constructed by selecting the Swin Transformer block based on the sliding window mechanism; the sensor data encoder network is constructed by repeatedly stacking the Swin Transformer blocks and inserting downsampling blocks between each layer; in: The input and output of the continuous Swin Transformer block are represented as follows: In the formula, represents the output of W-MSA and SW-MSA at layer l; f l represents the output of MLP in layer l; LN represents the LayerNorm layer; S2.3: Construct a multimodal feature fusion network based on multi-head attention mechanism; The encoder part of the Transformer model is selected and the MSA module is optimized. The optimization of the MSA module includes: deleting the three QKV matrices of the original MSA module input; using the extracted sensor data features as the Q matrix, splicing the sensor data features and the infrared image features as the K matrix and the V matrix to replace the originally deleted QKV matrices and input them into the MSA module to calculate attention; stacking the optimized Transformer models to build a multimodal feature fusion network; in: The process is described as: In the formula, E s and E i Defined as sensor features and infrared image features after pooling and linear layers; Represents three weight matrices; represents the three recalculated matrices; MCA represents the multi-head attention layer; F mca represents the features initially fused by the multi-head attention layer; F fus Represents the output of the multimodal fusion network.

2. The harmful gas leakage detection method based on industrial multimodal data fusion according to claim 1, characterized in that: The step S1 specifically includes the following sub-steps: S1.1: Clean the acquired and captured sensor data and infrared image data; S1.2: Perform image enhancement on the cleaned infrared image data; S1.3: Match and annotate the cleaned sensor data and enhanced infrared image data respectively.

3. The harmful gas leakage detection method based on industrial multimodal data fusion according to claim 1, characterized in that: The step S3 specifically includes the following sub-steps: S3.1: Determine the loss function L; Select cross entropy loss as the loss function for encoder network training; Among them: The expression of the loss function L is: In the formula, y represents the true value of the multi-label, p i and i Represent the predicted value and true value of the i-th label respectively, N represents the number of multiple labels, and CE represents the cross entropy loss; S3.2: Train the sensor data encoder network based on the preprocessed sensor data; Input the preprocessed sensor data into the constructed sensor data encoder network, obtain the loss value of the output of the sensor data encoder network under the loss function L, update the encoder network parameters according to the loss value, and save the trained sensor data encoder network weight parameters; S3.3: Train the infrared image encoder network based on the preprocessed infrared image data; Load the intermediate weights of the SwinTransformer network pre-trained on ImageNet21k; input the pre-processed infrared image into the constructed infrared image encoder network; obtain the loss value of the output of the infrared image encoder network under the loss function L; update the encoder network parameters according to the loss value; save the trained infrared image encoder network weight parameters; S3.4: Train a multimodal fusion network based on pre-trained weights; Load the trained infrared sensor data encoder network weight parameters; load the trained infrared image encoder network weight parameters; input sensor data and infrared image data to extract and fuse features; obtain the loss value of the output of the multimodal fusion network under the loss function L; update the multimodal fusion network parameters according to the loss value; save the trained infrared multimodal fusion network weight parameters.

4. The harmful gas leakage detection method based on industrial multimodal data fusion according to claim 1, characterized in that: The step S4 specifically includes the following sub-steps: S4.1: Load the saved network weight parameters; S4.2: Input the preprocessed sensor data into the constructed sensor data encoder network to extract sensor data features; S4.3: Input the preprocessed infrared image into the constructed infrared image encoder network to extract infrared image features; S4.4: Input the extracted sensor data features and infrared image features into the constructed multimodal fusion network for feature fusion, and output the fused features.

5. The harmful gas leakage detection method based on industrial multimodal data fusion according to claim 1, characterized in that: The step S5 specifically includes: inputting the fused features into a classifier to obtain an identification result of the harmful gas leakage.

6. A harmful gas leakage detection system based on the method according to any one of claims 1 to 5, characterized in that: include: A data acquisition module configured to acquire gas sensor data and thermal imager data in an industrial plant environment; A data processing module, configured to pre-process the gas sensor data and the thermal imager data to obtain sensor data and infrared images that have undergone data cleaning and data enhancement; a feature extraction module configured to extract effective features from the sensor data and the infrared image to obtain a high-dimensional, distinguishable feature representation; A network training module is configured to learn the preprocessed sensor data and infrared images, calculate the overall loss value through the loss function, and update the network parameters through the loss value; A feature fusion module configured to fuse the sensor data and the high-dimensional feature representation in the infrared image to obtain a unified high-dimensional feature representation after fusion; The gas detection module is configured to obtain a detection result of harmful gas leakage according to the unified high-dimensional feature representation after fusion.

Images