LNG low-temperature experiment center safety early warning method and system based on multi-modal data fusion

Abstract

The invention discloses an LNG low-temperature experiment center safety early warning method based on multi-modal data fusion, and the method specifically comprises the following steps: S1, carrying out the preprocessing of multi-source monitoring data, and carrying out the data cleaning and denoising of the multi-source monitoring data; performing time sequence synchronization on the data of different acquisition frequencies by using a timestamp alignment algorithm and an interpolation compensation algorithm; s2, performing feature extraction on the multi-source data; s3, performing deep fusion on the extracted multi-source data features by using a multi-level cross attention module, and modeling correlation among the video, the audio and the sensor data; and S4, decoding the fused features through a Transform decoding framework based on an attention mechanism, so as to realize safety risk early warning of the LNG low-temperature experiment center. According to the method, a Transform architecture and a multi-stage cross attention module are introduced, deep feature extraction and fusion are performed on video, audio and multi-class sensor data, and efficient identification and early warning of security risks are realized.

Description

Technical Field

[0001] This invention belongs to the field of industrial safety monitoring and artificial intelligence technology, specifically involving a safety early warning method and system for LNG cryogenic test centers based on multimodal data fusion. Background Technology

[0002] Liquefied natural gas (LNG) cryogenic testing centers are crucial facilities for the natural gas industry to conduct cryogenic characteristic research, equipment testing, and safety assessments. These centers typically include critical equipment such as LNG storage tanks, pipelines, cryogenic pumps, and heat exchangers. During operation and testing, they frequently face potential risks such as temperature fluctuations, abnormal pressure, gas leaks, and equipment malfunctions. Failure to identify and issue timely warnings can easily lead to safety accidents, potentially resulting in serious consequences such as fires or explosions.

[0003] Currently, most common safety monitoring technologies rely on single-type sensors, such as temperature, pressure, or gas detectors, or employ relatively simple data fusion methods, such as numerical stitching or weighted summation. These methods have significant shortcomings: First, a single data source cannot fully cover complex operating conditions; under the interaction of multiple factors, single-source information often cannot support high-precision risk identification. Second, the operating environment of experimental centers is complex and noisy, with drastic temperature and humidity changes, frequent equipment vibrations, and electromagnetic interference, which can easily lead to signal anomalies, resulting in false alarms or missed alarms. Third, different modal data differ significantly in structure, distribution, and time dimensions; existing simple stitching strategies cannot effectively model the dependencies between them, resulting in limited fusion effects.

[0004] In recent years, artificial intelligence has made significant breakthroughs in fields such as computer vision and natural language processing, providing new approaches to solving the aforementioned problems. In particular, the Transformer framework based on self-attention mechanisms, capable of dynamically capturing global correlations in sequence data, has been widely applied to tasks such as machine translation and image generation. However, most existing multimodal information fusion methods rely on strategies such as summation, maximization, or simple merging, making it difficult to fully align and utilize data from different modalities. In complex experimental environments, simple fusion operations often lead to an excessive proportion of certain modal features in the final result, while the role of other modalities is weakened or even ignored. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a safety early warning method for LNG cryogenic test centers based on multimodal data fusion. This invention introduces the Transformer architecture and multi-level cross-attention modules to perform deep feature extraction and fusion of video, audio and multi-type sensor data to obtain higher-dimensional and more accurate feature representations, thereby achieving efficient identification and early warning of safety risks.

[0006] To achieve the above-mentioned objectives, the technical solution provided by this invention patent is as follows: A safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion, the method specifically includes the following steps: S1 preprocesses the multi-source monitoring data by using Kalman filtering, wavelet transform, image enhancement, and audio spectrum analysis to clean and denoise the data; and uses timestamp alignment and interpolation compensation algorithms to synchronize the data from different acquisition frequencies. Kalman filtering and wavelet transform are used to suppress noise and smooth signals from sensor data; image enhancement methods are used to improve the quality of video data; and audio spectrum analysis is used to extract acoustic features and filter noise from sound data, thereby achieving cleaning and denoising of multi-source data. S2 extracts features from multi-source data by using autoencoders, convolutional neural networks, and long short-term memory networks to extract features based on the characteristics of different data. An autoencoder is used to learn features from time-series data collected by sensors and obtain low-dimensional feature representations; a convolutional neural network is used to extract spatial features from video surveillance data and gas imaging data; and a long short-term memory network is used to extract time-series features from sound monitoring data to obtain feature representations of different modal data. S3 utilizes a multi-level cross-attention module to deeply fuse the extracted multi-source data features and model the correlation between video, audio, and sensor data. S4 uses an attention-based Transformer decoding framework to decode the fused features, enabling early warning of safety risks at the LNG cryogenic experimental center.

[0007] Furthermore, the multi-source monitoring data includes sensor data, video data, and audio data; the sensor data includes temperature sensor data, pressure sensor data, vibration sensor data, and gas concentration sensor data; the video data includes video surveillance data and optical gas imaging data; and the audio data includes sound data.

[0008] Furthermore, the sensor data preprocessing specifically involves: identifying and correcting data singularities in the sensor data using Kalman filtering and wavelet transform methods, and performing noise reduction processing on the sensor data to achieve physical rationality correction of the sensor and avoid misjudgment caused by instantaneous signal fluctuations or uneven sampling.

[0009] Furthermore, the video data preprocessing specifically involves: the data collected by the monitoring and optical gas imaging equipment deployed in the LNG cryogenic experimental center are all video data. By adopting adaptive histogram equalization, the image is adaptively enhanced according to the contrast of local areas, avoiding exposure or underexposure caused by global equalization, and reducing the image quality degradation caused by factors such as ambient light and reflection; at the same time, the bilateral filtering method is used to remove environmental noise in the video while protecting the image edge information and retaining important structural information.

[0010] Furthermore, the audio data preprocessing specifically involves: performing a Fourier transform on the original sound signal to convert it to the frequency domain, and then using a bandpass filter to filter the sound signal, thereby removing interference from environmental noise and completing the audio data preprocessing.

[0011] Furthermore, the sensor data feature extraction specifically involves: converting the time-series signal in the sensor data into a two-dimensional feature map of frequency, time, and energy using a short-time Fourier transform; then compressing and encoding the two-dimensional feature map using an autoencoder; and extracting potential and key operating state features.

[0012] Furthermore, the video surveillance data feature extraction specifically involves: using a target detection algorithm driven by a convolutional neural network to identify people in the scene and their safety attire in real time, and extracting visual features related to people's behavior and safety status; the optical gas imaging data feature extraction specifically involves: using the YOLO model to accurately identify gas leakage phenomena in the image, and using it as the core feature of this modality.

[0013] Furthermore, the audio data feature extraction specifically involves: constructing an abnormal sound detection model based on a long short-term memory network, capturing abnormal sounds appearing in the audio data, and treating them as independent audio features.

[0014] Furthermore, the multi-source data features in S3 specifically involve: internal fusion of sensor data, normalizing the acquired temperature sensor data features, pressure sensor data features, vibration sensor data features, and gas concentration sensor data features, and then merging them into a single feature vector through a splicing operation, integrating the information from all sensor data into a whole; normalizing the acquired video surveillance data features and optical gas imaging data features; and normalizing the acquired audio data features; and aligning and fusing features from different modalities through a multi-level cross-attention mechanism; thereby achieving correlation fusion between video, audio, and sensor data.

[0015] A safety early warning system for an LNG cryogenic experimental center based on multimodal data fusion is proposed. This system includes a multi-source monitoring data preprocessing module, a multi-source data feature extraction module, a multi-source data feature fusion module, and a feature decoding module. The multi-source monitoring data preprocessing module is used to clean and denoise the acquired sensor data, video data, and audio data; to suppress noise and smooth signals using Kalman filtering and wavelet transform for sensor time-series data; to improve image quality using image enhancement methods for video data; to process acoustic signals and filter noise using spectral analysis methods for audio data; and to synchronize data at different sampling frequencies using timestamp alignment and interpolation compensation algorithms. The multi-source data feature extraction module is used to extract features from different types of data; to learn features from sensor time-series data through an autoencoder and obtain low-dimensional feature representations; to extract spatial features from video surveillance data and optical gas imaging data through a convolutional neural network; and to extract temporal features from audio data through a long short-term memory network. The multi-source data feature fusion module is used to fuse features of different modal data. It aligns and fuses sensor data features, video data features, and audio data features through a multi-level cross-attention mechanism to establish the correlation between different modal data. The feature decoding module is used by the attention-based Transformer decoding framework to decode the fused features and output the safety risk warning results of the LNG cryogenic experimental center based on the decoding results.

[0016] Based on the above technical solutions, the safety early warning method for LNG cryogenic test centers based on multimodal data fusion has achieved the following technical advantages through practical application: The safety early warning method for LNG cryogenic test centers based on multimodal data fusion effectively solves the feature alignment problem in multimodal data fusion through a multi-level cross-attention framework. Through a hierarchical and refined attention mechanism, it realizes deep interaction and collaboration of different modal information, thereby achieving a more comprehensive safety early warning for LNG cryogenic test centers. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the preprocessing of multimodal data in a safety early warning method for LNG cryogenic experimental centers based on multimodal data fusion, as described in this invention.

[0018] Figure 2 This is a schematic diagram of feature extraction in a safety early warning method for LNG cryogenic experimental centers based on multimodal data fusion, as described in this invention.

[0019] Figure 3This is a schematic diagram of cross-attention calculation in Embodiment 2 of the present invention, which is a safety early warning method for LNG cryogenic experimental centers based on multimodal data fusion.

[0020] Figure 4 This is a multimodal data feature fusion framework diagram in the LNG cryogenic experimental center safety early warning method based on multimodal data fusion of the present invention.

[0021] Figure 5 This is a schematic diagram of the model optimization process in Embodiment 2 of the present invention, which is a safety early warning method for LNG cryogenic experimental centers based on multimodal data fusion. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is described below with reference to specific examples shown in the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0023] Example 1: A safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion. The method specifically includes the following steps: S1 preprocesses the multi-source monitoring data by using Kalman filtering, wavelet transform, image enhancement, and audio spectrum analysis to clean and denoise the data; and uses timestamp alignment and interpolation compensation algorithms to synchronize the data from different acquisition frequencies. Kalman filtering and wavelet transform are used to suppress noise and smooth signals from sensor data; image enhancement methods are used to improve the quality of video data; and audio spectrum analysis is used to extract acoustic features and filter noise from sound data, thereby achieving cleaning and denoising of multi-source data. S2 extracts features from multi-source data by using autoencoders, convolutional neural networks, and long short-term memory networks to extract features based on the characteristics of different data. An autoencoder is used to learn features from time-series data collected by sensors and obtain low-dimensional feature representations; a convolutional neural network is used to extract spatial features from video surveillance data and gas imaging data; and a long short-term memory network is used to extract time-series features from sound monitoring data to obtain feature representations of different modal data. S3 utilizes a multi-level cross-attention module to deeply fuse the extracted multi-source data features and model the correlation between video, audio, and sensor data. S4 uses an attention-based Transformer decoding framework to decode the fused features, enabling early warning of safety risks at the LNG cryogenic experimental center.

[0024] The multi-source monitoring data includes sensor data, video data, and audio data; the sensor data includes temperature sensor data, pressure sensor data, vibration sensor data, and gas concentration sensor data; the video data includes video surveillance data and optical gas imaging data; and the audio data includes sound data.

[0025] The sensor data preprocessing specifically involves: identifying and correcting data singularities in the sensor data using Kalman filtering and wavelet transform, and performing noise reduction on the sensor data to achieve physical rationality correction of the sensor and avoid misjudgment caused by instantaneous signal fluctuations or uneven sampling.

[0026] Sensor data such as temperature, pressure, vibration, and gas concentration have strong temporal sequence and numerical continuity. Therefore, in the process of processing sensor data, this invention mainly uses Kalman filtering and wavelet transform to identify and correct data singularities in the sensor data, perform noise reduction on the sensor data, and perform physical rationality correction on different data to avoid misjudgment caused by instantaneous signal fluctuations or uneven sampling, thereby improving the stability and reliability of multimodal fusion. The video data preprocessing specifically involves: the data collected by the monitoring and optical gas imaging equipment deployed in the LNG cryogenic experimental center are all video data. By adopting adaptive histogram equalization, the image is adaptively enhanced according to the contrast of local areas, avoiding exposure or underexposure caused by global equalization, and reducing the image quality degradation caused by factors such as ambient light and reflection. At the same time, the bilateral filtering method is used to remove environmental noise in the video while protecting the image edge information and retaining important structural information.

[0027] For the monitoring and optical gas imaging equipment deployed in the LNG cryogenic experimental center, which collects video data, this invention combines data enhancement and denoising methods to ensure data quality and support subsequent multimodal fusion. This invention employs adaptive histogram equalization, which adaptively enhances the image based on the contrast of local areas, avoiding overexposure or underexposure caused by global equalization and reducing image quality degradation due to ambient lighting, reflections, and other factors. Simultaneously, it utilizes bilateral filtering to remove environmental noise from the video while protecting image edge information and preserving important structural information. The audio data preprocessing specifically involves: performing a Fourier transform on the original sound signal to convert it to the frequency domain, and then using a bandpass filter to filter the sound signal, removing interference from environmental noise, thus completing the audio data preprocessing.

[0028] Because the LNG cryogenic test center contains a large number of heavy mechanical equipment and the on-site environment is relatively noisy, this invention performs spectrum analysis on the original audio signal. By performing Fourier transform on the original sound signal to convert the sound signal to the frequency domain, and using a bandpass filter to filter the sound signal, the interference of environmental noise is effectively filtered out, providing high-quality and clean original data for subsequent data analysis and feature extraction.

[0029] To ensure accurate fusion analysis of data from different modalities, this invention establishes a unified global clock marker to precisely align all data points according to their timestamps, ensuring that each set of data to be fused comes from the same time. Since the sampling frequency of sensor data is often low, in order to reduce the impact of data sparsity on subsequent results, this invention uses spline interpolation to generate missing data points on the time axis.

[0030] The sensor data feature extraction specifically involves: converting the time-series signal in the sensor data into a two-dimensional feature map of frequency, time, and energy using a short-time Fourier transform; then compressing and encoding the two-dimensional feature map using an autoencoder; and extracting potential and key operational state features.

[0031] The video surveillance data feature extraction specifically involves: using a target detection algorithm driven by a convolutional neural network to identify people in the scene and their safety attire in real time, and extracting visual features related to people's behavior and safety status; the optical gas imaging data feature extraction specifically involves: using the YOLO model to accurately identify gas leakage phenomena in the image, and using it as the core feature of this modality.

[0032] The specific audio data extraction feature is as follows: an abnormal sound detection model is constructed based on a long short-term memory network to capture abnormal sounds appearing in the audio data and treat them as independent audio features.

[0033] The multi-source data features in S3 specifically involve: internal fusion of sensor data, normalizing the acquired temperature sensor data features, pressure sensor data features, vibration sensor data features, and gas concentration sensor data features, and then merging them into a single feature vector through a splicing operation, integrating the information from all sensor data into a whole; normalizing the acquired video surveillance data features and optical gas imaging data features; and normalizing the acquired audio data features; and aligning and fusing features from different modalities through a multi-level cross-attention mechanism; thus achieving correlation fusion between video, audio, and sensor data.

[0034] As shown in Figure 2, the method of this invention aims to solve the problem of feature alignment for multimodal data. In particular, the alignment of different modal data (such as sensor, video, and audio) in the feature space requires a large amount of data to ensure accuracy, which greatly increases the difficulty of model training. To this end, we propose a strategy: to use existing pre-trained models for specific tasks to extract features from data of different modalities. This approach fully leverages the powerful capabilities of existing models to transform raw data into high-quality feature representations, thereby simplifying subsequent fusion tasks and reducing reliance on large-scale labeled data. In the feature extraction stage, we employ tailored methods to efficiently acquire core features for different modalities of data. For sensor data, we first utilize Short-Time Fourier Transform (STFT) to convert its time-series signal into a two-dimensional feature map of frequency-time-energy, then compress and encode it using an autoencoder to extract its potential and critical operational state features. For video data, we utilize a Convolutional Neural Network (CNN)-driven target detection algorithm to extract visual features related to personnel behavior and safety status by identifying people and their safety attire in real-time. For optical gas imaging data, we rely on the powerful detection capabilities of the YOLO (You Only Look Once) model to accurately identify gas leaks in images and use them as the core feature of this modality. Finally, for audio data, we construct an abnormal sound detection model based on a Long Short-Term Memory (LSTM) network, which can effectively capture abnormal sounds such as unusual noises and friction sounds occurring during equipment operation and treat them as independent audio features. Through these diverse and targeted feature extraction strategies, we have laid a solid foundation for subsequent multimodal data fusion.

[0035] A safety early warning system for an LNG cryogenic experimental center based on multimodal data fusion is proposed. This system includes a multi-source monitoring data preprocessing module, a multi-source data feature extraction module, a multi-source data feature fusion module, and a feature decoding module. The multi-source monitoring data preprocessing module is used to clean and denoise the acquired sensor data, video data, and audio data; to suppress noise and smooth signals using Kalman filtering and wavelet transform for sensor time-series data; to improve image quality using image enhancement methods for video data; to process acoustic signals and filter noise using spectral analysis methods for audio data; and to synchronize data at different sampling frequencies using timestamp alignment and interpolation compensation algorithms. The multi-source data feature extraction module is used to extract features from different types of data; to learn features from sensor time-series data through an autoencoder and obtain low-dimensional feature representations; to extract spatial features from video surveillance data and optical gas imaging data through a convolutional neural network; and to extract temporal features from audio data through a long short-term memory network. The multi-source data feature fusion module is used to fuse features of different modal data. It aligns and fuses sensor data features, video data features, and audio data features through a multi-level cross-attention mechanism to establish the correlation between different modal data. The feature decoding module is used by the attention-based Transformer decoding framework to decode the fused features and output the safety risk warning results of the LNG cryogenic experimental center based on the decoding results.

[0036] Example 2: Feature fusion of multi-source data; This invention utilizes a multi-level cross-attention framework to efficiently fuse extracted features from multi-source data. The specific process is as follows: Figure 4 As shown.

[0037] The fusion process consists of the following key steps: 1. Sensor data internal fusion: First, the features of different types of sensor data (such as temperature, pressure, vibration, etc.) are normalized. Then, they are merged into a unified feature vector through concatenation. This step integrates the information of all sensor data into a whole, preparing for subsequent fusion with other modalities.

[0038] 2. Initial Fusion of Multimodal Features: At this level, we introduce a cross-attention mechanism, using sensor feature vectors as query vectors (Query, Q) and features from other modalities (such as audio, video, and gas imaging) as key-value pairs (Key, K and Value, V). This allows the model to learn the interdependencies between different modal data. This process enables each modal feature to "pay attention" to the most relevant information in other modalities, thereby achieving preliminary feature alignment and fusion.

[0039] 3. Multi-level attention fusion: Figure 4 This demonstrates that the fusion is not accomplished all at once, but rather through a multi-layered cross-attention mechanism. This multi-level structure allows the model to capture and fuse complex relationships between modalities at different levels of abstraction, ensuring that information is fully interacted and integrated, and avoiding information loss or conflict caused by simple splicing.

[0040] In this invention, cross-attention is an important component of multi-source feature fusion, and its working principle is as follows: Figure 3 As shown; firstly, the query (Q1,Q2), key (K1,K2), and pointer (V1,V2) of the input features (x1,x2) are obtained through different linear layers; next, the cross-attention of the input values ​​is calculated, as shown in the following formula: , Will , The parts are spliced ​​together and then passed through a linear layer to obtain the fused feature y.

[0041] Based on the attention mechanism, the feature decoding and fusion features are processed in depth through the Transformer decoder framework. The self-attention mechanism inside the Transformer framework can further capture the dependencies within the fused features, and finally output the final prediction result through the linear layer and the softmax layer, which serves as a safety warning result for the LNG cryogenic test center.

[0042] The multi-level cross-attention framework proposed in this invention effectively solves the feature alignment problem in multimodal data fusion. Through a hierarchical and refined attention mechanism, it realizes deep interaction and collaboration of information from different modalities, thereby enabling more comprehensive safety early warning for LNG cryogenic test centers.

[0043] The initially trained model (M0) often performs poorly in specific scenarios due to incomplete data collection in the early stages. Furthermore, its performance can change over time due to equipment aging, changes in operating modes, and other factors. To ensure system performance, this invention proposes a localized automatic model optimization process. This process autonomously optimizes the deployed LNG cryogenic experimental center's multimodal safety monitoring model (M0) by collecting user feedback. This process forms a closed loop, ensuring the model can continuously adapt to the complexity and changes in real-world applications. The specific process is as follows: P1, User Feedback Information Collection: The system first deploys the initial security monitoring model M0. During model operation, the system autonomously and continuously collects user feedback information. This feedback includes not only manually entered user evaluations (such as annotations for false alarms or missed alarms), but may also include user behavior data automatically captured by the system, such as how users handle alarm events and the processing time. These serve as important bases for model optimization. In P2, using the user feedback information collected in P1, the system further optimizes M0 to generate a new, improved model M1. This optimization process may be based on reinforcement learning technology, enabling the model to learn from user feedback to reduce false alarms and improve accuracy, thereby obtaining the optimized security monitoring model M1. P3, Model Performance Validation and Deployment This stage is a crucial step in ensuring that the new model is superior to the old model. It verifies the optimization effect through a series of comparisons and tests. P31; Pre-deployment and effect comparison: Pre-deploy the optimized model M1 and compare its output with the output of the original model M0. If M1 is statistically significantly better than M0 (e.g., lower false alarm rate, higher accuracy), the process proceeds to the next stage P32. If M1 performs poorly or does not show a significant improvement compared to M0, it is considered that there is a problem with the model optimization, and it is necessary to return to P1 to collect more feedback information again. P32, Task Decentralization and Online Evaluation If M1 passes the verification in P31, the system will deploy and test using a task splitting approach. Initially, only 10% of tasks will flow to M1, while 90% will still be handled by M0. As time progresses and M1's performance stabilizes, the proportion of tasks allocated to M1 will gradually increase, for example, from 10% to 20%, and eventually to 100%. During this process, the system will continuously monitor M1's performance and evaluate it in real time based on user feedback. If a problem is found with M1 during this period: if the problem is minor, the process returns to P2, and the model is fine-tuned using the new feedback information; if the problem is serious, the process returns directly to P1, restarting the entire optimization process. The entire process ends when M1's performance fully meets the preset standards. This closed-loop mechanism ensures that the model's performance can continuously improve with changes in actual application and environment, achieving true localized autonomous optimization.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion, characterized in that, The method specifically includes the following steps: S1 preprocesses the multi-source monitoring data by performing data cleaning and noise reduction through Kalman filtering, wavelet transform, image enhancement, and audio spectrum analysis. It also utilizes timestamp alignment and interpolation compensation algorithms to synchronize data from different acquisition frequencies. Kalman filtering and wavelet transform are used to suppress noise and smooth signals in sensor data; image enhancement methods are used to improve the quality of video data. Audio spectrum analysis is used to extract acoustic features and filter noise from sound data, thereby achieving cleaning and denoising of multi-source data; S2 extracts features from multi-source data by using autoencoders, convolutional neural networks, and long short-term memory networks to extract features based on the characteristics of different data. An autoencoder is used to learn features from time-series data collected by sensors and obtain low-dimensional feature representations; a convolutional neural network is used to extract spatial features from video surveillance data and gas imaging data; and a long short-term memory network is used to extract time-series features from sound monitoring data to obtain feature representations of different modal data. S3 utilizes a multi-level cross-attention module to deeply fuse the extracted multi-source data features and model the correlation between video, audio, and sensor data. S4 uses an attention-based Transformer decoding framework to decode the fused features, enabling early warning of safety risks at the LNG cryogenic experimental center.

2. The safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion according to claim 1, characterized in that, The multi-source monitoring data includes sensor data, video data, and audio data; the sensor data includes temperature sensor data, pressure sensor data, vibration sensor data, and gas concentration sensor data; the video data includes video surveillance data and optical gas imaging data; and the audio data includes sound data.

3. The safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion according to claim 2, characterized in that, The sensor data preprocessing specifically involves: identifying and correcting data singularities in the sensor data using Kalman filtering and wavelet transform, and performing noise reduction on the sensor data to achieve physical rationality correction of the sensor and avoid misjudgment caused by instantaneous signal fluctuations or uneven sampling.

4. The safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion according to claim 3, characterized in that, The video data preprocessing specifically involves: the data collected by the monitoring and optical gas imaging equipment deployed in the LNG cryogenic experimental center are all video data. By adopting adaptive histogram equalization, the image is adaptively enhanced according to the contrast of local areas, avoiding exposure or underexposure caused by global equalization, and reducing the image quality degradation caused by factors such as ambient light and reflection. At the same time, the bilateral filtering method is used to remove environmental noise in the video while protecting the image edge information and retaining important structural information.

5. A safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion according to claim 4, characterized in that, The audio data preprocessing specifically involves: performing a Fourier transform on the original sound signal to convert it to the frequency domain, and then using a bandpass filter to filter the sound signal, removing interference from environmental noise, thus completing the audio data preprocessing.

6. A safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion as described in claim 5, characterized in that, The sensor data feature extraction specifically involves: converting the time-series signal in the sensor data into a two-dimensional feature map of frequency, time, and energy using a short-time Fourier transform; then compressing and encoding the two-dimensional feature map using an autoencoder; and extracting potential and key operational state features.

7. A safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion as described in claim 6, characterized in that, The video surveillance data feature extraction specifically involves: using a target detection algorithm driven by a convolutional neural network to identify people in the scene and their safety attire in real time, and extracting visual features related to people's behavior and safety status; the optical gas imaging data feature extraction specifically involves: using the YOLO model to accurately identify gas leakage phenomena in the image, and using it as the core feature of this modality.

8. A safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion as described in claim 7, characterized in that, The specific audio data extraction feature is as follows: an abnormal sound detection model is constructed based on a long short-term memory network to capture abnormal sounds appearing in the audio data and treat them as independent audio features.

9. A safety early warning method for an LNG cryogenic experimental center based on multimodal data fusion as described in claim 8, characterized in that, The multi-source data features in S3 specifically involve: internal fusion of sensor data, normalizing the acquired temperature sensor data features, pressure sensor data features, vibration sensor data features, and gas concentration sensor data features, and then merging them into a single feature vector through a splicing operation, integrating the information from all sensor data into a whole; normalizing the acquired video surveillance data features and optical gas imaging data features; and normalizing the acquired audio data features; and aligning and fusing features from different modalities through a multi-level cross-attention mechanism; thus achieving correlation fusion between video, audio, and sensor data.

10. A safety early warning system for an LNG cryogenic experimental center based on multimodal data fusion, characterized in that, The system includes a multi-source monitoring data preprocessing module, a multi-source data feature extraction module, a multi-source data feature fusion module, and a feature decoding module. The multi-source monitoring data preprocessing module is used to clean and denoise the acquired sensor data, video data, and audio data; to suppress noise and smooth signals using Kalman filtering and wavelet transform for sensor time-series data; to improve image quality using image enhancement methods for video data; to process acoustic signals and filter noise using spectral analysis methods for audio data; and to synchronize data at different sampling frequencies using timestamp alignment and interpolation compensation algorithms. The multi-source data feature extraction module is used to extract features from different types of data; to learn features from sensor time-series data through an autoencoder and obtain low-dimensional feature representations; to extract spatial features from video surveillance data and optical gas imaging data through a convolutional neural network; and to extract temporal features from audio data through a long short-term memory network. The multi-source data feature fusion module is used to fuse features of different modal data. It aligns and fuses sensor data features, video data features, and audio data features through a multi-level cross-attention mechanism to establish the correlation between different modal data. The feature decoding module is used by the attention-based Transformer decoding framework to decode the fused features and output the safety risk warning results of the LNG cryogenic experimental center based on the decoding results.

Images