Heating station environment monitoring and early warning method based on bimodal image sample training

Abstract

The present application relates to the technical field of intelligent monitoring, in particular to a heating station environment monitoring and early warning method based on bimodal image sample training, which synchronously collects video frame sequences and audio frame sequences with time stamps through image acquisition equipment and sound acquisition equipment, and constructs bimodal image sample data according to time stamp pairing after pretreatment; visual modality confidence and auditory modality confidence are obtained, and bimodal image sample features are extracted; when the consistency check result is inconsistent, the heating station environment monitoring result is obtained according to the bimodal image sample features; when the consistency check result is consistent, the bimodal image sample features are weighted and fused according to the confidence, the fusion classifier is used to identify and obtain the heating station environment monitoring result; early warning is carried out according to the heating station environment monitoring result. The method can adapt to special working conditions such as high humidity steam shielding, equipment vibration blur and high reverberation in the heating station, effectively improving the accuracy and reliability of the heating station environment monitoring.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology, specifically to a method for environmental monitoring and early warning of heating stations based on dual-modal image sample training. Background Technology

[0002] Heating stations are the core component of urban centralized heating systems, and their equipment operation directly affects the quality of heating. Heating stations typically house boilers, heat exchangers, circulating pumps, valves, and other equipment. The operating environment is characterized by high temperature and humidity, posing various safety risks such as steam leaks, pipe ruptures, bearing wear, electrical fires, and unauthorized intrusion. Any abnormalities can range from affecting heating efficiency to causing serious safety accidents.

[0003] Currently, the daily management of heating stations still relies heavily on regular manual inspections. This method is not only labor-intensive but also difficult to achieve real-time monitoring, and missed inspections are common. With advancements in sensor technology and artificial intelligence algorithms, visual image-based intelligent monitoring systems are gradually being applied in industrial scenarios. These systems deploy cameras in the monitoring area to capture video footage and then use deep learning models to identify abnormal events such as intrusions and smoke / fire. Some solutions also attempt to introduce sound acquisition devices, improving the comprehensiveness of monitoring by fusing image and sound information. Theoretically, multimodal fusion can utilize the complementarity between different signals, offering advantages over relying on only one signal.

[0004] However, in practical applications, the environmental conditions of heating stations pose significant challenges to monitoring systems. The interiors of heating stations are often filled with high-temperature water vapor, which reduces the effectiveness of sunlight and can directly obstruct the camera's field of view. Furthermore, the continuous vibrations caused by equipment operation easily lead to motion blur in video footage, significantly degrading image quality. When visual signals become unreliable, the accuracy of monitoring methods relying solely on video analysis decreases drastically. Sound signals also face interference. The densely packed equipment and relatively enclosed space of heating stations cause sound to reflect multiple times from walls and equipment surfaces during propagation, resulting in significant reverberation. This blurs the extracted acoustic features, affecting the accuracy of auditory modality analysis.

[0005] In multimodal fusion, most existing methods employ feature concatenation or fixed-weight addition to merge image and sound features before feeding them into the classifier. This approach assumes that both modalities are equally reliable at all times, which is not the case in reality. When both visual and auditory signals are of good quality, the fusion effect is acceptable; however, when one modality is disturbed or its output contradicts that of the other, the fixed-weight fusion method introduces erroneous information into the final judgment, resulting in worse outcomes than single-modal fusion. Currently available technical solutions rarely include mechanisms for consistency checks on the judgment results of different modalities before fusion, nor do they design corresponding processing paths for modal conflicts.

[0006] Furthermore, deep learning-based monitoring models are often a "black box," capable of outputting anomaly categories and probabilities, but unable to explain the basis for the judgment, the specific area of ​​the image where the anomaly occurred, or its location in space. In scenarios with high reliability requirements, such as heating stations, if maintenance personnel only see a warning result without knowing the cause, it is difficult for them to make quick decisions and may lead to distrust of the system. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a heating station environmental monitoring and early warning method based on dual-modal image sample training, which solves the problem of poor detection accuracy in existing technologies that rely on single-modal or fixed-weight multimodal fusion analysis.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a heating station environmental monitoring and early warning method based on dual-modal image sample training, comprising: S101. Acquire video frame sequences and audio frame sequences with timestamps by at least one image acquisition device in the monitoring area within the heating station and at least one sound acquisition device. S102. Preprocess the video frame sequence and the audio frame sequence to obtain a preprocessed video frame sequence and a preprocessed audio frame sequence. S103. Based on the timestamp, pair the preprocessed video frame sequence with the preprocessed audio frame sequence to obtain bimodal image sample data. S104. Obtain the confidence level of the bimodal image samples and obtain the features of the bimodal image samples based on the bimodal image sample data, and perform cross-modal consistency verification based on the features of the bimodal image samples to obtain the consistency verification result. S105. If the consistency verification result is inconsistent, the environmental monitoring result of the heating station is obtained based on the dual-modal image sample features. S106. If the consistency verification result is consistent, the features of the dual-modal image samples are weighted and fused according to the confidence level of the dual-modal image samples to obtain joint data features, and the joint data features are identified by a pre-set fusion classifier to obtain the environmental monitoring results of the heating station. S107. Issue an early warning based on the environmental monitoring results of the heating station, and complete the environmental monitoring and early warning of the heating station based on dual-modal image sample training.

[0009] By simultaneously acquiring video and audio frame sequences in the monitoring area of ​​the heating station, and constructing dual-modal image sample data after preprocessing and timestamp pairing, a cross-modal consistency verification mechanism is introduced. When the consistency verification results are consistent, weighted fusion is performed based on confidence level; when inconsistent, single-modal diversion decision is adopted. This effectively avoids the problem of forced fusion false alarms caused by modal conflict or single-modal degradation. Early warning is implemented based on the environmental monitoring results of the heating station, which significantly improves the detection accuracy and robustness of environmental monitoring of the heating station under complex operating conditions, effectively reduces the false alarm rate and missed alarm rate, and provides reliable technical support for the safe and stable operation of the heating station.

[0010] Preferably, the video frame sequence and the audio frame sequence are preprocessed to obtain a preprocessed video frame sequence and a preprocessed audio frame sequence, including: The video frame sequence is subjected to frame extraction processing to obtain target video frames, and multiple target video frames constitute a preprocessed video frame sequence. The audio frame sequence is subjected to a windowed short-time Fourier transform to generate a log-Mel spectrogram, which yields the target audio spectrum segments. Multiple target audio spectrum segments constitute the preprocessed audio frame sequence.

[0011] This invention generates log-Mel spectrograms by performing frame extraction on video frame sequences and windowed short-time Fourier transform on audio frame sequences. This enables the efficient conversion of raw video and audio streams into standard input formats that can be processed by deep learning models. Frame extraction reduces data redundancy and computational overhead, while log-Mel spectrograms can better simulate the characteristics of human hearing, laying the foundation for accurate extraction of subsequent bimodal features.

[0012] Preferably, based on the timestamp, the preprocessed video frame sequence and the preprocessed audio frame sequence are paired to obtain bimodal image sample data, including: For any target video frame in the preprocessed video frame sequence, the time difference between the target video frame and the target audio spectrum segment in the preprocessed audio frame sequence is obtained based on the timestamp. When the time difference is less than a preset time threshold, the corresponding target video frame and the target audio spectrum segment are constructed into bimodal image sample data.

[0013] This invention calculates the time difference between the target video frame and the target audio spectrum segment by timestamp and compares it with a preset threshold, thereby achieving precise time alignment between the video frame and the audio spectrum segment. This ensures the consistency of the bimodal image sample data in the time dimension and provides data quality assurance for the accuracy of subsequent cross-modal consistency verification and fusion analysis.

[0014] Preferably, after pairing the preprocessed video frame sequence with the preprocessed audio frame sequence, the method further includes: Unpaired target video frames and target audio spectrum segments are used as asynchronous data, and corresponding modal analysis is performed on the asynchronous data to output independent modal detection results. The independent modal detection results are used as auxiliary environmental monitoring data.

[0015] This invention uses unpaired target video frames and target audio spectrum segments as asynchronous data, performs independent analysis of the corresponding modalities, and outputs independent modal detection results as auxiliary environmental monitoring data. This ensures that even in the event of time alignment failure, the effective information of each modality can still be fully utilized instead of being directly discarded, thereby improving data utilization and the system's fault tolerance.

[0016] Preferably, obtaining the confidence level of the bimodal image samples and obtaining the bimodal image sample features based on the bimodal image sample data includes: The visual modality confidence score is obtained from the video frame sequence, and the auditory modality confidence score is obtained from the audio frame sequence. The auditory modality confidence score and the visual modality confidence score are used together as the bimodal image sample confidence score. The target video frames in the bimodal image sample data are input into a visual deep learning model for analysis, and visual feature vectors and visual event category probability distributions are extracted to obtain visual modal image sample features. The target audio spectrum segment in the bimodal image sample data is input into the auditory deep learning model for analysis, and auditory feature vectors and auditory event category probability distributions are extracted to obtain auditory modal image sample features. The visual modality image sample features and the auditory modality image sample features are used together as bimodal image sample features.

[0017] This invention obtains visual modal confidence scores from video frame sequences and auditory modal confidence scores from audio frame sequences, respectively. It then inputs the target video frames and target audio spectrum segments into visual and auditory deep learning models to extract feature vectors and event category probability distributions. This enables dynamic quantitative evaluation and accurate extraction of deep semantic features for the quality of various modal signals under special operating conditions such as high humidity steam obstruction, equipment vibration ambiguity, and high reverberation in heating stations. This provides reliable foundational data for subsequent consistency verification and weighted fusion.

[0018] Preferably, cross-modal consistency verification is performed based on the features of the dual-modal image samples to obtain the consistency verification result, including: Obtain the KL divergence values ​​of the auditory event category probability distribution and the visual event category probability distribution in the bimodal image sample features; Determine whether the KL divergence value is less than or equal to a preset divergence threshold. If it is, determine that the consistency check result is consistent; otherwise, determine that the consistency check result is inconsistent.

[0019] This invention performs cross-modal consistency verification by obtaining the KL divergence values ​​of the probability distributions of auditory and visual event categories and comparing them with a preset divergence threshold. This can accurately quantify the distribution differences between the bimodal recognition results, providing an objective quantitative basis for triage decisions. When the bimodal judgments are contradictory, triage is automatically performed, effectively avoiding false alarms caused by forced fusion.

[0020] Preferably, if the consistency verification result is inconsistent, the environmental monitoring result of the heating station is obtained based on the features of the dual-modal image samples, including: If the consistency check result is inconsistent, then visual monomodal classification is performed using the visual modality image sample features in the bimodal image sample features to obtain the visual modality event probability distribution. Auditory monomodal classification is performed using the auditory modality image sample features from the bimodal image sample features to obtain the probability distribution of auditory modality events; The visual monomodal warning type and corresponding confidence level are determined based on the probability distribution of the visual modal events, and the auditory monomodal warning type and corresponding confidence level are determined based on the probability distribution of the auditory modal events. The visual monomodal warning type and its corresponding confidence level, and the auditory monomodal warning type and its corresponding confidence level are used together as the environmental monitoring results of the heating station.

[0021] In cases where the consistency verification results are inconsistent, this invention performs single-modal classification using visual modal image sample features and auditory modal image sample features respectively, and determines the single-modal warning type and corresponding confidence level based on their respective event probability distributions. This can avoid the introduction of erroneous information through forced fusion when visual and auditory modalities conflict, while making full use of the effective recognition results of each modality, thus improving the monitoring reliability of the system under modal conflict conditions.

[0022] Preferably, the features of the bimodal image samples are weighted and fused based on the confidence level of the bimodal image samples to obtain joint data features, and a pre-set fusion classifier is used to identify the joint data features to obtain the environmental monitoring results of the heating station, including: The visual weighting and auditory weighting are obtained based on the confidence scores of the bimodal image samples, respectively. The bimodal image sample features are weighted according to the visual weighting weight and the auditory weighting weight, and the weighted bimodal image sample features are concatenated into a vector to obtain joint data features; The joint data features are identified using a pre-set fusion classifier to obtain the joint event probability distribution; The probability distribution of the joint events is determined as the joint modal early warning type and the corresponding confidence level, and the environmental monitoring results of the heating station are obtained.

[0023] When the consistency verification result is consistent, this invention obtains visual weighting and auditory weighting based on the confidence level of dual-modal image samples, weights the features, and concatenates them into joint data features. The joint event probability distribution and corresponding warning type are identified by a fusion classifier, realizing dynamic adaptive weighted fusion based on signal quality. Compared with the fixed weight fusion method, it can effectively cope with the dynamic changes in the signal quality of each modality under different operating conditions of the heating station, and improves the adaptability of the fusion strategy and the accuracy of the fusion results.

[0024] Preferably, early warning is issued based on the environmental monitoring results of the heating station, including: If the environmental monitoring results of the heating station are a joint modal warning type and its corresponding confidence level, and the joint modal warning type is an abnormal type, a graded warning shall be issued based on the confidence level of the joint modal warning type. If the environmental monitoring results of the heating station are a visual single-modal warning type and its corresponding confidence level, an auditory single-modal warning type and its corresponding confidence level, and both the visual single-modal warning type and the auditory single-modal warning type are abnormal types, a graded warning is issued based on the confidence level of the visual single-modal warning type and the corresponding confidence level of the auditory single-modal warning type.

[0025] This invention implements a differentiated graded early warning strategy by providing graded early warnings based on confidence levels when the joint modal early warning type is an anomaly type, and by providing graded early warnings based on their respective confidence levels when all single modal early warning types are anomaly types. This enables operation and maintenance personnel to quickly determine the severity of the anomaly based on the early warning level and take corresponding response measures, thereby improving the availability of early warning information and the efficiency of emergency response.

[0026] Preferably, it further includes: Based on the fact that the joint modal warning type is an anomaly type, the fault causal chain tracing process is triggered; When the fault causal chain tracing process is triggered, gradient backpropagation and upsampling are performed according to the joint event probability distribution to obtain visual heatmaps and auditory heatmaps. Visual spatial mapping is performed based on the visual heatmap, and auditory spatial mapping is performed based on the auditory heatmap to obtain bimodal anomaly location information; Causal consistency verification is performed based on the bimodal anomaly location information to obtain the causal consistency verification result, and a diagnostic report is output based on the bimodal anomaly location information and the causal consistency verification result.

[0027] This invention triggers a fault causal chain tracing process when the joint modal warning type is an anomaly type. It generates visual and auditory heat maps through gradient backpropagation and integral gradient attribution methods. Combined with camera calibration and microphone array arrival time difference positioning, it achieves accurate location of anomaly points in the three-dimensional space of the heating station and verifies causal consistency. It also outputs a visual diagnostic report, presenting the judgment basis of the black box model intuitively in the form of heat maps and spatial positioning, which significantly improves the interpretability of monitoring results and helps operation and maintenance personnel quickly locate the root cause of the fault and make accurate handling decisions.

[0028] Compared with existing technologies, this invention has the following advantages: By simultaneously acquiring video and audio frame sequences in the monitoring area of ​​a heating station, and constructing dual-modal image sample data after preprocessing and timestamp pairing, a visual modal confidence function based on physical parameters such as effective illuminance, motion blur, and steam obstruction rate, and an auditory modal confidence function based on physical parameters such as signal-to-noise ratio and reverberation time are introduced. This enables dynamic quantitative evaluation of the quality of each modal signal under special operating conditions such as high humidity steam obstruction, equipment vibration blur, and high reverberation in heating stations. Cross-modal consistency verification is performed using KL divergence. When the consistency verification result is consistent, weighted fusion is performed based on confidence level; otherwise, cross-modal consistency is not performed. The system employs a single-modal triage decision-making mechanism, effectively avoiding false alarms caused by forced fusion due to modal conflicts or single-modal degradation. Simultaneously, during the early warning phase, a tiered early warning strategy is implemented based on confidence levels. Upon anomaly triggering, visual and auditory heat maps are generated using gradient backpropagation and integral gradient attribution methods. Combined with camera calibration and microphone array time-difference positioning, the system achieves precise location and causal consistency verification of anomaly points in the three-dimensional space of the heating station, outputting a visualized diagnostic report. This significantly improves the accuracy, robustness, and interpretability of environmental monitoring in heating stations under complex operating conditions, effectively reducing false alarm and missed alarm rates, and providing reliable technical support for the safe and stable operation of heating stations. Attached Figure Description

[0029] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0030] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0031] like Figure 1 As shown in the figure, this application provides a heating station environmental monitoring and early warning method based on dual-modal image sample training, including: S101. Acquire video frame sequences and audio frame sequences with timestamps by at least one image acquisition device in the monitoring area within the heating station and at least one sound acquisition device. For example, at least one image acquisition device and at least one audio acquisition device can be deployed in the monitoring area within the heating station. Based on a unified network clock, video frame sequences and audio frame sequences are acquired, and acquisition timestamps are recorded simultaneously. For instance, the image acquisition device can be configured as a camera with a fixed frame rate of 25 frames per second and adaptively adjustable exposure time; the audio acquisition device can be configured as a microelectromechanical system (MEMS) digital microphone array with a sampling rate of 48,000 samples per second and a quantization bit depth of 24 bits; time synchronization is achieved using the Precision Time Protocol (PTP), with a synchronization error not exceeding 1 millisecond.

[0032] S102. Preprocess the video frame sequence and the audio frame sequence to obtain a preprocessed video frame sequence and a preprocessed audio frame sequence. In some possible implementations, the video frame sequence and the audio frame sequence are preprocessed to obtain a preprocessed video frame sequence and a preprocessed audio frame sequence, including: S102.1. The video frame sequence is subjected to frame extraction processing to obtain target video frames, and multiple target video frames constitute a preprocessed video frame sequence. The target video frame can be obtained by extracting one video frame every other frame. For example, assuming the size of the video frame is 640 pixels × 480 pixels, the video frame sequence can be processed by extracting frames, keeping the frame rate after extraction at 12.5 frames per second, and outputting an RGB color image with a size of 640 pixels × 480 pixels to obtain the target video frame.

[0033] S102.2 Perform a windowed short-time Fourier transform on the audio frame sequence to generate a log-Mel spectrum, obtain the target audio spectrum segment, and multiple target audio spectrum segments constitute the preprocessed audio frame sequence.

[0034] It can perform windowed short-time fourier transform (STFT) on audio frame sequences, using a Hanning window function with a window length of 1024 sampling points, a window shift of 512 sampling points, and an overlap rate of 50%. After transformation, it generates a log-Mel spectrogram with a size of 128 pixels × 128 pixels, covering a frequency range of 0 to 24 kHz.

[0035] S103. Based on the timestamp, pair the preprocessed video frame sequence with the preprocessed audio frame sequence to obtain bimodal image sample data. In some possible implementations, the preprocessed video frame sequence and the preprocessed audio frame sequence are paired according to the timestamp to obtain bimodal image sample data, including: For any target video frame in the preprocessed video frame sequence, the time difference between the target video frame and the target audio spectrum segment in the preprocessed audio frame sequence is obtained based on the timestamp. When the time difference is less than a preset time threshold, the corresponding target video frame and the target audio spectrum segment are constructed into bimodal image sample data.

[0036] Assume the timestamp of video frame i is Assume the timestamp of the target audio spectrum segment j is , then when | - |< When this happens, the corresponding target video frame and the target audio spectrum segment are used to construct bimodal image sample data. Here, v represents the visual modality, and a represents the auditory modality. This indicates a preset time threshold. For example, The timeout can be set to 10 milliseconds, derived from the minimum value of the typical vibration period of heating station equipment, to ensure it is less than one-fifth of the equipment's vibration period and avoid alignment errors introduced by motion blur. It's worth noting that after the short-time Fourier transform, each target audio spectrum segment corresponds to a windowed segment in the original time-domain signal. This segment occupies an interval on the time axis, rather than a single point in time. Therefore, the timestamp of the spectrum segment is usually taken as the midpoint (or left or right endpoint) of this time interval to establish a correspondence with the timestamp of the video frame.

[0037] In some possible implementations, after pairing the preprocessed video frame sequence with the preprocessed audio frame sequence, the method further includes: Unpaired target video frames and target audio spectrum segments are used as asynchronous data, and corresponding modal analysis is performed on the asynchronous data to output independent modal detection results. The independent modal detection results are used as auxiliary environmental monitoring data.

[0038] For asynchronous data, it is sent to a single-modal backup branch for processing and is not used as input for fusion decision, thus ensuring the time alignment quality of fusion.

[0039] S104. Obtain the confidence level of the bimodal image samples and obtain the features of the bimodal image samples based on the bimodal image sample data, and perform cross-modal consistency verification based on the features of the bimodal image samples to obtain the consistency verification result. In some possible implementations, obtaining the confidence level of bimodal image samples and obtaining bimodal image sample features based on the bimodal image sample data includes: S104.1 Obtain the visual modality confidence score based on the video frame sequence and the auditory modality confidence score based on the audio frame sequence, and use the auditory modality confidence score and the visual modality confidence score together as the dual-modality image sample confidence score; Optionally, the visual modality confidence score is obtained based on the video frame sequence as follows: ; in, Indicates the confidence level of the visual modality. Represents the first visual modality coefficient. Represents the second visual modality coefficients. Represents the third visual modality coefficient. The coefficients representing the fourth visual modality can be set to the following reference values ​​in one optional implementation: , , as well as These can be set to 0.95, 0.12, 0.08, and 0.15 respectively. Represents the natural constant. The effective illuminance is expressed in lux and is collected from an illuminance sensor deployed at the location of the image acquisition equipment in the heating station. Those skilled in the art can make adaptive adjustments to the above coefficients according to the actual environment of the heating station (such as illuminance range and steam concentration), without departing from the protection scope of this invention. It represents the global motion vector magnitude of the image, with the unit being pixels per frame. The average optical flow modulus between two consecutive frames is calculated using the Farneback optical flow method, which characterizes the degree of image blurring caused by equipment vibration or steam flow. The value represents the steam obstruction rate, dimensionless (ranging from 0 to 1). The lightweight semantic segmentation model BiSeNet V2 is used to classify the image pixel-by-pixel, identifying steam or water mist categories, and calculating the proportion of pixels in that category out of the total pixels. It's worth noting that the above data can also be dimensionless, for example... It can be set to To perform dimensionless processing on the effective illuminance.

[0040] The confidence level of auditory modality can be obtained from the audio frame sequence as follows: ; in, Indicates auditory modality confidence. Indicates the confidence level of the first auditory modality. This indicates the confidence level of the second auditory modality. Indicates the confidence level of the third auditory modality. , as well as It can also be set to 0.92, 3.0, and 0.35 respectively; It represents the audio signal-to-noise ratio, with the unit being decibels (dB). It can use the minimum statistics tracking algorithm (Martin algorithm) to estimate the ambient noise power spectrum in real time, calculate the ratio of the target signal power to the noise power, and the output range is 0 to 30 dB. The ambient reverberation time (SNR) is expressed in seconds (s) and can be estimated from the audio signal using the Schroeder inverse integration method. It represents the time required for the sound pressure level to drop by 60 dB after the sound source stops emitting sound; a typical value for heating stations is 0.5 to 1.2 seconds. It is worth noting that the above parameters can also be dimensionless. When the SNR approaches 0 (signal much weaker than noise), the audio signal is completely unreliable, and its reliability should approach 0; when the SNR approaches infinity (signal much stronger than noise), the audio signal is completely reliable, and its reliability should approach 1. The signal-to-noise ratio can be mapped to a fixed range to determine auditory confidence.

[0041] Once the calculation functions for visual modality confidence and auditory modality confidence are solidified, the confidence level will be higher for more reliable data and lower for less reliable data, thus providing prior experience for deep learning recognition.

[0042] S104.2 Input the target video frame in the dual-modal image sample data into the visual deep learning model for analysis, extract the visual feature vector and the probability distribution of visual event categories, and obtain the visual modal image sample features. Optionally, target video frames from the dual-modal image sample data can be input into a visual deep learning model for analysis, outputting a 512-dimensional visual feature vector. Probability distribution of visual event categories , Indicates the category of visual events, and ∈{Leakage, Bearing Wear, Fire, Intrusion, Instrument Abnormality, Normal}. The visual deep learning model can be set as a YOLO model. For example, using YOLOv8s as the backbone network, extracting the output feature map of layer P5 of its backbone network CSP-Darknet, and performing Global Average Pooling (GAP) to obtain a 512-dimensional visual feature vector. The classification header outputs the Softmax probability distributions for the six categories.

[0043] S104.3. Input the target audio spectrum segment in the dual-modal image sample data into the auditory deep learning model for analysis, extract auditory feature vectors and auditory event category probability distributions, and obtain auditory modal image sample features; Optionally, the target audio spectrum segment from the bimodal image sample data can be input into the auditory deep learning model for analysis, outputting a 256-dimensional auditory feature vector. Probability distribution of auditory event categories For example, a 128-pixel × 128-pixel log-Mel spectrogram can be input, passed through 6 convolutional layers, and then through 2 fully connected layers (the first layer flattens the convolutional output and maps it to 256 dimensions, and the output is an auditory feature vector). The second layer maps to 6 dimensions, and finally the probability distribution of the 6 categories is output through the Softmax activation function.

[0044] S104.4 The visual modality image sample features and the auditory modality image sample features are used together as bimodal image sample features.

[0045] In some possible implementations, cross-modal consistency verification is performed based on the features of the bimodal image samples to obtain a consistency verification result, including: S104.5 Obtain the KL (Kullback-Leibler) divergence values ​​of the probability distribution of auditory event categories and the probability distribution of visual event categories in the bimodal image sample features; For example, the KL divergence values ​​of the probability distributions of auditory event categories and the visual event category categories can be obtained as follows: ; in, Represents the KL divergence value. The function representing the KL divergence value is logarithm with base e.

[0046] S104.6 Determine whether the KL divergence value is less than or equal to the preset divergence threshold. If so, determine that the consistency check result is consistent; otherwise, determine that the consistency check result is inconsistent.

[0047] Assuming the divergence threshold is set to Then, when the KL divergence value is less than or equal to the preset divergence threshold... If the divergence threshold is met, then consistency is determined. The KL divergence value of each sample pair can be calculated by using sample pairs labeled as modally consistent in the training set (assuming a total of 3000 sets of synchronized frame-spectrum pairs, labeled as consistent with visual and auditory categories). The 95th percentile is taken as the threshold, and after confirmation by 10-fold cross-validation, the divergence threshold can be determined. (like =10).

[0048] S105. If the consistency verification result is inconsistent, the environmental monitoring result of the heating station is obtained based on the dual-modal image sample features. In some possible implementations, if the consistency check result is inconsistent, the environmental monitoring results of the heating station are obtained based on the features of the dual-modal image samples, including: S105.1 If the consistency check result is inconsistent, then visual monomodal classification is performed using the visual modal image sample features in the bimodal image sample features to obtain the visual modal event probability distribution. S105.2. Perform auditory monomodal classification using the auditory modality image sample features in the bimodal image sample features to obtain the probability distribution of auditory modality events; S105.3 Determine the visual monomodal warning type and corresponding confidence level based on the probability distribution of the visual modal events, and determine the auditory monomodal warning type and corresponding confidence level based on the probability of the auditory modal events; For example, if the probability distribution of visual modal events includes the probabilities of six categories: leakage, bearing wear, fire, intrusion, instrument malfunction, and normal, then the category with the highest probability can be taken as the visual monomodal warning type, and the corresponding probability can be taken as the confidence level. The principle for obtaining auditory monomodal warning types and corresponding confidence levels is similar, and will not be elaborated here.

[0049] S105.4 The visual single-modal warning type and its corresponding confidence level, and the auditory single-modal warning type and its corresponding confidence level are used together as the environmental monitoring results of the heating station.

[0050] Optionally, based on the environmental monitoring results of the heating station, if the confidence level of each single mode is higher than 0.7, a single-mode early warning will be output; if the confidence level of each dual mode is lower than 0.3, a sensor self-test command will be triggered.

[0051] S106. If the consistency verification result is consistent, the features of the dual-modal image samples are weighted and fused according to the confidence level of the dual-modal image samples to obtain joint data features, and the joint data features are identified by a pre-set fusion classifier to obtain the environmental monitoring results of the heating station. In some possible implementations, the features of the bimodal image samples are weighted and fused based on the confidence level of the bimodal image samples to obtain joint data features, and a pre-set fusion classifier is used to identify the joint data features to obtain the environmental monitoring results of the heating station, including: S106.1 Obtain the visual weighting weight and auditory weighting weight respectively based on the confidence scores of the bimodal image samples; Optionally, the visual weighting and auditory weighting can be obtained based on the confidence levels of the bimodal image samples, respectively: ; ; in, Indicates visual weighting. Indicates auditory weighting. This represents an exponential function with the natural constant e as its base.

[0052] S106.2. The bimodal image sample features are weighted according to the visual weighting weight and the auditory weighting weight, and the weighted bimodal image sample features are concatenated into a vector to obtain joint data features; Optionally, joint data features can be represented as: ; in, Represent joint data features; Assumptions It is a 512-dimensional visual feature vector. Given a 256-dimensional auditory feature vector, then It should then be a 768-dimensional joint feature vector.

[0053] S106.3. The joint data features are identified using a pre-set fusion classifier to obtain the joint event probability distribution; Joint feature vectors can be Input the fusion classifier and output the final classification result. The fusion classifier consists of a two-layer fully connected network: the first layer maps the 768-dimensional input to 256 dimensions, the second layer maps the 256-dimensional input to 6 dimensions, and the last layer is a softmax activation function. Optionally, the fusion classifier can also be set to some more conventional classifiers, such as a backpropagation (BP) neural network classifier.

[0054] S106.4. Determine the joint event probability distribution as the joint modal early warning type and corresponding confidence level to obtain the environmental monitoring results of the heating station.

[0055] S107. Issue an early warning based on the environmental monitoring results of the heating station, and complete the environmental monitoring and early warning of the heating station based on dual-modal image sample training.

[0056] In some possible implementations, early warning is issued based on the environmental monitoring results of the heating station, including: S107.1 When the environmental monitoring results of the heating station are a joint modal warning type and its corresponding confidence level, and the joint modal warning type is an abnormal type, a graded warning shall be issued according to the confidence level of the joint modal warning type; S107.2. When the environmental monitoring results of the heating station are a visual single-modal warning type and its corresponding confidence level, an auditory single-modal warning type and its corresponding confidence level, and both the visual single-modal warning type and the auditory single-modal warning type are abnormal types, a graded warning shall be issued based on the confidence level of the visual single-modal warning type and the corresponding confidence level of the auditory single-modal warning type.

[0057] The environmental monitoring results of the heating station include joint modal identification results and / or single modal identification results, which can be used to determine whether there are any abnormalities. That is, anything other than the normal category is considered abnormal, and this can be used as a basis for early warning.

[0058] For example, assuming a three-level warning system is set, it can be as follows: Level 1 warning: The trigger condition is max( )≥0.9, or simultaneously triggering both visual and auditory independent warnings with confidence levels ≥0.8. Applicable event types: fire, serious leak, personnel intrusion. Response measures: Immediately push to the management platform, and simultaneously notify management personnel via SMS and application push to trigger linked responses (such as closing valves, starting the ventilation system). Wherein, max( ) indicates the confidence level of the joint modal warning type.

[0059] Level 2 warning: The trigger condition is 0.7 ≤ max( <0.9, or only meets the single-modal independent early warning requirement with a confidence level ≥0.8. Applicable event types: abnormal instrument readings, minor leaks. Response measures: push to the management platform, record logs, and generate a maintenance work order.

[0060] Level 3 Warning: Triggered by modal conflict and both modal confidence scores < 0.5. Applicable event type: Sensor degradation. Response measure: Trigger a sensor self-test command and push a notification to check the image acquisition device or sound acquisition device. It is worth noting that max( When the value is less than 0.7, it can usually be considered a normal situation, and no warning needs to be issued, or a text message reminder can be set.

[0061] Aside from the three situations mentioned above that require alerts, no alerts will be triggered in any other situation. It's worth noting that other types of alerts can also be set based on the joint modality recognition results and / or the single modality recognition results. For example, different types of alerts can be triggered based on the anomaly category confidence levels in the joint modality recognition results and / or the anomaly category confidence levels in the single modality recognition results.

[0062] In some possible implementations, it also includes: Based on the fact that the joint modal warning type is an anomaly type, the fault causal chain tracing process is triggered; When the fault causal chain tracing process is triggered, gradient backpropagation and upsampling are performed according to the joint event probability distribution to obtain visual heatmaps and auditory heatmaps. Visual spatial mapping is performed based on the visual heatmap, and auditory spatial mapping is performed based on the auditory heatmap to obtain bimodal anomaly location information; Causal consistency verification is performed based on the bimodal anomaly location information to obtain the causal consistency verification result, and a diagnostic report is output based on the bimodal anomaly location information and the causal consistency verification result.

[0063] For example, obtaining a visual heatmap by performing gradient backpropagation and upsampling based on the joint event probability distribution may include: Obtain the output logit value of the joint modality warning type, which is the input value before the Softmax activation function.

[0064] Set a baseline joint feature vector with the same dimensions as the joint data features. Baseline joint eigenvectors The parameter for each dimension is 0; From baseline joint feature vector If m interpolation points (e.g., 50) are uniformly sampled along the path to the joint data features, then the first interpolation point can be obtained. k The feature vectors of the interpolation points are: ; in, Indicates the first k The feature vector of each interpolation point k =1,2,...,m; Obtain the output logit value against the feature vector. The gradient is: ; in, Indicates the type of joint modal early warning The output logit value, Representing the eigenvector The gradient.

[0065] According to the feature vector The gradient attribution vector obtained from the integral gradient is: ; in, Represents element-wise product. This represents the approximate gradient mean along the integration path.

[0066] The integral gradient attribution vector is split according to the dimensions of the visual feature vector and the auditory feature vector. Assuming... It is a 512-dimensional visual feature vector. Given a 256-dimensional auditory feature vector, then Therefore, it should be a 768-dimensional joint eigenvector. The first 512 elements are used as the visual attribution vector, and the remaining 256 elements are used as the auditory attribution vector.

[0067] Assume the output feature map size of layer P5 in the backbone network CSP-Darknet of the YOLOv8 visual deep neural network is 20×20×512. Let this feature map... After global average pooling, a 512-dimensional visual feature vector is obtained. ; ; in, Represents the visual feature vector at the th c Elements on each channel Indicates height index, Indicates the width index. Indicates altitude, Indicates width; Then the visual attribution vector can be reconstructed from the feature map. The channel-dimensional weight vector for matching is: ; in, Represents the weight vector. Let c be the c-th element of the visual attribution vector.

[0068] Using bilinear sampling method Upsampling to the original input image size yields the following visual heatmap:

[0069] in, This represents a visual heatmap, where each element represents the pixel pair at the corresponding location and the joint modality warning type. Contribution Indicates bilinear sampling. Indicates the height of the original input image. This represents the width of the original input image. For example, if the original input image size is 640×480, then... It is 640. It is 480.

[0070] Assume the feature map of the last layer of the auditory deep neural network (output of the 6th convolutional layer) has a size of 16×16×256 (after 6 layers of convolutional downsampling with a stride of 2, the input 128×128 becomes 128 / 2 after 6 times of 2x downsampling). 6 =2, but it actually needs to be adjusted to 16×16, or a convolution with a stride of 1 can be used to preserve the size. Let the feature map of this last layer be... The dimensions are R16×16×256, after flattening ( After the operation, the input is fed into the fully connected layer, resulting in a 256-dimensional auditory feature vector: ; in, Indicates the weights of the fully connected layer. Indicates the bias of the fully connected layer. This indicates a flattening operation.

[0071] The auditory attribution vector is backpropagated to the flattened feature map space via the transpose of the weight matrix as follows: ; in, This represents the auditory attribution vector after backpropagation. Let T denote the auditory attribution vector, and T denote the transpose. This step uses a linear approximation method to propagate the auditory attribution vector to the feature map space. Theoretically, accurate propagation of the attribution value requires recalculating the integral gradient at the feature map level. However, considering that the fully connected layers in the mapping from the feature map to the final score are linear layers, and the nonlinearity of Softmax is mainly achieved through exponential normalization, its monotonicity is well maintained. Therefore, using a linear approximation has acceptable accuracy in engineering practice. Those skilled in the art will understand that this approximation is sufficient and effective for the interpretability requirements of anomaly detection in heating stations.

[0072] The auditory attribution vector after backpropagation can then be reconstructed into three-dimensional space as follows: ; Here, Reshape represents the reshaping operation. This represents the features after reshaping. The Reshape operation reshapes the 65536-dimensional one-dimensional vector Attributionflat into a three-dimensional tensor with a shape of 16×16×256.

[0073] Will Summing along the channel dimensions yields the following two-dimensional spatial attribution map: ; in, Represents a two-dimensional spatial attribution diagram; Finally, bilinear interpolation was used to... The spectrum graph from 16×16 upsampled to the original input is as follows:

[0074] in, Represents an auditory heatmap. Indicates the height of the original input spectrogram. This represents the width of the original input spectrogram; assuming the size of the original input spectrogram is 128×128, then... It is 128. It is 128; Based on the above heatmap scheme, the information on bimodal anomaly locations can be obtained as follows: The pixel region with the highest response value (top 10 percentile) in the visual heat map is projected onto the three-dimensional spatial coordinates of the heating station. , , The projection uses the camera intrinsic and extrinsic parameter matrices pre-calibrated using the Zhang Zhengyou calibration method, and is calculated through inverse perspective projection transformation. During the calibration process, a checkerboard calibration board (30 mm side length) is used to acquire 20 images in different poses, and the calibration reprojection error is controlled within 0.5 pixels.

[0075] The highest-response time-frequency region in the auditory spectrum is mapped to the spatial azimuth and distance of the sound source. Sound source localization employs the Time Difference of Arrival (TDOA) method, calculating the time delay difference based on the Generalized Cross-Correlation-Phase Transform (GCC-PHAT) algorithm of the microphone array, and solving for the sound source coordinates using the hyperbolic intersection method. The microphone array can consist of four microelectromechanical system (MEMS) digital microphones arranged in a diamond shape, with an adjacent microphone spacing of 0.15 meters. The maximum time difference of the array is approximately 440 microseconds. In a heating station environment (reverberation time R ≤ 1.2 seconds), the theoretical azimuth resolution is approximately 5 degrees, and the localization error does not exceed 0.3 meters.

[0076] Therefore, causal consistency verification can be performed based on the bimodal anomaly location information, and the causal consistency verification result is as follows: ; in, If the physical deviation is less than a preset distance threshold, the causal consistency verification result is determined to be consistent; otherwise, it is determined to be inconsistent. , , () represents the three-dimensional spatial coordinates of the heating station. Distance from the sound source The azimuth angle of the sound source. Indicates the installation height of the microphone array.

[0077] It is worth noting that the diagnostic report is an enhanced solution of the embodiments of this application. Even if this part of the technical solution is not used, it will still not affect the environmental monitoring and early warning of heating stations based on dual-modal image sample training.

[0078] The beneficial effects of the technical solution described in this application are as follows: This application's embodiments introduce physical parameters such as effective illuminance, motion ambiguity, steam obstruction rate, signal-to-noise ratio, and reverberation time to construct a modal reliability function specifically for heating station environments, solving the problem of performance degradation in fusion under high humidity and high noise conditions in existing technologies. In the field of heating monitoring, KL divergence is introduced as a quantitative indicator of modal consistency, automatically separating visual and auditory judgments to avoid false alarms caused by forced fusion. This mechanism also possesses active learning capabilities, with conflicting samples feeding back into model optimization. Through gradient backpropagation and spatial coordinate mapping, the black-box model output is transformed into a visualized heat map and physical spatial location, allowing maintenance personnel to intuitively understand the model's judgment basis and improving system reliability.

[0079] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that the invention is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for environmental monitoring and early warning of heating stations based on dual-modal image sample training, characterized in that, include: S101. Acquire video frame sequences and audio frame sequences with timestamps by at least one image acquisition device and at least one sound acquisition device in the monitoring area within the heating station. S102. Preprocess the video frame sequence and the audio frame sequence to obtain a preprocessed video frame sequence and a preprocessed audio frame sequence. S103. Based on the timestamp, pair the preprocessed video frame sequence with the preprocessed audio frame sequence to obtain bimodal image sample data. S104. Obtain the confidence level of the bimodal image samples and obtain the features of the bimodal image samples based on the bimodal image sample data, and perform cross-modal consistency verification based on the features of the bimodal image samples to obtain the consistency verification result. S105. If the consistency verification result is inconsistent, the environmental monitoring result of the heating station is obtained based on the dual-modal image sample features. S106. If the consistency verification result is consistent, the features of the dual-modal image samples are weighted and fused according to the confidence level of the dual-modal image samples to obtain joint data features, and the joint data features are identified by a pre-set fusion classifier to obtain the environmental monitoring results of the heating station. S107. Issue an early warning based on the environmental monitoring results of the heating station, and complete the environmental monitoring and early warning of the heating station based on dual-modal image sample training; Obtaining the confidence level of bimodal image samples and obtaining bimodal image sample features based on the bimodal image sample data includes: The visual modality confidence score is obtained from the video frame sequence, and the auditory modality confidence score is obtained from the audio frame sequence. The auditory modality confidence score and the visual modality confidence score are used together as the bimodal image sample confidence score. The target video frames in the bimodal image sample data are input into a visual deep learning model for analysis, and visual feature vectors and visual event category probability distributions are extracted to obtain visual modal image sample features. The target audio spectrum segment in the bimodal image sample data is input into the auditory deep learning model for analysis, and auditory feature vectors and auditory event category probability distributions are extracted to obtain auditory modal image sample features. The visual modality image sample features and the auditory modality image sample features are used together as bimodal image sample features; The visual modality confidence level is obtained based on the video frame sequence as follows: ； in, Indicates the confidence level of the visual modality. Represents the first visual modality coefficient. Represents the second visual modality coefficients. Represents the third visual modality coefficient. The coefficients for the fourth visual modality are represented by the following reference values: , , as well as The values ​​were set to 0.95, 0.12, 0.08, and 0.15 respectively. Represents the natural constant. Indicates effective illuminance. Represents the magnitude of the global motion vector of the image. Indicates the vapor blocking rate; The auditory modality confidence score is obtained from the audio frame sequence as follows: ； in, Indicates auditory modality confidence. Represents the first auditory modality coefficient. Indicates the second auditory modality coefficients. Represents the third auditory modality coefficient. , as well as The values ​​were set to 0.92, 3.0, and 0.35 respectively. Indicates the audio signal-to-noise ratio; Indicates the ambient reverberation time; Cross-modal consistency verification is performed based on the features of the bimodal image samples to obtain the consistency verification result, including: Obtain the KL divergence values ​​of the auditory event category probability distribution and the visual event category probability distribution in the bimodal image sample features; Determine whether the KL divergence value is less than or equal to a preset divergence threshold. If it is, determine that the consistency check result is consistent; otherwise, determine that the consistency check result is inconsistent.

2. The heating station environmental monitoring and early warning method based on dual-modal image sample training according to claim 1, characterized in that, Preprocessing the video frame sequence and the audio frame sequence to obtain preprocessed video frame sequence and preprocessed audio frame sequence includes: The video frame sequence is subjected to frame extraction processing to obtain target video frames, and multiple target video frames constitute a preprocessed video frame sequence. The audio frame sequence is subjected to a windowed short-time Fourier transform to generate a log-Mel spectrogram, which yields the target audio spectrum segments. Multiple target audio spectrum segments constitute the preprocessed audio frame sequence.

3. The heating station environmental monitoring and early warning method based on dual-modal image sample training according to claim 1, characterized in that, Based on the timestamp, the preprocessed video frame sequence and the preprocessed audio frame sequence are paired to obtain bimodal image sample data, including: For any target video frame in the preprocessed video frame sequence, the time difference between the target video frame and the target audio spectrum segment in the preprocessed audio frame sequence is obtained based on the timestamp. When the time difference is less than a preset time threshold, the corresponding target video frame and the target audio spectrum segment are constructed into bimodal image sample data.

4. The heating station environmental monitoring and early warning method based on dual-modal image sample training according to claim 3, characterized in that, After pairing the preprocessed video frame sequence with the preprocessed audio frame sequence, the process further includes: Unpaired target video frames and target audio spectrum segments are used as asynchronous data, and corresponding modal analysis is performed on the asynchronous data to output independent modal detection results. The independent modal detection results are used as auxiliary environmental monitoring data.

5. The heating station environmental monitoring and early warning method based on dual-modal image sample training according to claim 1, characterized in that, If the consistency verification result is inconsistent, the environmental monitoring result of the heating station is obtained based on the features of the dual-modal image samples, including: If the consistency check result is inconsistent, then visual monomodal classification is performed using the visual modality image sample features in the bimodal image sample features to obtain the visual modality event probability distribution. Auditory monomodal classification is performed using the auditory modality image sample features from the bimodal image sample features to obtain the probability distribution of auditory modality events; The visual monomodal warning type and corresponding confidence level are determined based on the probability distribution of the visual modal events, and the auditory monomodal warning type and corresponding confidence level are determined based on the probability distribution of the auditory modal events. The visual monomodal warning type and its corresponding confidence level, and the auditory monomodal warning type and its corresponding confidence level are used together as the environmental monitoring results of the heating station.

6. The heating station environmental monitoring and early warning method based on dual-modal image sample training according to claim 1, characterized in that, The features of the bimodal image samples are weighted and fused based on the confidence scores of the bimodal image samples to obtain joint data features. A pre-set fusion classifier is then used to identify these joint data features to obtain environmental monitoring results for the heating station, including: The visual weighting and auditory weighting are obtained based on the confidence scores of the bimodal image samples, respectively. The bimodal image sample features are weighted according to the visual weighting weight and the auditory weighting weight, and the weighted bimodal image sample features are concatenated into a vector to obtain joint data features; The joint data features are identified using a pre-set fusion classifier to obtain the joint event probability distribution; The probability distribution of the joint events is determined as the joint modal early warning type and the corresponding confidence level, and the environmental monitoring results of the heating station are obtained.

7. The heating station environmental monitoring and early warning method based on dual-modal image sample training according to claim 1, characterized in that, Early warnings are issued based on the environmental monitoring results of the heating station, including: If the environmental monitoring results of the heating station are a joint modal warning type and its corresponding confidence level, and the joint modal warning type is an abnormal type, a graded warning shall be issued based on the confidence level of the joint modal warning type. If the environmental monitoring results of the heating station are a visual single-modal warning type and its corresponding confidence level, an auditory single-modal warning type and its corresponding confidence level, and both the visual single-modal warning type and the auditory single-modal warning type are abnormal types, a graded warning is issued based on the confidence level of the visual single-modal warning type and the corresponding confidence level of the auditory single-modal warning type.

8. The heating station environmental monitoring and early warning method based on dual-modal image sample training according to claim 7, characterized in that, Also includes: Based on the joint modal warning type being an anomaly type, the fault causal chain tracing process is triggered; When the fault causal chain tracing process is triggered, gradient backpropagation and upsampling are performed based on the joint event probability distribution to obtain visual heatmaps and auditory heatmaps. Visual spatial mapping is performed based on the visual heatmap, and auditory spatial mapping is performed based on the auditory heatmap to obtain bimodal anomaly location information; Causal consistency verification is performed based on the bimodal anomaly location information to obtain the causal consistency verification result, and a diagnostic report is output based on the bimodal anomaly location information and the causal consistency verification result.

Images