Power plant area inspection method and inspection system based on multi-modal perception fusion

Abstract

The application provides a power plant area inspection method and inspection system based on multi-modal perception fusion, comprising: acquiring infrared images, electric signals, vibrations and environmental parameters by using multi-modal perception equipment, performing space-time labeling and signal denoising preprocessing; mining cross-modal joint features through a deep learning encoder, modeling multi-modal causal relationships by using mutual information and time sequence gradient sensitivity, dynamically generating an optimizable causal attention weight matrix to drive the generation of abnormal classification and root cause contribution integral graph; combining with a device mechanism model library to realize semantic matching of abnormal root causes and output of a structured diagnosis report, and supporting continuous adaptive optimization of the model based on actual feedback, which improves the accuracy and explainability of power plant equipment abnormality recognition, and realizes efficient intelligent diagnosis for complex coupled scenes.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection and multimodal anomaly analysis technology for power systems, and particularly to a power plant area inspection method and system based on multimodal perception fusion. Background Technology

[0002] In the current field of intelligent operation and maintenance and power plant inspection systems in the power industry, with the continuous development of digitalization and artificial intelligence technologies, automated anomaly identification solutions based on multimodal perception and intelligent analysis have been widely adopted. Mainstream technologies utilize the deployment of multiple types of sensors (such as infrared thermal imaging, partial discharge monitoring, vibration and environmental parameter acquisition) to achieve real-time monitoring and data acquisition of the operating status of key equipment. Existing multimodal fusion methods mostly rely on deep learning networks, such as convolutional neural networks (CNN) and temporal convolutional networks (TCN), to improve the accuracy of equipment fault detection and adaptability to multiple scenarios through joint encoding of different data sources. Simultaneously, to enhance the interpretability of the system, some research and products integrate knowledge graphs, expert rules, or causal relationship modeling techniques to generate anomaly attribution and auxiliary diagnostic conclusions. Despite continuous technological advancements, existing solutions still generally suffer from the following significant problems: (1) Most mainstream anomaly identification systems adopt static knowledge graphs or causal networks with pre-defined structures. Their attribution analysis capabilities are limited by prior rule bases, making it difficult to dynamically cope with complex changes in equipment operating conditions and nonlinear coupling between multimodal data. (2) In actual fault diagnosis scenarios, existing methods can only provide the anomaly category and appearance judgment, lacking the interpretability to trace the root physical cause and key impact path of the anomaly, making it difficult for maintenance personnel to accurately locate the root cause of the fault and formulate efficient handling plans. (3) Most system attribution analysis only supports forward reasoning path and does not have the ability to trace back from the output abnormal results to the input source data, resulting in insufficient transparency of diagnostic results and low trust in AI decision-making. (4) Due to the lack of closed-loop learning and adaptive optimization mechanism of causal structure, the existing model has difficulty absorbing and utilizing the latest operation and maintenance empirical experience, thus restricting the long-term robustness and generalization performance of the model in highly coupled and nonlinear scenarios. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, the present invention provides a power plant area inspection method and inspection system based on multimodal perception fusion.

[0004] The technical solution of this invention is implemented as follows: a power plant area inspection method based on multimodal perception fusion, comprising: S1: Acquire environmental data collected by the multimodal sensing device in the power plant inspection system. The environmental data includes infrared thermal imaging images, partial discharge electrical signals, vibration acceleration sequences, and environmental temperature and humidity parameters. Record the equipment component labels and sampling timestamps corresponding to each data channel. S2: Denoising and time-aligned preprocessing is performed on the multimodal environmental data. Linear interpolation and sliding window mean filtering are performed based on the timestamp sequences of each sensor to generate a time-synchronized multi-channel signal sequence, which serves as the input data for subsequent feature extraction. S3: Input the time-synchronized multi-channel signal sequence into the deep feature encoder for nonlinear mapping, extract the high-dimensional latent space representation, and generate a multimodal joint feature vector, where each dimension corresponds to the abstract representation of different physical modes in the compressed latent space. S4: Based on the multimodal joint feature vector, the dynamic influence intensity between each feature subspace is jointly calculated by mutual information estimation and temporal gradient sensitivity analysis, and an evolvable causal attention weight matrix is ​​constructed. This matrix represents the asymmetric causal relationship structure between multimodal variables as the working conditions change. S5: Embed the causal attention weight matrix into the forward classification network. When an abnormal event is detected, use the abnormal classification score as the backpropagation target, execute the gradient weighted class activation mapping extension algorithm, backtrack along the network layers and accumulate the sensitive region responses of the feature maps of each layer in a weighted manner to generate the root cause contribution integral map. S6: Based on the root cause contribution integral map, identify the modal type and time segment that contributes the most to the anomaly discrimination result in the original input channel, and form a key influence path sequence. This sequence contains the sensing modality, time window and spatial location information that dominates the anomaly performance. S7: Perform semantic similarity matching between the key impact path sequence and the typical fault modes pre-stored in the equipment mechanism model library, output the most likely root cause type based on the maximum matching degree, and generate a structured diagnostic report containing anomaly category, confidence level, impact path and recommended investigation items. S8: Based on the actual handling results reported by maintenance personnel, update the learning parameters of the causal attention weight matrix, adjust the modeling accuracy of the dynamic causal structure through online gradient optimization, and realize the adaptive evolution capability of the anomaly recognition model in complex coupled scenarios.

[0005] The present invention also provides a power plant area inspection system based on multimodal perception fusion, which uses the above-mentioned power plant area inspection method based on multimodal perception fusion to inspect the power plant area.

[0006] The power plant area inspection method and system based on multimodal perception fusion provided by this invention have the following beneficial effects: (1) This invention, by constructing a dynamic causal attention module, abandons the traditional analysis paradigm that relies on static knowledge graphs or preset causal rules, and realizes time-varying modeling of complex coupling relationships between devices. This module takes multimodal joint features as input, combines mutual information estimation and temporal gradient sensitivity analysis, automatically learns the dynamic influence strength between each potential variable node, and generates a causal attention weight matrix that evolves with the operating conditions, effectively overcoming the shortcomings of fixed structure networks in terms of insufficient adaptability when facing nonlinear and highly coupled systems. Compared with traditional methods that can only provide isolated abnormal alarms or forward inference results, this mechanism can capture the instantaneous response relationship and propagation path changes between device states, significantly improving the accuracy of fault correlation analysis and scenario adaptability, especially maintaining stable causal inference under complex operating conditions such as load fluctuations and start-stop switching; (2) This invention introduces a root cause inversion scoring mechanism to construct a reverse tracing channel from the output anomaly to the original input. When an abnormal event is detected, the system responds with the classification score as the target, applies an extended gradient-weighted class activation mapping algorithm layer by layer along the feedforward network, and integrates dynamic causal attention weights for path backtracking, accurately calculating the contribution integral of each input channel in the time dimension, forming a quantifiable "influence transmission chain". This mechanism not only reveals which sensor signals or modal features dominate the current discrimination decision, but also performs semantic matching between high contribution segments and typical fault modes in the mechanism model library, automatically labeling the most likely root cause type and its propagation path, giving the AI ​​judgment a clear physical meaning and engineering explanation basis. Compared with the traditional black box model that only outputs confidence, this method greatly improves the readability and operability of the diagnostic results, enabling maintenance personnel to quickly understand the AI ​​logic and formulate targeted handling strategies, significantly shortening the fault response time; (3) The design of this invention has good online learning and continuous optimization capabilities, supports incremental updates of causal attention parameters, and dynamically adjusts and optimizes the learning effect of the causal structure by continuously absorbing real cases verified by humans, thus avoiding model failure due to operational drift. The entire mechanism does not require manual pre-construction of a complete causal rule system, reducing the deployment threshold and maintenance cost, and has strong adaptability and generalization performance. The output structured diagnostic report contains multi-dimensional information such as anomaly type, confidence level, key impact modalities, key time windows, and recommended investigation items, which can be flexibly integrated into edge computing nodes or central analysis platforms, improving the system's intelligence level while ensuring real-time performance. Attached Figure Description

[0007] Figure 1 The flowchart shows the power plant area inspection method based on multimodal perception fusion according to the present invention. Figure 2 This is a sub-flowchart of the power plant area inspection method based on multimodal perception fusion of the present invention; Figure 3 This is another sub-flowchart of the power plant area inspection method based on multimodal perception fusion of the present invention. Detailed Implementation

[0008] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0009] The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the invention. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0010] like Figure 1 As shown, this invention provides a power plant area inspection method based on multimodal perception fusion, specifically including: S1: Acquire environmental data collected by the multimodal sensing device in the power plant inspection system. The environmental data includes infrared thermal imaging images, partial discharge electrical signals, vibration acceleration sequences, and environmental temperature and humidity parameters. Record the equipment component labels and sampling timestamps corresponding to each data channel. S2: Denoising and time-aligned preprocessing is performed on the multimodal environmental data. Linear interpolation and sliding window mean filtering are performed based on the timestamp sequences of each sensor to generate a time-synchronized multi-channel signal sequence, which serves as the input data for subsequent feature extraction. S3: Input the time-synchronized multi-channel signal sequence into the deep feature encoder for nonlinear mapping, extract the high-dimensional latent space representation, and generate a multimodal joint feature vector, where each dimension corresponds to the abstract representation of different physical modes in the compressed latent space. S4: Based on the multimodal joint feature vector, the dynamic influence intensity between each feature subspace is jointly calculated by mutual information estimation and temporal gradient sensitivity analysis, and an evolvable causal attention weight matrix is ​​constructed. This matrix represents the asymmetric causal relationship structure between multimodal variables as the working conditions change. S5: Embed the causal attention weight matrix into the forward classification network. When an abnormal event is detected, use the abnormal classification score as the backpropagation target, execute the gradient weighted class activation mapping extension algorithm, backtrack along the network layers and accumulate the sensitive region responses of the feature maps of each layer in a weighted manner to generate the root cause contribution integral map. S6: Based on the root cause contribution integral map, identify the modal type and time segment that contributes the most to the anomaly discrimination result in the original input channel, and form a key influence path sequence. This sequence contains the sensing modality, time window and spatial location information that dominates the anomaly performance. S7: Perform semantic similarity matching between the key impact path sequence and the typical fault modes pre-stored in the equipment mechanism model library, output the most likely root cause type based on the maximum matching degree, and generate a structured diagnostic report containing anomaly category, confidence level, impact path and recommended investigation items. S8: Based on the actual handling results reported by maintenance personnel, update the learning parameters of the causal attention weight matrix, adjust the modeling accuracy of the dynamic causal structure through online gradient optimization, and realize the adaptive evolution capability of the anomaly recognition model in complex coupled scenarios.

[0011] Step S1: Acquire environmental data collected by the multimodal sensing device in the power plant inspection system. This environmental data includes infrared thermal imaging images, partial discharge electrical signals, vibration acceleration sequences, and environmental temperature and humidity parameters. Record the device component labels and sampling timestamps corresponding to each data channel. Specifically, this includes: S1.1: Based on the multimodal sensing device network deployed in the power plant area, four types of physical measurement data are acquired: infrared thermal imaging images, partial discharge electrical signals, vibration acceleration sequences, and environmental temperature and humidity parameters. Each type of data is continuously collected by the corresponding type of sensor at a preset sampling frequency to form the raw sensing signal stream. Based on the multimodal sensing equipment network deployed within the power plant area, infrared thermal imaging cameras, high-frequency partial discharge sensors, triaxial vibration accelerometers, and integrated temperature and humidity sensors are selected as environmental data acquisition nodes to establish a complete set of physical measurement channels. The device network access control protocol (parameters: node ID, communication frequency, bandwidth allocation) is adopted to enable each sensor to access the inspection data bus at high speed and stably. By adjusting the spectral response of the infrared thermal imaging camera (parameters: wavelength range 8-14μm, sampling resolution 640×480 pixels), the temperature radiation field covering the transformer and switch cabinet shell area is collected, forming a continuous infrared thermal image data stream. By using the pulse response mode of a high-frequency partial discharge sensor (parameters: sampling frequency 10MHz, signal-to-noise ratio > 60dB), transient discharge pulse waveforms near the terminal block and insulator are acquired, forming a high-time-resolution partial discharge electrical signal flow. Furthermore, by acquiring the vector components of a triaxial vibration accelerometer (parameters: sampling frequency 5kHz, dynamic range ±50g), the vibration acceleration time-series data at the bearing housing of the rotating machinery is captured, and a mechanical response signal stream is generated. Furthermore, by using dual-channel sampling (parameters: temperature resolution 0.1℃, humidity resolution 0.1%RH, sampling frequency 1Hz) of the integrated temperature and humidity sensor, the temperature and humidity scalar parameters of the air monitoring point in the power distribution room are recorded to form an environmental status data stream; Through the continuous sampling triggering mechanism of the data acquisition controller, the above four types of physical measurement data are sampled in real time at their respective preset sampling frequencies and cached in a local circular buffer to eliminate signal loss caused by short-term communication anomalies and achieve stable generation of the original sensor signal stream. Through a unified acquisition interface protocol, the data streams output by each sensor are encapsulated into raw data frames with modal identifiers, providing input basis for subsequent channel label binding. For example, in an inspection of the cooling system of a power plant unit, an infrared thermal imaging camera was set with a wavelength response of 8-14μm, a resolution of 640×480 pixels, and a sampling frequency of 25Hz, continuously covering the cooling water pump casing and outlet pipe; a high-frequency partial discharge sensor was set to a 10MHz high-frequency sampling mode to monitor the terminal block area of ​​the high-voltage switchgear; a triaxial vibration accelerometer was installed on the drive bearing housing of the cooling water pump, with a sampling frequency of 5kHz and a range of ±50g; a temperature and humidity sensor was fixed at the air duct inlet of the power distribution room, with a sampling frequency of 1Hz. Under the trigger condition of the data acquisition unit, four data streams were simultaneously sampled to the ring buffer. The infrared data stream showed a significant increase in the average temperature field of the outlet pipe outer wall, the partial discharge signal stream detected intermittent high-amplitude pulses, the vibration acceleration signal stream showed a low-frequency resonance peak, and the temperature and humidity parameters stabilized within the set range. This raw multimodal signal stream was buffered and output to the inspection analysis server, realizing high-fidelity, multi-source real-time data acquisition, providing a complete input basis for the channel tag binding of S1.2. S1.2: Perform channel tag binding operation on the acquired raw sensor signal streams, associate infrared thermal imaging images with the outer shell area of ​​the inspected power transformer, map partial discharge electrical signals to the terminal blocks of the high-voltage switchgear, assign vibration acceleration sequences to the rotating machinery bearing housing, and label temperature and humidity parameters to the air monitoring points in the power distribution room, generating a set of tagged data channels with spatial location semantics; For the raw sensor signal stream data generated by step S1.1, a channel label binding algorithm (parameters: sensor mode corresponding device component index table, spatial position mapping matrix) is used to achieve accurate association between each modal signal and its physical spatial position. Furthermore, by using a region segmentation and matching method based on the equipment layout diagram (parameters: infrared thermal image two-dimensional pixel coordinates, transformer shell geometric boundary), spatial binding between infrared thermal imaging image channel data and the inspected power transformer shell area is achieved, and the bound infrared image spatial identification data is obtained. Furthermore, by using a sensor installation location database query algorithm (parameter: partial discharge sensor number and high-voltage switchgear terminal block mapping table), the mapping between partial discharge electrical signal channels and high-voltage switchgear terminal block locations is realized, and a set of partial discharge signal channels with terminal block location identifiers is generated. Furthermore, by using the vibration feature indexing method for mechanical equipment components (parameters: vibration sensor installation point coordinates, rotating machinery bearing seat identification code), the vibration acceleration sequence is bound to the location of the rotating machinery bearing seat, forming a set of vibration signal channels with the semantic meaning of the mechanical component location; Furthermore, an environmental sensing point labeling method (parameters: spatial coordinates of temperature and humidity sensors, identifier of air monitoring point in power distribution room) is adopted to bind the temperature and humidity parameter channels with the labels of air monitoring points in power distribution room, thereby obtaining a set of temperature and humidity channels that integrate spatial location information; By binding channel labels, the results of the previous step are transformed into a set of labeled channels containing multimodal data of infrared thermal imaging, partial discharge, motor vibration, and temperature and humidity, so as to realize explicit semantic labeling of data in the spatial location dimension. For example, in a low-voltage substation scenario at a power plant, an infrared thermal imager with a resolution of 640×480 pixels, deployed on the side of the transformer, has its acquisition area matching the boundary of the transformer casing area with a threshold of 5 pixels. A partial discharge sensor, numbered PD-12, is bound to the high-voltage switchgear terminal block numbered SWG-T5 by querying the terminal block mapping table. A vibration acceleration sensor with a sampling frequency of 5kHz has its installation location coordinates matched with the bearing housing numbered BM-AX3 in the mechanical parts library. A temperature and humidity sensor, numbered TH-07, has its spatial coordinates matched with the distribution room location MC-R1. The mapping matrix uses a coordinate difference of less than 0.2 meters as a successful binding condition. After executing the above tag binding algorithm, the spatial identification data of the infrared channels includes transformer casing boundary information, the partial discharge channel is identified as the SWG-T5 terminal block, the vibration channel as the BM-AX3 bearing housing, and the temperature and humidity channel as the MC-R1 monitoring point, forming a four-modal data channel set with location tags for the entire plant area, which can be directly used for timestamp binding processing in step S1.3. S1.3: Based on the timestamp sequence recorded by the built-in clock module of each sensor, an absolute time stamp accurate to the millisecond level is added to each data sampling point to construct a spatiotemporally aligned tuple structure containing a time dimension index, ensuring that subsequent cross-modal analysis can be compared and fused under a unified time reference; Based on the timestamp sequence output by the high-precision clock module built into various sensors, an absolute time stamp generation algorithm with millisecond resolution is adopted (parameters: time source synchronization protocol is IEEE 1588 PTP, resolution is set to 1ms) to add a unique time identifier to each data sampling point; Furthermore, by using a clock signal consistency check method (parameter: maximum tolerance threshold of 2ms across devices), the alignment and correction of the timestamps of each sensor are achieved, and the corrected time series data structure is obtained. Furthermore, by using a time-series indexing construction algorithm (parameters: index dimensions include time, spatial location, and modality type), the correspondence between timestamps and channel labels is encoded into a multi-dimensional index, and a set of candidate data tuples containing the time dimension index is generated. Furthermore, by using a spatiotemporal binding mapping method (parameter: mapping rules are based on the device topology table and the perception modality binding table), candidate data tuples are transformed into spatiotemporally aligned tuple structures with physical location semantics, and a unified data interface is generated for subsequent cross-modal comparison. By using the above-mentioned millisecond-level timestamp appending and correction, indexing, and mapping processing methods, the original sampled data stream with inconsistent time distribution is transformed into a unified spatiotemporal data structure that can be aligned and analyzed across modalities, thereby achieving the expected technical effects of cross-sensor modal time reference consistency and subsequent fusion accuracy improvement. For example, the infrared thermal imaging camera, partial discharge sensor, vibration accelerometer, and temperature and humidity module deployed in the power plant inspection system output raw data containing built-in clock timestamps. The infrared image frame rate is 10Hz, the partial discharge sampling rate is 50kHz, the accelerometer sampling rate is 1kHz, and the temperature and humidity sampling period is 5s. The IEEE 1588PTP protocol is used to synchronize all sensors with the plant's master clock, setting millisecond-resolution timestamps and adding absolute time to each infrared frame, each group of discharge pulse waveforms, each acceleration sampling point, and each temperature and humidity measurement. Sampling points with a time difference exceeding 2ms are eliminated through cross-device clock consistency checks. An indexing algorithm binds the timestamp with the device label and mode type, forming a three-dimensional index <device component label, mode type, timestamp> tuple. For example, the infrared image frame of a transformer casing area at time... The sampling records of high-amplitude pulses of partial discharge signals, abrupt changes in accelerometer readings, and increases in ambient temperature at the same time point (ms) are bound into the same spatiotemporally aligned tuple for subsequent cross-modal fusion comparison. After this processing step, the output spatiotemporally unified data structure exhibits a significant reduction in event alignment differences in multimodal time matching tests, meeting the timing accuracy requirements for subsequent causal analysis and dynamic anomaly identification. S1.4: The tagged data channel set and its corresponding timestamp sequence are structurally encapsulated to generate an original multimodal data packet with a three-dimensional index feature of 'device component tag-sensor mode type-sampling timestamp', which serves as the input object for the next stage of noise reduction and time synchronization preprocessing; S1.5: Verify the integrity and consistency of data in each channel of the original multimodal data packet, detect whether there is packet loss, time misordering or missing tags. If an anomaly is found, trigger the resampling request mechanism and send a data supplementation instruction to the corresponding sensing device to ensure the spatiotemporal continuity and topological integrity of the input data.

[0012] Step S2: The multimodal environmental data undergoes denoising and time-alignment preprocessing. Linear interpolation and sliding window mean filtering are performed based on the timestamp sequences of each sensor to generate a time-synchronized multi-channel signal sequence, which serves as input data for subsequent feature extraction. Specifically, this includes: S2.1: Based on the raw environmental data collected by each sensor, including infrared thermal imaging image sequences, partial discharge electrical signal waveforms, vibration acceleration time series and temperature and humidity scalar parameters, obtain a data stream with device component labels and original sampling timestamps, which serves as the input object for noise reduction and alignment processing; S2.2: Perform modal-adaptive denoising processing on the original signal of each mode: For infrared images, a two-dimensional wavelet threshold denoising algorithm is used to suppress random noise in thermal imaging to obtain a denoised temperature distribution matrix; for partial discharge electrical signals, empirical mode decomposition (EMD) combined with kurtosis criteria is applied to screen intrinsic mode function components and extract effective discharge pulse components; for vibration acceleration sequences, bandpass filtering and singular value removal are performed to retain effective mechanical response signals within the equipment's resonant frequency band; for temperature and humidity parameters, sliding window mean filtering is used to eliminate transient jump interference, generating a clean signal stream after denoising for each mode; For the infrared thermal imaging image matrix, a two-dimensional wavelet threshold denoising algorithm (parameters: wavelet basis type is Daubechies 4th order, decomposition level is 3, threshold calculation method is Rigrsure) is used to suppress high-frequency random noise; Furthermore, the high-frequency coefficients are amplitude-shrinked at each decomposition scale by using a soft threshold function to generate denoised coefficient matrices at each scale, and the temperature distribution matrix is ​​reconstructed by inverse wavelet transform to obtain a high-fidelity infrared imaging signal. For the partial discharge electrical signal waveform, the empirical mode decomposition (EMD) algorithm (parameters: stopping criterion is a root mean square error of less than 1e-4, and the maximum number of modal components is 10) is used to decompose it into several intrinsic mode functions (IMFs). Furthermore, the peak sharpness of each IMF is calculated using the kurtosis criterion. The formula is as follows:

[0013] in, For IMF sample sequences, The sample mean. Standard deviation This is a mathematical expectation operator. IMF components with kurtosis values ​​exceeding a preset threshold are selected as valid discharge pulse components, and the denoised discharge signal sequence is reconstructed. For vibration acceleration time series data, bandpass filtering is performed (parameters: filter type is FIR, cutoff frequency range is 10Hz to 2kHz, order is 64) to preserve the response signal in the mechanical resonance frequency band; Furthermore, singular value decomposition (SVD) is performed to remove outlier peak components. By comparing each singular value with a set anomaly detection threshold, the corresponding noise vector is removed, generating clean time-series data of the mechanical response. For scalar parameters of temperature and humidity, a sliding window mean filter (parameters: window length of 10 seconds, step size of 1 second) is used to smooth transient jumps. Furthermore, through the mean within the window Calculation formula:

[0014] in, For the i-th sample value within the window, Given the number of samples, generate a smoothed temperature and humidity time series signal; Through the above modal-adaptive denoising processing method, the multimodal raw signal stream of the previous step is transformed into an infrared thermal imaging temperature distribution matrix, an effective discharge pulse sequence, a mechanical response timing signal, and a smoothed temperature and humidity curve, thereby achieving the expected technical effects of cross-modal noise suppression and signal fidelity improvement. For example, in a power plant inspection application, the infrared thermal imaging sampling frequency was 5Hz, the image resolution was 640×480 pixels, and the denoising parameters were configured as a Daubechies 4th order wavelet basis, a decomposition level of 3, and a Rigrsure threshold strategy. After denoising, the high-frequency noise amplitude of the infrared image was significantly reduced, and the root mean square error of the temperature distribution matrix was reduced by 0.8 degrees Celsius compared to the original data. The partial discharge signal sampling frequency was 100kHz, and EMD decomposition yielded 8 IMF components. The kurtosis threshold was set to 30, and 2 effective IMF reconstructed pulse sequences were selected. After comparison, the deviation between the peak value of the effective pulse and the peak value of the original waveform was reduced to 0.02 mV. The vibration acceleration signal sampling frequency was 5kHz, the FIR filter cutoff frequency range was set to 10Hz to 2kHz, the order was 64, and the singular value removal threshold was set to 1.5 times the standard deviation of the signal energy. After processing, the noise energy in the same frequency band was significantly reduced. The temperature and humidity signal sampling period is 1 second, the sliding window length is 10 seconds, and the step size is 1 second. After smoothing, the maximum jump amplitude is reduced from the original 2.3℃ to 0.5℃. In this scenario, the clean signals after denoising of each modality maintain high spatiotemporal consistency in the subsequent time alignment and feature extraction stages, effectively ensuring the input quality of dynamic causal modeling; S2.3: Based on the timestamp sequence corresponding to the denoised signals of each mode, a unified reference clock framework is constructed, and all sensor data are resampled to a common time base; a linear interpolation algorithm is used to numerically estimate the asynchronous sampling points, fill the time gaps caused by the difference in sampling frequency, and generate time-aligned signal segments with consistent time granularity. S2.4: Spatial registration of the time-aligned multi-channel signals according to the device component labels to ensure that the multimodal observations corresponding to the same physical component are strictly corresponding in time and space; the signal after alignment based on sliding window mean filtering is further smoothed to suppress high-frequency oscillations introduced by resampling and output a multi-channel synchronization signal block with uniform time resolution and stable amplitude characteristics. S2.5: Organize the processed multi-channel synchronization signal sequence into a structured tensor format, where the dimensions represent the time step, sensing mode type, spatial location index, and device component identifier, respectively, to generate a standardized multimodal joint signal volume that can be used as input for the depth feature encoder, serving as the input data basis for the next stage of nonlinear mapping.

[0015] like Figure 2 As shown, step S3 involves inputting a time-synchronized multi-channel signal sequence into a deep feature encoder for nonlinear mapping, extracting a high-dimensional latent space representation, and generating a multimodal joint feature vector, where each dimension corresponds to the abstract representation of different physical modes in the compressed latent space. Specifically, this includes: S3.1: Obtain the preprocessed multi-channel synchronization signal sequence, which includes an infrared thermal imaging image matrix, a partial discharge electrical signal time sequence, vibration acceleration time sequence data, and environmental temperature and humidity scalar parameters. Modal marking is performed based on the device component labels corresponding to each data channel to form an original input tensor with spatial-temporal-modal triple attributes. The structured multimodal joint signal volume output from step S2.5 is used as the input object of the deep feature encoder. A data channel attribute parsing method (parameters: device component label, time step, modality type index) is adopted to realize the physical modality identification and spatial location binding of each channel, and to ensure that the parsing results are stored in the form of tensor dimension mapping. Furthermore, by using a modal label injection algorithm (parameters: spatial location index, time series identifier), physical spatial information and time series characteristics are synchronously embedded into the metadata field of each element inside the input tensor, and a tensor index matrix with spatial-temporal binding attributes is obtained. Furthermore, a normalized pre-calibration method (parameters: amplitude range within the modality, data unit dimensions) is adopted to standardize the amplitude of each modality data of the input tensor, so as to eliminate feature shifts caused by dimensional differences between modalities. Furthermore, a temporal integrity verification algorithm (parameters: time step consistency threshold, missing value marking rules) is used to verify the integrity of the temporal dimension of the input tensor and generate a missing segment marking table to ensure the temporal continuity of the input data in the deep encoding stage. Through the above processing method, the multi-channel synchronization signal sequence of the previous step is transformed into a raw input tensor with complete spatial-temporal-modal triple attributes and uniform amplitude, thereby achieving the technical effect of providing structural consistency and semantic parsingability for multimodal deep feature encoding; For example, in a power plant inspection task, the input data consists of an infrared thermal imaging matrix with a size of 64×64×100 frames processed by step S2. The partial discharge signal sampling rate is 100kHz with a segment length of 1000 points, the vibration acceleration sampling rate is 10kHz with a segment length of 200 points, and the temperature and humidity parameter sampling rate is 1Hz with 100 continuous acquisitions. Equipment component labels include four categories: "main transformer casing," "high-voltage switchgear terminal," "turbine bearing housing," and "distribution room air monitoring point," with spatial location indices being integer points in a two-dimensional coordinate system. The modal label injection algorithm encodes the label information of each data channel into the third-dimensional index of the corresponding tensor and binds it to the time dimension number through a mapping table. The normalized pre-calibration method sets the pixel temperature range of the infrared matrix to normalized values. Within the range, the amplitude of the partial discharge signal is normalized to the maximum absolute value. The vibration acceleration signal amplitude is shifted to zero and the variance is scaled to the specified range. Temperature and humidity scalars are uniformly based on Celsius temperature and percentage humidity units. In time sequence integrity verification, a time step consistency threshold is set to... For missing value locations, zero-padding is inserted and a marker is recorded to ensure that all modalities are strictly aligned over 100 time steps. The original input tensor size after the above processing is [100, 4, number of spatial locations]. The amplitude of each modal data has been standardized, and the label information and temporal attributes have been embedded, meeting the input specifications of the deep feature encoder. During operation, it shows significantly improved cross-modal alignment accuracy and encoding stability. S3.2: Based on the original input tensor, perform modality-specific encoding processing on different modal data: use a two-dimensional convolutional neural network (2D-CNN) to extract spatial texture feature maps for the infrared thermal imaging image matrix; use a one-dimensional temporal convolutional network (1D-TCN) to capture local temporal patterns for vibration acceleration and partial discharge signals; and use a fully connected layer to nonlinearly extend the temperature and humidity scalars to generate independent intermediate feature representations for each modality. Based on the original input tensor with spatial-temporal-modal attributes obtained via S3.1, a two-dimensional convolutional neural network is employed (parameter: kernel size). × Step length The same padding strategy is used to perform spatial feature extraction operations on the infrared thermal imaging image matrix to achieve joint encoding of local texture and global contour of temperature distribution pattern, and output a primary feature map containing channel dimension and spatial dimension. One-dimensional temporal convolutional network is used (parameter: kernel length) Step length expansion coefficient The vibration acceleration signal and the partial discharge electrical signal are convolved in the time domain to realize the local time domain pattern capture of the equipment operation status. The network structure includes a multi-scale receptive field to enhance the pattern detection capability at different time scales and outputs a time domain feature sequence arranged in time order. Employs a fully connected layer (parameters: input dimension matches a single value of temperature and humidity scalars, output dimension...). The activation function is ReLU) to perform a nonlinear extended mapping on the temperature and humidity parameters, thereby achieving feature dimensionality upgrade from low-dimensional scalar to high-dimensional embedding space and generating extended feature vectors with nonlinear separability. Furthermore, through batch normalization (parameter: momentum) , ε= The amplitude of the primary features of each modality is standardized to unify the numerical distribution of different modal features, reduce the risk of gradient explosion or vanishing during network training, and lay a stable numerical foundation for subsequent cross-modal fusion. Through the above modality-specific encoding process, the original input tensor of the previous step is mapped to the corresponding intermediate feature representation, realizing the independent nonlinear compressed representation of each physical modality, and providing a high-quality basic feature set for the dimensional unification and positional encoding injection of S3.3; For example, in a power plant inspection system, the infrared thermal imaging modality uses a 2D-CNN with a convolution kernel size of [missing information]. × Step length The activation function is LeakyReLU, and the layers are two convolutional layers followed by one max pooling layer (pooling kernel size). × After that, the output feature map size is changed from... × Reduce to × The number of channels has been expanded to The vibration acceleration mode input length is Sampling points, 1D-TCN uses kernel length Expansion coefficient The three-layer structure allows you to experience the wild coverage. Sampling points, output 3D time-series characteristics. The processing method for partial discharge signals is the same as that for vibration acceleration, but the expansion coefficient is set to... To emphasize long-range dependency patterns, output Dimensional temporal characteristics. The temperature and humidity scalar inputs are two floating-point values ​​(temperature, humidity), and the fully connected layer extends to... The feature vectors are 3D and standardized to make the mean value 1. variance is After batch normalization, the numerical range of each modal feature is within... to This process ensures that the attention distribution will not be skewed due to amplitude differences during fusion in the subsequent S3.3 step. Results show that this step significantly improves the quality of fused features on the validation set, and significantly enhances the separability and recognition stability of anomalous patterns. S3.3: The intermediate feature representations of each modality are dimensionally unified. The learnable linear projection matrix is ​​used to map features of different dimensions to a unified latent space dimension. Modal identity information is injected through modal position encoding to generate an equal-dimensional feature embedding sequence with modal discrimination capability, which serves as the pre-expression for cross-modal fusion. S3.4: Based on the equal-dimensional feature embedding sequence, input it into the stacked Transformer encoder to perform cross-modal self-attention fusion calculation, use the multi-head self-attention mechanism to capture the long-range dependencies and interaction response strengths between modalities, and generate a context-aware fusion feature sequence, where each node represents the multimodal joint semantic state; Based on the equal-dimensional feature embedding sequence output by S3.3, a stacked Transformer encoder is used to perform cross-modal self-attention fusion computation (parameters: number of encoder layers L, number of attention heads H, hidden dimension). This enables global modeling and interaction feature capture of long-range dependencies between different modal embeddings; Furthermore, through a multi-head self-attention mechanism (parameter: scaling factor, where...) For each attention head (key vector dimension), the query, key, and value projection vectors of each modality embedding are computed in parallel. The dot product of the query and key vectors is normalized by Softmax to form the attention coefficient matrix, which is then mapped to the corresponding value vector to achieve weighted aggregation of multimodal information. Furthermore, residual connections and layer normalization (LayerNorm) are used to optimize the structure preservation and numerical stability of the weighted aggregation vectors of each attention head output, and a feedforward fully connected network (parameters: ReLU activation function, number of hidden units equal to 4) is used. The fused features are subjected to nonlinear transformation to enhance the discriminative power of multimodal semantic representation; Furthermore, the self-attention-feedforward network module described above is repeated in the stacked multi-layer Transformer encoder, so that different modalities are embedded in multiple layers to iteratively exchange and fuse information. Position encoding vectors are introduced and superimposed on the feature representation to explicitly preserve temporal structure information and track the interaction change trend between modalities in data spanning a long time. Furthermore, by accumulating the cross-modal attention weight tensor layer by layer, a context-aware fusion feature is generated for each embedded node. This feature encodes both temporal dependency and modal interaction strength in the semantic space, forming a fusion feature sequence containing multimodal joint semantic state. By using a stacked Transformer encoder and a multi-head self-attention mechanism, the isodimensional feature embedding sequence is transformed into a fusion feature sequence that reflects multimodal long-range dependencies and contextual interactions, achieving high-fidelity joint representation and providing semantically rich and structurally complete input data for subsequent feature compression and causal relationship modeling. For example, in a power plant inspection scenario, the input tensor The Transformer encoder depth L is set to 256, with each layer containing 8 attention heads (H=8), and the key vector dimension of each head is... =32. Using a position encoding length equal to the time step of 64, the embedding vector of the infrared thermal imaging modality is projected to generate a query. Vibration mode projection generation key and Through calculation Divide by , and then apply Softmax to obtain the attention weight matrix. →Vibration, weighted to The cross-modal response features are generated and then added to the residual of the original infrared modal vector and normalized. After 6 layers of stacked computation, the length of the fused feature sequence at each time step is 64, and the feature dimension remains at 256. Analysis shows that the temporal coupling mode between the transformer's infrared hotspot and vibration anomaly can be captured. The generated fused features significantly improve the accuracy of inter-modal directionality determination in the subsequent causal weight matrix construction. S3.5: Perform global average pooling on the fused feature sequence to compress the temporal and spatial dimensions, and output a fixed-length high-dimensional vector as a multimodal joint feature vector. This vector is a nonlinear coupling representation of each physical mode in the compressed latent space, and is used for subsequent evolution modeling of the dynamic causal attention weight matrix.

[0016] like Figure 3As shown, step S4 involves: based on the multimodal joint feature vector, jointly calculating the dynamic influence strength between each feature subspace using mutual information estimation and temporal gradient sensitivity analysis, and constructing an evolvable causal attention weight matrix. This matrix characterizes the asymmetric causal relationship structure among multimodal variables as the operating conditions change. Specifically, this includes: S4.1: Based on the multimodal joint feature vector output by S3, it is divided into several feature subspaces according to modality type. Each feature subspace corresponds to a high-dimensional representation sequence of a physical perception mode in the latent space, which serves as the basic analysis unit for dynamic causal modeling. S4.2: Estimate the mutual information of the nonlinear dependencies between each feature subspace. Use the K-nearest neighbor algorithm based on kernel density estimation to calculate the mutual information value between each pair of feature subspaces and obtain a symmetric mutual information matrix that reflects the statistical coupling strength between variables, which serves as a preliminary basis for causal directionality judgment. S4.3: Based on the feature subspace pairs with significant coupling relationships in the mutual information matrix, perform temporal gradient sensitivity analysis, use the forward difference method to calculate the gradient response of the source modal features to the target modal features in future time steps, and extract the instantaneous influence coefficient sequence across modes to capture potential causal driving directions; S4.4: The mutual information value and the temporal gradient sensitivity result are weighted and fused to construct an asymmetric dynamic influence intensity scoring function, in which mutual information is used as the correlation confidence weight and temporal gradient directionality is used as the causal arrow criterion to generate an initial causal attention weight matrix, whose elements represent the dynamic causal influence intensity of one modality feature on another modality feature. Based on the symmetric mutual information matrix output by S4.2 and the cross-modal instantaneous influence coefficient sequence output by S4.3, a weighted fusion method (parameters: correlation confidence weight α, causal directionality weight β) is adopted to unify the two types of indicators into the same dynamic influence intensity dimension. Furthermore, by using a normalization algorithm (parameter: L2 norm constraint), the range of the mutual information value matrix is ​​standardized to obtain the mutual information confidence matrix. And maintain matrix symmetry to reflect the bidirectional characteristics of statistical dependence; Furthermore, by using a normalization algorithm (parameter: maximum absolute value normalization), the directional distribution of the temporal gradient sensitivity result matrix is ​​normalized, thus obtaining the causal directionality matrix. And maintain matrix asymmetry to characterize the time-driving effect of different modes on another mode; Furthermore, a weighted fusion formula is used to calculate the asymmetric dynamic influence intensity scoring function. :

[0017] in, For the relevance confidence weight, For causal directional weights, It is a symmetric mutual information confidence matrix. It is an asymmetric causal directionality matrix; Furthermore, by threshold sparsification (parameter: dynamic threshold τ), elements with low influence intensity are pruned to generate an initial causal attention weight matrix, whose elements directly represent the dynamic causal influence intensity of a source modality feature on the target modality feature. Through the above weighted fusion and sparsification processing, the mutual information and temporal gradient results of the previous step are transformed into a structured and quantified initial causal attention weight matrix, so as to achieve the expected technical effect of characterizing the asymmetric causal relationship structure between multimodal variables as the working conditions change. For example, in a power plant inspection scenario, the mutual information values ​​of the infrared thermal imaging mode and the vibration acceleration mode, after normalization, are: The corresponding temporal gradient sensitivity coefficient is Set relevance confidence weights. = Causal directional weights = Substituting into the weighted fusion formula, we get:

[0018] The calculation result is When setting a dynamic threshold τ= Under these conditions, the element is retained in the initial causal attention weight matrix, reflecting the significant dynamic causal influence of the infrared thermal imaging mode on the vibration acceleration mode under the current operating conditions. This processing is performed on the full-mode pair matrix, which can significantly improve the sparsity and interpretability of the causal structure representation, ultimately supporting the subsequent causal sensing path modulation embedded in the classification network; S4.5: A trainable parameterization mechanism is introduced into the initial causal attention weight matrix. Differentiable optimization of the attention distribution is achieved through Softmax normalization and temperature coefficient adjustment. This mechanism is embedded into the end-to-end network framework, supporting online updating of weight parameters based on backpropagation error signals, so that the causal structure can be adaptively adjusted as the operating conditions evolve.

[0019] Step S5: Embed the causal attention weight matrix into the feedforward classification network. When an abnormal event is detected, the abnormal classification score is used as the backpropagation target. An extended algorithm of gradient-weighted class activation mapping is executed, backtracking along the network layers and weighting and accumulating the responses of sensitive regions of the feature maps of each layer to generate a root cause contribution integral map. Specifically, this includes: S5.1: Based on the evolvable causal attention weight matrix output from the preceding step S4, it is embedded as a dynamic gating factor into the feature propagation path between each hidden layer of the feedforward classification network. This weights and modulates the cross-layer propagation of the multimodal joint feature vector during the classification process, generating a discriminative feature representation constrained by the causal structure. This ensures that the classification decision process implicitly follows the asymmetric causal relationship structure between multimodal variables. Based on the evolvable causal attention weight matrix output by step S4, the elements of this matrix are loaded as dynamic gating coefficients into the feature transfer layer of the feedforward classification network to achieve directional weighted modulation of cross-modal feature streams. The feature channel weighted embedding method is adopted (parameter: the mapping relationship between source mode and target mode corresponding to the elements of the causal weight matrix). At the input of each hidden layer of the network, the feature vectors from different modes are multiplied by the weight coefficients one channel at a time. This suppresses the propagation path with low causal influence and strengthens the path with high causal influence, generating feature maps filtered by causal structure. Furthermore, through a cross-layer gating parameter regularization method (parameter: L2 norm constraint coefficient is taken as follows): This maintains the stability of the causal weight distribution between different layers, prevents the weight matrix from shifting due to local gradient fluctuations, and improves the consistency of cross-layer feature propagation. Furthermore, a hierarchical normalization algorithm is adopted (parameter: Softmax temperature coefficient is set to...). The embedded causal weight matrix is ​​normalized and its distribution is adjusted within each hidden layer to ensure that the weighted feature map maintains numerical stability in activation intensity and avoids numerical explosion during propagation. Furthermore, using the causal weight path mask generation method (parameter: the influence threshold is set to the global mean plus standard deviation of the weight matrix), zero-value masking is performed on cross-modal connections below the threshold during feature propagation to reduce the interference of invalid paths on discriminative features and retain causal connections with high contribution to support subsequent reverse root cause backtracking. Through the above embedding and modulation algorithms, the multimodal joint features after causal structure screening are transformed into discriminative feature representations that conform to the constraints of asymmetric causal relationships, so that the forward classification network implicitly follows the condition-driven causal dependency structure during the inference stage. For example, in a power plant inspection application scenario, the input multimodal joint feature dimension is 128, of which the infrared image mode accounts for 32 dimensions, the partial discharge signal mode accounts for 32 dimensions, the vibration acceleration mode accounts for 32 dimensions, and the temperature and humidity mode accounts for 32 dimensions. The causal attention weight matrix generated in step S4 is... Dimension, range of matrix elements Between. During embedding, matrix elements are assigned sequentially to the four hidden layers of the network according to the modal mapping relationship, with the average weight from infrared to vibration being... The average weight of partial discharge to temperature and humidity is Each layer's input feature vector is multiplied channel-by-channel with its corresponding weight coefficient to generate a new feature tensor. After Softmax temperature normalization, the numerical range stabilizes within a certain range. Between. The threshold mask is set to the global mean. Added standard deviation After masking, the propagation of low-weight paths is blocked, while high-weight paths are significantly amplified during reverse backtracking. Validation results show that the network with embedded causal weights significantly improves the root cause localization accuracy in multimodal anomaly recognition tasks, and the discriminative feature response maps can be concentrated in mechanism-related modal channels, thereby enhancing the causal interpretability of anomaly recognition. S5.2: When the output layer of the classification network generates an anomaly event judgment and its classification score exceeds the preset confidence threshold, the predicted score of the anomaly category is used as the target response variable for backpropagation, the gradient backpropagation mechanism is started, and the gradient tensor of the loss function on the feature map of the layer is calculated based on the chain rule to obtain the local sensitivity intensity distribution of the current layer to anomaly judgment. Based on the judgment results of the output layer of the feedforward classification network after pre-embedding the causal attention weight matrix, the anomaly category prediction score is analyzed. The input objects are the classification output distribution of multimodal joint features in the last layer of the network and the corresponding anomaly category index, and the reverse sensitivity calculation process is executed. A threshold comparison method (parameter: prediction score threshold = 0.85) is used to determine whether the output layer anomaly category score meets the preset confidence condition, thereby realizing the filtering function for anomaly event activation. Furthermore, by using the target substitution method (parameter: target category = anomaly category prediction index), the output anomaly category score is replaced with the target response variable of backpropagation, and the outputs of other categories are fixed as constants, ensuring that the gradient calculation process focuses on the discriminative contribution of a single anomaly category; Furthermore, by utilizing the chain rule, the gradient tensor of the loss function relative to the feature map of each layer in the classification network is calculated, thereby achieving a quantitative analysis of the local sensitivity intensity. The formula is as follows:

[0020] in, The single-output loss function is constructed based on the anomaly category prediction scores. For the network Feature map tensors of the layer; Furthermore, by using the automatic differential calculation engine (parameter: enable element-wise gradient accumulation), the distribution matrix of the gradient tensor in the spatial dimension and channel dimension is obtained, and gradient magnitude normalization is performed to suppress the interference of numerical scale differences on subsequent response accumulation. Furthermore, a gradient cutoff rule (parameter: cutoff threshold = gradient mean ± twice the standard deviation) is applied to remove extreme outlier gradient values, ensuring the stability and interpretability of the sensitivity distribution; By using gradient backpropagation and normalization, the anomaly category scores selected in the previous step are transformed into layer-by-layer local sensitivity intensity distribution data, thereby achieving accurate acquisition of the spatial-channel two-dimensional response matrix and providing highly reliable input for the subsequent generation of causal path weighting and root cause contribution integrals. For example, in a scenario involving abnormal vibration detection of a power plant cooling pump, the forward classification network outputs a predicted score of 0.91 for the category "mechanical abnormality." When the abnormality activation threshold is set to 0.85, the backpropagation sensitivity calculation process is triggered. The target permutation method locks the abnormality category index as 3, corresponding to the 3rd component in the output vector as the backpropagation target response variable, while the remaining category components are set to a constant of 1. In the chain-like differentiation, it is assumed that the feature map of the 5th convolutional layer of the network is... loss function Constructing a negative prediction score, and using an automatic differential calculation engine to generate a gradient tensor. This yielded a sensitivity matrix with a spatial dimension of 64×64 and 128 channels. After channel-based normalization of the gradient matrix, the vibration mode-related channels showed response amplitudes higher than the normalized mean. With the cutoff threshold set at ±2 standard deviations, 80 stable response channels were retained, outputting the precise local sensitivity distribution of the cooling pump vibration anomaly, providing a valid basis for the next step of Grad-CAM extension and causal weighting. S5.3: Based on the element-wise multiplication operation of the gradient tensor and the corresponding layer feature map, perform the extension operation of gradient weighted class activation mapping (Grad-CAM) to generate a spatial-channel joint attention heatmap for each layer. Combine the heatmap with the causal attention weight matrix embedded in S5.1 to perform path weighting, so that the feature channels with high causal influence can obtain higher response gain, forming a causal-aware sensitive region response map. The input consists of the gradient tensor and the corresponding hidden layer feature map obtained in step S5.2. The gradient tensor represents the local sensitivity intensity distribution of the features of this layer to the anomaly discrimination output, and the feature map is the multi-channel representation of this layer after convolution or temporal coding. An element-wise multiplication method (parameters: gradient tensor G, feature map F) is used to fuse gradient sensitivity and feature activation intensity to obtain a weighted activation matrix M. The formula for calculating the element-wise multiplication is as follows:

[0021] Furthermore, by extending the Gradient Weighted Class Activation Mapping (Grad-CAM) algorithm (parameters: number of convolutional layer channels C, spatial size H×W), global average pooling is performed on the weighted activation matrix M along the channel dimension, and the weight coefficient α of each channel is calculated, where the formula is:

[0022] in Let c be the weight coefficient of the c-th channel. These represent the spatial height and width of the feature map, respectively. Let M be the element value of the fusion matrix at the c-th channel and the (i, j)-th spatial position. Furthermore, by multiplying the weight coefficient α by the feature activation map of the corresponding channel and summing the results for all channels, a spatial attention heatmap L is obtained, which is used to characterize the spatial importance distribution of the features of this layer. The calculation formula is as follows:

[0023] in Let C be the c-th channel of the feature map, where C is the number of channels in the feature map. Furthermore, by combining the causal attention weight matrix embedded in step S5.1 (Each element represents the causal influence strength of each feature channel), implementing path weighting to proportionally amplify the response values ​​of channels with high causal influence. The formula is:

[0024] in, A heatmap of response to causal perception. Let c be the causal attention weight for the c-th channel; By using the above causal weighting processing method, the original Grad-CAM spatial heatmap is transformed into a causal perception spatial-channel joint attention heatmap, thereby enhancing the saliency of regions with high causal influence. By using the causal weighted Grad-CAM extension algorithm, the gradient-feature fusion result of the previous step is transformed into a causally sensitive spatial attention distribution, which enables visualization of high-contribution regions of each modality in enhanced anomaly detection and provides a more accurate response map for subsequent influence path backtracking. For example, considering the partial discharge signal and infrared thermal imaging features of the terminal block of a high-voltage switchgear in a power plant inspection system as input, a convolutional layer with 64 channels (C) and a spatial size of 32×32 pixels is selected. The local sensitivity intensity obtained by chain rule differentiation of the gradient tensor is within the range of [0.05, 0.8]. Element-wise multiplication is used to calculate M, obtaining the weighted activation matrix for each channel. The channel weights are then calculated using global average pooling. The channel corresponding to the maximum value represents the infrared modal high-temperature anomaly feature, with a weight of 0.72. Multiplying the feature map of this channel by the weight and summing it with the results of other channels generates a spatial thermal map L. A high response peak is observed at the terminal connection location in L. A causal attention weight matrix is ​​then introduced. A gain of 1.5x was applied to the high-temperature characteristic channel of the infrared mode, and a gain of 1.2x was applied to the characteristic channel of the discharge signal, generating a causal sensing response heatmap. The peak value is significantly concentrated at the terminal block contact points. This response map can accurately locate the starting region of anomaly propagation in subsequent spatiotemporal backtracking, achieving significant identification and enhanced visualization of the anomaly source; S5.4: Traverse the causal perception sensitive region response maps of each hidden layer from top to bottom along the network depth direction, perform spatial averaging and channel normalization on the response map output of each layer, extract its significant response components, and map them back to the time segment of the original input signal based on the sliding window matching mechanism in the time dimension to generate a hierarchical influence transmission sequence. S5.5: Weighted cumulative fusion of the influence transmission sequences of all layers is performed, where the fusion weight is determined by the depth position of each layer in the network and the propagation attenuation coefficient of the causal attention weight. Finally, a unified root cause contribution integral map is generated, which is the comprehensive contribution distribution of each original input channel on the time-modal two-dimensional plane, and is used for subsequent extraction and attribution analysis of key influence paths.

[0025] Step S6: Based on the root cause contribution integral map, identify the modality type and time segment with the highest contribution to the anomaly discrimination result in the original input channel, forming a key influence path sequence. This sequence includes the sensing modality, time window, and spatial location information that dominates the anomaly performance. Specifically, it includes: S6.1: Based on the root cause contribution integral map output from the previous steps, this integral map represents the response intensity of the sensitive region after the feature map of each layer is accumulated by causal attention weighting during backpropagation. The normalization processing algorithm is used to perform cross-modal amplitude standardization on this integral map to eliminate the contribution bias caused by the difference in dimensions of different sensing modes, and generate a standardized root cause response heatmap as a basis for comparable influence intensity among multiple modes. S6.2: Based on the standardized root cause response heatmap, a dynamic threshold segmentation algorithm is used to extract significant activation regions, wherein the dynamic threshold is set to the global mean plus two standard deviations. High response segments with contributions higher than the threshold are identified and mapped back to the time axis and modal dimension of the original input space to obtain the initially located high contribution time windows and corresponding sensing modal sets, forming a candidate influence factor list. Based on the standardized root cause response heatmap input, a dynamic threshold segmentation algorithm (parameter setting: threshold = global mean + twice the standard deviation) is used to achieve automatic detection of significant activation regions across modalities; Furthermore, the mean value of the root cause response heatmap is calculated through a global amplitude scan. and standard deviation and according to the formula Generate dynamic threshold This is used to determine the starting conditions of high-response segments; Furthermore, binarization is used to extract values ​​greater than a threshold from the heatmap. The pixel positions are marked as activation points, a saliency mask matrix is ​​generated, and high-response regions are grouped and identified through a connected component analysis algorithm (parameter: 8-neighborhood connection rule). Furthermore, a cross-modal mapping operation is performed on each connected component to map its spatial coordinates to the modal index dimension of the input tensor, and its temporal coordinate range is parsed and mapped to the original signal time axis to obtain the high contribution time window and the corresponding sensing mode set; Furthermore, by filtering out active connected components whose area or response peak is lower than a preset threshold, the set of responses with high significance is retained to form a list of candidate influencing factors; By using a dynamic threshold segmentation algorithm and a spatiotemporal mapping processing method, the standardized heatmap from the previous step is transformed into a high-response candidate factor with modal and time labels, thus achieving the preliminary localization effect of abnormal events in the multimodal input space. For example, in a power plant transformer inspection scenario, the standardized root cause response heatmap size is 128×256, and its global mean is... The standard deviation is The dynamic threshold is calculated as follows: equal After binarization, the number of high-response pixels in the saliency mask matrix was 3542. Eight-neighbor connected component analysis yielded 12 high-response regions, of which 9 were assigned to the infrared thermal imaging mode and 3 to the vibration acceleration mode via modal mapping. In the time-axis mapping results, the high-response time window for the infrared mode was concentrated between 12.4 s and 15.7 s, while that for the vibration mode was concentrated between 14.2 s and 14.9 s. After screening using area thresholds (minimum area 50 pixels) and peak response thresholds (minimum peak value 0.75), 7 candidate influencing factors were ultimately retained, including 5 time windows for the infrared mode and 2 time windows for the vibration mode. This result can be bound to specific device component tags in subsequent steps to achieve high-precision anomaly root cause localization. S6.3: Based on the candidate impact factor list, combined with the device component tags and sampling timestamps recorded by each data channel, perform a spatiotemporal alignment mapping operation to bind the high contribution time window with the specific device location and physical structure, and generate a key impact unit containing a triplet of 'sensing mode - time window - spatial location' as the basic element constituting the key impact path; Based on the input of the candidate impact factor list, a spatiotemporal alignment mapping algorithm (parameters: equipment component label, sampling timestamp, modality identifier) ​​is used to accurately bind high contribution time windows with physical equipment spatial locations; Furthermore, by using the tag index matching method (parameter: 3D index structure 'device component tag-modal type-time stamp'), a one-to-one correspondence between candidate factors and equipment component records is achieved, and a binding mapping table is obtained for subsequent path construction; Furthermore, a sampling timestamp normalization algorithm (parameter: millisecond-level time base) is adopted to realize the synchronization adjustment of time windows of different modes and generate time-aligned mode-position mapping data; Furthermore, by using a spatial topology retrieval method (parameters: device structure topology diagram, location code), the coordinates of the sensing position corresponding to the time window in the device physical model are located, and a mapping output containing spatial location information is generated. Furthermore, a data fusion encapsulation algorithm (parameters: sensing mode, normalized time window, spatial location coordinates) is adopted to generate key influencing units of the 'sensing mode-time window-spatial location' triple and form the basic elements of the key influencing path; By binding mapping and coordinate positioning processing, the candidate influencing factors in the previous step are transformed into structured key influencing units, realizing accurate mapping of the anomaly propagation chain on physical devices and the availability of subsequent path sequence construction; For example, in a high-voltage side inspection scenario of a power transformer, the candidate influencing factor list includes the high contribution time window of the infrared thermal imaging mode from sampling time 2024-05-08 14:23:12.532 to 2024-05-08 14:23:15.876, and the response of the partial discharge signal from sampling time 2024-05-08 14:23:14.105 to 2024-05-08 14:23:16.402. The label index matching method binds the infrared mode window to the record row corresponding to the component label "Transformer Shell Area HV-A", and the partial discharge mode window to "High Voltage Terminal Block HV-Terminal-1". The sampling timestamp normalization process uses a millisecond-level absolute clock as a reference to align the two time windows to a unified time base, forming an aligned mode-time mapping result. The spatial topology retrieval method locates the three-dimensional coordinates of the HV-A region in the equipment structure topology map as (x=3.2m, y=1.4m, z=2.5m), and the HV-Terminal-1 coordinates as (x=3.5m, y=1.2m, z=2.7m). The data fusion encapsulation algorithm encapsulates the infrared mode—time window 14:23:12.532 to 14:23:15.876—position (3.2, 1.4, 2.5) and the partial discharge mode—time window 14:23:14.105 to 14:23:16.402—position (3.5, 1.2, 2.7)—as triplet key influence units, which are used to support the priority ranking of subsequent anomaly propagation paths and the summarization of physical causes. In this scenario, the generated key influence units accurately characterize the spatiotemporal distribution characteristics of the modal response, providing sufficient basis for matching the fault mechanism on the high-voltage side, and significantly improving the accuracy of attribution and the credibility of the explanation. S6.4: Sort the multiple key influencing units according to their response intensity in the root cause contribution integral map, select the top K units with the highest contribution, construct an ordered sequence based on their temporal sequence and physical coupling logic, and generate a key influence path sequence. This sequence reflects the multimodal inputs and their spatiotemporal evolution trajectory that dominate the abnormal propagation process. S6.5: Perform redundancy pruning on the key impact path sequence, use sliding window correlation analysis to detect repeated response intervals between adjacent impact units, merge overlapping time windows and retain the maximum contribution mode, optimize the compactness of path expression, and finally output a structured, non-redundant key impact path sequence to support subsequent semantic matching with the device mechanism model library.

[0026] Step S7: Perform semantic similarity matching between the key impact path sequence and the typical failure modes pre-stored in the equipment mechanism model library. Output the most likely root cause type based on the maximum matching degree, and generate a structured diagnostic report containing the anomaly category, confidence level, impact path, and recommended investigation items. Specifically, this includes: S7.1: Construct an equipment mechanism model library based on equipment operation mechanism and historical failure cases. The equipment mechanism model library stores several typical failure mode templates. Each template includes a preset combination of multimodal anomaly representation features, a corresponding failure type label, a physical cause description, and a recommended handling strategy. The combination of multimodal anomaly representation features consists of the dominant sensing mode, typical temporal evolution law, and spatial distribution pattern, which serve as the knowledge benchmark for subsequent semantic similarity matching. Based on the structured, non-redundant key impact path sequence output by S6 and its included sensing modalities, time windows and spatial location identifiers, an equipment mechanism model library is constructed using equipment operation mechanisms and historical fault cases to generate a knowledge benchmark for multimodal semantic matching. Using data extraction and feature summarization methods (parameters: fault case database, equipment operation log, inspection image archive), multimodal abnormal performance data of each equipment type under different fault conditions are extracted to form an original mechanism case set; Furthermore, through feature pattern analysis methods (parameters: modality type, temporal trajectory, spatial distribution matrix), the abnormal manifestations in the original case set are normalized to obtain feature combinations containing dominant sensing modes, typical temporal evolution laws and spatial distribution patterns, which constitute the core feature units of the mechanism template; Furthermore, using a fault classification and labeling algorithm (parameters: equipment type label, fault category dictionary), a corresponding fault type label is bound to each feature combination, and the physical cause description in the accident investigation report is extracted to supplement the causal explanation part of the mechanism template. Furthermore, a disposal strategy extraction method (parameters: operation and maintenance procedure library, maintenance work order record) is adopted to add recommended investigation items and disposal steps to each mechanism template, and improve the template into a complete multi-field structure containing feature combination, type label, physical cause and disposal strategy; By using a structured storage method (parameter: JSON Schema definition), the resulting template set is organized into an indexable and searchable device mechanism model library, which provides the basic support for subsequent semantic similarity matching. For example, in the operation and maintenance scenario of a generator unit in a coal-fired power plant, three types of historical faults—inter-turn short circuit in the generator stator winding, loose cooling water pump coupling, and poor contact in the high-voltage switchgear—were selected as case data sources. Corresponding infrared thermal imaging peak temperature curves, partial discharge pulse density sequences, vibration spectrum peak trajectories, and environmental temperature and humidity change records were extracted. The multimodal sequences of the three cases were normalized, the temperature curves were compressed into time-series vectors of length 50, the discharge density trend was fitted to quadratic curve parameters, the vibration spectrum peaks were represented by frequency-amplitude pairs, and the temperature and humidity fluctuation ranges before and after the fault were converted into standard deviation indices. For instance, the vibration spectrum peak of the loose cooling water pump coupling appeared at 47Hz with an amplitude of 0.8g; the corresponding infrared thermal image showed a bearing housing temperature rise of 68℃. This feature combination was tagged with "cooling water pump mechanical looseness," the extraction mechanism was explained as "the loose coupling caused imbalance in rotating parts, triggering vibration and generating frictional heat," and the handling strategy was "immediately stop the machine, check the coupling tightness, and replace the damaged parts." The three mechanism templates mentioned above are stored in structured JSON. Each template field includes the dominant mode (vibration, infrared, discharge, temperature and humidity), time evolution vector, spatial location identifier (equipment part code), fault type, physical cause description, and recommended treatment strategy, providing a high-confidence knowledge basis for semantic similarity matching of subsequent key impact paths. S7.2: Perform structured encoding on the key impact path sequence output by S6, extract the dominant sensing mode type, start and end times of high contribution time window, spatial location identifier and signal change trend features contained therein, and generate a standardized impact path feature vector, which serves as the query input for comparison with the equipment mechanism model library; S7.3: Calculate the semantic similarity score between the feature vector of the key impact path and the templates of each typical fault mode in the equipment mechanism model library based on the cosine similarity algorithm. The calculation process performs weighted fusion of modal consistency, temporal alignment deviation and spatial overlap to obtain a comprehensive matching score, so as to quantify the degree of closeness between the current abnormal path and various known fault modes. S7.4: Determine the typical fault mode template corresponding to the maximum value based on all matching scores, output the fault type label associated with it as the most likely root cause type, and extract the pre-stored physical cause description and recommended investigation items in the template to form a preliminary diagnostic conclusion set. S7.5: Integrate elements such as anomaly category, classification confidence level, key impact path information, root cause type and its confidence matching degree, and recommended investigation items to generate a structured diagnostic report in a unified format. The report uses JSON Schema to define the field structure, supports visual presentation and automatic integration with the operation and maintenance work order system, and improves the usability and integration efficiency of AI diagnostic results.

[0027] Step S8: Based on the actual handling results reported by maintenance personnel, update the learning parameters of the causal attention weight matrix, and adjust the modeling accuracy of the dynamic causal structure through online gradient optimization to achieve the adaptive evolution capability of the anomaly recognition model in complex coupled scenarios. Specifically, this includes: S8.1: Obtain the actual handling results reported by the operation and maintenance personnel. The actual handling results include the equipment fault type confirmed on-site, the root cause location conclusion, the handling measures record and the corresponding timestamp information, which are used as the supervision signal input for model correction to establish a data closed loop for human-machine collaborative verification. S8.2: Based on the actual handling results, compare the differences with the original structured diagnostic report output by the system, calculate the misjudgment rate of abnormal categories and the deviation of root cause paths, and generate a quantitative index of model deviation as the target optimization direction for updating the parameters of the causal attention weight matrix; S8.3: The backpropagation algorithm is used to map the model bias quantification index to the causal attention embedding layer in the forward classification network. Based on the gradient change trend of the loss function with respect to the causal attention weight matrix, the parameter sensitivity coefficients corresponding to each multimodal feature subspace are extracted to form a gradient guidance field for parameter tuning. S8.4: Based on the gradient-guided field, perform online gradient descent optimization to incrementally update the asymmetric correlation parameters in the causal attention weight matrix. An adaptive learning rate strategy is used to control the parameter adjustment range to avoid forgetting historical knowledge and generate an updated dynamic causal attention structure. S8.5: The updated dynamic causal attention structure is injected into the anomaly identification process of the next cycle, and the parameter evolution trajectory is recorded for subsequent analysis of the trend of causal relationship with the operating conditions, so as to realize the continuous adaptation and robustness enhancement of the anomaly identification model in highly coupled, nonlinear power plant systems.

[0028] The present invention also provides a power plant area inspection system based on multimodal perception fusion, which uses the above-mentioned power plant area inspection method based on multimodal perception fusion to inspect the power plant area.

[0029] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0030] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and rules of the present invention should be included within the scope of protection of the present invention.

Claims

1. A power plant area inspection method based on multimodal perception fusion, characterized in that, Includes the following steps: S1: Acquire multimodal environmental data collected by sensing devices in the power plant inspection system, and record the device component labels and sampling timestamps corresponding to each data channel; S2: Denoising and time-aligned preprocessing is performed on the multimodal environmental data. Linear interpolation and sliding window mean filtering are performed based on the timestamp sequences of each sensor to generate a time-synchronized multi-channel signal sequence. S3: Input the multi-channel signal sequence into a deep feature encoder for nonlinear mapping, extract the high-dimensional latent space representation, and generate a multimodal joint feature vector; S4: Based on the multimodal joint feature vector, the dynamic influence intensity between each feature subspace is jointly calculated using mutual information estimation and temporal gradient sensitivity analysis, and a causal attention weight matrix is ​​constructed. S5: Embed the causal attention weight matrix into the feedforward classification network. When an abnormal event is detected, the abnormal classification score is used as the backpropagation target. The gradient weighted class activation mapping expansion operation is performed. The network layers are backtracked step by step and the sensitive region responses of the feature maps of each layer are weighted and accumulated to generate the root cause contribution integral map. S6: Based on the root cause contribution integral map, identify the modality type and time segment that contributes the most to the anomaly discrimination result in the original input channel, and form a key influence path sequence; S7: Perform semantic similarity matching between the key impact path sequence and the typical failure modes pre-stored in the equipment mechanism model library, output the most likely root cause type based on the maximum matching degree, and generate a structured diagnostic report.

2. The power plant area inspection method based on multimodal perception fusion according to claim 1, characterized in that, Following step S7, the following is also included: S8: Based on the actual handling results reported by the operation and maintenance personnel, update the learning parameters of the causal attention weight matrix, and adjust the modeling accuracy of the dynamic causal structure through online gradient optimization to realize the adaptive evolution capability of the anomaly recognition model in complex coupled scenarios.

3. The power plant area inspection method based on multimodal perception fusion according to claim 1, characterized in that, The multimodal environmental data includes infrared thermal imaging images, partial discharge electrical signals, vibration acceleration sequences, and environmental temperature and humidity parameters.

4. The power plant area inspection method based on multimodal perception fusion according to claim 1, characterized in that, Step S3 specifically includes: The preprocessed multi-channel synchronization signal sequence is obtained, and modal labeling is performed based on the device component labels corresponding to each data channel to form the original input tensor; Based on the original input tensor, modality-specific encoding processing is performed on different modal data to generate independent intermediate feature representations for each modality. The intermediate feature representations of each modality are subjected to dimensional unification processing, and features of different dimensions are mapped to a unified latent space dimension. Modal identity information is injected through modal position encoding to generate an equal-dimensional feature embedding sequence. Based on the equidimensional feature embedding sequence, it is input into the stacked Transformer encoder to perform cross-modal self-attention fusion calculation. The multi-head self-attention mechanism is used to capture the long-range dependencies and interaction response strengths between modes and generate a fused feature sequence. A global average pooling operation is performed on the fused feature sequence to compress the temporal and spatial dimensions, and a fixed-length high-dimensional vector is output as the multimodal joint feature vector.

5. The power plant area inspection method based on multimodal perception fusion according to claim 4, characterized in that, The specific steps for performing modality-specific encoding on data of different modalities are as follows: A two-dimensional convolutional neural network is used to extract spatial texture feature maps from the infrared thermal imaging image matrix; a one-dimensional temporal convolutional network is used to capture local temporal patterns for vibration acceleration and partial discharge signals; and temperature and humidity scalars are nonlinearly extended through fully connected layers.

6. The power plant area inspection method based on multimodal perception fusion according to claim 1, characterized in that, Step S4 specifically includes: Based on the multimodal joint feature vector output in step S3, it is divided into several feature subspaces according to modality type. Each feature subspace corresponds to a high-dimensional representation sequence of a physical perception modality in the latent space. Mutual information estimation is performed on the nonlinear dependencies between each feature subspace, and the mutual information value between each pair of feature subspaces is calculated to obtain a symmetric mutual information matrix. Based on the feature subspace pairs with significant coupling relationships in the symmetric mutual information matrix, perform temporal gradient sensitivity analysis, calculate the gradient response of the source modal features to the target modal features in future time steps, and extract the instantaneous influence coefficient sequence across modes; The mutual information values ​​are weighted and fused with the temporal gradient sensitivity analysis results to construct an asymmetric dynamic influence intensity scoring function and generate an initial causal attention weight matrix. A trainable parameterization mechanism is introduced into the initial causal attention weight matrix. The attention distribution is optimized by Softmax normalization and temperature coefficient adjustment, and then embedded into the end-to-end network framework.

7. The power plant area inspection method based on multimodal perception fusion according to claim 6, characterized in that, In the initial causal attention weight matrix, the matrix elements represent the dynamic causal influence strength of one modality feature on another modality feature.

8. The power plant area inspection method based on multimodal perception fusion according to claim 1, characterized in that, Step S5 specifically includes: Based on the causal attention weight matrix output in step S4, it is embedded as a dynamic gating factor into the feature propagation path between hidden layers of the feedforward classification network to perform weighted modulation on the cross-layer propagation of the multimodal joint feature vector in the classification process, thereby generating a discriminative feature representation constrained by the causal structure. When the output layer of the classification network generates an anomaly event judgment and its classification score exceeds the preset confidence threshold, the predicted score of the anomaly category is used as the target response variable for backpropagation, the gradient backpropagation mechanism is activated, and the gradient tensor of the loss function with respect to the feature map corresponding to the anomaly event is calculated to obtain the local sensitivity intensity distribution of the current layer to anomaly discrimination. Based on the element-wise multiplication operation between the gradient tensor and the corresponding layer feature map, the extension operation of gradient weighted activation mapping is performed to generate the spatial and channel joint attention heatmap of each layer. The spatial and channel joint attention heatmap is then path-weighted by the embedded causal attention weight matrix to form a causal perception sensitive region response map. The causal perception sensitive region response maps of each hidden layer are traversed from top to bottom along the network depth direction. Spatial averaging and channel normalization are performed on the response map output by each layer to extract its significant response components. Based on the sliding window matching mechanism in the time dimension, these components are mapped back to the time segments of the original input signal to generate a hierarchical influence transmission sequence. The influence transmission sequences of all layers are weighted, accumulated, and fused to generate a root cause contribution integral map.

9. The power plant area inspection method based on multimodal perception fusion according to claim 1, characterized in that, The key impact path sequence includes the sensing modality, time window, and spatial location information that dominates the abnormal behavior.

10. A power plant area inspection system based on multimodal perception fusion, characterized in that: The power plant area inspection method based on multimodal perception fusion as described in any one of claims 1-9 is used to conduct power plant area inspections.

Images