Power equipment auxiliary monitoring system and method based on multi-modal data fusion

Abstract

The invention provides a power equipment auxiliary monitoring system and method based on multi-modal data fusion. The method comprises the following steps: acquiring three types of data of an infrared image, a sound signal and a partial discharge signal of to-be-monitored power equipment; when each modal feature is extracted, a weight is generated according to a sound signal reliability index, a space attention mask is generated according to a time-frequency characteristic, infrared image feature extraction is enabled after weight adjustment, then three types of modal features are input into a shared encoder and a private encoder, common modal features and specific modal features are separated, and the modal features are extracted. And synchronously applying modal invariance and inter-modal difference constraints, constructing a multi-modal weight according to each data reliability index, performing weighted adjustment on each specific feature, inputting the specific features and the common features into a time sequence information fusion module together, outputting a fusion state feature sequence, and judging the operation state of the power equipment according to the fusion state feature sequence.

Description

Technical Field

[0001] This application belongs to the field of monitoring, and in particular relates to an auxiliary monitoring system and method for power equipment based on multimodal data fusion. Background Technology

[0002] Power equipment is the foundation for the stable operation of power systems. Traditional power equipment condition monitoring, such as manual inspections, uses infrared thermal imaging technology to diagnose overheating faults by detecting the temperature distribution on the equipment surface. However, this method is susceptible to interference from environmental factors such as weather and light reflection, and is insensitive to non-thermal effects such as internal discharges and mechanical loosening. Sound monitoring, especially ultrasonic testing, can identify problems such as corona discharge and mechanical vibration, but in noisy industrial environments, the signal-to-noise ratio is low, feature extraction is difficult, and misdiagnosis is common. Partial discharge detection is a means of assessing the insulation condition of equipment and can sensitively detect early insulation defects, but it is extremely sensitive to electromagnetic interference and cannot reflect the mechanical or overall thermal condition of the equipment. These single-mode monitoring methods have limited information dimensions, weak anti-interference capabilities, and suffer from incomplete information acquisition and limited diagnostic accuracy.

[0003] Using multiple sensors to collect data in parallel and determining device status through feature layer stitching or decision layer fusion can reduce the problems mentioned above. However, the data quality and reliability of different modalities can vary. For example, audio signals may be distorted by sudden noise, and infrared images may show false hotspots due to sudden changes in illumination. Most existing methods treat all modal data equally, lacking an evaluation and weighting mechanism for the reliability of information from each modality, leading to low-quality data affecting the fusion results. Existing fusion strategies, such as the time-frequency characteristics of audio signals, may have a spatial correspondence with local abnormal hotspots in infrared images, but traditional methods cannot use the features of one modality to guide and optimize the feature extraction process of another modality. Different modal data contain both common information reflecting the same device status and unique supplementary information. Existing technologies struggle to reliably separate and utilize these two characteristics, leading to information redundancy or loss of key information, limiting further improvements in multimodal fusion performance. Summary of the Invention

[0004] This invention proposes an auxiliary monitoring system for power equipment based on multimodal data fusion, which addresses the shortcomings of existing technologies in reflecting the mechanical or overall thermal state of equipment and lacks an assessment and weighting mechanism for the reliability of information from each modality. The system includes the following modules:

[0005] The acquisition module is used to acquire infrared image data, sound signal data, and partial discharge signal data of the power equipment to be monitored. The generation module is used to extract the infrared image data, sound signal data, and partial discharge signal data respectively to obtain infrared modal features, sound modal features, and partial discharge modal features; wherein, the process of generating the infrared modal features includes: generating sound modal reliability weights based on the reliability index of the sound signal data, generating a spatial attention mask based on the time-frequency characteristics of the sound signal data, adjusting the spatial attention mask using the sound modal reliability weights, and applying the adjusted spatial attention mask to the feature extraction process of the infrared image data; The input module is used to input the infrared modal features, acoustic modal features, and partial discharge modal features to a shared encoder and multiple private encoders, and separate the modal common features and modal specific features corresponding to each modal feature; modal invariance constraints are applied to the modal common features, and difference constraints are applied between the modal specific features; A construction module is used to construct multimodal reliability weights based on the reliability indices of the infrared image data, sound signal data, and partial discharge signal data; to use the multimodal reliability weights to weight and adjust the modal-specific features; to input the modal common features and the weighted modal-specific features into a time-series information fusion module to obtain a fused state feature sequence; and to determine the operating state of the power equipment based on the fused state feature sequence.

[0006] Optionally, generating sound modal reliability weights based on the reliability index of the sound signal data includes: The signal-to-noise ratio of the audio signal data is calculated as the reliability index. The signal-to-noise ratio is input into the activation function to obtain a value normalized to a preset range [α,β], which is then used as the reliability weight of the sound mode.

[0007] Optionally, generating a spatial attention mask based on the time-frequency characteristics of the sound signal data includes: Perform time-frequency analysis on the sound signal data to obtain the time spectrum of the sound signal; The temporal spectrum of the sound signal is input into a feature extraction network to extract low-dimensional spatial codes. The low-dimensional spatial encoding is upsampled and reshaped into a matrix with the same spatial dimension as the feature map of the infrared image data, which serves as the spatial attention mask.

[0008] Optionally, the step of adjusting the spatial attention mask using the sound modality reliability weights and applying the adjusted spatial attention mask to the feature extraction process of the infrared image data includes: The spatial attention mask is multiplied by the sound modality reliability weight to obtain the adjusted spatial attention mask. When using a feature extraction network to extract features from the infrared image data, the adjusted spatial attention mask is multiplied element-wise with the feature map output by the intermediate layer of the feature extraction network.

[0009] Optionally, applying mode invariance constraints to the modal common features includes: An adversarial training strategy is adopted, and a modality discriminator is used to classify the common modal features. The modality discriminator is trained to maximize modality classification accuracy, while the shared encoder is trained to minimize the modality classification accuracy of the modality discriminator.

[0010] Optionally, the imposition of differential constraints on the modality-specific features includes: The unique features of different modalities are combined in pairs; Calculate the cosine similarity or other similarity measures between the features of each combination; Minimize the similarity metric during model training to enhance the differences between modality-specific features.

[0011] Optionally, the process of constructing multimodal reliability weights based on the respective reliability indices of the infrared image data, sound signal data, and partial discharge signal data includes: The quality assessment indices for the infrared image data, sound signal data, and partial discharge signal data are calculated separately as their respective reliability indices. ,in ; Reliability indicators for each mode After standardization, a reliability vector is formed; The reliability vector is processed using the Softmax function to obtain the reliability weights corresponding to each mode. The calculation formula is:

[0012] All weights Together they constitute the multimodal reliability weights.

[0013] Optionally, the multimodal reliability weights are used to weight and adjust the modality-specific features; the modality-shared features and the weighted modality-specific features are input into the time-series information fusion module to obtain a fused state feature sequence, including: Each weight value in the multimodal reliability weight is multiplied by its corresponding modality-specific feature to obtain the weighted modality-specific features. The modality-shared features are concatenated with all weighted modality-specific features along the feature dimension to form a combined feature; The combined features are input into a long short-term memory network or other temporal processing network, and the output of the network is used as the fused state feature sequence.

[0014] Furthermore, this invention also relates to an auxiliary monitoring method for power equipment based on multimodal data fusion, comprising the following steps: Acquire infrared image data, sound signal data, and partial discharge signal data of the power equipment to be monitored; The infrared image data, sound signal data, and partial discharge signal data are extracted respectively to obtain infrared modal features, sound modal features, and partial discharge modal features. The process of generating the infrared modal features includes: generating sound modal reliability weights based on the reliability index of the sound signal data; generating a spatial attention mask based on the time-frequency characteristics of the sound signal data; adjusting the spatial attention mask using the sound modal reliability weights; and applying the adjusted spatial attention mask to the feature extraction process of the infrared image data. The infrared modal features, acoustic modal features, and partial discharge modal features are input to a shared encoder and multiple private encoders to separate the common modal features and the modality-specific features corresponding to each modal feature. Modality invariance constraints are applied to the common modal features, and difference constraints are applied to the modality-specific features. Based on the reliability indices of the infrared image data, sound signal data, and partial discharge signal data, a multimodal reliability weight is constructed; the multimodal reliability weight is used to weight and adjust the modal-specific features; the modal common features and the weighted modal-specific features are input into the time-series information fusion module to obtain a fused state feature sequence; based on the fused state feature sequence, the operating state of the power equipment is determined.

[0015] Preferably, generating sound modal reliability weights based on the reliability index of the sound signal data includes: The signal-to-noise ratio of the audio signal data is calculated as the reliability index. The signal-to-noise ratio is input into the activation function to obtain a value normalized to a preset range [α,β], which is then used as the reliability weight of the sound mode.

[0016] Preferably, generating a spatial attention mask based on the time-frequency characteristics of the sound signal data includes: Perform time-frequency analysis on the sound signal data to obtain the time spectrum of the sound signal; The temporal spectrum of the sound signal is input into a feature extraction network to extract low-dimensional spatial codes. The low-dimensional spatial encoding is upsampled and reshaped into a matrix with the same spatial dimension as the feature map of the infrared image data, which serves as the spatial attention mask.

[0017] Preferably, the step of adjusting the spatial attention mask using the sound modality reliability weights and applying the adjusted spatial attention mask to the feature extraction process of the infrared image data includes: The spatial attention mask is multiplied by the sound modality reliability weight to obtain the adjusted spatial attention mask. When using a feature extraction network to extract features from the infrared image data, the adjusted spatial attention mask is multiplied element-wise with the feature map output by the intermediate layer of the feature extraction network.

[0018] Preferably, applying mode invariance constraints to the modal common features includes: An adversarial training strategy is adopted, and a modality discriminator is used to classify the common modal features. The modality discriminator is trained to maximize modality classification accuracy, while the shared encoder is trained to minimize the modality classification accuracy of the modality discriminator.

[0019] Preferably, the imposition of differential constraints on the modality-specific features includes: The unique features of different modalities are combined in pairs; Calculate the cosine similarity or other similarity measures between the features of each combination; Minimize the similarity metric during model training to enhance the differences between modality-specific features.

[0020] Preferably, the process of constructing multimodal reliability weights based on the reliability indices of the infrared image data, sound signal data, and partial discharge signal data includes: The quality assessment indices for the infrared image data, sound signal data, and partial discharge signal data are calculated separately as their respective reliability indices. ,in ; Reliability indicators for each mode After standardization, a reliability vector is formed; The reliability vector is processed using the Softmax function to obtain the reliability weights corresponding to each mode. The calculation formula is:

[0021] All weights Together they constitute the multimodal reliability weights.

[0022] Preferably, the multimodal reliability weights are used to weight and adjust the modal-specific features; the modal common features and the weighted modal-specific features are input to the time-series information fusion module to obtain a fused state feature sequence, including: Each weight value in the multimodal reliability weight is multiplied by its corresponding modality-specific feature to obtain the weighted modality-specific features. The modality-shared features are concatenated with all weighted modality-specific features along the feature dimension to form a combined feature; The combined features are input into a long short-term memory network or other temporal processing network, and the output of the network is used as the fused state feature sequence.

[0023] This invention improves the performance of power equipment condition monitoring through a strategy of deep fusion of multimodal information. It utilizes the time-frequency characteristics of sound signals to guide feature extraction of key areas in infrared images, establishing correlations between cross-modal information and reliably detecting subtle thermal anomalies related to faults. By separating and reconstructing common and unique features across modalities, it achieves deep analysis and complementarity of multi-source information. This allows for cross-verification using redundant information from multiple modalities while retaining unique fault indication information for each modality, thus constructing a more comprehensive profile of the equipment's condition. In the fusion decision-making stage, reliability evaluations of each modal data are input, assigning different weights to information of varying quality to suppress the impact of distortion caused by interference from single sensor data on the overall judgment result. By fully exploring the synergistic and complementary value among multimodal data, the operational status judgment results are made more reliable. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of multimodal data acquisition. Figure 2 A schematic diagram of infrared feature enhancement guided by sound; Figure 3 A schematic diagram showing the separation of common features and unique features; Figure 4 This is a schematic diagram of adversarial training for modal invariance constraints. Figure 5 A schematic diagram is provided to illustrate the equipment's operating status; Figure 6 A schematic diagram for constructing multimodal reliability weights. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In the first embodiment, the present invention proposes an auxiliary monitoring system for power equipment based on multimodal data fusion, comprising the following modules: S1, Acquisition module, used to acquire infrared image data, sound signal data and partial discharge signal data of the power equipment to be monitored; By pre-installing infrared thermal imagers, ultrasonic microphone arrays, and transient ground voltage sensors at key locations of the power equipment to be monitored, such as transformers or switchgear, infrared thermal image sequences, multi-channel sound waveform signals, and partial discharge pulse signals are collected synchronously and periodically. Figure 1 .

[0027] S2, the generation module, is used to extract the infrared image data, sound signal data, and partial discharge signal data respectively to obtain infrared modal features, sound modal features, and partial discharge modal features; wherein, the process of generating the infrared modal features includes: generating sound modal reliability weights based on the reliability index of the sound signal data, generating a spatial attention mask based on the time-frequency characteristics of the sound signal data, adjusting the spatial attention mask using the sound modal reliability weights, and applying the adjusted spatial attention mask to the feature extraction process of the infrared image data; A convolutional neural network (CNN), such as ResNet, is used to process infrared images. Another CNN processes the time-spectrum image generated from the short-time Fourier transform of the sound signal. A third CNN processes the phase-resolved partial discharge spectrum generated from the partial discharge signal, thus initially extracting the features of each mode. For the generation of infrared mode features, the signal-to-noise ratio (SNR) of the sound signal is calculated as a reliability index for that infrared mode feature, and this index is mapped to a reliability weight between 0 and 1 using a Sigmoid function. Simultaneously, using the sound signal received by the microphone array, the direction of the sound source in three-dimensional space is calculated using a beamforming algorithm, and this direction is projected onto the two-dimensional plane of the infrared image. A two-dimensional Gaussian distribution map is generated with the projection point as the center as a spatial attention mask. The sound mode reliability weights are multiplied element-wise with this spatial attention mask to obtain an adjusted mask. In an intermediate layer of the CNN extracting infrared image features, the feature map of this layer is multiplied with the adjusted spatial attention mask, so that the network pays more attention to the infrared features of the physical region indicated by the reliable sound signal, such as... Figure 2 .

[0028] In an optional embodiment, generating sound modal reliability weights based on the reliability index of the sound signal data includes: The signal-to-noise ratio of the audio signal data is calculated as the reliability index. The signal-to-noise ratio is input into the activation function to obtain a value normalized to a preset range. The value is used as the reliability weight of the sound mode.

[0029] Acquire a raw audio signal, such as a 1-second audio waveform. To calculate the signal-to-noise ratio (SNR), it's necessary to distinguish between the signal and noise segments. A simple approach is to consider the initial portion of the signal, such as the first 100ms, as background noise, and the remaining portion as the signal. Calculate the average power of each segment separately. and the average power of noise The signal-to-noise ratio (SNR) is calculated using the formula... The signal-to-noise ratio (SNR) is calculated. For example, if the signal power is 100 times the noise power, the calculated SNR is 20 dB, which is the reliability index of the audio signal.

[0030] To map the signal-to-noise ratio to a standardized weight range, such as [0.1, 1.0], an activation function is used for transformation. A commonly used activation function is the Sigmoid function or its variants. For example, a scaled and translated Sigmoid function can be used. ,in and These are the lower and upper limits of the preset interval, namely 0.1 and 1.0, respectively. k and c are hyperparameters that adjust the shape of the function. If the input signal-to-noise ratio is high at 20dB, the output after processing by this function may be 0.98; if the input signal-to-noise ratio is low at -5dB, the output may be closer to 0.1. The output of 0.98 is the reliability weight of the sound mode, representing the quality of the current sound signal.

[0031] In an optional embodiment, generating a spatial attention mask based on the time-frequency characteristics of the sound signal data includes: Perform time-frequency analysis on the sound signal data to obtain the time spectrum of the sound signal; The temporal spectrum of the sound signal is input into a feature extraction network to extract low-dimensional spatial codes. The low-dimensional spatial encoding is upsampled and reshaped into a matrix with the same spatial dimension as the feature map of the infrared image data, which serves as the spatial attention mask.

[0032] Time-frequency analysis of input audio signals typically employs the Short-Time Fourier Transform (STFT). For example, a 16kHz audio signal is segmented using a Hanning window with 1024 sampling points and 512 overlapping sampling points. A Fast Fourier Transform (FFT) is applied to each segment, and the amplitude is calculated to generate a two-dimensional matrix, i.e., the time-frequency spectrum of the audio signal. The dimensions of this time-frequency spectrum, such as 513 frequency channels multiplied by 64 time frames, illustrate the distribution of audio signal energy over time and frequency.

[0033] The time-spectrum of the sound is fed into a feature extraction network. This network can be a lightweight convolutional neural network (CNN), with a structure including two convolutional layers, activation function layers, and pooling layers. For example, the first layer might use 16 3×3 convolutional kernels, and the second layer 32 3×3 convolutional kernels. After processing by this network, the high-dimensional information of the time-spectrum is compressed and extracted into a low-dimensional feature vector, such as a 128-dimensional vector. This vector represents the low-dimensional spatial encoding of the sound signal, including key time-frequency information of the sound event. To generate a spatial attention mask aligned with the feature map of the infrared image, the 128-dimensional vector needs to be upsampled and reshaped. Assuming the feature map spatial dimension of the infrared image at a certain intermediate layer is 14×14, a fully connected layer maps the 128-dimensional vector to 196 dimensions, i.e., 14×14. Then, the 196-dimensional vector is reshaped into a 14×14 matrix. This matrix is ​​the spatial attention mask, where each element represents the degree of attention indicated by the sound signal at the corresponding spatial location.

[0034] The sound feature extraction network is a lightweight convolutional neural network, comprising an input layer, a convolutional layer with 16 3×3 kernels followed by a ReLU activation function, a 2×2 max-pooling layer, a convolutional layer with 32 3×3 kernels followed by a ReLU activation function, a global average pooling layer, and a fully connected layer with an output dimension of 128. The network input is the temporal spectrum matrix of the sound signal, for example, a 513×64 matrix. The network output is a 128-dimensional low-dimensional spatial encoding vector. The training set is the entire multimodal dataset, containing pairs of multimodal data and corresponding device state labels. The network is trained end-to-end as part of the overall model, with weights updated via backpropagation and using the Adam optimizer based on the overall model's total loss function.

[0035] In an optional embodiment, the step of adjusting the spatial attention mask using the acoustic modality reliability weights and applying the adjusted spatial attention mask to the feature extraction process of the infrared image data includes: The spatial attention mask is multiplied by the sound modality reliability weight to obtain the adjusted spatial attention mask. When using a feature extraction network to extract features from the infrared image data, the adjusted spatial attention mask is multiplied element-wise with the feature map output by the intermediate layer of the feature extraction network.

[0036] The system receives two inputs: the generated spatial attention mask (a matrix, for example, 14×14) and the reliability weights of the sound modes (a scalar, for example, 0.98). The adjustment operation is performed by multiplying these scalar weights by the attention mask matrix, multiplying each element by 0.98. This operation yields an adjusted spatial attention mask. If the sound signal quality is good and the reliability weights are high (close to 1), the mask remains essentially unchanged; conversely, if the sound signal is noisy and the reliability weights are low (e.g., 0.2), the entire mask value is weakened, thus reducing its impact on subsequent processes.

[0037] In the feature extraction process of infrared images, an adjusted spatial attention mask is used to guide the network to focus on specific regions. An infrared image feature extraction network, such as a ResNet model, generates a feature map at an intermediate stage of the processing flow. Assume this feature map has dimensions of 14×14×256, where 14×14 is the spatial dimension and 256 is the number of channels. To apply attention, the adjusted 14×14 attention mask is broadcast along the channel dimension, copied 256 times, making the dimension 14×14×256, exactly the same as the feature map. The broadcast attention mask is then element-wise multiplied with the feature map. This operation enhances the feature response of regions in the feature map with higher mask values ​​and suppresses the feature response of regions with lower mask values. The attention-weighted feature map is then fed into subsequent layers of the network for further processing.

[0038] The infrared image feature extraction network is structured as a ResNet-50-based convolutional neural network, pre-trained on the ImageNet dataset. The structure consists of a series of stacked residual blocks, each containing multiple convolutional layers, batch normalization layers, and ReLU activation functions, with a cross-layer connection. The network input is a pre-processed infrared image, such as a three-channel image resized to 224×224 pixels. The network output is the feature maps of the intermediate layers, such as a tensor with dimensions of 14×14×256. The training set is an infrared image dataset specifically designed for device status monitoring, with each image associated with a status label. Training involves fine-tuning the pre-trained weights using a small learning rate and updating the network parameters via backpropagation to minimize the overall model's loss function.

[0039] In an optional embodiment, the partial discharge signal is preprocessed to generate a 128×360 single-channel PRPD map, which is then input into a feature extraction network to obtain partial discharge modal features. The network structure is as follows: after the input layer, two convolutional blocks are connected in sequence, each containing 16 / 32 3×3 convolutional kernels, a batch normalization layer and a ReLU activation function, a 2×2 max pooling layer, a global average pooling layer, and finally, a 64-dimensional fully connected layer to output the partial discharge modal feature vector. During training, an end-to-end approach is adopted. The training set contains PRPD maps labeled with device status, which are randomly cropped and rotated for enhancement. The model is initialized with a He normal distribution, using the Adam optimizer with a learning rate of 1e-4. The loss function combines cross-entropy loss and inter-modal consistency constraint loss. Iterative training is performed until the classification accuracy on the validation set converges.

[0040] S3, the input module, is used to input the infrared modal features, acoustic modal features, and partial discharge modal features to a shared encoder and multiple private encoders, to separate the modal common features and the modal specific features corresponding to each modal feature; to apply modal invariance constraints to the modal common features, and to apply difference constraints between the modal specific features; The three modal features obtained above are input into a shared encoder network and three independent private encoder networks, respectively. The encoders can be implemented using a multilayer perceptron or Transformer architecture. The output of the shared encoder represents the modal common features, while the outputs of the three private encoders are defined as modal-specific features for infrared, sound, and partial discharge, respectively. During model training, the maximum mean difference (MMD) loss between any two pairs of common features generated by different modalities is calculated and minimized in the total loss function. This achieves modal invariance constraints, forcing the shared encoder to learn cross-modal general information. Simultaneously, the cosine similarity between any two modal-specific feature vectors is calculated, and the square of this similarity is added to the total loss function as a difference loss. Minimizing this loss encourages each private encoder to learn orthogonal or differentiated modal-specific information, such as... Figure 3 .

[0041] In an optional embodiment, applying mode invariance constraints to the modal common features includes: An adversarial training strategy is adopted, and a modality discriminator is used to classify the common modal features. The modality discriminator is trained to maximize modality classification accuracy, while the shared encoder is trained to minimize the modality classification accuracy of the modality discriminator.

[0042] The model structure includes a shared encoder for all modes and a dedicated mode discriminator. The shared encoder's task is to extract a common feature vector from inputs of any mode, whether it be an infrared image, a sound signal, or a partial discharge signal. The modality discriminator is a classification network, such as a multilayer perceptron containing two fully connected layers and a softmax output layer. Its task is to receive feature vectors. And determine which mode it originally originated from.

[0043] The modality discriminator is trained to classify modalities as accurately as possible. Network parameters are updated by minimizing the cross-entropy loss of the classification, aiming to improve recognition accuracy. The shared encoder, on the other hand, is trained to deceive the modality discriminator. A gradient inversion layer is typically inserted between the shared encoder and the modality discriminator. During backpropagation, this layer inverts the sign of the gradient as it propagates back from the discriminator's loss. Therefore, the shared encoder updates its parameters to maximize the discriminator's classification loss, making the discriminator unable to distinguish features. The source of this is that, through multiple rounds of adversarial training, the shared encoder learns to extract essential features that do not contain any modality-specific information and are common across all modalities, thereby achieving modality invariance.

[0044] The modality discriminator is a multilayer perceptron neural network, comprising an input layer, a fully connected hidden layer with 256 neurons and a LeakyReLU activation function, a fully connected hidden layer with 128 neurons and a LeakyReLU activation function, and a fully connected output layer with an output dimension equal to the number of modality categories, followed by a Softmax activation function. The input to this network is a shared modality feature vector generated by a shared encoder. The output is a probability distribution vector, representing the probability that the input features belong to each modality. The shared encoder is also a neural network, its structure adapted to different modal inputs, mapping different modal inputs to the same feature space. The two networks are trained adversarially using a training set containing data from all modalities. The training process alternates: with the shared encoder parameters fixed, the modality discriminator is trained by minimizing the cross-entropy loss; with the modality discriminator parameters fixed, the shared encoder is trained by maximizing the cross-entropy loss through a gradient inversion layer, as shown below. Figure 4 .

[0045] In an optional embodiment, imposing differential constraints on the modality-specific features includes: The unique features of different modalities are combined in pairs; Calculate the cosine similarity or other similarity measures between the features of each combination; Minimize the similarity metric during model training to enhance the differences between modality-specific features.

[0046] The model assigns a unique encoder to each mode to extract information specific to that mode. Assuming there are three modes—infrared, acoustic, and partial discharge—each receives its own unique feature vector after passing through its encoder. , and To ensure that the features are indeed mode-specific, differential constraints are imposed on them. These unique features are then paired to form three combinations: infrared and sound, infrared and partial discharge, and sound and partial discharge.

[0047] For each feature pair, calculate the similarity between them. A commonly used metric is cosine similarity, with results ranging from -1 to 1. Values ​​closer to 0 indicate that the two vectors are more orthogonal or uncorrelated. For example, calculate the cosine similarity between infrared and sound-specific features. Combine all the similarity measures of the pairs into a single difference loss term; for example, you could take the sum of the absolute values ​​of the similarities of each pair. Throughout the training process of the model, the differential loss... This loss is minimized as part of the overall loss function. By minimizing this loss, the model parameters are adjusted to force the feature vectors generated by the modality-specific encoders to be as far apart as possible in the feature space, becoming as orthogonal as possible, thereby ensuring that each modality-specific feature detects non-redundant modal information.

[0048] Each modality-specific encoder is a dedicated neural network designed to match the characteristics of the corresponding modality data. The infrared modality-specific encoder employs a small two-dimensional convolutional neural network (CNN). Its input layer receives a 14×14×256-dimensional infrared basic feature map output from the S2 module, which is then sequentially connected to a convolutional layer with 64 3×3 kernels, a batch normalization layer, a ReLU activation function, and a 2×2 max-pooling layer. This is followed by a convolutional layer with 128 3×3 kernels, a batch normalization layer, a ReLU activation function, and a 2×2 max-pooling layer. The large pooling layer compresses the 3×3×128-dimensional features into a 128-dimensional vector through a global average pooling layer. Finally, it outputs 256-dimensional infrared-specific features through a 256-dimensional fully connected layer and a dropout layer with a dropout rate of 0.5. The acoustic modality-specific encoder is a two-dimensional lightweight CNN adapted to the temporal spectrum of sound, taking 16×16×64-dimensional basic acoustic features as input. Its structure is similar to that of the infrared encoder, only adjusting the input dimension to adapt to the temporal spectrum characteristics. The partial discharge modality-specific encoder uses a one-dimensional CNN, with pre-processed data as input. The processed 1×1024-dimensional partial discharge time-series signal is sequentially passed through 64 convolutional layers with 1×3 kernels, batch normalization layers, ReLU activation functions, and one-dimensional max pooling layers with kernel_size=2. Then, it passes through 128 convolutional layers with 1×3 kernels, batch normalization layers, ReLU activation functions, and one-dimensional max pooling layers with kernel_size=2. Finally, it undergoes global average pooling and a 256-dimensional fully connected layer to output 256-dimensional partial discharge-specific features. The input to each encoder is the basic features processed for the corresponding modality, and the output is a feature vector specific to each modality. During the training phase, a dataset containing labeled data for all modalities is used. Each specific encoder is trained end-to-end as part of the overall model. The training objective is to minimize a composite loss function that includes the main task loss and the difference loss. The Adam optimizer is used as the optimization strategy, with a learning rate of 1e-4. The network parameters are iteratively updated using the backpropagation algorithm, thereby ensuring low similarity between the specific feature vectors while completing the main task.

[0049] S4, a construction module, is used to construct multimodal reliability weights based on the reliability indices of the infrared image data, sound signal data, and partial discharge signal data; to use the multimodal reliability weights to weight and adjust the modal-specific features; to input the modal common features and the weighted modal-specific features into the time-series information fusion module to obtain a fused state feature sequence; and to determine the operating state of the power equipment based on the fused state feature sequence.

[0050] The average gradient or sharpness of the infrared image, the signal-to-noise ratio of the sound signal, and the pulse peak signal-to-noise ratio of the partial discharge signal are calculated separately as reliability indicators for each of the three modalities. These three indicators are combined into a three-dimensional vector and input into a Softmax function to generate a set of multimodal reliability weights that sum to 1. This set of weights is then multiplied by the three modality-specific feature vectors obtained in the previous step to achieve weighted adjustment of the specific features. At each time step, the common modal features and the three weighted modality-specific features are concatenated along the feature dimension to form a unified pre-fusion feature vector. This chronologically ordered pre-fusion feature vector sequence is input into a Long Short-Term Memory (LSTM) network. The LSTM network detects and fuses the time-varying patterns of these features through its internal gating units. The hidden layer outputs at each time step constitute the fused state feature sequence.

[0051] The output vector of the last time step of the fused state feature sequence output by the time-series information fusion module (i.e., the LSTM network) is taken and input into a classifier consisting of several fully connected layers. After passing through a Softmax activation function, the output vector determines the probability of the device being in various predefined operating states, such as normal, moderately defective, and severely defective. (See [link to relevant documentation]). Figure 5 The state with the highest probability value is selected as the monitoring and diagnostic result.

[0052] In an optional embodiment, the process of constructing multimodal reliability weights based on the respective reliability indices of the infrared image data, sound signal data, and partial discharge signal data includes: The quality assessment indices for the infrared image data, sound signal data, and partial discharge signal data are calculated separately as their respective reliability indices. ,in ; Reliability indicators for each mode After standardization, a reliability vector is formed; The reliability vector is processed using the Softmax function to obtain the reliability weights corresponding to each mode. The calculation formula is:

[0053] All weights Together they constitute the multimodal reliability weights.

[0054] A quality assessment score is calculated for the input data of each modality as a reliability metric. For example, for infrared image data, image sharpness can be used as the metric, represented by the variance of the image's Laplacian operator. A high variance value, such as 200.0, indicates a sharp image and a high reliability metric. That is, 200.0. For audio signal data, the signal-to-noise ratio (SNR) mentioned above can be used as an indicator. A higher SNR value, such as 18.0 dB, represents good audio signal quality and reliability. That is, 18.0. For partial discharge signal data, the kurtosis of the signal can be used as an indicator. A kurtosis value such as 9.5 indicates that there are prominent pulse spikes in the signal, indicating high reliability. This is a reliability indicator. That is, 9.5.

[0055] After obtaining the reliability indices for each modality, they are standardized and combined into a reliability vector, for example, [0.8, 0.2, 0.6]. The vector is then normalized using the Softmax function, transforming it into a probability distribution with a sum of 1, i.e., the reliability weights of each modality. According to the formula... For the vector [0.8, 0.2, 0.6], the calculated weight vector is [0.41, 0.18, 0.41]. It can be seen that the weights of the lower-quality sound modalities are reduced. These weight vectors are the multimodal reliability weights used for subsequent feature fusion. Figure 6 .

[0056] In an optional embodiment, the multimodal reliability weights are used to weight and adjust the modality-specific features; the modality-shared features and the weighted modality-specific features are input to the time-series information fusion module to obtain a fused state feature sequence, including: Each weight value in the multimodal reliability weight is multiplied by its corresponding modality-specific feature to obtain the weighted modality-specific features. The modality-shared features are concatenated with all weighted modality-specific features along the feature dimension to form a combined feature; The combined features are input into a long short-term memory network or other temporal processing network, and the output of the network is used as the fused state feature sequence.

[0057] Assume the model has extracted modal common features of dimension 512. The infrared, acoustic, and partial discharge characteristics, each with dimensions of 256, are unique. , and And a multimodal reliability weight vector, such as [0.8, 0.15, 0.05]. The weighting adjustment step involves performing a scalar multiplication of each weight value with its corresponding idiomatic feature vector. For example, the weighted infrared idiomatic feature is... The weighted sound features are This operation adjusts the contribution of each modality-specific information to the decision, amplifying the unique information of high-quality modalities and suppressing that of low-quality modalities.

[0058] After weighted adjustment, the modality-shared features and all weighted modality-specific features are concatenated along the feature dimension, also known as concatenation. This results in a longer combined feature vector. The dimension is the sum of all component dimensions, i.e., 1280. The combined feature vector integrates common information between modalities and reliability-adjusted individual information at a single time point. For a time series dataset, the above process is repeated at each time step to obtain a sequence of combined feature vectors. This feature sequence is input to a time-series information fusion module, such as a Long Short-Term Memory (LSTM) network. This LSTM network processes the sequence data through a gating mechanism to detect the dependencies of features over time. The hidden state of the last time step of the LSTM, or the sequence of hidden states from all time steps, is considered the fused state feature sequence, used for subsequent device status diagnosis or prediction tasks.

[0059] The preferred structure of the Long Short-Term Memory (LSTM) network is a network containing two stacked LSTM layers. The hidden state dimension of each LSTM layer is 512. A dropout layer with a dropout rate of 0.5 is applied between the two layers to prevent overfitting. The input to the network is a feature sequence with a dimension of T x 1280, where T is the sequence length and 1280 is the dimension of the combined features at each time step. The output of the network is the hidden state vector of the last time step, with a dimension of 512, representing the fused temporal information of the entire input sequence. The training set for the network consists of time-varying serialized multimodal data, with each sequence corresponding to a device state label. As part of the overall model, the LSTM is trained using a backpropagation algorithm, and parameter updates are designed to minimize the task's loss function.

[0060] In an optional embodiment, determining the operating state of the power equipment based on the fused state feature sequence includes: The fused feature sequence is input into a classifier, which outputs the probability of the device being in various predefined operating states, such as normal, moderately defective, and severely defective. The state with the highest probability value is selected as the monitoring and diagnostic result.

[0061] The classifier employs a three-layer fully connected neural network structure. The input layer receives a 512-dimensional fused state feature vector from an LSTM network. The first hidden layer contains 256 neurons and uses the ReLU activation function, followed by a Dropout layer with a dropout rate of 0.5 to prevent overfitting. The second hidden layer contains 128 neurons and also uses the ReLU activation function. Finally, the output is passed through an output layer, the number of neurons in which is equal to the predefined number of device operating state categories, and the output value is normalized to a probability distribution using the Softmax activation function. During training, classification cross-entropy is used as the loss function, and the Adam optimizer is used with a learning rate of 1e-4 for end-to-end training until the classification accuracy on the validation set converges. During inference, the input feature vector is fed into the trained classifier to obtain a probability vector, for example, [0.02, 0.15, 0.8, 0.02, 0.01], corresponding to normal, general defect A, severe defect, general defect B, and critical defect, respectively. The severe defect with the highest probability value is selected as the diagnosis result.

[0062] In a second embodiment, the present invention also provides a power equipment auxiliary monitoring method based on multimodal data fusion, comprising the following steps: Acquire infrared image data, sound signal data, and partial discharge signal data of the power equipment to be monitored; The infrared image data, sound signal data, and partial discharge signal data are extracted respectively to obtain infrared modal features, sound modal features, and partial discharge modal features. The process of generating the infrared modal features includes: generating sound modal reliability weights based on the reliability index of the sound signal data; generating a spatial attention mask based on the time-frequency characteristics of the sound signal data; adjusting the spatial attention mask using the sound modal reliability weights; and applying the adjusted spatial attention mask to the feature extraction process of the infrared image data. The infrared modal features, acoustic modal features, and partial discharge modal features are input to a shared encoder and multiple private encoders to separate the common modal features and the modality-specific features corresponding to each modal feature. Modality invariance constraints are applied to the common modal features, and difference constraints are applied to the modality-specific features. Based on the reliability indices of the infrared image data, sound signal data, and partial discharge signal data, a multimodal reliability weight is constructed; the multimodal reliability weight is used to weight and adjust the modal-specific features; the modal common features and the weighted modal-specific features are input into the time-series information fusion module to obtain a fused state feature sequence; based on the fused state feature sequence, the operating state of the power equipment is determined.

[0063] In this specification, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise limited, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the associated listed items.

[0064] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0065] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A power equipment auxiliary monitoring system based on multimodal data fusion, characterized in that, Includes the following modules: The acquisition module is used to acquire infrared image data, sound signal data, and partial discharge signal data of the power equipment to be monitored. The generation module is used to extract the infrared image data, sound signal data, and partial discharge signal data respectively to obtain infrared modal features, sound modal features, and partial discharge modal features; wherein, the process of generating the infrared modal features includes: generating sound modal reliability weights based on the reliability index of the sound signal data, generating a spatial attention mask based on the time-frequency characteristics of the sound signal data, adjusting the spatial attention mask using the sound modal reliability weights, and applying the adjusted spatial attention mask to the feature extraction process of the infrared image data; The input module is used to input the infrared modal features, acoustic modal features, and partial discharge modal features to a shared encoder and multiple private encoders, and separate the modal common features and modal specific features corresponding to each modal feature; modal invariance constraints are applied to the modal common features, and difference constraints are applied between the modal specific features; A construction module is used to construct multimodal reliability weights based on the reliability indices of the infrared image data, sound signal data, and partial discharge signal data; to use the multimodal reliability weights to weight and adjust the modal-specific features; to input the modal common features and the weighted modal-specific features into a time-series information fusion module to obtain a fused state feature sequence; and to determine the operating state of the power equipment based on the fused state feature sequence.

2. The system according to claim 1, characterized in that, The step of generating sound modal reliability weights based on the reliability index of the sound signal data includes: The signal-to-noise ratio of the audio signal data is calculated as the reliability index. The signal-to-noise ratio is input into the activation function to obtain a value normalized to a preset range. The value is used as the reliability weight of the sound mode.

3. The system according to claim 1, characterized in that, The generation of a spatial attention mask based on the time-frequency characteristics of the sound signal data includes: Perform time-frequency analysis on the sound signal data to obtain the time spectrum of the sound signal; The temporal spectrum of the sound signal is input into a feature extraction network to extract low-dimensional spatial codes. The low-dimensional spatial encoding is upsampled and reshaped into a matrix with the same spatial dimension as the feature map of the infrared image data, which serves as the spatial attention mask.

4. The system according to claim 1, characterized in that, The process of adjusting the spatial attention mask using the sound modality reliability weights and applying the adjusted spatial attention mask to the feature extraction of the infrared image data includes: The spatial attention mask is multiplied by the sound modality reliability weight to obtain the adjusted spatial attention mask. When using a feature extraction network to extract features from the infrared image data, the adjusted spatial attention mask is multiplied element-wise with the feature map output by the intermediate layer of the feature extraction network.

5. The system according to any one of claims 1-4, characterized in that, The application of mode invariance constraints to the common modal features includes: An adversarial training strategy is adopted, and a modality discriminator is used to classify the common modal features. The modality discriminator is trained to maximize modality classification accuracy, while the shared encoder is trained to minimize the modality classification accuracy of the modality discriminator.

6. The system according to any one of claims 1-4, characterized in that, The imposition of differential constraints on the modality-specific features includes: The unique features of different modalities are combined in pairs; Calculate the cosine similarity or other similarity measures between the features of each combination; Minimize the similarity metric during model training to enhance the differences between modality-specific features.

7. The system according to any one of claims 1-4, characterized in that, The process of constructing multimodal reliability weights based on the reliability indices of the infrared image data, sound signal data, and partial discharge signal data includes: The quality assessment indices for the infrared image data, sound signal data, and partial discharge signal data are calculated separately as their respective reliability indices. ,in ; Reliability indicators for each mode After standardization, a reliability vector is formed; The reliability vector is processed using the Softmax function to obtain the reliability weights corresponding to each mode. The calculation formula is: All weights Together they constitute the multimodal reliability weights.

8. The system according to any one of claims 1-4, characterized in that, The multimodal reliability weights are used to weight and adjust the modality-specific features; the common modality features and the weighted modality-specific features are input into the time-series information fusion module to obtain a fused state feature sequence, including: Each weight value in the multimodal reliability weight is multiplied by its corresponding modality-specific feature to obtain the weighted modality-specific features. The modality-shared features are concatenated with all weighted modality-specific features along the feature dimension to form a combined feature; The combined features are input into a long short-term memory network or other temporal processing network, and the output of the network is used as the fused state feature sequence.

9. A power equipment auxiliary monitoring method based on multimodal data fusion, characterized in that, Includes the following steps: Acquire infrared image data, sound signal data, and partial discharge signal data of the power equipment to be monitored; The infrared image data, sound signal data, and partial discharge signal data are extracted respectively to obtain infrared modal features, sound modal features, and partial discharge modal features. The process of generating the infrared modal features includes: generating sound modal reliability weights based on the reliability index of the sound signal data; generating a spatial attention mask based on the time-frequency characteristics of the sound signal data; adjusting the spatial attention mask using the sound modal reliability weights; and applying the adjusted spatial attention mask to the feature extraction process of the infrared image data. The infrared modal features, acoustic modal features, and partial discharge modal features are input to a shared encoder and multiple private encoders to separate the common modal features and the modality-specific features corresponding to each modal feature. Modality invariance constraints are applied to the common modal features, and difference constraints are applied to the modality-specific features. Based on the reliability indices of the infrared image data, sound signal data, and partial discharge signal data, a multimodal reliability weight is constructed; the multimodal reliability weight is used to weight and adjust the modal-specific features; the modal common features and the weighted modal-specific features are input into the time-series information fusion module to obtain a fused state feature sequence; based on the fused state feature sequence, the operating state of the power equipment is determined.

10. The method according to claim 9, characterized in that, The step of generating sound modal reliability weights based on the reliability index of the sound signal data includes: The signal-to-noise ratio of the audio signal data is calculated as the reliability index. The signal-to-noise ratio is input into the activation function to obtain a value normalized to a preset range. The value is used as the reliability weight of the sound mode.

Images