A substation equipment defect intelligent identification method based on multi-modal fusion

Abstract

The application discloses a kind of multimodal fusion substation equipment defect intelligent identification method, comprising: visible light image, infrared thermal imaging, local discharge ultrasonic signal, vibration signal and oil chromatogram data are synchronously collected, and time-space alignment and registration are carried out after being preprocessed;Multi-scale features of image are extracted using double-branch network, time-frequency and component features of non-image mode are extracted;Through cross-modal attention, bidirectional enhancement of visible light and infrared features is realized, and dynamic weighting strategy is used to adaptively fuse image features and non-image features;The fusion features are input into the neural network containing thermodynamic, structure and multimodal consistency physical constraints for defect identification;Based on multimodal cross-validation, the comprehensive confidence is calculated and the defect type, location, level and confidence are output.The application significantly improves the accuracy, robustness and explainability of substation equipment defect identification through multimodal complementary perception and physical constraint mechanism.

Description

Technical Field

[0001] This invention relates to the field of technology, and more specifically, to a multimodal fusion method for intelligent identification of defects in substation equipment. Background Technology

[0002] The safe and stable operation of substation equipment is crucial for ensuring reliable power supply to the power system. With the continuous advancement of smart grid construction and the sustained growth in the number and types of substation equipment, traditional maintenance methods relying on manual inspections are no longer sufficient to meet the requirements of real-time performance, accuracy, and safety. Automated intelligent inspection and defect identification technologies have become an important development direction in the field of power equipment operation and maintenance.

[0003] Currently, deep learning and multimodal fusion technologies have received widespread attention and application in substation equipment defect detection. Existing technologies include schemes that extract features from infrared and visible light images in substation scenarios using deep neural networks, establishing target detection networks based on the fusion of infrared and visible light images, and utilizing cross-attention structures to achieve bidirectional feature reorganization under deep multimodal fusion. In addition, some schemes employ joint acquisition of infrared and visible light images, achieving spatiotemporal alignment of multimodal images through registration algorithms that match multi-source features with prior equipment structure, and generating fused features using dynamic weighted fusion strategies. Other research utilizes cross-modal attention mechanisms to fuse infrared images, oil chromatography data, and audio signals for substation fault early warning. These methods have improved the automation level and recognition accuracy of equipment defect detection to a certain extent.

[0004] However, existing technologies still have the following shortcomings: First, existing fusion methods mostly focus on fusing infrared and visible light image modalities, failing to fully utilize important modal data that reflect the internal operating status of equipment, such as partial discharge ultrasonic signals, vibration signals, and oil chromatography data, thus neglecting to leverage the complementary sensing advantages of multi-source modalities. Second, existing feature fusion methods have weak semantic discriminative power, easily leading to feature loss or redundancy during the fusion process. Furthermore, scene structure information in visible light images is easily masked by infrared temperature information, resulting in a decline in the quality of the fused image. Third, existing deep learning methods are mostly purely data-driven, lacking constraints and verification mechanisms for the physical characteristics of the equipment, resulting in insufficient robustness under complex operating conditions and poor interpretability of the detection results. Fourth, multi-modal data collected by different sensors exhibit deviations in the spatiotemporal dimensions, and existing registration methods are still imperfect in combining multi-source features with prior knowledge of equipment structure.

[0005] To address the aforementioned issues, a method for intelligent identification of substation equipment defects needs to be proposed that can fully integrate multiple modal data, possess physical constraint mechanisms, and exhibit good robustness and interpretability. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a multimodal fusion intelligent identification method for substation equipment defects, addressing the above-mentioned deficiencies of the prior art.

[0007] The technical solution adopted by this invention to solve its technical problem is: on the one hand A multimodal fusion-based intelligent identification method for substation equipment defects includes the following steps: S1. Simultaneously acquire visible light image data, infrared thermal imaging data, partial discharge ultrasonic signals, equipment vibration signals, and oil chromatography data of the target substation equipment, and preprocess the acquired data. S2. Perform spatiotemporal alignment and registration on the preprocessed multimodal data, including spatial registration based on multi-source features and prior knowledge of device structure and temporal alignment based on unified timestamps. S3. Extract the features of each modal data respectively. A dual-branch feature extraction network is used and a multi-dimensional cross-connection mechanism is introduced to extract multi-scale features of visible light images and infrared thermal imaging images. The corresponding time-frequency features and component features are extracted for the ultrasonic signal, the vibration signal and the oil chromatography data respectively. S4. Achieve bidirectional information enhancement of visible light features and infrared features through a cross-modal attention module, and adopt a dynamic weighted fusion strategy to adaptively fuse the enhanced image features with non-image modal features to generate a multimodal comprehensive feature vector. S5. Input the multimodal integrated feature vector into a deep neural network that integrates physical priors for defect identification and classification. The physical priors include thermodynamic consistency constraints, structural consistency constraints, and multimodal consistency constraints. S6. Calculate the overall confidence level of the defect identification results based on multimodal cross-validation, and output the defect type, defect location, defect level and overall confidence level.

[0008] Preferably, the synchronous acquisition in S1 specifically includes: acquiring visible light image data through a high-definition visible light camera, acquiring infrared thermal imaging data through an infrared thermal imaging sensor, acquiring partial discharge ultrasonic signals through an ultrasonic sensor, acquiring equipment vibration signals through a vibration sensor, and acquiring oil chromatographic data through an online oil chromatographic monitoring device; each sensor is synchronously triggered to acquire data using a unified timestamp. The preprocessing in S1 includes: denoising, enhancing, and normalizing the visible light image and the infrared thermal imaging image; filtering and denoising the partial discharge ultrasonic signal to remove environmental electromagnetic interference; detrending and filtering the vibration signal; and outlier detection and missing value imputation of the oil chromatographic data.

[0009] Preferably, the spatial registration in S2 specifically includes: obtaining spatial transformation parameters between each sensor through calibration, the spatial transformation parameters including rotation matrix and translation vector; constructing a priori structural map of the device based on the CAD structural model or 3D point cloud data of the device; using the priori structural map of the device to guide feature point matching, mapping the pixel coordinates of the infrared image to the visible light image coordinate system, and achieving sub-pixel level spatial alignment; The time alignment in S2 specifically includes: synchronizing various types of sensor data in the time dimension based on the unified timestamp; and for sensors with different sampling frequencies, using interpolation methods to resample the data to a unified time reference.

[0010] Preferably, step S3 employs a dual-branch feature extraction network and introduces a multi-dimensional cross-connection mechanism to extract multi-scale features from visible light images and infrared thermal imaging images, specifically including: The visible light branch utilizes a deep convolutional neural network to extract texture features, structural features, and scene semantic features from the visible light image, generating a multi-scale visible light feature map. ; The infrared branch uses a deep convolutional neural network to extract temperature distribution and thermal anomaly features from the infrared thermal imaging image, generating a multi-scale infrared feature map. ; A multidimensional cross-connection mechanism is introduced during the feature extraction process to enhance the information transfer between shallow and deep features.

[0011] Preferably, in step S3, the extraction of corresponding time-frequency features and component features from the ultrasonic signal, the vibration signal, and the oil chromatographic data specifically includes: Time-frequency analysis was performed on the partial discharge ultrasonic signal to extract time-domain statistical features and frequency-domain features, and an ultrasonic feature vector was constructed. The time-domain statistical features include peak value, root mean square value, and peak-to-peak value; the frequency-domain features include power spectral density, center frequency, and bandwidth. Empirical mode decomposition (EMD) is performed on the vibration signal to extract the energy and spectral features of the intrinsic mode functions, and a vibration feature vector is constructed. ; The concentration data of dissolved gases in the oil in the oil chromatography data are normalized to construct an oil chromatography feature vector. The dissolved gases in the oil include H2, CH4, C2H2, C2H4, C2H6, CO, and CO2.

[0012] Preferably, in S4, bidirectional information enhancement of visible light and infrared features is achieved through a cross-modal attention module, specifically including: The visible light feature and the infrared feature are input into the cross-modal attention module. The visible light feature is used as the query and the infrared feature is used as the key and value to calculate the cross-modal attention weight and generate infrared-enhanced visible light feature. The infrared feature is used as the query and the visible light feature is used as the key and value to calculate the cross-modal attention weight and generate visible light-enhanced infrared feature. By employing a bidirectional cross-modal attention mechanism, the semantic space alignment of infrared and visible light images is achieved. In step S4, a dynamic weighted fusion strategy is used to adaptively fuse enhanced image features with non-image modality features, specifically including: The visible light features enhanced by cross-modal attention are concatenated with infrared features, and the fusion weights of each feature channel are calculated through an adaptive weight learning network to generate fused image features. ; The fused image features With ultrasound feature vector Vibration characteristic vector Oil chromatographic feature vectors The features are concatenated to form a multimodal comprehensive feature vector. ; The calculation of the fusion weights is dynamically adjusted based on the confidence level of each modal feature under the current input.

[0013] Preferably, the deep neural network in S5 that incorporates physical priors includes a fully connected layer, a batch normalization layer, an activation layer, a Dropout layer, and an output layer; The thermodynamic consistency constraint ensures that the infrared temperature anomaly region remains consistent with the equipment's heating physical model. The structural consistency constraint is that the defect location is within a reasonable area of ​​the equipment's structure. The multimodal consistency constraint is that the judgment results of different modes on the same defect corroborate each other; The deep neural network that integrates physical priors is trained using a joint loss function that includes physical constraint priors. The joint loss function is:

[0014] in, For classifying losses, For the aforementioned thermodynamic consistency loss, The structural consistency loss is... For the aforementioned multimodal consistency loss, These are the corresponding weighting coefficients; The defect types in S5 include equipment overheating, partial discharge, insulation deterioration, abnormal mechanical vibration, and abnormal dissolved gas in oil; the defect levels are divided into four levels according to the severity of the defect: normal, attention, abnormal, and severe.

[0015] Preferably, the comprehensive confidence level of the defect identification results calculated based on multimodal cross-validation in S6 specifically includes: Calculate the confidence level of each single mode for identifying the same defect. ; The overall confidence level C_total is calculated using a multimodal cross-validation mechanism: when the recognition results of multiple modalities are consistent, the overall confidence level increases; when there is a conflict between modalities, the overall confidence level decreases and a manual review mechanism is triggered. The defect locations in S6 are simultaneously marked in the visible light image and the infrared image.

[0016] on the other hand A multimodal fusion intelligent identification system for substation equipment defects includes: A multimodal data acquisition module is used to simultaneously acquire visible light image data, infrared thermal imaging data, partial discharge ultrasonic signals, equipment vibration signals, and oil chromatography data of the target substation equipment; the multimodal data acquisition module is deployed on an intelligent inspection robot or a fixed multi-sensor terminal; The data preprocessing module is used to preprocess various types of collected data; The spatiotemporal alignment and registration module is used to perform spatial registration based on multi-source features and device structure priors, and temporal alignment based on a unified timestamp on preprocessed multimodal data. The multimodal feature extraction module is used to extract features of each modality of data. It employs a dual-branch feature extraction network and introduces a multidimensional cross-connection mechanism to extract multi-scale features of visible light images and infrared thermal imaging images. It also extracts corresponding time-frequency features and component features from the ultrasonic signal, the vibration signal, and the oil chromatography data. The cross-modal attention fusion module is used to enhance the information of visible light features and infrared features in both directions through the cross-modal attention mechanism, and adopts a dynamic weighted fusion strategy to adaptively fuse the enhanced image features with non-image modal features to generate a multimodal comprehensive feature vector. The physical constraint defect identification module is used to input the multimodal integrated feature vector into a deep neural network that fuses physical priors for defect identification and classification. The confidence assessment and output module is used to calculate the overall confidence of the defect identification results based on multimodal cross-validation, and output the defect type, defect location, defect level and overall confidence.

[0017] The beneficial effects of this invention are as follows: 1. This invention breaks through the limitations of traditional single-modal detection by simultaneously acquiring visible light images, infrared thermal imaging, partial discharge ultrasonic signals, equipment vibration signals, and oil chromatography data. This enables comprehensive three-dimensional perception of the appearance, thermodynamic state, insulation discharge, mechanical vibration, and internal chemical composition of substation equipment. Through simultaneous monitoring covering multiple physical dimensions—electrical, mechanical, thermal, and chemical—this method can capture abnormal signals in the early stages of equipment failure (such as the initial stage of partial discharge or slight overheating), rather than relying solely on significant abrupt changes in a single physical quantity. This significantly improves the early warning capability for latent, complex, and interconnected defects, providing a rich and complete data foundation for condition-based maintenance of power equipment.

[0018] 2. Addressing the technical challenge of aligning multi-sensor data across spatial coordinates and temporal scales, this project innovatively introduces the device's CAD structural model or 3D point cloud data as a prior knowledge graph to guide feature point matching. This achieves high-precision spatial mapping from infrared image pixel coordinates to the visible light image coordinate system, reaching sub-pixel-level spatial alignment accuracy. Simultaneously, a time alignment strategy driven by a unified timestamp unifies sensor data from different sampling frequencies to the same time reference through interpolation resampling. This dual guarantee of sub-pixel spatial alignment and temporal synchronization completely resolves the fusion distortion problem caused by errors or spatiotemporal misalignments in multi-source data, ensuring the accuracy and reliability of the input data for subsequent feature extraction and fusion stages.

[0019] 3. A dual-branch feature extraction network was designed for visible light and infrared images, and a multi-dimensional cross-connection mechanism was introduced. The visible light branch focuses on high-resolution texture details, structural contours, and scene semantics, while the infrared branch accurately captures temperature gradient distribution and thermal anomaly regions. Through multi-dimensional cross-transmission of shallow and deep features, the network can retain local details such as fine cracks and stains while also taking into account global temperature field distribution information, significantly enhancing its ability to represent equipment defects at multiple scales. Simultaneously, refined time-frequency analysis (such as power spectral density and intrinsic mode function energy) was performed on ultrasonic and vibration signals, and the seven components of oil chromatography were normalized and encoded. This created a deep complementarity between image appearance features and non-image physical features, solving the problem of insufficient descriptive power of single features under complex operating conditions.

[0020] 4. By constructing a bidirectional cross-modal attention module, visible light features and infrared features can mutually query and enhance each other in the semantic space. Infrared thermal anomalies can guide visible light to focus on potential heat-generating areas, while the clear outlines of visible light can correct edge blurring in infrared images. Furthermore, a dynamic weighted fusion strategy is adopted, adaptively allocating fusion weights based on the confidence level of each modality feature under the current input. When the confidence level of a certain modality decreases due to environmental interference (such as changes in illumination or electromagnetic noise), the system automatically reduces its contribution and enhances the weights of other reliable modalities. This mutual enhancement and adaptive mechanism ensures that the model maintains extremely high recognition stability and generalization ability even under non-ideal conditions such as severe weather and sensor degradation.

[0021] 5. This method deeply couples the powerful fitting capabilities of deep learning with the physical mechanisms of power equipment, explicitly introducing thermodynamic consistency constraints, structural consistency constraints, and multimodal consistency constraints during neural network training. Thermodynamic constraints ensure that identified hotspots follow heat transfer laws, reducing non-physical heating judgments caused by pixel-level statistical bias; structural constraints ensure that defect locations are within reasonable areas of the equipment structure, eliminating false alarms caused by background interference; multimodal constraints require that the judgment results of different physical quantities for the same defect corroborate each other. Combined with multi-objective optimization of the joint loss function, the method produces identification results that are not only accurate but also rigorously supported by physical logic.

[0022] 6. In the final decision output stage, in addition to outputting the defect category, specific location, and severity level, the system innovatively calculates the overall confidence level through a multimodal cross-validation mechanism. When the identification conclusions of each single modality, such as visible light, infrared, ultrasound, vibration, and oil chromatography, are highly consistent, the overall confidence level increases accordingly, enhancing the maintenance personnel's trust in the diagnostic results. When there are significant conflicts or dispersed votes between modalities, the system automatically lowers the confidence level and triggers manual review or secondary in-depth detection warnings. This avoids invalid alarms caused by misjudgments of a single modality and provides a scientific and reliable probabilistic basis for maintenance decisions, greatly improving the fault tolerance and security of field applications.

[0023] 7. The entire identification process is highly integrated, allowing for convenient deployment on intelligent inspection robots or fixed multi-sensor terminals, enabling fully automated, non-contact, and uninterrupted online real-time monitoring in substation scenarios. Through fully automated data processing and intelligent diagnostics, the tedious manual operation of carrying multiple instruments for individual testing is significantly reduced, substantially lowering the workload of maintenance personnel and the risk of high-voltage electric shock. Simultaneously, standardized data flow and decision output processes ensure the consistency and comparability of diagnostic results across different sites and equipment, providing core technological support for the large-scale application of digital transformation in power systems and the comprehensive construction of smart substations. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a flowchart illustrating the steps of the multimodal fusion-based intelligent identification method for substation equipment defects according to an embodiment of this application.

[0025] Figure 2 This is a schematic diagram of the structure of the multimodal fusion intelligent identification system for substation equipment defects according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0027] Example 1 Preferred embodiments of the present invention, for example Figure 1 As shown, a multimodal fusion-based intelligent identification method for substation equipment defects includes the following steps: S1. Simultaneously acquire visible light image data, infrared thermal imaging data, partial discharge ultrasonic signals, equipment vibration signals, and oil chromatography data of the target substation equipment, and preprocess the acquired data.

[0028] Synchronous data acquisition specifically includes: acquiring visible light image data using a high-definition visible light camera with a resolution of no less than 1920×1080 pixels and a frame rate of no less than 30fps; acquiring infrared thermal imaging data using an infrared thermal imaging sensor with a spectral response range of 8μm~14μm, a temperature measurement accuracy of no less than ±2°C or ±2% (whichever is greater), and a resolution of no less than 640×512 pixels; acquiring partial discharge ultrasonic signals using an ultrasonic sensor with a frequency response range of 20kHz~300kHz and a sampling rate of no less than 5MHz; acquiring equipment vibration signals using a vibration sensor, specifically a piezoelectric accelerometer with a frequency response range of 0.5Hz~10kHz and a sampling rate of no less than 25.6kHz; and acquiring oil chromatographic data using an online oil chromatographic monitoring device with a detection limit of no less than 0.1μL / L. All sensors are synchronously triggered for acquisition using a unified timestamp (based on the IEEE 1588 precise time protocol or a GPS / BeiDou time synchronization module), with a time synchronization accuracy better than 1ms.

[0029] Preprocessing includes: denoising visible light and infrared thermal images (using nonlocal mean filtering or bilateral filtering algorithms), enhancement (using adaptive histogram equalization or contrast-limited adaptive histogram equalization algorithms), and size normalization (bicubic interpolation resampling to a uniform size, such as 640×480 pixels) for visible light and infrared thermal images; filtering and denoising partial discharge ultrasonic signals (using bandpass filtering combined with wavelet thresholding to remove environmental electromagnetic interference and power frequency harmonics); detrending vibration signals (using polynomial fitting to remove baseline drift) and filtering (using bandpass filtering to retain effective frequency bands); and outlier detection (using a detection method based on local outliers) and missing value imputation (using multiple interpolation or K-nearest neighbor imputation methods) for oil chromatographic data.

[0030] S2. Perform spatiotemporal alignment and registration on the preprocessed multimodal data, including spatial registration based on multi-source features and prior knowledge of device structure, and temporal alignment based on unified timestamps.

[0031] Spatial registration specifically includes: obtaining spatial transformation parameters between each sensor through calibration, including rotation matrices and translation vectors; for a binocular optical system with an infrared camera and a visible light camera, the Zhang Zhengyou checkerboard calibration method is used to obtain the intrinsic and extrinsic parameter matrices and establish the infrared pixel coordinates. To visible light pixel coordinates The projection mapping relationship is as follows:

[0032] in These are the intrinsic parameter matrices for the infrared and visible light cameras, respectively. Let be a rotation matrix. It is a translation vector.

[0033] A priori structural map of the equipment is constructed based on the equipment's CAD structural model or 3D point cloud data. This priori structural map includes the geometric contours, edge features, and spatial topological relationships between the components. Feature point matching is guided by this priori structural map, mapping the pixel coordinates of the infrared image to the visible light image coordinate system to achieve sub-pixel-level spatial alignment. Specifically, under the constraints of the priori structural map, feature point pairs corresponding to the same physical location of the equipment (such as equipment edges, bolts, heat sink corners, etc.) are extracted from the infrared and visible light images. Mismatched point pairs that do not conform to the priori structural map are eliminated. A random sampling consensus algorithm is used to estimate the fine registration transformation parameters, mapping the pixel coordinates of the infrared image to the visible light image coordinate system to achieve sub-pixel-level spatial alignment with a registration accuracy better than one pixel.

[0034] Time alignment specifically includes: synchronizing various types of sensor data in the time dimension based on a unified timestamp; for sensors with different sampling frequencies, using interpolation methods to resample the data to a unified time reference (e.g., 100Hz). The time deviation of the synchronized data is controlled within 10ms. Specifically, let sensor A at time... Data is collected by sensor B at time... Collect data and use interpolation functions Calculating a unified time base The following data values: .

[0035] S3. Extract the features of each modal data respectively. A dual-branch feature extraction network is used and a multi-dimensional cross-connection mechanism is introduced to extract multi-scale features of visible light images and infrared thermal imaging images. The corresponding time-frequency features and component features are extracted for ultrasonic signals, vibration signals and oil chromatography data respectively.

[0036] A dual-branch feature extraction network is employed, and a multi-dimensional cross-connection mechanism is introduced to extract multi-scale features from visible light images and infrared thermal imaging images, specifically including: The visible light branch utilizes a deep convolutional neural network (preferably ResNet-50 or EfficientNet-B4, pre-trained on the ImageNet dataset) to extract texture, structural, and scene semantic features from visible light images. The visible light branch employs a feature pyramid structure, generating four scales of visible light feature maps through top-down paths and lateral connections. These correspond to multi-scale representations ranging from shallow detail features (edges, textures) to deep semantic features (device component semantics). .

[0037] The infrared branch utilizes a deep convolutional neural network (which shares the same backbone network architecture as the visible light branch, but trains its weights independently) to extract temperature distribution and thermal anomaly features from infrared thermal imaging images, generating corresponding infrared feature maps at four scales. .

[0038] A multi-dimensional cross-connection mechanism is introduced during feature extraction to enhance information transfer between shallow and deep features. This mechanism includes: same-scale skip connections—combining visible light and infrared features at the same scale through element-wise addition or gated fusion; cross-scale cross-connections—fusing shallow feature maps with deep feature maps through upsampling, and fusing deep feature maps with shallow feature maps through downsampling; and bidirectional feature propagation—establishing bidirectional information flow from shallow to deep and from deep to shallow, with the feature recalibration module adaptively adjusting the contribution weights of features at each scale.

[0039] The corresponding time-frequency features and component features were extracted from the ultrasonic signal, vibration signal, and oil chromatography data, respectively, including: Time-frequency analysis was performed on the partial discharge ultrasonic signal to extract time-domain statistical features and frequency-domain features, and an ultrasonic feature vector was constructed. The time-domain statistical features include peak value, root mean square (RMS), peak-to-peak value, skewness, kurtosis, impulse factor, and margin factor. Frequency-domain features are extracted by performing a Fast Fourier Transform (FFT) on the ultrasonic signal to obtain the power spectral density, center frequency, bandwidth, peak frequency, and corresponding amplitude. Simultaneously, a Short-Time Fourier Transform (SFT) or Continuous Wavelet Transform (CWT) is performed on the ultrasonic signal to extract the energy distribution features on the time-frequency spectrum, serving as joint time-frequency features.

[0040] Empirical mode decomposition (EMD) is performed on the vibration signal, decomposing the vibration signal x(t) into the sum of several eigenmode functions and residuals: .

[0041] Extracting the energy characteristics of each intrinsic mode function And spectral characteristics (center frequency and bandwidth of each IMF component), construct vibration feature vector Among them, energy characteristics Calculation: .

[0042] Meanwhile, the time-domain statistical features of the vibration signal (peak value, root mean square, peak-to-peak value, etc.) are extracted as a supplement.

[0043] The concentration data of dissolved gases in oil in the oil chromatography data were normalized (using Z-score normalization or Min-Max normalization) to construct the oil chromatography feature vector. The dissolved gases in the oil include H2, CH4, C2H2, C2H4, C2H6, CO, and CO2. Furthermore, the ratios of key characteristic gases are calculated, including... These, etc., are incorporated into the oil chromatography feature vector as extended features.

[0044] S4. Achieve bidirectional information enhancement of visible light features and infrared features through a cross-modal attention module, and adopt a dynamic weighted fusion strategy to adaptively fuse the enhanced image features with non-image modal features to generate a multimodal comprehensive feature vector.

[0045] Bidirectional information enhancement of visible light and infrared features is achieved through a cross-modal attention module, specifically including: Visible light characteristics With infrared features Input a cross-modal attention module. Use visible light features as the query. Infrared features as a key Sum Calculate cross-modal attention weights to generate infrared-enhanced visible light features. The calculation method is as follows: .

[0046] Using infrared features as a query Visible light characteristics as bonds Sum Calculate cross-modal attention weights to generate visible light-enhanced infrared features. The calculation method is as follows: .in The feature dimension is defined by a bidirectional cross-modal attention mechanism, which aligns infrared and visible light images in the semantic space. This allows visible light features to be guided by infrared temperature anomaly information, while infrared features are supplemented by visible light texture structure information. The cross-modal attention module employs a multi-head attention mechanism, followed by a feedforward network and layer normalization after attention calculation, forming a standard Transformer encoder structure.

[0047] A dynamic weighted fusion strategy is employed to adaptively fuse enhanced image features with non-image modality features, specifically including: Visible light features enhanced by cross-modal attention With infrared features The features are cascaded along the channel dimension, and the fusion weights for each feature channel are calculated using an adaptive weight learning network. The adaptive weight learning network consists of a global average pooling layer, two fully connected layers (with ReLU activation in the middle layer), and a sigmoid activation function. Image features are then fused. The calculation method is as follows: .

[0048] in The channel-level weight vector obtained through adaptive learning. This represents element-wise multiplication (Hadamard product). The weights are dynamically adjusted based on the confidence level of each modality's features under the current input. Specifically, for the current input sample, the information entropy and signal-to-noise ratio of the features extracted from the visible light branch and the infrared branch are calculated. Modalities with lower information entropy and higher signal-to-noise ratio are assigned higher fusion weights.

[0049] Fusing image features With ultrasound feature vector Vibration characteristic vector Oil chromatographic feature vectors The features are concatenated to form a multimodal comprehensive feature vector. .

[0050] The calculation of the fusion weights is dynamically adjusted based on the confidence level of each modal feature under the current input.

[0051] S5. Input the multimodal integrated feature vector into a deep neural network that integrates physical priors for defect identification and classification. The physical priors include thermodynamic consistency constraints, structural consistency constraints, and multimodal consistency constraints.

[0052] The deep neural network incorporating physical priors consists of an input layer (receiving multimodal integrated feature vectors), three fully connected layers (with 1024, 512, and 256 neurons respectively, each followed by a batch normalization layer and a ReLU activation layer), a dropout layer (with a dropout rate of 0.3–0.5 to prevent overfitting), and an output layer (softmax activation, outputting the probability distribution of various defects). The network is trained using the Adam optimizer with an initial learning rate of [missing information]. A cosine annealing learning rate scheduling strategy is adopted.

[0053] Thermodynamic consistency constraints ensure that regions of abnormal infrared temperature remain consistent with the physical model of equipment heating. Specifically, this is based on the heat transfer model of normal equipment operation (such as the heat conduction equation). Predict the theoretical temperature distribution on the surface of the device. The actual temperature distribution measured by infrared thermal imaging. Compared with theoretical predictions, the thermodynamic consistency loss The overheating defects identified by the constraint network do not violate the basic thermodynamic laws.

[0054] Structural consistency constraints require that defects be located within a structurally reasonable area of ​​the equipment. Specifically, based on the prior CAD structural drawings of the equipment, the geometric spatial range of each component (such as transformer windings, bushings, heat sinks, tank walls, etc.) is defined; structural consistency loss... The penalty is to locate the defect in an area where it is physically impossible for the defect to occur (such as a hollow area inside the equipment or the space outside the equipment).

[0055] The multimodal consistency constraint ensures that the judgments of different modalities on the same defect corroborate each other. Specifically, the prediction results of each single modality are obtained by using independent shallow classifiers through each single modality feature. Multimodal consistency loss, i.e. This constrains the prediction results of each mode to tend to be consistent.

[0056] The deep neural network incorporating physical priors is trained using a joint loss function that includes physical constraint priors. The joint loss function is: .

[0057] in, This is the classification loss (cross-entropy loss). This is the thermodynamic consistency loss (mean square error of the temperature field). This is a structural consistency loss (spatial location penalty). This is the multimodal consistency loss (prediction discrepancies between modes). These are the corresponding weighting coefficients.

[0058] Defect types include equipment overheating, partial discharge, insulation degradation, abnormal mechanical vibration, and abnormal dissolved gas in oil; defect levels are classified into four levels according to the severity of the defect: normal, warning, abnormal, and severe; the classification is based on the degree of deviation of various defect characteristic parameters (such as the percentage of temperature deviation from the normal value, the multiple of the partial discharge amplitude relative to the baseline value, the multiple of the oil chromatographic component concentration relative to the warning value, etc.).

[0059] S6. Calculate the overall confidence level of the defect identification results based on multimodal cross-validation, and output the defect type, defect location, defect level and overall confidence level.

[0060] The comprehensive confidence level of defect identification results is calculated based on multimodal cross-validation, specifically including: Calculate the confidence level of each single mode for identifying the same defect. .

[0061] The overall confidence level is calculated using a multimodal cross-validation mechanism. When the recognition results of multiple modalities are consistent, the overall confidence level increases; when there are conflicts between modalities, the overall confidence level decreases and triggers a manual review mechanism. Defect locations are simultaneously labeled in visible light and infrared images; specifically, the defect area is labeled in the visible light image using a rectangular detection box or semantic segmentation mask, and the corresponding thermal anomaly area is labeled in the infrared image using a temperature pseudo-color overlay. The labeling results of the two images are then pixel-level aligned using spatial registration parameters to form a fused labeling result, which is output to the maintenance terminal.

[0062] Example 2 A multimodal fusion intelligent identification system for substation equipment defects, referring to Figure 2 ,include: The multimodal data acquisition module is used to simultaneously acquire visible light image data, infrared thermal imaging data, partial discharge ultrasonic signals, equipment vibration signals, and oil chromatography data of the target substation equipment. The multimodal data acquisition module is deployed on intelligent inspection robots (wheeled or track-mounted robots equipped with a gimbal and multiple sensors) or fixed multi-sensor terminals (installed on poles near the equipment or on the equipment itself). The acquisition module has a built-in IEEE 1588 precision time protocol synchronization clock, ensuring that all sensors trigger acquisition with a unified time reference, achieving a time synchronization accuracy better than 1ms.

[0063] The data preprocessing module is used to preprocess various types of collected data, including image denoising and enhancement and normalization, signal filtering and noise reduction, and data anomaly detection and missing value imputation. The preprocessing module uses FPGA or GPU acceleration to meet real-time processing requirements.

[0064] The spatiotemporal alignment and registration module is used to perform spatial registration based on multi-source features and prior knowledge of device structure, and temporal alignment based on a unified timestamp, on preprocessed multimodal data. The spatial registration module has a built-in calibration parameter library for each sensor, supporting rapid calibration and replacement of different sensor models; the temporal alignment module uses a high-precision interpolation algorithm to resample data at different sampling rates to a unified time reference.

[0065] The multimodal feature extraction module is used to extract features from each modality of data. It employs a dual-branch feature extraction network and introduces a multi-dimensional cross-connection mechanism to extract multi-scale features from visible light and infrared thermal imaging images. Time-frequency and component features are extracted from ultrasonic signals, vibration signals, and oil chromatography data, respectively. The feature extraction module is deployed based on a deep learning inference engine (such as TensorRT or OpenVINO) to achieve efficient forward inference computation.

[0066] The cross-modal attention fusion module enhances both visible light and infrared features through a cross-modal attention mechanism. It then employs a dynamic weighted fusion strategy to adaptively fuse the enhanced image features with non-image modal features, generating a multimodal comprehensive feature vector. The fusion module dynamically adjusts the fusion weights based on the real-time confidence level of each modality feature, ensuring stable system operation even when the quality of some modal data degrades.

[0067] The physical constraint defect identification module is used to input multimodal integrated feature vectors into a deep neural network that integrates physical priors for defect identification and classification. The physical priors include thermodynamic consistency constraints, structural consistency constraints, and multimodal consistency constraints. The identification module is trained using a joint loss function that incorporates physical constraint priors and outputs the defect type, defect location, and defect level.

[0068] The confidence assessment and output module is used to calculate the overall confidence of defect identification results based on multimodal cross-validation, and output the defect type, defect location, defect level, and overall confidence. The output formats include a visual interface display (overlaying labeled boxes and temperature information on a fused visible light and infrared image), structured data reports (JSON or XML format), and early warning signals (audio-visual alarms, SMS push notifications, and maintenance platform notifications).

[0069] The implementation principle of the multimodal fusion-based intelligent identification method for substation equipment defects in this application is as follows: Five modalities of sensor data (visible light, infrared, ultrasound, vibration, and oil chromatography) are simultaneously collected, and after preprocessing and spatiotemporal alignment, a multimodal dataset is formed. A dual-branch feature extraction network is used to extract multi-scale features from visible light and infrared images, and a multi-dimensional cross-connection mechanism is used to enhance feature transfer. A cross-modal attention module is used to achieve bidirectional information enhancement between image modalities, and a dynamic weighted fusion strategy adaptively fuses image features and non-image features into a comprehensive feature vector. This comprehensive feature vector is input into a deep neural network that incorporates physical priors, and defect identification and classification are performed under the constraints of a joint loss function. Finally, multimodal cross-validation is used to calculate the comprehensive confidence score and output the identification result. This method fully utilizes the complementary advantages of multimodal data and ensures the physical rationality of the identification result through physical prior constraints, effectively improving the accuracy, robustness, and interpretability of substation equipment defect identification.

[0070] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multimodal fusion-based intelligent identification method for substation equipment defects, characterized in that, Includes the following steps: S1. Simultaneously acquire visible light image data, infrared thermal imaging data, partial discharge ultrasonic signals, equipment vibration signals, and oil chromatography data of the target substation equipment, and preprocess the acquired data. S2. Perform spatiotemporal alignment and registration on the preprocessed multimodal data, including spatial registration based on multi-source features and prior knowledge of device structure and temporal alignment based on unified timestamps. S3. Extract the features of each modal data respectively. A dual-branch feature extraction network is used and a multi-dimensional cross-connection mechanism is introduced to extract multi-scale features of visible light images and infrared thermal imaging images. The corresponding time-frequency features and component features are extracted for the ultrasonic signal, the vibration signal and the oil chromatography data respectively. S4. Achieve bidirectional information enhancement of visible light features and infrared features through a cross-modal attention module, and adopt a dynamic weighted fusion strategy to adaptively fuse the enhanced image features with non-image modal features to generate a multimodal comprehensive feature vector. S5. Input the multimodal integrated feature vector into a deep neural network that integrates physical priors for defect identification and classification. The physical priors include thermodynamic consistency constraints, structural consistency constraints, and multimodal consistency constraints. S6. Calculate the overall confidence level of the defect identification results based on multimodal cross-validation, and output the defect type, defect location, defect level and overall confidence level.

2. The multimodal fusion-based intelligent identification method for substation equipment defects according to claim 1, characterized in that, The synchronous acquisition in S1 specifically includes: acquiring visible light image data through a high-definition visible light camera, acquiring infrared thermal imaging data through an infrared thermal imaging sensor, acquiring partial discharge ultrasonic signals through an ultrasonic sensor, acquiring equipment vibration signals through a vibration sensor, and acquiring oil chromatographic data through an online oil chromatographic monitoring device; each sensor is synchronously triggered to acquire data using a unified timestamp. The preprocessing in S1 includes: denoising, enhancing, and normalizing the visible light image and the infrared thermal imaging image; filtering and denoising the partial discharge ultrasonic signal to remove environmental electromagnetic interference; detrending and filtering the vibration signal; and outlier detection and missing value imputation of the oil chromatographic data.

3. The multimodal fusion-based intelligent identification method for substation equipment defects according to claim 1, characterized in that, The spatial registration in S2 specifically includes: obtaining spatial transformation parameters between each sensor through calibration, the spatial transformation parameters including rotation matrix and translation vector; constructing a priori structural map of the device based on the CAD structural model or 3D point cloud data of the device; using the priori structural map of the device to guide feature point matching, mapping the pixel coordinates of the infrared image to the visible light image coordinate system, and achieving sub-pixel level spatial alignment. The time alignment in S2 specifically includes: synchronizing various types of sensor data in the time dimension based on the unified timestamp; and for sensors with different sampling frequencies, using interpolation methods to resample the data to a unified time reference.

4. The multimodal fusion-based intelligent identification method for substation equipment defects according to claim 1, characterized in that, The S3 method employs a dual-branch feature extraction network and introduces a multi-dimensional cross-connection mechanism to extract multi-scale features from visible light images and infrared thermal imaging images, specifically including: The visible light branch utilizes a deep convolutional neural network to extract texture features, structural features, and scene semantic features from the visible light image, generating a multi-scale visible light feature map. ; The infrared branch uses a deep convolutional neural network to extract the temperature distribution and thermal anomaly features of the infrared thermal imaging image, generating a multi-scale infrared feature map. ; A multidimensional cross-connection mechanism is introduced during the feature extraction process to enhance the information transfer between shallow and deep features.

5. The multimodal fusion-based intelligent identification method for substation equipment defects according to claim 1, characterized in that, In step S3, corresponding time-frequency features and component features are extracted from the ultrasonic signal, the vibration signal, and the oil chromatography data, respectively, specifically including: Time-frequency analysis was performed on the partial discharge ultrasonic signal to extract time-domain statistical features and frequency-domain features, and an ultrasonic feature vector was constructed. The time-domain statistical features include peak value, root mean square value, and peak-to-peak value; the frequency-domain features include power spectral density, center frequency, and bandwidth. Empirical mode decomposition (EMD) is performed on the vibration signal to extract the energy and spectral features of the intrinsic mode functions, and a vibration feature vector is constructed. ; The concentration data of dissolved gases in the oil in the oil chromatography data are normalized to construct an oil chromatography feature vector. The dissolved gases in the oil include H2, CH4, C2H2, C2H4, C2H6, CO, and CO2.

6. The multimodal fusion-based intelligent identification method for substation equipment defects according to claim 1, characterized in that, S4 utilizes a cross-modal attention module to enhance both visible and infrared features, specifically including: The visible light feature and the infrared feature are input into the cross-modal attention module. The visible light feature is used as the query and the infrared feature is used as the key and value to calculate the cross-modal attention weight and generate infrared-enhanced visible light feature. The infrared feature is used as the query and the visible light feature is used as the key and value to calculate the cross-modal attention weight and generate visible light-enhanced infrared feature. By employing a bidirectional cross-modal attention mechanism, the semantic space alignment of infrared and visible light images is achieved.

7. The multimodal fusion-based intelligent identification method for substation equipment defects according to claim 1, characterized in that, In step S4, a dynamic weighted fusion strategy is used to adaptively fuse enhanced image features with non-image modality features, specifically including: The visible light features enhanced by cross-modal attention are concatenated with infrared features, and the fusion weights of each feature channel are calculated through an adaptive weight learning network to generate fused image features. ; The fused image features With ultrasound feature vector Vibration characteristic vector Oil chromatographic feature vectors The features are concatenated to form a multimodal comprehensive feature vector. ; The calculation of the fusion weights is dynamically adjusted based on the confidence level of each modal feature under the current input.

8. The multimodal fusion-based intelligent identification method for substation equipment defects according to claim 1, characterized in that, The deep neural network in S5 that incorporates physical priors includes fully connected layers, batch normalization layers, activation layers, Dropout layers, and an output layer. The thermodynamic consistency constraint ensures that the infrared temperature anomaly region remains consistent with the equipment's heating physical model. The structural consistency constraint is that the defect location is within a reasonable area of ​​the equipment's structure. The multimodal consistency constraint is that the judgment results of different modes on the same defect corroborate each other; The deep neural network that integrates physical priors is trained using a joint loss function that includes physical constraint priors. The joint loss function is: ； in, For classifying losses, For the aforementioned thermodynamic consistency loss, The structural consistency loss is... For the aforementioned multimodal consistency loss, These are the corresponding weighting coefficients; The defect types in S5 include equipment overheating, partial discharge, insulation deterioration, abnormal mechanical vibration, and abnormal dissolved gas in oil; the defect levels are divided into four levels according to the severity of the defect: normal, attention, abnormal, and severe.

9. The multimodal fusion-based intelligent identification method for substation equipment defects according to claim 1, characterized in that, The comprehensive confidence level of defect identification results calculated in S6 based on multimodal cross-validation includes: Calculate the confidence level of each single mode for identifying the same defect. ; The overall confidence level C_total is calculated using a multimodal cross-validation mechanism: when the recognition results of multiple modalities are consistent, the overall confidence level increases; when there is a conflict between modalities, the overall confidence level decreases and a manual review mechanism is triggered. The defect locations in S6 are simultaneously marked in the visible light image and the infrared image.

10. A multimodal fusion intelligent identification system for substation equipment defects, characterized in that, include: A multimodal data acquisition module is used to simultaneously acquire visible light image data, infrared thermal imaging data, partial discharge ultrasonic signals, equipment vibration signals, and oil chromatography data of the target substation equipment; the multimodal data acquisition module is deployed on an intelligent inspection robot or a fixed multi-sensor terminal; The data preprocessing module is used to preprocess various types of collected data; The spatiotemporal alignment and registration module is used to perform spatial registration based on multi-source features and device structure priors, and temporal alignment based on a unified timestamp on preprocessed multimodal data. The multimodal feature extraction module is used to extract features of each modality of data. It employs a dual-branch feature extraction network and introduces a multidimensional cross-connection mechanism to extract multi-scale features of visible light images and infrared thermal imaging images. It also extracts corresponding time-frequency features and component features from the ultrasonic signal, the vibration signal, and the oil chromatography data. The cross-modal attention fusion module is used to enhance the information of visible light features and infrared features in both directions through the cross-modal attention mechanism, and adopts a dynamic weighted fusion strategy to adaptively fuse the enhanced image features with non-image modal features to generate a multimodal comprehensive feature vector. The physical constraint defect identification module is used to input the multimodal integrated feature vector into a deep neural network that fuses physical priors for defect identification and classification. The confidence assessment and output module is used to calculate the overall confidence of the defect identification results based on multimodal cross-validation, and output the defect type, defect location, defect level and overall confidence.

Images