Power transformation equipment defect detection method based on multi-modal input in unmanned aerial vehicle scene

Abstract

The invention discloses a substation equipment defect detection method based on multi-modal normal sample learning, which is suitable for unmanned aerial vehicle data acquisition. The method comprises the following steps: step 1, acquiring multi-modal data of substation equipment and preprocessing the multi-modal data; 2, extracting and fusing features from the multi-modal data preprocessed in the step 1 to obtain a feature set; dividing the feature set into a training set and a verification set; 3, adopting a machine learning model, and performing normal sample learning by utilizing the training set and the verification set to obtain a normal classification model; 4, features are extracted from the to-be-detected substation equipment multi-modal data and then compared with the normal classification model, and whether defect features exist in the to-be-detected substation equipment multi-modal data or not is judged through difference analysis. The detection accuracy can be improved, manual intervention is reduced, dependence on defect samples is reduced, and the method has the advantage of being high in adaptability.

Description

Technical Field

[0001] The present invention relates to the field of substation equipment defect detection methods, and specifically to a substation equipment defect detection method based on multi-modal input in an unmanned aerial vehicle scenario. Background Art

[0002] With the continuous development of the power system, the safe and stable operation of substation equipment is crucial to the reliability of the entire power grid. Traditional substation equipment detection methods mainly rely on manual inspections, which are inefficient and have certain safety risks. In order to solve the problems of manual inspections, there are currently existing substation equipment defect detection algorithms to achieve substation equipment defect detection, but most of the existing substation equipment defect detection algorithms are based on single-modality data (such as images) and rely heavily on defect samples. However, in practical applications, it is often difficult and costly to obtain a large number of defect samples. In addition, single-modality data may not be able to fully and accurately reflect the status of substation equipment.

[0003] In recent years, drone technology has been widely used in the field of power inspection. Drones equipped with a variety of sensors can collect multimodal data of substation equipment. Therefore, if multimodal data can be used to realize substation equipment defect detection, it will undoubtedly solve the problems of existing technologies that realize substation equipment defect detection methods based on single model data. Summary of the invention

[0004] The present invention provides a substation equipment defect detection method based on multi-modal input in a drone scenario, so as to solve the problem that the prior art substation equipment defect detection algorithm is highly dependent on defect samples based on single modal data.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: The defect detection method of substation equipment based on multi-modal input in the UAV scenario includes the following steps: Step 1: Obtain multimodal data of substation equipment through multiple sensors carried on the drone, and pre-process the multimodal data; Step 2: Use the feature extraction network to process the multimodal data preprocessed in step 1 to extract and fuse features, thereby obtaining a feature set; then divide the feature set into a training set and a validation set; Step 3: adopt a machine learning model, and the machine learning model uses the training set and the verification set obtained in step 2 to perform normal sample learning. During learning, the machine learning model is trained by the training set, and the prediction result of the trained machine learning model is verified by the verification set after training, and the training parameters are adjusted according to the prediction result. The above process is repeated for multiple rounds of training until the loss function of the machine learning model converges, and the training is terminated, thereby obtaining a normal classification model; Step 4: After the multimodal data of the substation equipment to be detected is subjected to feature extraction by the feature extraction network, it is compared with the normal classification model, and whether the multimodal data of the substation to be detected contains defect features is determined through difference analysis.

[0006] Furthermore, in step 1, the multimodal data includes corresponding data of substation equipment collected by an optical camera, a multispectral camera, a thermal imaging camera, and a lidar respectively mounted on the drone.

[0007] Furthermore, in step 1, the preprocessing process includes spatial alignment, temporal synchronization, resolution matching, radiometric calibration and normalization, data fusion, and data quality check.

[0008] Furthermore, in step 2, the feature extraction network is a Meta-Transformer network.

[0009] Furthermore, in step 2, the feature set is divided into a training set and a validation set by random partitioning, wherein the validation set must contain abnormal data.

[0010] Furthermore, in step 3, the machine learning model is SwAV.

[0011] Compared with the prior art, the present invention has the following advantages: 1. Improve detection efficiency and accuracy. Make full use of multimodal data collected by drones or other methods to fully and accurately reflect the status of substation equipment, thereby improving detection accuracy. The automated detection process greatly improves detection efficiency and reduces manual intervention.

[0012] 2. Reduce the dependence on defective samples. Based on multimodal normal sample learning, it reduces the dependence on scarce defective samples and reduces the cost and difficulty of data collection.

[0013] 3. Strong adaptability. When using multimodal data collected by drones equipped with multiple types of sensors, it can adapt to various complex situations when drones collect data, such as different flight attitudes, lighting conditions, and shooting angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the SwAV process used in the embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0016] This embodiment discloses a method for detecting defects in substation equipment based on multimodal input in a drone scenario. First, the multimodal data of substation equipment collected by multiple types of sensors carried by the drone is preprocessed, and then feature extraction is performed based on the preprocessed data. Then, a normal classification model of the normal state is established by learning a large number of multimodal normal samples. Finally, in the actual detection process, the data to be detected is compared with the normal classification model, and the difference analysis is performed to determine whether there is a defect. This embodiment includes the following steps: Step 1: Obtain multimodal data of substation equipment and preprocess the multimodal data.

[0017] In this embodiment, multimodal data of substation equipment is obtained through multiple types of sensors, and multiple types of sensors are mounted on drones to collect corresponding data during drone flight inspections. The multiple types of sensors in this embodiment include optical cameras, multispectral cameras, thermal imaging cameras, and laser radars (LiDAR). During drone flight inspections, the optical cameras, multispectral cameras, thermal imaging cameras, and laser radars mounted on drones collect corresponding data of substation equipment, thereby obtaining multimodal data. Then, the multimodal data is preprocessed to improve data quality and availability.

[0018] In this embodiment, the process of preprocessing multimodal data includes denoising, spatial alignment, time synchronization, resolution matching, radiation calibration and normalization, data fusion, and data quality inspection. During preprocessing, denoising is performed first, and then other preprocessing is performed. The other preprocessing parts are specifically described as follows: (1) Spatial alignment In this embodiment, the coordinate system and reference of each type of sensor (i.e., optical camera, multispectral camera, thermal imaging camera, lidar) are determined, which usually involves obtaining the installation parameters of each type of sensor, such as position, attitude, focal length, etc. Then, the geographic coordinate system is established using GPS data as geographic reference information, and the corresponding data collected by each type of sensor is converted into a unified geographic coordinate system. For image data, common feature points are found through feature matching and image registration technology, and the transformation matrix is ​​calculated to achieve spatial alignment.

[0019] Specifically, first, for visible light cameras, calibrate the internal and external parameters of each camera; use ICP or other calibration methods to calibrate the relative position between the lidar and the camera; project the lidar point cloud to the geographic coordinate system as a basic reference; use external parameters to project the images of optical, multispectral and thermal imaging cameras to the same coordinate system; apply feature point matching or depth-assisted registration methods to ensure the alignment accuracy of each data; then fuse the data from all sensors and ensure time synchronization.

[0020] (2) Time synchronization In order to ensure accurate synchronization of the acquisition time of various sensors (i.e., optical cameras, multispectral cameras, thermal imaging cameras, and lidars), this embodiment usually uses a high-precision clock synchronization device or sets a synchronization mechanism in the data acquisition software to achieve accurate synchronization of the acquisition time when the optical camera, multispectral camera, thermal imaging camera, and lidar each collect data.

[0021] Alternatively, this embodiment adjusts the time sequence of data collected by different sensors with time deviations by interpolation or extrapolation according to timestamps, thereby achieving precise synchronization of the collection time of optical cameras, multispectral cameras, thermal imaging cameras, and lidars.

[0022] (3) Resolution matching In this embodiment, based on the original resolution of each type of sensor, resampling or interpolation is performed as needed to make the data collected by the optical camera, multispectral camera, thermal imaging camera, and lidar as close as possible in spatial resolution, thereby achieving resolution matching.

[0023] Specifically, the interpolation method is used to upsample the low-resolution data and downsample the high-resolution data; under the premise of spatial alignment, the depth information of the lidar point cloud data is combined to generate a depth map, which is matched with the resolution of the two-dimensional image.

[0024] The relative calibration method is used to calibrate the different sensors radiometrically. First, a reference plate (usually a white or gray panel with known reflectivity) is placed in the scene when the sensor is observing. The sensor's response to the reference plate is recorded and combined with the known reflectivity of the reference plate to infer the sensor's actual response. This information is used to correct the observations.

[0025] (4) Radiation calibration and normalization In this embodiment, radiation calibration is performed for optical cameras and multispectral cameras to convert the pixel values ​​of the data they collect into actual radiation brightness or reflectivity values ​​to eliminate the differences in the sensor's own response characteristics, environmental influences (such as atmospheric interference) and data collection conditions, and to ensure that the output values ​​can reflect real physical phenomena. The radiation calibrated data collected by the optical camera and multispectral camera, as well as the data collected by the thermal imaging camera and lidar, are normalized to have the same numerical range and unit. For example, the temperature data of the thermal imaging camera can be converted into standardized temperature units and matched with the data range of other sensors.

[0026] (5) Data fusion In this embodiment, a suitable data fusion algorithm is used to fuse the data of different sensors. Specifically, the data fusion algorithm used in this embodiment includes weighted average, feature-based fusion, and model-based fusion, which enriches and unifies the subsequent algorithm inputs through fusion.

[0027] (6) Data quality check In this embodiment, the quality of the data after spatial alignment, time synchronization, resolution matching, radiation calibration and normalization, and data fusion is checked, including checking outliers, missing values, and data consistency. The outlier check process is to calculate the histogram and observe the statistical results, the missing value check process is to check through data visualization, and the data consistency check process is to set range constraints and observe the calculation results.

[0028] Then, a known standard data set or field measurement data is used to verify the accuracy of the data quality check results. The specific process is to calculate the mean and variance of the collected data, and then compare it with the labeled data set. If the comparison result is less than the preset threshold, it is considered accurate.

[0029] Step 2: Use the feature extraction network to process the multimodal data preprocessed in step 1 to perform feature extraction and fusion, thereby obtaining a feature set; then divide the feature set into a training set and a validation set.

[0030] In this embodiment, the feature extraction network uses a Meta-Transformer network. The Meta-Transformer network is a framework that can encode 12 types of modal data, including images, natural language, point clouds, audio spectrograms, videos, infrared, hyperspectral, x-rays, IMUs, tables, graphs, and time series data.

[0031] The Meta-Transformer network mainly consists of a unified data tokenizer, a modality-shared Transformer encoder, and a head specific to downstream tasks. Meta-Transformer uses a frozen-weight encoder for multimodal perception without any paired multimodal training data, maps the original input data from different modalities into a shared token space, and then uses a Transformer encoder with frozen parameters to extract high-level semantic features of the input data.

[0032] In this embodiment, the multimodal data preprocessed in step 1 is input into the Meta-Transformer network for feature extraction and fusion. The process is as follows: (1) Constructing the input sequence: Split and flatten the data of each modality into appropriate sequence elements, and add specific tags or to the data of each modality to distinguish different modalities.

[0033] (2) Design of modal fusion layer: Introduce modal fusion layers at the shallow, middle, and deep levels of the Meta-Transformer network to fuse features of different modalities to enrich the expressive power of features. Use concatenation, weighted summation, and attention-based fusion methods to fuse features corresponding to optical cameras, multispectral cameras, thermal imaging cameras, and lidars. For the data of optical cameras and multispectral cameras, flatten them into one-dimensional sequences and then process them in the Transformer layer. In the modal fusion layer, different weights are assigned to the two modal data according to their importance and weighted summation is performed. For the data of thermal imaging cameras and lidars, similar preprocessing and sequence construction are performed, and then they are integrated with the features of optical and multispectral data in the fusion layer and jointly optimized to extract a unified feature representation.

[0034] The Meta-Transformer network is used to process multimodal data to obtain a feature set, which is then randomly divided into a training set and a validation set, where the validation set must contain abnormal data.

[0035] Step 3: Use a machine learning model. The machine learning model uses the training set and verification set obtained in step 2 to perform normal sample learning. During learning, the machine learning model is trained with the training set. After training, the prediction results of the trained machine learning model are verified with the verification set, and the training parameters are adjusted according to the prediction results. The above process is repeated for multiple rounds of training until the loss function of the machine learning model converges, then the training is terminated, thereby obtaining a normal classification model.

[0036] In this embodiment, the machine learning model uses SwAV. The following describes the process of using SwAV to learn normal samples and obtain a normal classification model: The architecture of SwAV (Swapping Assignments between Views) is as follows Figure 1 As shown, as a basic algorithm, SwAV uses the idea of ​​clustering for comparative learning, avoiding dependence on a large number of negative samples. In a multimodal scenario, this embodiment uses SwAV to cluster and compare features of different modalities.

[0037] SwAV first performs multiple data augmentation operations on the input image to generate different views, and then extracts the features of each view through a convolutional neural network. Next, these features are compared with the learnable prototype vector to calculate the soft assignment of each feature. The core innovation of SwAV is to exchange the assignment results of different views to ensure that different enhanced views of the same image have consistent feature representations. By contrasting the loss function, the model learns meaningful feature representations under unsupervised conditions.

[0038] A cluster center named C is constructed in the algorithm framework of SwAV, which combines self-supervised learning with clustering methods. The similarity matrix can be obtained by multiplying the output z1 of the encoder f in the figure above with the cluster center vector c, so as to predict Q2 (Q1 and Q2 are obtained by the sinkhorn algorithm), and vice versa, so as to achieve the effect of exchange prediction. The clustering method has obvious advantages. First, by comparing with the cluster center, the demand for negative samples can be reduced; second, the cluster center has a certain meaning, while random sampling may draw positive samples or the categories are unbalanced.

[0039] In this embodiment, the specific process of SwAV using the training set and the validation set to perform normal sample learning is as follows: Assume that the data in the training set is D, and use D to train the SwAV model to optimize the parameters of the SwAV model. After training, input the data in the validation set into the trained SwAV model to test the prediction results of the trained SwAV model, and adjust the training parameters according to the prediction results. Repeat the above process until the loss function of the SwAV model converges. The loss function of the SwAV model is shown as follows:

[0040] Among them: For the same original image, z t This is the result of a data augmentation (t) on this image; z S This is the result after another data augmentation (s) for this image; q S For this Figure 1 The code corresponding to the data enhancement (t); q t The code corresponding to another data enhancement(s) for this picture; l (z t ,q S ) l (z S ,q t ) is the same formula, and the specific calculation is shown below.

[0041] Here l(z,q) measures the relationship between feature z and code q, and zt and zs represent different enhancement results from the same original image. Specifically, it is calculated by the cross entropy between code and z and C after softmax, and the formula is expressed as

[0042] in,

[0043] Among them: k represents the number of categories; k' is the cumulative variable of k; q s (k) When the category is k, q is the code of s data enhancement, p t (k) When the category is k, z t The probability of k is the true cluster center when the category is K; c k ' is the cluster center value under different k; Represents the temperature parameter (see Unsupervised feature learning via non-parametricinstance discrimination.).

[0044] Step 4: After the multimodal data of the substation equipment to be detected is subjected to feature extraction by the feature extraction network, it is compared with the normal classification model, and whether the multimodal data of the substation to be detected contains defect features is determined through difference analysis.

[0045] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. The embodiments described in the present invention are merely descriptions of the preferred embodiments of the present invention, and do not limit the concept and scope of the present invention. The various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction, and such combinations should also be regarded as the contents disclosed in the present disclosure as long as they do not violate the concept of the present invention. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0046] The present invention is not limited to the specific details of the above-mentioned embodiments. Within the technical concept of the present invention and without departing from the design concept of the present invention, various modifications and improvements made to the technical solution of the present invention by technical personnel in this field should fall within the protection scope of the present invention. The technical contents for which protection is sought in the present invention have been fully recorded in the claims.

Claims

1. A defect detection method for substation equipment based on multimodal input in UAV scenario, characterized in that: The following steps are involved: Step 1: Obtain multimodal data of substation equipment through multiple sensors carried on the drone, and pre-process the multimodal data; Step 2: Use the feature extraction network to process the multimodal data preprocessed in step 1 to perform feature extraction and fusion, thereby obtaining a feature set; then divide the feature set into a training set and a validation set; Step 3: adopt a machine learning model, and the machine learning model uses the training set and the verification set obtained in step 2 to perform normal sample learning. During learning, the machine learning model is trained by the training set, and the prediction result of the trained machine learning model is verified by the verification set after training, and the training parameters are adjusted according to the prediction result. The above process is repeated for multiple rounds of training until the loss function of the machine learning model converges, and the training is terminated, thereby obtaining a normal classification model; Step 4: After the multimodal data of the substation equipment to be detected is subjected to feature extraction by the feature extraction network, it is compared with the normal classification model, and whether the multimodal data of the substation to be detected contains defect features is determined through difference analysis.

2. According to the method for detecting defects of substation equipment based on multimodal input in a drone scenario according to claim 1, it is characterized in that: In step 1, the multimodal data includes corresponding data of substation equipment collected by an optical camera, a multispectral camera, a thermal imaging camera, and a lidar respectively mounted on a drone.

3. According to claim 1, the method for detecting defects of substation equipment based on multimodal input in a drone scenario is characterized in that: In step 1, the preprocessing process includes spatial alignment, temporal synchronization, resolution matching, radiometric calibration and normalization, data fusion, and data quality check.

4. According to claim 1, the method for detecting defects of substation equipment based on multimodal input in a drone scenario is characterized in that: In step 2, the feature extraction network is a Meta-Transformer network.

5. According to the method for detecting defects of substation equipment based on multimodal input in a drone scenario according to claim 1, it is characterized in that: In step 2, the feature set is divided into a training set and a validation set by random partitioning, wherein the validation set must contain abnormal data.

6. The method for detecting defects of substation equipment based on multimodal input in a drone scenario according to claim 1 is characterized in that: In step 3, the machine learning model is SwAV.

Images