Power transformer defect diagnosis method based on multi-modal acoustic image fusion

Abstract

The present invention belongs to the technical field of transformer defect diagnosis, and specifically relates to a method for diagnosing defects in power transformers based on multimodal sound and image fusion, the steps of which include obtaining image data and voiceprint data of a target power transformer; constructing a transformer defect diagnosis model and a transformer classification model; detecting whether the transformer has defects, determining the location of the defects, and judging the defect type of the transformer; synthesizing the results of defect detection and the results of defect type judgment to obtain a predicted score value of image data; constructing a voiceprint diagnosis model, and pre-training the voiceprint diagnosis model; processing the voiceprint data, extracting features, and performing fault type diagnosis to obtain a predicted score value of voiceprint data; based on the obtained predicted score value of image data and the predicted score value of voiceprint data, adopting an attention mechanism to fuse the predicted score value to perform decision fusion, and obtain the final defect diagnosis result. The present invention can accurately diagnose the defects of transformers.

Description

Technical Field

[0001] The present invention belongs to the technical field of transformer defect diagnosis, and in particular relates to a power transformer defect diagnosis method based on multi-modal sound image fusion. Background Art

[0002] The power transformer is one of the most critical equipment in the power system. It undertakes the tasks of voltage conversion, power distribution and transmission, and provides various power services. However, various faults will occur in the operation of the power transformer, such as cracks on the surface of porcelain parts and porcelain bushings, rupture of explosion-proof membranes, damage to oil level gauges or oil leakage, which are visible to the naked eye, or excessive temperature caused by load operation, cooling system failure, etc., which rely on infrared judgment, or obvious abnormal noise or vibration caused by loose fasteners, magnetic bias, etc., which rely on soundprint judgment. These fault types affect and transform each other, causing the transformer to shut down or even completely damaged. Therefore, how to accurately and efficiently determine the cause of such faults based on the abnormal appearance and abnormal sound of the transformer is conducive to the maintenance of power transformers, reducing maintenance costs and improving the efficiency of inspection personnel.

[0003] In order to diagnose transformer defects quickly and efficiently, some existing studies include: CN201611158965.9 proposed a power transformer defect diagnosis method using oil chromatography monitoring data, using a normalized hypercube mapping method to diagnose defects; CN202211687682.9 proposed a transformer defect diagnosis method based on heterogeneous data, pre-training the correlation between the simulation model and the field sensor data to obtain the transformer defect diagnosis results; CN201811307816.3 proposed a transformer partial discharge signal classification device and fault detection device, which can effectively realize the classification of discharge signals and accurately diagnose the defect information caused by partial discharge.

[0004] However, the above-mentioned power transformer defect diagnosis method has the following two problems: first, the existing method often only targets a certain defect type, and the fault diagnosis is relatively single, which may lead to the omission of transformer defects on site, incomplete maintenance, and hidden dangers; second, the existing diagnosis is generally based on the expert system of the fault tree. Although the method is simple, the diagnosis speed is slow, and the single-layer network structure also leads to longer training time, poor adaptability, inconvenient maintenance, and inefficient diagnosis process. Summary of the invention

[0005] In view of the above deficiencies in the prior art, the present invention provides a method for diagnosing defects in power transformers based on multimodal sound and image fusion. A multimodal fusion method is adopted to pre-train image data and voiceprint data respectively, and an attention mechanism is used to weight the prediction scores extracted from image and voiceprint features. After decision fusion, accurate diagnosis of transformer defects can be achieved.

[0006] To achieve the above objectives, the present invention provides a power transformer defect diagnosis method based on multi-modal sound image fusion, comprising the following steps:

[0007] S1. Acquire image data and voiceprint data of a target power transformer;

[0008] S2. construct a transformer defect diagnosis model and a transformer classification model, and pre-train the transformer defect diagnosis model and the transformer classification model;

[0009] S3, using the pre-trained transformer defect diagnosis model to process the image data, detect whether the transformer has defects, and determine the location of the defects, and use the pre-trained transformer classification model to process the output of the transformer defect diagnosis model to determine the defect type of the transformer;

[0010] S4, combining the defect detection results and the defect type determination results to obtain a predicted score value of the image data;

[0011] S5. Build a voiceprint diagnosis model and pre-train the voiceprint diagnosis model;

[0012] S6. Process the voiceprint data using a pre-trained voiceprint diagnosis model, extract features and perform fault type diagnosis, and obtain a predicted score value for the voiceprint data;

[0013] S7. Based on the predicted score values ​​of the image data and the predicted score values ​​of the voiceprint data, decision fusion is performed by fusing the predicted score values ​​using an attention mechanism to obtain the final defect diagnosis result.

[0014] Preferably, in S1, image data of the target power transformer is acquired based on on-site shooting, drone shooting or infrared thermal imaging; voiceprint data of the target power transformer is acquired based on on-site recording or data acquisition system DAQ; preprocessing including cropping, scaling and rotation is performed on the acquired image data; and preprocessing including denoising and filtering is performed on the acquired voiceprint data.

[0015] Preferably, in S2, the multimodal transformer defect diagnosis model includes an input layer, a feature extraction layer, a fusion layer, a positioning layer, a classification layer and an output layer; after the input layer acquires the image data, the feature extraction layer uses a fully connected neural network and a convolutional neural network to extract features from the image data, and connects the feature vectors of each mode together in the fusion layer to form a comprehensive feature vector, the positioning layer determines the location of the defect through a region proposal network, outputs the probability of each type of defect through the classification layer, and finally outputs the defect detection result in the output layer;

[0016] The transformer classification model includes an input layer, multiple consecutive 3×3 convolution kernels and 2×2 maximum pooling layers, and an output layer; the input layer obtains the defect detection results, and through consecutive 3×3 convolution kernels and 2×2 maximum pooling operations, continuously reduces the feature map and increases the number of channels to obtain the classification of transformer defects, and finally outputs the defect type judgment results in the output layer.

[0017] Preferably, in S2, during pre-training, the existing image data containing defects is expanded and then used as training data for training, specifically:

[0018] For the visible light images in the existing image data, we selected transformer defect images including surface cracks of porcelain parts and porcelain bushings, discoloration of desiccant, and external injuries caused by lightning strikes. Based on the generative adversarial network, we comprehensively considered the diversity of background object information and occlusion structure, collected texture images containing transformer defect areas and corresponding real defect texture information, and after preprocessing the images, we used the deep learning feature extraction method to extract texture features in the images.

[0019] The histogram similarity measurement method is used to calculate the similarity between the defect area texture image and the real defect texture information, so as to establish a similarity evaluation system between the transformer defect area texture image and the real defect texture information.

[0020] The defect mask is generated using the convolutional neural network model architecture, and the similarity evaluation results are added to the training process as auxiliary information to build a transformer defect mask generation model, narrow the distribution difference between the generated data and the real data in the instance space and feature space, and achieve rapid expansion of scarce samples;

[0021] Based on the transformer defect mask generation model, data augmentation is performed on the image data used for training;

[0022] Before using the expanded image data as training data, defects are annotated. For visible light images, the surface of the transformer is observed, and the areas with obvious abnormalities in the transformer defect image are intercepted, and the areas are annotated and the defect categories are marked.

[0023] For infrared images, the temperature of the transformer surface is monitored to determine whether the transformer has overheating defects and mark the defect location.

[0024] Preferably, in S4, the method for obtaining the predicted score value of the image data is to obtain the predicted score value of the image data by weighted averaging based on the confidence score of the defect existence in the defect detection result and the confidence scores of different types of defects in the defect type judgment result.

[0025] Preferably, in S5, the voiceprint diagnosis model adopts the Wav2Vec2.0 model, and based on the Wav2Vec2.0 model, the voiceprint data is processed, features are extracted, and fault type diagnosis is performed, and the confidence score of the fault type is used as the prediction score value of the voiceprint data.

[0026] Preferably, in said S5, during pre-training, the existing voiceprint data containing defects is expanded and then used as training data for training, specifically:

[0027] First, the voiceprint data is convolved in the time domain or frequency domain to extract features. Then, based on the simulation of real scene noise, the expectation and standard deviation are adjusted according to the signal-to-noise ratio distribution function to generate a probability density curve that meets expectations. Then, the signal quality is improved through filtering, and the noise data is superimposed on the filtered voiceprint data according to different signal-to-noise ratios to obtain the expanded voiceprint data.

[0028] Before using the expanded image data as training data, defects are annotated. The voiceprint features under normal operating conditions of the transformer are used as baseline feature values. The expanded voiceprint data are compared with the baseline feature values ​​to determine the fault state of the transformer. The voiceprint features include at least one of time domain features, frequency domain features, and time-frequency domain features.

[0029] Preferably, in S7, decision fusion is performed by fusing the prediction score value with the attention mechanism, specifically: matrix operations are performed on the prediction score value of the image data and the prediction score value of the voiceprint data, and the attention weight of each feature or sample is obtained by using the attention module, and the attention weight is multiplied by the corresponding prediction score value, and the loss function is calculated for each of them for training to obtain the final defect diagnosis result.

[0030] Preferably, it also includes S8, performing autonomous question-and-answer advanced intelligent analysis, and providing contextual text information related to the problem in response to a request about the defect diagnosis problem based on the final defect diagnosis result.

[0031] Preferably, in S8, the autonomous question-answering advanced intelligent analysis includes a personnel feedback mechanism and a knowledge database injection technology:

[0032] The personnel feedback mechanism adopts three methods: sorting-based, problem-based, and rule-based, to collect personnel feedback and preference data from transformer maintenance personnel. The sorting-based method prioritizes different problems according to their urgency and importance by designing and optimizing sorting algorithms; the problem-based method classifies problems and performs personalized processing and responses to different problems; the rule-based method establishes feedback loops and monitoring mechanisms to timely evaluate and adjust strategies;

[0033] The knowledge database injection technology injects the transformer's operating procedures and historical fault handling records into the database. When faced with question-and-answer requests, it quickly performs vector matching in the database and uses the matched text to form context related to the question.

[0034] The algorithm involved in the present invention can be executed by an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The algorithm is implemented by executing software through the processor.

[0035] The beneficial effects of the present invention are:

[0036] In the data processing stage, the present invention performs data expansion on the image data through an adversarial network, and performs filtering and noise processing on the voiceprint data to simulate more realistic noise data in real scenes, thereby achieving the purpose of data enhancement.

[0037] In order to obtain the diagnosis results of transformer defects, the present invention adopts a multimodal fusion method, pre-trains the image data and voiceprint data respectively, and uses the attention mechanism to weight the prediction scores extracted from the image and voiceprint features. After decision fusion, accurate diagnosis of transformer defects is achieved.

[0038] The present invention adopts autonomous question-and-answer advanced intelligent analysis, introduces a personnel feedback mechanism and knowledge database injection technology to achieve accurate question-and-answer, can better avoid answering irrelevant questions, make the final result more accurate, and improve the work efficiency of operation and maintenance personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a schematic diagram of the process of the method in Example 1 of the present invention;

[0040] Figure 2 is a schematic diagram of the process of the method in Embodiment 2 of the present invention;

[0041] Figure 3 It is a schematic diagram of the structure of the system in Example 3 of the present invention. DETAILED DESCRIPTION

[0042] The embodiments of the present invention are further described below in conjunction with the accompanying drawings:

[0043] Example 1

[0044] like Figure 1 As shown, the power transformer defect diagnosis method based on multi-modal sound image fusion includes the following steps:

[0045] S1. Acquire image data and voiceprint data of a target power transformer;

[0046] S2. construct a transformer defect diagnosis model and a transformer classification model, and pre-train the transformer defect diagnosis model and the transformer classification model;

[0047] S3, using the pre-trained transformer defect diagnosis model to process the image data, detect whether the transformer has defects, and determine the location of the defects, and use the pre-trained transformer classification model to process the output of the transformer defect diagnosis model to determine the defect type of the transformer;

[0048] S4, combining the defect detection results and the defect type determination results to obtain a predicted score value of the image data;

[0049] S5. Build a voiceprint diagnosis model and pre-train the voiceprint diagnosis model;

[0050] S6. Process the voiceprint data using a pre-trained voiceprint diagnosis model, extract features and perform fault type diagnosis, and obtain a predicted score value for the voiceprint data;

[0051] S7. Based on the predicted score values ​​of the image data and the predicted score values ​​of the voiceprint data, decision fusion is performed by fusing the predicted score values ​​using an attention mechanism to obtain the final defect diagnosis result.

[0052] In S1, the image data of the target power transformer is obtained based on on-site shooting, drone shooting or infrared thermal imaging; the voiceprint data of the target power transformer is obtained based on on-site recording or data acquisition system DAQ; the obtained image data is preprocessed including cropping, scaling and rotation; the obtained voiceprint data is preprocessed by denoising and filtering.

[0053] In S2, the multimodal transformer defect diagnosis model includes an input layer, a feature extraction layer, a fusion layer, a positioning layer, a classification layer, and an output layer. After the input layer obtains the image data, the feature extraction layer uses a fully connected neural network and a convolutional neural network to extract features from the image data, and connects the feature vectors of each mode together in the fusion layer to form a comprehensive feature vector. The positioning layer determines the location of the defect through the region proposal network, outputs the probability of each type of defect through the classification layer, and finally outputs the defect detection result in the output layer.

[0054] The transformer classification model includes an input layer, multiple consecutive 3×3 convolution kernels and 2×2 maximum pooling layers, and an output layer; the input layer obtains the defect detection results, and through consecutive 3×3 convolution kernels and 2×2 maximum pooling operations, continuously reduces the feature map and increases the number of channels to obtain the classification of transformer defects, and finally outputs the defect type judgment results in the output layer.

[0055] In the multimodal transformer defect diagnosis model, data from different sensors and monitoring systems, including but not limited to vibration signals, temperature data, and dissolved gas analysis in oil, can also be used as input for processing to improve the accuracy of defect detection. Defect types usually include abnormal phenomena such as lightning damage and oil leakage.

[0056] This embodiment also provides an intelligent generation and expansion method for scarce image data and voiceprint data, which achieves data expansion and solves the problem of inaccurate results caused by insufficient sample training data.

[0057] In S2, during pre-training, the existing image data containing defects is augmented and used as training data for training, specifically:

[0058] For the visible light images in the existing image data, various transformer defect images including surface cracks of porcelain parts and porcelain bushings, discoloration of desiccant, and external injuries caused by lightning strikes are selected. Based on the generative adversarial network, the diversity of background object information and occlusion structure is comprehensively considered, and the texture images containing the defective areas of the transformer and the corresponding real defect texture information are collected. After preprocessing the images, the texture features in the images are extracted using the deep learning feature extraction method.

[0059] The histogram similarity measurement method is used to calculate the similarity between the defect area texture image and the real defect texture information, so as to establish a similarity evaluation system between the transformer defect area texture image and the real defect texture information.

[0060] The defect mask is generated using the convolutional neural network model architecture, and the similarity evaluation results are added to the training process as auxiliary information to build a transformer defect mask generation model, narrow the distribution difference between the generated data and the real data in the instance space and feature space, and achieve rapid expansion of scarce samples;

[0061] Based on the transformer defect mask generation model, data augmentation is performed on the image data used for training;

[0062] Before using the expanded image data as training data, defects are annotated. For visible light images, the surface of the transformer is observed, and the areas with obvious abnormalities in the transformer defect image are intercepted, and the areas are annotated and the defect categories are marked.

[0063] For infrared images, the temperature of the transformer surface is monitored to determine whether the transformer has overheating defects and mark the defect location.

[0064] The surface temperature of transformer equipment is usually maintained at 40-60 degrees Celsius. When the transformer is aged or damaged or short-circuit current and load current increase, the transformer will heat up and the temperature will rise sharply. Therefore, for the intelligent analysis of transformer equipment defect diagnosis based on infrared images, one way is to monitor the temperature of the transformer surface to determine whether the transformer has an overheating defect abnormality. Another way is to use an infrared imager to more intuitively find the part where the transformer temperature rises and further locate the defect.

[0065] For image pre-training, the ResNet model can be used. This model introduces the structure of residual blocks, which can effectively avoid the gradient vanishing problem. A basic residual block contains two convolutional layers, each followed by a batch normalization and ReLU activation function. The input is directly added to the output of the residual block to form a residual connection. Specifically, given the input X, the residual block calculates:

[0066] ;

[0067] in, It is a combination of convolution, batch normalization and ReLU; W i is the weight parameter of the i-th layer; Y is the output.

[0068] In S4, the method for obtaining the predicted score value of the image data is to obtain the predicted score value of the image data by weighted average based on the confidence score of the defect existence in the defect detection result and the confidence scores of different types of defects in the defect type judgment result.

[0069] In S5, the voiceprint diagnosis model adopts the Wav2Vec2.0 model. Based on the Wav2Vec2.0 model, it processes the voiceprint data, extracts features, and performs fault type diagnosis. The confidence score of the fault type is used as the prediction score value of the voiceprint data.

[0070] The Wav2Vec2.0 model extracts low-level features of audio through CNN. These features contain rich time-frequency information, and use the Transformer network to model long-term dependencies. It can capture global context information in audio signals and has efficient feature extraction capabilities. The architecture of Wav2Vec2.0 can be divided into three main parts: Feature Encoder, Transformer Network, and Quantization Module. The model can process a large amount of unlabeled data and use this data for self-supervised pre-training.

[0071] The voiceprint data (voiceprint signal) of the transformer contains characteristic information reflecting the internal defects and faults of the transformer. Therefore, the voiceprint characteristics of the transformer under normal operation are used as the benchmark, and the voiceprint data obtained by monitoring is compared with the benchmark characteristic value to qualitatively diagnose the fault state of the transformer. These characteristics include time domain characteristics, frequency domain characteristics, and time-frequency domain characteristics.

[0072] In S5, during pre-training, the existing voiceprint data containing defects is augmented and used as training data for training, specifically:

[0073] First, the voiceprint data is convolved in the time domain or frequency domain to extract features. Then, based on the simulation of real scene noise, the expectation and standard deviation are adjusted according to the signal-to-noise ratio distribution function to generate a probability density curve that meets expectations. Then, the signal quality is improved through filtering, and the noise data is superimposed on the filtered voiceprint data according to different signal-to-noise ratios to obtain the expanded voiceprint data.

[0074] Before using the expanded image data as training data, defects are annotated. The voiceprint features under normal operating conditions of the transformer are used as baseline feature values. The expanded voiceprint data are compared with the baseline feature values ​​to determine the fault state of the transformer. The voiceprint features include at least one of time domain features, frequency domain features, and time-frequency domain features.

[0075] For voiceprint information, since there is a lot of noise in the real environment, the voiceprint data needs to be processed after filtering. The acquired voiceprint information is subjected to time domain convolution or frequency domain convolution operation, which is called filtering. Take the noise data in the real scene, such as car horns, wind noise, etc., and use the signal-to-noise ratio SNR distribution function to superimpose such noise data on the filtered data. The SNR distribution function is:

[0076] ;

[0077] In the formula, x represents a specific SHR value, that is, the signal-to-noise ratio at a certain moment or under a certain situation; is the mean (expected value) of the signal-to-noise ratio distribution, that is, the average of all possible SNR values; is the standard deviation of the signal-to-noise ratio distribution, describing the SNR values ​​relative to the mean The signal-to-noise ratio can be calculated by the following formula:

[0078] ;

[0079] In the formula, is the signal power, is the noise power. The power of the signal and the noise can be calculated by the mean square value (MSV) of the signal and the noise. and To obtain a probability density curve that better meets expectations. According to the signal-to-noise ratio SNR, the acquired noise fragments are superimposed with the data obtained after filtering to obtain the expanded voiceprint data.

[0080] In this embodiment, the voiceprint data is divided into training samples and test samples in a ratio of 7:3, and the training samples are used as the input of the voiceprint diagnosis model, and the pre-training parameters are set. During the training process, the error between the predicted result and the actual result is compared. When the model Loss value (loss value) converges, the model parameters of the optimal round are saved and the model training is stopped. If the evaluation accuracy of some defect types is poor, the prediction result of the voiceprint diagnosis model is used as the input vector of the meta-learner, and the prediction result of each sample obtained by predicting the training data is the classification probability. The meta-learner learns the best combination strategy or weight by using the prediction results of the basic model to optimize the performance of the voiceprint data diagnosis model. The meta-learning algorithm used in this model architecture is the ensemble learning method (Boosting). The best combination strategy or weight learned by the meta-learner is then applied to the voiceprint diagnosis model to form an integrated model. This method can further improve the accuracy and robustness of the voiceprint data diagnosis model by adjusting the parameters or integration strategy of the meta-learner.

[0081] The expanded visible light image, infrared image and voiceprint information are analyzed and processed to further locate the defect position, which is conducive to more accurate analysis of the defect type and defect diagnosis.

[0082] In S7, the decision fusion is performed by fusing the prediction score value with the attention mechanism. Specifically, matrix operations are performed on the prediction score value of the image data and the prediction score value of the voiceprint data. The attention weight of each feature or sample is obtained by using the attention module. The attention weight is multiplied by the corresponding prediction score value, and the loss function is calculated for each of them to train them respectively to obtain the final defect diagnosis result.

[0083] This fusion method is a multimodal attention mechanism that fuses data of different modes by assigning weights. The key architecture is to add an attention layer between the feature extraction layer and the classification layer, weight the features of different modes through the attention mechanism, and input the weighted features into the classification layer to output the probability of various defects, thereby realizing the diagnosis of transformer defect types.

[0084] After obtaining the predicted scores of defective images and abnormal voiceprint information respectively, the attention mechanism is used to fuse the predicted scores for decision fusion. The predicted scores are input into the attention module after matrix operation to obtain the attention weights. The attention weights are multiplied by the predicted scores of the original model and the loss functions are calculated for the results respectively. This can enhance the model's attention to important data inputs. After obtaining the weighted loss function, back propagation is performed to update the model parameters, and the optimizer is used to update the parameters to ensure that the model can update the parameters in the direction of reducing losses. When the performance of the validation set no longer improves or begins to decline, training is stopped to avoid overfitting, thereby obtaining the final defect diagnosis results.

[0085] The matrix operations are:

[0086] ;

[0087] In the formula, A is the attention weight matrix or vector used to calculate the attention weight of each feature or sample. The dimension of A is usually m×k, and each element a in A ij represents the attention weight corresponding to the input feature or sample, where k is the dimension of the attention weight, which is used to measure the importance of the corresponding sample or feature. B is a matrix or vector used to weight the attention weight, b ij For the elements, i and j are indices, representing the rows and columns in the matrix, and m, n, p, q are the number of rows and columns (dimensions). The dimension of B depends on the specific application, and is usually a matrix or vector used to weight the attention weights to the prediction scores.

[0088] Example 2

[0089] like Figure 2 As shown, based on Example 1, it also includes S8, performing autonomous question-and-answer advanced intelligent analysis, and providing contextual text information related to the problem in response to a request about the defect diagnosis problem based on the final defect diagnosis result.

[0090] In S8, autonomous question-answering advanced intelligent analysis includes personnel feedback mechanism and knowledge database injection technology:

[0091] The personnel feedback mechanism adopts three methods: sorting-based, problem-based, and rule-based, to collect personnel feedback and preference data from transformer maintenance personnel. The sorting-based method prioritizes different problems according to their urgency and importance by designing and optimizing sorting algorithms; the problem-based method classifies problems and performs personalized processing and responses to different problems; the rule-based method establishes feedback loops and monitoring mechanisms to timely evaluate and adjust strategies;

[0092] The knowledge database injection technology injects the transformer's operating procedures and historical fault handling records into the database. When faced with question-and-answer requests, it quickly performs vector matching in the database and uses the matched text to form context related to the question.

[0093] For example, when the transformer oil leakage defect image or the transformer sound is abnormal, according to the method of Example 1, it can be judged that the transformer has defects and the cause of the oil leakage may be poor welding quality, cold welding, desoldering, pinholes and sand holes in the weld, etc. By analyzing the feature weights, checking the coefficient of the feature or the importance score of the feature to evaluate its contribution to the model prediction, it can be judged that the cause of the transformer abnormal sound may be overload, loose individual parts, ferromagnetic resonance, etc. These reasons can help operation and maintenance personnel to timely investigate and solve such problems. However, in the field of precise knowledge question and answer, there will be a phenomenon of answering questions that are not asked. For example, when the input is "How to detect the internal insulation condition of the transformer?" The model answers "The internal insulation condition of the transformer can be judged by the change of insulating oil." Although the answer mentions that the change of insulating oil is related to the insulation condition, it does not provide a specific method for detection. Therefore, the introduction of personnel feedback mechanism and knowledge database injection technology can improve the professional matching of fault handling question and answer response results to a certain extent.

[0094] The personnel feedback mechanism mainly uses three methods: sorting-based, problem-based, and rule-based to collect personnel feedback and preference data from transformer maintenance personnel. The sorting-based method designs and optimizes the sorting algorithm to prioritize different problems according to their urgency, importance, and other criteria, ensuring that a large amount of user feedback can be efficiently managed and processed; the problem-based method classifies problems and performs personalized processing and responses to different problems; the rule-based method establishes feedback loops and monitoring mechanisms to timely evaluate and adjust strategies. The personnel feedback mechanism can be constructed based on the incentive model and the strategy model. The personnel feedback mechanism is adapted to the personnel feedback through learning incentives, and expert information is incorporated into the training cycle. The incentive model and the strategy model can be updated iteratively, so that the incentive can more accurately describe the quality of the model output, and the output text is more and more consistent with the logic of the transformer defect factors. In the strategy model, the ReLU activation function is used in the hidden layer and the output layer to increase the nonlinear characteristics of the network. The loss function in the incentive model uses the cross entropy loss to measure the difference between the defect results predicted by the incentive model and the actual defects.

[0095] The knowledge database injection technology injects the transformer's operating procedures and historical fault handling records into the database. When faced with a question-and-answer request, it can quickly perform vector matching in the database and use the matched text to form context related to the question. Through retrieval enhancement, while ensuring the credibility and authority of transformer equipment knowledge, the large model provides interpretability in the field of transformer defects to achieve knowledge injection. First, a knowledge base containing information about transformer defects is constructed, including the types and characteristics of defects, detection methods and tools, processing and repair suggestions, etc. Then, the data in the database is structured and stored using a specific data structure, and the knowledge base content is annotated. The knowledge base is then combined with the defect diagnosis method as a corpus resource for model content generation, enhancing the interpretability of model-generated content and achieving dynamic knowledge injection.

[0096] Example 3

[0097] like Figure 3 As shown, this embodiment provides a power transformer defect diagnosis system based on multi-modal sound image fusion that can implement embodiment 2, including:

[0098] Data acquisition module: acquires image data and voiceprint data of the target power transformer;

[0099] Image model building module: builds transformer defect diagnosis model and transformer classification model, and pre-trains the transformer defect diagnosis model and transformer classification model;

[0100] Image processing module: Use the pre-trained transformer defect diagnosis model to process image data, detect whether the transformer has defects, and determine the location of the defects. Use the pre-trained transformer classification model to process the output of the transformer defect diagnosis model and determine the defect type of the transformer.

[0101] Score value acquisition module: It integrates the defect detection results and defect type judgment results to obtain the predicted score value of the image data;

[0102] Voiceprint model construction module: construct a voiceprint diagnosis model and pre-train the voiceprint diagnosis model;

[0103] Voiceprint processing module: Use the pre-trained voiceprint diagnosis model to process the voiceprint data, extract features and perform fault type diagnosis, and obtain the predicted score value of the voiceprint data;

[0104] Decision fusion module: Based on the predicted scores of the image data and the voiceprint data, the decision fusion is performed by fusing the predicted scores with the attention mechanism to obtain the final defect diagnosis result.

[0105] Autonomous question-and-answer advanced intelligent analysis module: performs autonomous question-and-answer advanced intelligent analysis, responds to requests about defect diagnosis issues based on the final defect diagnosis results, and provides contextual text information related to the issue.

Claims

1. A power transformer defect diagnosis method based on multi-modal sound image fusion, characterized in that The following steps are involved: S1. Acquire image data and voiceprint data of a target power transformer; S2. construct a transformer defect diagnosis model and a transformer classification model, and pre-train the transformer defect diagnosis model and the transformer classification model; S3, using the pre-trained transformer defect diagnosis model to process the image data, detect whether the transformer has defects, and determine the location of the defects, and use the pre-trained transformer classification model to process the output of the transformer defect diagnosis model to determine the defect type of the transformer; S4. Combining the defect detection results and the defect type determination results, obtaining a predicted score value of the image data; S5. Build a voiceprint diagnosis model and pre-train the voiceprint diagnosis model; S6. Process the voiceprint data using a pre-trained voiceprint diagnosis model, extract features and perform fault type diagnosis, and obtain a predicted score value for the voiceprint data; S7, based on the obtained prediction score value of the image data and the prediction score value of the voiceprint data, the decision fusion is performed by fusing the prediction score value with the attention mechanism to obtain the final defect diagnosis result; In the above S2, the multimodal transformer defect diagnosis model includes an input layer, a feature extraction layer, a fusion layer, a positioning layer, a classification layer and an output layer; after the input layer acquires the image data, the feature extraction layer uses a fully connected neural network and a convolutional neural network to extract features from the image data, and connects the feature vectors of each mode together in the fusion layer to form a comprehensive feature vector, the positioning layer determines the location of the defect through the region proposal network, outputs the probability of each type of defect through the classification layer, and finally outputs the defect detection result in the output layer; The transformer classification model includes an input layer, multiple consecutive 3×3 convolution kernels and 2×2 maximum pooling layers, and an output layer. The input layer obtains the defect detection results, and through consecutive 3×3 convolution kernels and 2×2 maximum pooling operations, the feature map is continuously reduced and the number of channels is increased to obtain the classification of transformer defects. Finally, the defect type judgment result is output in the output layer. In the above S2, during pre-training, the existing image data containing defects is expanded and then used as training data for training, specifically: For the visible light images in the existing image data, we selected transformer defect images including surface cracks of porcelain parts and porcelain bushings, discoloration of desiccant, and external injuries caused by lightning strikes. Based on the generative adversarial network, we comprehensively considered the diversity of background object information and occlusion structure, collected texture images containing transformer defect areas and corresponding real defect texture information, and after preprocessing the images, we used the deep learning feature extraction method to extract texture features in the images. The histogram similarity measurement method is used to calculate the similarity between the defect area texture image and the real defect texture information, so as to establish a similarity evaluation system between the transformer defect area texture image and the real defect texture information. The defect mask is generated using the convolutional neural network model architecture, and the similarity evaluation results are added to the training process as auxiliary information to build a transformer defect mask generation model, narrow the distribution difference between the generated data and the real data in the instance space and feature space, and achieve rapid expansion of scarce samples; Based on the transformer defect mask generation model, data augmentation is performed on the image data used for training; Before using the expanded image data as training data, defects are annotated. For visible light images, the surface of the transformer is observed, and the areas with obvious abnormalities in the transformer defect image are intercepted, and the areas are annotated and the defect categories are marked. For infrared images, determine whether the transformer has overheating defects by monitoring the temperature of the transformer surface and mark the defect location; In the above S5, during pre-training, the existing voiceprint data containing defects is expanded and used as training data for training, specifically: First, the voiceprint data is convolved in the time domain or frequency domain to extract features. Then, based on the simulation of real scene noise, the expectation and standard deviation are adjusted according to the signal-to-noise ratio distribution function to generate a probability density curve that meets expectations. Then, the signal quality is improved through filtering, and the noise data is superimposed on the filtered voiceprint data according to different signal-to-noise ratios to obtain the expanded voiceprint data. Before using the expanded image data as training data, defects are marked, and the voiceprint features of the transformer in normal operation are used as reference feature values. The expanded voiceprint data is compared with the reference feature values ​​to determine the fault state of the transformer, and the voiceprint features include at least one of time domain features, frequency domain features, and time-frequency domain features; In the above-mentioned S7, decision fusion is performed by adopting the attention mechanism to fuse the predicted score value, specifically: matrix operation is performed on the predicted score value of the image data and the predicted score value of the voiceprint data, the attention weight of each feature or sample is obtained by using the attention module, the attention weight is multiplied by the corresponding predicted score value, and the loss function is calculated for each of them for training to obtain the final defect diagnosis result.

2. The power transformer defect diagnosis method based on multi-modal sound image fusion according to claim 1 is characterized in that: In the S1, image data of the target power transformer is obtained based on on-site shooting, drone shooting or infrared thermal imaging; voiceprint data of the target power transformer is obtained based on on-site recording or data acquisition system DAQ; the obtained image data is preprocessed including cropping, scaling and rotation; the obtained voiceprint data is preprocessed by denoising and filtering.

3. The power transformer defect diagnosis method based on multi-modal sound image fusion according to claim 1 is characterized in that: In the above-mentioned S4, the method for obtaining the predicted score value of the image data is to obtain the predicted score value of the image data by weighted average based on the confidence score of the defect existence in the defect detection result and the confidence scores of different types of defects in the defect type judgment result.

4. The power transformer defect diagnosis method based on multi-modal sound image fusion according to claim 1 is characterized in that: In the above-mentioned S5, the voiceprint diagnosis model adopts the Wav2Vec2.0 model. Based on the Wav2Vec2.0 model, the voiceprint data is processed, features are extracted, and fault type diagnosis is performed. The confidence score of the fault type is used as the prediction score value of the voiceprint data.

5. The power transformer defect diagnosis method based on multi-modal sound image fusion according to claim 1 is characterized in that: It also includes S8, which performs autonomous question-and-answer advanced intelligent analysis and provides contextual text information related to the problem in response to a request about the defect diagnosis problem based on the final defect diagnosis result.

6. The power transformer defect diagnosis method based on multi-modal sound image fusion according to claim 5 is characterized in that: In the above-mentioned S8, the autonomous question-answering advanced intelligent analysis includes a personnel feedback mechanism and a knowledge database injection technology: The personnel feedback mechanism adopts three methods: sorting-based, problem-based, and rule-based, to collect personnel feedback and preference data from transformer maintenance personnel. The sorting-based method prioritizes different problems according to their urgency and importance by designing and optimizing sorting algorithms; the problem-based method classifies problems and performs personalized processing and responses to different problems; the rule-based method establishes feedback loops and monitoring mechanisms to timely evaluate and adjust strategies; The knowledge database injection technology injects the transformer's operating procedures and historical fault handling records into the database. When faced with question-and-answer requests, it quickly performs vector matching in the database and uses the matched text to form context related to the question.

Images