Power equipment cross-modal data fusion method and system based on deep learning

Abstract

The invention discloses a power equipment cross-modal data fusion method and system based on deep learning, and relates to the technical field of data processing. Performing stress value extraction on the electrical data and the voiceprint data to obtain an electrical stress value and a voiceprint stress value; performing feature extraction on the image data to obtain target image features, and performing feature extraction on the electrical data and the voiceprint data through a full-connection neural network to obtain target electrical features and target voiceprint features; and performing feature fusion on the target image feature, the target electrical feature and the target voiceprint feature according to the electrical stress value and the voiceprint stress value to obtain a target total feature, and determining a fault type of the target equipment according to the target total feature. According to the method, the stress values are extracted from the electrical and voiceprint data, the modal features are extracted by using the neural network, the electrical, voiceprint and image target features are fused into a unified total feature, the fault type is determined according to the total feature, and the overall stability and operation efficiency of the power system are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and specifically relates to a method and system for cross-modal data fusion of power equipment based on deep learning. Background Art

[0002] With the rapid development of smart grid and Internet of Things technologies, equipment status monitoring and operation and maintenance in power systems are gradually developing towards intelligence, real-time and multi-modal integration. Traditional means of power equipment status monitoring mainly rely on manual regular inspections and regular experiments. The inspection items usually include observation of the equipment appearance, recording and comparison of instrument data, which is a typical passive detection method. This method is not only unable to obtain the operating status of the equipment in real time, but is also limited by the frequency and coverage of manual operations, and it is difficult to meet the needs of modern smart grids for efficient, accurate and real-time monitoring.

[0003] Modern power system equipment has gradually integrated sensors of various modes such as electrical, voiceprint, and image. Various parameter data of operating status and performance are obtained through different types of sensors, so that the operation of equipment can be fully perceived from multiple angles and dimensions. However, in practical applications, traditional data collection still has a lot of separation and fragmentation. Data of different modes are often collected independently, lacking unified time alignment and spatial alignment. Due to the lack of effective multimodal data fusion, the relationship between different data is difficult to be accurately captured, which makes the subsequent power equipment status prediction and fault diagnosis based on these data low in accuracy. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that due to the lack of effective multimodal data fusion, the correlation between different data is difficult to be accurately captured, which makes the subsequent power equipment status prediction and fault diagnosis based on these data have low accuracy, and proposes a cross-modal data fusion method and system for power equipment based on deep learning.

[0005] In a first aspect of the implementation of the present invention, a method for cross-modal data fusion of power equipment based on deep learning is first proposed, and the method comprises:

[0006] Collecting image data, electrical data and voiceprint data of the target device in a preset time period, and extracting stress values ​​from the electrical data and the voiceprint data to obtain electrical stress values ​​and voiceprint stress values;

[0007] Extracting features from the image data to obtain target image features, and extracting features from the electrical data and the voiceprint data using a fully connected neural network to obtain target electrical features and target voiceprint features;

[0008] The target image feature, the target electrical feature and the target voiceprint feature are fused according to the electrical stress value and the voiceprint stress value to obtain a target total feature, and a preset database is searched according to the target total feature to determine the fault type of the target device.

[0009] Optionally, the electrical data includes voltage data, current data and power data; the voiceprint data includes amplitude data and frequency data; and extracting stress values ​​from the electrical data and the voiceprint data to obtain electrical stress values ​​and voiceprint stress values ​​includes:

[0010] Preprocessing the electrical data and the voiceprint data to obtain first electrical data and first voiceprint data respectively;

[0011] The target class data is mapped to a preset target interval to obtain a discrete point set of target data, the target average relative deviation of the target class data is obtained according to the discrete point set of target data, and the target average relative deviation of all classes in the target data set is normalized to obtain the stress value of each class; if the target data set is the first electrical data, the target class data is any one of voltage data, current data or power data; if the target data set is the first voiceprint data; the target class data is any one of amplitude data or frequency data; the target class data is any one of the target data sets.

[0012] Optionally, the image data is an infrared image with a temperature color bar, and extracting features from the image data to obtain target image features includes:

[0013] Gray-scale processing is performed on the image data to obtain a first gray-scale image, and an area where the target device is located in the first gray-scale image is extracted by an edge detection algorithm to obtain a target image;

[0014] For each pixel in the target image, a target temperature value corresponding to the pixel is obtained according to the gray value of the pixel and the temperature color bar;

[0015] Obtaining a temperature value with the highest target temperature value in the target image as a first temperature value, and a temperature value with the lowest target temperature value as a second temperature value;

[0016] The third temperature value is obtained by calculating the average value of all target temperature values ​​in the target image, and the target image feature is obtained by calculating the average value of the first temperature value, the second temperature value and the third temperature value.

[0017] Optionally, extracting features from the electrical data and the voiceprint data by a fully connected neural network to obtain target electrical features and target voiceprint features includes:

[0018] For the electrical data, wavelet transform is performed on the electrical data by using wavelet packets to obtain voltage wavelet coefficients, current wavelet coefficients and power wavelet coefficients;

[0019] For each type of electrical data, a feature data set corresponding to the type of electrical data is obtained according to the wavelet coefficient of the type, the feature data set is substituted into a fully connected neural network for feature extraction to obtain a target feature vector, and the target feature vectors of all types of electrical data in the electrical data are obtained to obtain electrical features; the electrical features include a voltage feature vector, a current feature vector and a power feature vector;

[0020] For the amplitude data and the frequency data in the voiceprint data, convert the amplitude data and the frequency data into a first amplitude spectrum and a first frequency spectrum by short-time Fourier transform;

[0021] The target spectrum is converted into Mel scale, and discrete cosine transform is performed to obtain the target coefficient, the preset number of coefficients are obtained to obtain the preset amplitude feature, and the preset amplitude feature is substituted into the fully connected neural network to obtain the target amplitude feature; if the target spectrum is the first amplitude spectrum, the target amplitude feature is the amplitude feature vector; if the target spectrum is the first frequency spectrum, the target amplitude feature is the frequency feature vector; the amplitude feature vector and the frequency feature vector are recorded as voiceprint features.

[0022] Optionally, performing feature fusion on the target image feature, the target electrical feature, and the target voiceprint feature according to the electrical stress value and the voiceprint stress value to obtain a target total feature includes:

[0023] By the formula T = T t +(w1I+w2U+w3P)+(w4A+w5f) to get the total target features;

[0024] Where T is the total characteristic of the target, T t is the target image feature, w1, w2, w3, w4, w5 are respectively the stress value of the current data, the stress value of the voltage data, the stress value of the power data, the stress value of the amplitude data, and the stress value of the frequency data, I is the current feature vector, U is the voltage feature vector, P is the power feature vector, A is the amplitude feature vector, and f is the frequency feature vector.

[0025] In a second aspect of the present invention, a cross-modal data fusion system for power equipment based on deep learning is proposed, comprising:

[0026] A stress value extraction module, used to collect image data, electrical data and voiceprint data of the target device during a preset time period, and extract stress values ​​from the electrical data and the voiceprint data to obtain electrical stress values ​​and voiceprint stress values;

[0027] A feature extraction module, used to extract features from the image data to obtain target image features, and to extract features from the electrical data and the voiceprint data through a fully connected neural network to obtain target electrical features and target voiceprint features;

[0028] A feature fusion module is used to perform feature fusion on the target image feature, the target electrical feature and the target voiceprint feature according to the electrical stress value and the voiceprint stress value to obtain a target total feature, and to search a preset database according to the target total feature to determine the fault type of the target device.

[0029] Optionally, the electrical data includes voltage data, current data and power data; the voiceprint data includes amplitude data and frequency data; and the stress value extraction module includes:

[0030] A preprocessing module, used for performing data preprocessing on the electrical data and the voiceprint data to obtain first electrical data and first voiceprint data;

[0031] A stress value calculation module is used to map the target class data to a preset target interval to obtain a target data discrete point set, obtain the target average relative deviation of the target class data according to the target data discrete point set, and normalize the target average relative deviation of all classes in the target data set to obtain the stress value of each class; if the target data set is the first electrical data, the target class data is any one of voltage data, current data or power data; if the target data set is the first voiceprint data; the target class data is any one of amplitude data or frequency data; the target class data is any one of the target data sets.

[0032] Optionally, the feature extraction module includes:

[0033] A target image production module, configured to perform grayscale processing on the image data to obtain a first grayscale image, and to extract the area where the target device is located in the first grayscale image by using an edge detection algorithm to obtain a target image;

[0034] A target temperature value calculation module is used to obtain, for each pixel in the target image, a target temperature value corresponding to the pixel according to the gray value of the pixel and the temperature color bar;

[0035] A temperature extreme value determination module, used for obtaining a temperature value with the highest target temperature value in the target image as a first temperature value, and a temperature value with the lowest target temperature value as a second temperature value;

[0036] The target image feature generation module is used to calculate the average value of all target temperature values ​​in the target image to obtain a third temperature value, and calculate the average value of the first temperature value, the second temperature value and the third temperature value to obtain a target image feature.

[0037] Optionally, the feature extraction module further includes:

[0038] A wavelet transform module, for performing wavelet transform on the electrical data through wavelet packets to obtain voltage wavelet coefficients, current wavelet coefficients and power wavelet coefficients;

[0039] An electrical feature generation module is used to obtain a feature data set corresponding to each type of electrical data according to the wavelet coefficients of each type of electrical data, substitute the feature data set into a fully connected neural network for feature extraction to obtain a target feature vector, and obtain the target feature vectors of all types of electrical data in the electrical data to obtain electrical features; the electrical features include voltage feature vectors, current feature vectors, and power feature vectors;

[0040] A spectrum conversion module, for converting the amplitude data and the frequency data in the voiceprint data into a first amplitude spectrum and a first frequency spectrum through short-time Fourier transform;

[0041] The voiceprint feature generation module is used to convert the target spectrum into Mel scale, perform discrete cosine transform to obtain the target coefficient, obtain a preset number of coefficients to obtain a preset amplitude feature, substitute the preset amplitude feature into the fully connected neural network to obtain the target amplitude feature; if the target spectrum is the first amplitude spectrum, the target amplitude feature is an amplitude feature vector; if the target spectrum is the first frequency spectrum, the target amplitude feature is a frequency feature vector; the amplitude feature vector and the frequency feature vector are recorded as voiceprint features.

[0042] Optionally, the feature fusion module includes:

[0043] The target total feature calculation module is used to calculate the total feature of the target through the formula

[0044] T=T t +(w1I+w2U+w3P)+(w4A+w5f) to get the total target features;

[0045] Where T is the total characteristic of the target, T tis the target image feature, w1, w2, w3, w4, w5 are respectively the stress value of the current data, the stress value of the voltage data, the stress value of the power data, the stress value of the amplitude data, and the stress value of the frequency data, I is the current feature vector, U is the voltage feature vector, P is the power feature vector, A is the amplitude feature vector, and f is the frequency feature vector.

[0046] Beneficial effects of the present invention:

[0047] The present invention proposes a cross-modal data fusion method for power equipment based on deep learning. The method collects image data, electrical data and voiceprint data of the target equipment through a preset time period, and extracts stress values ​​of the electrical data and voiceprint data to obtain electrical stress values ​​and voiceprint stress values ​​respectively; performs feature extraction on the image data to obtain target image features, and performs feature extraction on the electrical data and voiceprint data through a fully connected neural network to obtain target electrical features and target voiceprint features; performs feature fusion on the target image features, target electrical features and target voiceprint features according to the electrical stress values ​​and voiceprint stress values ​​to obtain target total features, and searches a preset database according to the target total features to determine the fault type of the target equipment. Electrical, voiceprint, and image data are collected synchronously at a preset time period to ensure the alignment of time and space, extract stress values ​​from electrical and voiceprint data, and use neural networks to extract each modal feature separately to maximize the key information expression of each data. The target features of electrical, voiceprint, and image are then merged into a unified overall feature, reducing the limitations of single modal data, generating more comprehensive characterization information, and improving the correlation between different data. The target overall feature is then matched and compared with the preset database to quickly identify the fault type, thereby improving the overall stability and operating efficiency of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present invention will be further described below in conjunction with the accompanying drawings.

[0049] Figure 1 A flowchart of a method for cross-modal data fusion of power equipment based on deep learning is provided for an embodiment of the present invention;

[0050] Figure 2 A framework diagram of a cross-modal data fusion system for power equipment based on deep learning is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the present invention, the description of "first", "second", etc. is only used for descriptive purposes, and cannot be understood as indicating or implying its relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0052] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0053] The embodiment of the present invention provides a method for cross-modal data fusion of power equipment based on deep learning. Figure 1 , Figure 1 A flowchart of a method for cross-modal data fusion of power equipment based on deep learning provided in an embodiment of the present invention. The method comprises the following steps:

[0054] S101, collecting image data, electrical data and voiceprint data of a target device in a preset time period, and extracting stress values ​​from the electrical data and voiceprint data to obtain electrical stress values ​​and voiceprint stress values;

[0055] S102, extracting features from the image data to obtain target image features, and extracting features from the electrical data and the voiceprint data using a fully connected neural network to obtain target electrical features and target voiceprint features;

[0056] S103, performing feature fusion on the target image feature, the target electrical feature and the target voiceprint feature according to the electrical stress value and the voiceprint stress value to obtain a target total feature, and searching a preset database according to the target total feature to determine the fault type of the target device.

[0057] A cross-modal data fusion method for power equipment based on deep learning is provided based on an embodiment of the present invention. Electrical, voiceprint and image data are synchronously collected through a preset time period to ensure the alignment of time series and space, extract stress values ​​from the electrical and voiceprint data, and use a neural network to extract the features of each modality respectively, maximize the key information expression of each data, and then fuse the target features of the electrical, voiceprint and image into a unified overall feature, reducing the limitations of single modal data, generating more comprehensive characterization information, and improving the correlation between different data. Then, by matching and comparing the target overall feature with the preset database, the fault type can be quickly identified, thereby improving the overall stability and operation efficiency of the power system.

[0058] In one implementation, image, electrical and voiceprint data are synchronously collected at a preset time period, and feature extraction is performed on these data to ensure that data of different modes have consistent time and space alignment, thereby improving the correlation between data of each mode, providing a better basis for subsequent feature fusion, and enabling the model to comprehensively and integratedly analyze the status of the equipment; stress values ​​of electrical and voiceprint data are extracted separately, and further deep extraction of electrical, voiceprint and image features is performed through neural networks to refine key information in each modal data, which is helpful for subsequent feature fusion and fault identification, making the diagnosis results more accurate.

[0059] In one implementation, the stress values ​​in this scheme include electrical stress values ​​and voiceprint stress values, wherein the electrical stress values ​​include stress values ​​of current data, stress values ​​of voltage data, and stress values ​​of power data; the voiceprint stress values ​​include stress values ​​of amplitude data and stress values ​​of frequency data; the stress values ​​of current data, the stress values ​​of voltage data, and the stress values ​​of power data are respectively normalized values ​​of the average relative deviation of current data, the average relative deviation of voltage data, and the average relative deviation of power data, the stress values ​​of amplitude data and the stress values ​​of frequency data are normalized values ​​of the average relative deviation of amplitude data and the average relative deviation of frequency data, and the normalization method may be linear normalization, Z-score normalization, and the like.

[0060] In one implementation, the preset time period is determined by a technician, the preset database is used to store target overall features corresponding to different fault types; the target device is the power equipment that needs to be detected; and the image data is an infrared image taken containing the target device.

[0061] In one implementation, by considering multi-dimensional information such as electrical, voiceprint and image, the real operating status of the device can be more comprehensively characterized. Compared with single-mode or simple parallel multi-mode solutions, the fused feature information is richer, which improves the accuracy and robustness of fault diagnosis.

[0062] In one embodiment, the electrical data includes voltage data, current data and power data; the voiceprint data includes amplitude data and frequency data; and extracting stress values ​​from the electrical data and the voiceprint data to obtain electrical stress values ​​and voiceprint stress values ​​includes:

[0063] Preprocessing the electrical data and the voiceprint data to obtain first electrical data and first voiceprint data respectively;

[0064] The target class data is mapped to the preset target interval to obtain a discrete point set of target data, the target average relative deviation of the target class data is obtained according to the discrete point set of target data, and the target average relative deviation of all classes in the target data set is normalized to obtain the stress value of each class; if the target data set is the first electrical data, the target class data is any one of voltage data, current data or power data; if the target data set is the first voiceprint data; the target class data is any one of amplitude data or frequency data; the target class data is any one of the target data sets.

[0065] In one implementation, the target class data is mapped to a preset target interval to form a discrete point set, which helps to unify the scale and distribution characteristics of different types of data, avoids the differences in units and amplitudes of each modal data, and thus improves the overall consistency and quality of the model input data.

[0066] In one implementation, the target average relative deviation of the target data is obtained according to the discrete point set of the target data. Specifically, thresholds are set for current, voltage and power, respectively, as a voltage threshold, a current threshold and a power threshold. The voltage threshold, the current threshold and the power threshold are determined by the technicians. Specifically, taking the current as an example, for each current data collected within a preset time period, the current threshold is subtracted from the current data to obtain a first current difference, the absolute value of the first current difference is divided by the current threshold to obtain a local current difference, and the local current difference corresponding to all the current data is calculated. The average value of the current difference obtains the target average relative deviation. For example, the current threshold is 5 amperes, the collected current data is (4.9, 4.7, 4.8, 4.7, 5.2, 5.1), the first current difference is calculated to obtain (-0.1, -0.3, -0.2, -0.3, 0.2, 0.1), the local current difference is calculated to obtain (0.02, 0.06, 0.04, 0.06, 0.04, 0.02), the average value of the local current difference is calculated to obtain 0.04, and the target average relative deviation corresponding to the current is 0.04.

[0067] In one implementation, data preprocessing includes common operations such as data denoising and data outlier processing to improve data quality. The preset target range is the electrical characteristics of the power equipment, such as voltage and current, which have specific rated ranges in the equipment manual or industry standards. The upper and lower limits of the preset target range are set according to these rated ranges. If there is a large amount of data that is not in the target range, it means that the equipment is abnormal.

[0068] In one implementation, the problem of magnitude differences between modes and classes is effectively solved by normalizing the target average relative deviations of all classes, making each stress value comparable at the same scale, thereby improving the robustness of the model in multimodal data fusion.

[0069] In one embodiment, the image data is an infrared image with a temperature color bar, and feature extraction of the image data to obtain target image features includes:

[0070] Grayscale processing is performed on the image data to obtain a first grayscale image, and an area where a target device is located in the first grayscale image is extracted by an edge detection algorithm to obtain a target image;

[0071] For each pixel in the target image, the target temperature value corresponding to the pixel is obtained according to the gray value of the pixel and the temperature color bar;

[0072] The temperature value with the highest target temperature value in the target image is recorded as the first temperature value, and the temperature value with the lowest target temperature value is recorded as the second temperature value;

[0073] The third temperature value is obtained by calculating the average value of all target temperature values ​​in the target image, and the target image feature is obtained by calculating the average value of the first temperature value, the second temperature value and the third temperature value.

[0074] In one implementation, grayscale processing converts a color image into a single-channel image, significantly reducing the dimension of the data, making subsequent processing more efficient and feature extraction more accurate; by extracting the area where the target device is located through an edge detection algorithm, important information can be effectively separated from the image, background interference can be reduced, and the recognition rate of the target device and the accuracy of feature extraction can be improved.

[0075] In one implementation, the corresponding target temperature value is obtained based on the grayscale value of the pixel and the temperature color bar, and the temperature information in the image is quantified into usable data. The image information can be intuitively converted into numerical features, which enhances the interpretability and applicability of the image data and facilitates subsequent analysis and decision-making.

[0076] In one implementation, the target temperature value corresponding to the pixel is obtained according to the gray value of the pixel and the temperature color bar, specifically, the lower limit and the upper limit of the temperature range are obtained according to the temperature color bar.

[0077] By formula Determine the target temperature value corresponding to the pixel, where Z is the target temperature value corresponding to the pixel, and Z min is the lower limit of temperature, G is the gray value of the current pixel, Z max The upper limit of temperature.

[0078] In one implementation, the highest temperature, the lowest temperature and their average values ​​in the target image are obtained, so that the thermal characteristics of the device can be captured from multiple dimensions, so that the extracted target image characteristics not only reflect the extreme temperature conditions, but also show the overall temperature distribution, which helps to more comprehensively understand the operating status of the device; by calculating the average values ​​of the first temperature value, the second temperature value and the third temperature value, a relatively stable characteristic value can be obtained, which reduces the misjudgment caused by a single abnormal temperature value, enhances the model's adaptability to temperature data fluctuations, and thus improves the reliability of fault diagnosis.

[0079] In one embodiment, extracting features from electrical data and voiceprint data by a fully connected neural network to obtain target electrical features and target voiceprint features includes:

[0080] For electrical data, wavelet transform is performed on the electrical data through wavelet packets to obtain voltage wavelet coefficients, current wavelet coefficients and power wavelet coefficients;

[0081] For each type of electrical data, the wavelet coefficients are used to obtain the feature data set corresponding to the type of electrical data. The feature data set is substituted into the fully connected neural network for feature extraction to obtain the target feature vector. The target feature vectors of all types of electrical data in the electrical data are obtained to obtain the electrical features. The electrical features include voltage feature vectors, current feature vectors and power feature vectors.

[0082] For the amplitude data and frequency data in the voiceprint data, convert the amplitude data and the frequency data into a first amplitude spectrum and a first frequency spectrum through short-time Fourier transform;

[0083] The target spectrum is converted into Mel scale and discrete cosine transform is performed to obtain the target coefficient. The preset number of coefficients are obtained to obtain the preset amplitude features. The preset amplitude features are substituted into the fully connected neural network to obtain the target amplitude features. If the target spectrum is the first amplitude spectrum, the target amplitude feature is the amplitude feature vector. If the target spectrum is the first frequency spectrum, the target amplitude feature is the frequency feature vector. The amplitude feature vector and the frequency feature vector are recorded as voiceprint features.

[0084] In one implementation, wavelet transform is used to extract wavelet coefficients of voltage, wavelet coefficients and power, which can fully reflect the instantaneous characteristics, frequency characteristics and energy distribution of electrical signals; short-time Fourier transform can effectively process the time and frequency information in the voiceprint signal, convert the amplitude and frequency data into a spectrum, provide rich frequency domain features, and help to accurately identify voiceprint features.

[0085] In one implementation, the extracted feature data set may be mean, standard deviation, maximum value, minimum value, root mean square value, etc. By integrating amplitude features and frequency features to form a comprehensive voiceprint feature, the accuracy of equipment fault diagnosis and identification can be significantly improved.

[0086] In one implementation, the fully connected neural network has an input layer of 5 nodes, two hidden layers, 32 neurons in each layer, and an activation function of ReLU, and an output layer of 1 node, and an activation function of sigmoid.

[0087] In one implementation, by performing unified feature extraction on electrical data and voiceprint data, cross-modal learning can be achieved, providing the model with richer contextual information, which helps to more comprehensively understand the operating status of the device; the first preset number is the same as the number of electrical features. If the feature data set of the electrical features includes mean, standard deviation, maximum value, minimum value and root mean square value, the first preset number is 5.

[0088] In one embodiment, the target total feature is obtained by performing feature fusion on the target image feature, the target electrical feature and the target voiceprint feature according to the electrical stress value and the voiceprint stress value, including:

[0089] By the formula T = T t +(w1I+w2U+w3P)+(w4A+w5f) to get the total target features;

[0090] Among them, T is the total feature of the target, T t is the target image feature, w1, w2, w3, w4, and w5 are the stress values ​​of current data, voltage data, power data, amplitude data, and frequency data respectively, I is the current feature vector, U is the voltage feature vector, P is the power feature vector, A is the amplitude feature vector, and f is the frequency feature vector.

[0091] In one implementation, the features of different modes are associated with the corresponding electrical stress values ​​through a calculation formula, so that the feature fusion process does not simply superimpose the data of different modes, but determines the fusion ratio according to the weights corresponding to different data, which can more comprehensively reflect the operating status of the equipment.

[0092] In one implementation, the target image features, electrical features, and target voiceprint features are in the same dimension and have the same number of features; if the number of electrical features is 5, and the average of the first temperature value, the second temperature value, and the third temperature value is denoted as P, then the target image features are (P, P, P, P, P).

[0093] Based on the same inventive concept, the embodiment of the present invention also provides a cross-modal data fusion system for power equipment based on deep learning. Figure 2 , Figure 2 A framework diagram of a cross-modal data fusion system for power equipment based on deep learning provided by an embodiment of the present invention includes:

[0094] A stress value extraction module is used to collect image data, electrical data and voiceprint data of the target device through a preset time period, and extract stress values ​​from the electrical data and voiceprint data to obtain electrical stress values ​​and voiceprint stress values;

[0095] A feature extraction module is used to extract features from image data to obtain target image features, and to extract features from electrical data and voiceprint data through a fully connected neural network to obtain target electrical features and target voiceprint features;

[0096] The feature fusion module is used to fuse the target image features, target electrical features and target voiceprint features according to the electrical stress value and the voiceprint stress value to obtain the target total feature, and to search the preset database according to the target total feature to determine the fault type of the target device.

[0097] A cross-modal data fusion system for power equipment based on deep learning is provided based on an embodiment of the present invention. Electrical, voiceprint and image data are synchronously collected through a preset time period to ensure the alignment of time series and space, and stress values ​​are extracted from the electrical and voiceprint data. The neural network is used to extract the features of each modality respectively, and the key information expression of each data is maximized. The target features of the electrical, voiceprint and image are then fused into a unified overall feature, which reduces the limitations of single modal data, generates more comprehensive characterization information, and improves the correlation between different data. The target overall feature is then matched and compared with the preset database to quickly identify the fault type, thereby improving the overall stability and operation efficiency of the power system.

[0098] In one embodiment, the electrical data includes voltage data, current data and power data; the voiceprint data includes amplitude data and frequency data; and the stress value extraction module includes:

[0099] A preprocessing module, used for performing data preprocessing on the electrical data and the voiceprint data to obtain first electrical data and first voiceprint data;

[0100] A stress value calculation module is used to map the target class data to a preset target interval to obtain a discrete point set of target data, obtain the target average relative deviation of the target class data based on the discrete point set of target data, and normalize the target average relative deviation of all classes in the target data set to obtain the stress value of each class; if the target data set is the first electrical data, the target class data is any one of voltage data, current data or power data; if the target data set is the first voiceprint data; the target class data is any one of amplitude data or frequency data; the target class data is any one of the target data sets.

[0101] In one embodiment, the feature extraction module includes:

[0102] A target image production module, used for graying the image data to obtain a first gray image, and extracting the area where the target device is located in the first gray image by an edge detection algorithm to obtain the target image;

[0103] The target temperature value calculation module is used to obtain the target temperature value corresponding to each pixel in the target image according to the gray value of the pixel and the temperature color bar;

[0104] A temperature extreme value determination module, used for obtaining a temperature value with the highest target temperature value in the target image as a first temperature value, and a temperature value with the lowest target temperature value as a second temperature value;

[0105] The target image feature generation module is used to calculate the average value of all target temperature values ​​in the target image to obtain a third temperature value, and calculate the average value of the first temperature value, the second temperature value and the third temperature value to obtain the target image feature.

[0106] In one embodiment, the feature extraction module further includes:

[0107] The wavelet transform module is used to perform wavelet transform on the electrical data through wavelet packets to obtain voltage wavelet coefficients, current wavelet coefficients and power wavelet coefficients;

[0108] The electrical feature generation module is used to obtain the feature data set corresponding to each type of electrical data according to the wavelet coefficient of each type of electrical data, substitute the feature data set into the fully connected neural network for feature extraction to obtain the target feature vector, and obtain the target feature vectors of all types of electrical data in the electrical data to obtain the electrical features; the electrical features include voltage feature vectors, current feature vectors and power feature vectors;

[0109] A spectrum conversion module, for converting the amplitude data and frequency data in the voiceprint data into a first amplitude spectrum and a first frequency spectrum through short-time Fourier transform;

[0110] The voiceprint feature generation module is used to convert the target spectrum into Mel scale, perform discrete cosine transform to obtain the target coefficient, obtain the preset number of coefficients to obtain the preset amplitude feature, substitute the preset amplitude feature into the fully connected neural network to obtain the target amplitude feature; if the target spectrum is the first amplitude spectrum, the target amplitude feature is the amplitude feature vector; if the target spectrum is the first frequency spectrum, the target amplitude feature is the frequency feature vector; the amplitude feature vector and the frequency feature vector are recorded as voiceprint features.

[0111] In one embodiment, the feature fusion module includes:

[0112] The target total feature calculation module is used to calculate the total feature of the target through the formula

[0113] T=T t +(w1I+w2U+w3P)+(w4A+w5f) to get the total target features;

[0114] Among them, T is the total feature of the target, T t is the target image feature, w1, w2, w3, w4, and w5 are the stress values ​​of current data, voltage data, power data, amplitude data, and frequency data respectively, I is the current feature vector, U is the voltage feature vector, P is the power feature vector, A is the amplitude feature vector, and f is the frequency feature vector.

[0115] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A cross-modal data fusion method for power equipment based on deep learning, characterized in that: The method comprises: Collecting image data, electrical data and voiceprint data of the target device in a preset time period, and extracting stress values ​​from the electrical data and the voiceprint data to obtain electrical stress values ​​and voiceprint stress values; Extracting features from the image data to obtain target image features, and extracting features from the electrical data and the voiceprint data using a fully connected neural network to obtain target electrical features and target voiceprint features; The target image feature, the target electrical feature and the target voiceprint feature are fused according to the electrical stress value and the voiceprint stress value to obtain a target total feature, and a preset database is searched according to the target total feature to determine the fault type of the target device.

2. According to a method for cross-modal data fusion of power equipment based on deep learning according to claim 1, it is characterized in that: The electrical data includes voltage data, current data and power data; the voiceprint data includes amplitude data and frequency data; Extracting stress values ​​from the electrical data and the voiceprint data to obtain electrical stress values ​​and voiceprint stress values ​​includes: Preprocessing the electrical data and the voiceprint data to obtain first electrical data and first voiceprint data respectively; The target class data is mapped to a preset target interval to obtain a discrete point set of target data, the target average relative deviation of the target class data is obtained according to the discrete point set of target data, and the target average relative deviation of all classes in the target data set is normalized to obtain the stress value of each class; if the target data set is the first electrical data, the target class data is any one of voltage data, current data or power data; if the target data set is the first voiceprint data; the target class data is any one of amplitude data or frequency data; the target class data is any one of the target data sets.

3. According to a method for cross-modal data fusion of power equipment based on deep learning according to claim 2, it is characterized in that: The image data is an infrared image with a temperature color bar, and feature extraction of the image data to obtain target image features includes: Gray-scale processing is performed on the image data to obtain a first gray-scale image, and an area where the target device is located in the first gray-scale image is extracted by an edge detection algorithm to obtain a target image; For each pixel in the target image, a target temperature value corresponding to the pixel is obtained according to the gray value of the pixel and the temperature color bar; Obtaining a temperature value with the highest target temperature value in the target image as a first temperature value, and a temperature value with the lowest target temperature value as a second temperature value; The third temperature value is obtained by calculating the average value of all target temperature values ​​in the target image, and the target image feature is obtained by calculating the average value of the first temperature value, the second temperature value and the third temperature value.

4. According to claim 2, a method for cross-modal data fusion of power equipment based on deep learning is characterized in that: The target electrical features and the target voiceprint features are obtained by extracting the features of the electrical data and the voiceprint data respectively through a fully connected neural network, including: For the electrical data, wavelet transform is performed on the electrical data by using wavelet packets to obtain voltage wavelet coefficients, current wavelet coefficients and power wavelet coefficients; For each type of electrical data, a feature data set corresponding to the type of electrical data is obtained according to the wavelet coefficient of the type, the feature data set is substituted into a fully connected neural network for feature extraction to obtain a target feature vector, and the target feature vectors of all types of electrical data in the electrical data are obtained to obtain electrical features; the electrical features include a voltage feature vector, a current feature vector and a power feature vector; For the amplitude data and the frequency data in the voiceprint data, convert the amplitude data and the frequency data into a first amplitude spectrum and a first frequency spectrum by short-time Fourier transform; The target spectrum is converted into Mel scale, and discrete cosine transform is performed to obtain the target coefficient, the preset number of coefficients are obtained to obtain the preset amplitude feature, and the preset amplitude feature is substituted into the fully connected neural network to obtain the target amplitude feature; if the target spectrum is the first amplitude spectrum, the target amplitude feature is the amplitude feature vector; if the target spectrum is the first frequency spectrum, the target amplitude feature is the frequency feature vector; the amplitude feature vector and the frequency feature vector are recorded as voiceprint features.

5. According to claim 4, a method for cross-modal data fusion of power equipment based on deep learning is characterized in that: The target total feature is obtained by performing feature fusion on the target image feature, the target electrical feature and the target voiceprint feature according to the electrical stress value and the voiceprint stress value, including: By the formula T = T t +(w1I+w2U+w3P)+(w4A+w5f) to get the total target features; Where T is the total characteristic of the target, T t is the target image feature, w1, w2, w3, w4, w5 are respectively the stress value of the current data, the stress value of the voltage data, the stress value of the power data, the stress value of the amplitude data, and the stress value of the frequency data, I is the current feature vector, U is the voltage feature vector, P is the power feature vector, A is the amplitude feature vector, and f is the frequency feature vector.

6. A cross-modal data fusion system for power equipment based on deep learning, characterized in that: The system comprises: A stress value extraction module, used to collect image data, electrical data and voiceprint data of the target device during a preset time period, and extract stress values ​​from the electrical data and the voiceprint data to obtain electrical stress values ​​and voiceprint stress values; A feature extraction module, used to extract features from the image data to obtain target image features, and to extract features from the electrical data and the voiceprint data through a fully connected neural network to obtain target electrical features and target voiceprint features; A feature fusion module is used to perform feature fusion on the target image feature, the target electrical feature and the target voiceprint feature according to the electrical stress value and the voiceprint stress value to obtain a target total feature, and to search a preset database according to the target total feature to determine the fault type of the target device.

7. The deep learning-based cross-modal data fusion system for electric power equipment according to claim 6, characterized in that: The electrical data includes voltage data, current data and power data; the voiceprint data includes amplitude data and frequency data; The stress value extraction module comprises: A preprocessing module, used for performing data preprocessing on the electrical data and the voiceprint data to obtain first electrical data and first voiceprint data; A stress value calculation module is used to map the target class data to a preset target interval to obtain a target data discrete point set, obtain the target average relative deviation of the target class data according to the target data discrete point set, and normalize the target average relative deviation of all classes in the target data set to obtain the stress value of each class; if the target data set is the first electrical data, the target class data is any one of voltage data, current data or power data; if the target data set is the first voiceprint data, the target class data is any one of amplitude data or frequency data; the target class data is any one of the target data sets.

8. The deep learning-based cross-modal data fusion system for electric power equipment according to claim 7, characterized in that: The feature extraction module comprises: A target image production module, configured to perform grayscale processing on the image data to obtain a first grayscale image, and extract the area where the target device is located in the first grayscale image by using an edge detection algorithm to obtain a target image; A target temperature value calculation module is used to obtain, for each pixel in the target image, a target temperature value corresponding to the pixel according to the gray value of the pixel and the temperature color bar; A temperature extreme value determination module, used for obtaining a temperature value with the highest target temperature value in the target image as a first temperature value, and a temperature value with the lowest target temperature value as a second temperature value; The target image feature generation module is used to calculate the average value of all target temperature values ​​in the target image to obtain a third temperature value, and calculate the average value of the first temperature value, the second temperature value and the third temperature value to obtain a target image feature.

9. The deep learning-based cross-modal data fusion system for electric power equipment according to claim 7, characterized in that: The feature extraction module also includes: A wavelet transform module, for performing wavelet transform on the electrical data through wavelet packets to obtain voltage wavelet coefficients, current wavelet coefficients and power wavelet coefficients; An electrical feature generation module is used to obtain a feature data set corresponding to each type of electrical data according to the wavelet coefficients of each type of electrical data, substitute the feature data set into a fully connected neural network for feature extraction to obtain a target feature vector, and obtain the target feature vectors of all types of electrical data in the electrical data to obtain electrical features; the electrical features include voltage feature vectors, current feature vectors, and power feature vectors; A spectrum conversion module, for converting the amplitude data and the frequency data in the voiceprint data into a first amplitude spectrum and a first frequency spectrum through short-time Fourier transform; The voiceprint feature generation module is used to convert the target spectrum into Mel scale, perform discrete cosine transform to obtain the target coefficient, obtain a preset number of coefficients to obtain a preset amplitude feature, substitute the preset amplitude feature into the fully connected neural network to obtain the target amplitude feature; if the target spectrum is the first amplitude spectrum, the target amplitude feature is an amplitude feature vector; if the target spectrum is the first frequency spectrum, the target amplitude feature is a frequency feature vector; the amplitude feature vector and the frequency feature vector are recorded as voiceprint features.

10. A cross-modal data fusion system for power equipment based on deep learning according to claim 9, characterized in that: The feature fusion module includes: The target total feature calculation module is used to calculate the total feature of the target through the formula T=T t +(w1I+w2U+w3P)+(w4A+w5f) to get the total target features; Where T is the total characteristic of the target, T t is the target image feature, w1, w2, w3, w4, w5 are respectively the stress value of the current data, the stress value of the voltage data, the stress value of the power data, the stress value of the amplitude data, and the stress value of the frequency data, I is the current feature vector, U is the voltage feature vector, P is the power feature vector, A is the amplitude feature vector, and f is the frequency feature vector.

Images