Power equipment defect detection method, system and device based on multi-modal alignment

Abstract

The application discloses a power equipment defect detection method, system and device based on multi-modal alignment, and particularly relates to the technical field of power equipment detection, and the technical points are as follows: the method comprises the following steps: in the collected visible light image, infrared image and ultrasonic signal, the spatial feature difference and the time feature difference between every two modal data are obtained; the spatial feature difference and the time feature difference are subjected to logic regression processing, the alignable coefficient between every two modal data is obtained, the to-be-aligned data combination for power equipment defect detection is determined based on the alignable coefficient; the to-be-aligned data combination is subjected to data alignment, the aligned modal data is obtained, and according to the aligned modal data, the time sequence model is used for defect detection analysis on the power equipment, so that the power equipment defect detection result is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power equipment detection, in particular to a power equipment defect detection method, system and device based on multi-modal alignment. BACKGROUND

[0002] Power equipment inspection is a key link to ensure the safe operation of the power grid. In recent years, detection technologies based on computer vision and deep learning have been widely used, but existing methods mainly rely on single modal data (such as infrared images, visible light images or ultrasonic signals), which are easily disturbed in complex working conditions, resulting in reduced defect detection accuracy. Therefore, how to fuse multiple modal data and effectively align data features to achieve accurate and intelligent detection has become an important research direction in this field.

[0003] At the same time, the existing technology mainly has the following shortcomings: single modal recognition cannot capture complementary features of equipment operating state, false positive rate increases sharply in foggy weather, equipment shielding and other scenes, and existing multi-modal fusion lacks a deep alignment mechanism, especially in extreme weather conditions, the selection of multi-modal parameters in the feature interaction process is relatively vague, which easily causes loss of key discriminative information.

[0004] Therefore, the present application aims to provide a power equipment defect detection method, system and device based on multi-modal alignment to solve the above-mentioned related problems. SUMMARY

[0005] The technical problem to be solved by the present application is that existing multi-modal fusion lacks a deep alignment mechanism, the selection of multi-modal parameters is relatively vague, and key discriminative information is easily lost. The present application provides a power equipment defect detection method, system and device based on multi-modal alignment, which acquires multi-modal data including visible light images, infrared images and ultrasonic signals, obtains spatial feature differences and temporal feature differences between visible light images, infrared images and ultrasonic signals, and calculates the alignable coefficients between multi-modal data by substituting the spatial feature differences and temporal feature differences into a logistic regression algorithm, determines the data combination required for power equipment alignment, and constructs a distribution alignment standard. After feature calibration and fusion of data with different alignment standards using contrastive learning, a time series model is used to detect power equipment defects. Based on the detection results, the defect alarm frequency can be optimized, the accuracy of data fusion can be improved, the robustness and generalization ability of data alignment can be improved, and the accuracy and reliability of power equipment defect detection can be met.

[0006] The present application is achieved by the following technical solutions:

[0007] A power equipment defect detection method based on multi-modal alignment, the method comprising:

[0008] In the collected visible light image, infrared image and ultrasonic signal, the spatial feature difference and the time feature difference between each two modal data are obtained;

[0009] The spatial feature difference and the time feature difference are subjected to logistic regression processing to obtain the alignable coefficient between each two modal data, and the to-be-aligned data combination for power equipment defect detection is determined based on the alignable coefficient;

[0010] The to-be-aligned data combination is subjected to data alignment to obtain the aligned modal data, and the power equipment is subjected to defect detection analysis by using a time series model according to the aligned modal data, so as to obtain the power equipment defect detection result.

[0011] Further, in the collected visible light image, infrared image and ultrasonic signal, the spatial feature difference and the time feature difference between each two modal data are obtained, specifically as follows:

[0012] The visible light image time feature, the infrared image time feature and the ultrasonic signal time feature are collected respectively, and the time feature difference between each two modal data is obtained by difference calculation.

[0013] A graph structure among the visible light image, the infrared image and the ultrasonic signal is constructed, and the spatial feature difference between each two modal data is extracted from the graph structure.

[0014] Further, the spatial feature difference and the time feature difference are subjected to logistic regression processing to obtain the alignable coefficient between each two modal data, specifically as follows:

[0015] The spatial feature difference and the time feature difference between each two modal data are input into a logistic regression algorithm for logistic regression processing to obtain the alignable coefficient between each two modal data.

[0016] Further, the to-be-aligned data combination for power equipment defect detection is determined based on the alignable coefficient, specifically as follows:

[0017] The alignable coefficient between each two modal data is compared with a preset alignable threshold;

[0018] If the alignable coefficient between two modal data is not less than the preset alignable threshold, the modal weight of the two modal data is increased respectively;

[0019] If the alignable coefficient between two modal data is less than the preset alignable threshold, the modal weight of the two modal data is not increased;

[0020] After comparing all the alignable coefficients, the modal data with the maximum modal weight is extracted as the to-be-aligned data combination for power equipment defect detection.

[0021] Further, the data alignment is performed on the to-be-aligned data combination to obtain aligned modal data, specifically as follows:

[0022] Obtain the distribution difference between the time feature and the space feature of the to-be-aligned modal data in the to-be-aligned data combination and the time feature and the space feature of the remaining modal data other than the to-be-aligned modal data.

[0023] If the distribution difference is not less than a preset distribution threshold, data alignment is performed on the to-be-aligned data combination by using a pre-constructed alignment transformation function to obtain the aligned modal data.

[0024] Further, according to the aligned modal data, a time series model is used to perform defect detection analysis on the power equipment to obtain a power equipment defect detection result, wherein the time series model specifically adopts an ARIMAX model, and specifically as follows:

[0025] The ARIMAX model parameters are estimated by using a maximum likelihood estimation method, and the estimated ARIMAX model parameters are input into the ARIMAX model for fitting to obtain a fitted ARIMAX model.

[0026] Data features of the aligned modal data are extracted, and the data features are input into the fitted ARIMAX model to perform defect detection analysis on the power equipment to obtain the power equipment defect detection result.

[0027] The application also provides a power equipment defect detection system based on multi-modal alignment, which is used in the power equipment defect detection method based on multi-modal alignment.

[0028] The feature difference acquisition module is configured to acquire the spatial feature difference and the time feature difference between each two modal data in the acquired visible light image, infrared image and ultrasonic signal.

[0029] The to-be-aligned data combination acquisition module is configured to perform logistic regression processing on the spatial feature difference and the time feature difference to obtain an alignable coefficient between each two modal data, and determine a to-be-aligned data combination for power equipment defect detection based on the alignable coefficient.

[0030] The power equipment defect detection module is configured to perform data alignment on the to-be-aligned data combination to obtain aligned modal data, and perform defect detection analysis on the power equipment by using a time series model according to the aligned modal data to obtain a power equipment defect detection result.

[0031] Further, in the collected visible light image, infrared image and ultrasonic signal, the spatial feature difference and the time feature difference between each two modal data are obtained, specifically:

[0032] The visible light image time feature, the infrared image time feature and the ultrasonic signal time feature are collected respectively, and the time feature difference between each two modal data is calculated by difference calculation, so as to obtain the time feature difference between each two modal data.

[0033] The graph structure between the visible light image, the infrared image and the ultrasonic signal is constructed, and the spatial feature difference between each two modal data is extracted from the graph structure.

[0034] Further, the spatial feature difference and the time feature difference are subjected to logistic regression processing, so as to obtain the alignable coefficient between each two modal data, specifically:

[0035] The spatial feature difference and the time feature difference between each two modal data are input into the logistic regression algorithm for logistic regression processing, so as to obtain the alignable coefficient between each two modal data.

[0036] The application further provides a computer device comprising a system memory and a processor, wherein the system memory stores a computer program, and the processor realizes the steps of the method in any one of the above methods when executing the computer program.

[0037] The application further provides a computer readable storage medium, which stores a computer program, and the computer program realizes the steps of the method in any one of the above methods when executed by a processor.

[0038] The application further provides a computer program product comprising instructions, which, when executed by a computer device cluster, cause the computer device cluster to perform the method in any one of the above methods.

[0039] Compared with the prior art, the application has the following advantages and beneficial effects:

[0040] In the present application, by collecting multi-modal data including visible light images, infrared images and ultrasonic signals, spatial feature differences and time feature differences between visible light images, infrared images and ultrasonic signals are obtained, and the spatial feature differences and time feature differences are substituted into a logistic regression algorithm to calculate the alignable coefficients between the multi-modal data, determine the data combination required to be aligned by the power equipment, and construct a distribution alignment standard. After feature calibration and fusion of data with different alignment standards by using contrast learning, a time series model is used to detect defects of the power equipment, and based on the detection result, the defect alarm frequency can be optimized, the accuracy of data fusion is improved, the robustness and generalization ability of data alignment are improved, the accuracy and reliability of power equipment defect detection are met, and the technical problems that the existing multi-modal fusion lacks a deep alignment mechanism and the selection of multi-modal parameters is relatively fuzzy, which is easy to cause loss of key discriminative information are solved. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor. In the drawings:

[0042] Figure 1 The method flow diagram of a power equipment defect detection method based on multi-modal alignment in the present embodiment;

[0043] Figure 2 The module diagram of a power equipment defect detection system based on multi-modal alignment in the present embodiment;

[0044] Figure 3 The structure diagram of a computer device in the present embodiment. DETAILED DESCRIPTION

[0045] The exemplary embodiments of the present disclosure are described below in conjunction with the drawings, including various details of the embodiments of the present disclosure to help understanding, and they should be considered only as exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Also, in order to be clear and concise, the description in the following description omits the description of well-known functions and structures.

[0046] In the present disclosure, the terms "first", "second", etc. used in the description of various examples are not intended to limit the positional relationship, timing relationship or importance relationship of the elements, and such terms are only used to distinguish one element from another. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0047] The terms used in the description of various examples in the present disclosure are only for the purpose of describing the specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in the present disclosure encompasses any one of the listed items and all possible combinations.

[0048] Embodiment 1

[0049] The present embodiment provides a power equipment defect detection method based on multi-modal alignment, see Figure 1 , Figure 1 A method flow diagram of a power equipment defect detection method based on multi-modal alignment is shown, wherein the method comprises:

[0050] S1: In the collected visible light image, infrared image and ultrasonic signal, the spatial feature difference and the time feature difference between each two modal data are obtained.

[0051] It should be noted that in the present embodiment, the plurality of modal data includes visible light image, infrared image and ultrasonic signal; wherein the visible light image data is obtained by a high-resolution industrial camera or a visible light camera carried by a drone, which usually has automatic focusing, optical zoom, high dynamic range (HDR) imaging functions, and can ensure clear power equipment images under different lighting conditions. Optionally, multi-angle shooting or structured light three-dimensional reconstruction technology can be combined during the visible light image acquisition process to obtain more rich spatial information and improve the accuracy of equipment defect recognition;

[0052] The infrared image data is obtained by an infrared thermal imager or an infrared thermal imaging camera carried by a drone. Based on the principle of thermal radiation, this device can detect the temperature distribution on the surface of the power equipment. The infrared camera usually has thermal imaging capability of different wavebands (short wave, medium wave, long wave), and combines functions such as automatic temperature compensation and radiation rate correction to improve the accuracy of temperature measurement. In practical applications, infrared data acquisition can be carried out in a fixed-point inspection, mobile shooting or unmanned aerial vehicle automatic inspection manner to ensure coverage of the entire equipment area;

[0053] The ultrasonic signal data is acquired by an ultrasonic sensor or a partial discharge detection device (such as an ultrasonic detector, an array ultrasonic sensor, an ultrasonic microphone, etc.). The ultrasonic sensor can detect defects such as mechanical vibration, partial discharge, crack, air gap, etc. inside the equipment, and usually uses a contact or non-contact measurement method. Non-contact measurement relies on air-coupled ultrasonic or laser ultrasonic technology, which can detect partial discharge signals at a long distance, while contact measurement uses a piezoelectric sensor directly attached to the surface of the equipment for high-precision defect detection. In addition, ultrasonic signal acquisition is usually combined with time-frequency analysis (short-time Fourier transform STFT, wavelet transform, etc.) to extract features and improve defect recognition capability. Feature extraction uses conventional techniques in the art, which will not be described in detail here.

[0054] Meanwhile, it should be noted that the spatial features and temporal features common to visible light images, infrared images and ultrasonic signal data are as follows: they are all sampled or photographed after a certain time interval, i.e. the temporal feature, and they all have corresponding spatial features, i.e. point sampling or point photography for each region of the power equipment.

[0055] Specifically, in this embodiment, first, the visible light image temporal feature, the infrared image temporal feature and the ultrasonic signal temporal feature are collected respectively, and the temporal feature difference between each two modal data is calculated by difference calculation.

[0056] It should be noted that in this embodiment, the visible light image temporal feature refers to the visible light image photographing timestamp, the infrared image temporal feature refers to the infrared image photographing timestamp, and the ultrasonic signal temporal feature refers to the ultrasonic signal sampling time.

[0057] Meanwhile, the difference between the visible light image timestamp and the infrared image photographing timestamp refers to the time interval when the same power equipment is photographed by two sensors (a visible light camera and an infrared thermal imager) in the same monitoring task. The acquisition logic is to synchronize the time by using a unified time source, subtract the infrared camera photographing timestamp from the visible light camera photographing timestamp to obtain the difference between the visible light image and the infrared image photographing timestamp, i.e. to obtain the temporal feature difference between the visible light image and the infrared image.

[0058] The difference between the visible light image and the ultrasonic signal sampling time refers to the time interval between the time when the visible light camera photographs the image and the time when the ultrasonic sensor collects the signal in the same monitoring task. The acquisition logic is to synchronize the time by using a high-precision clock, subtract the ultrasonic sensor sampling timestamp from the visible light camera photographing timestamp to obtain the difference between the visible light image and the ultrasonic signal sampling time, i.e. to obtain the temporal feature difference between the visible light image and the ultrasonic signal.

[0059] The infrared image and ultrasonic signal sampling time difference refers to a time interval between an infrared image collection time and an ultrasonic signal sampling time in the same power equipment monitoring process, and the acquisition logic is to subtract an ultrasonic sensor sampling time stamp from an infrared image shooting time stamp through a unified clock source to obtain the infrared image and ultrasonic signal sampling time difference, that is, a time characteristic difference between the infrared image and the ultrasonic signal;

[0060] Meanwhile, it also needs to be explained that the above-mentioned time synchronization methods are prior art, and those skilled in the art can understand that the data collection of different sensors is affected by factors such as hardware trigger delay, signal transmission delay, and sensor sampling rate, and there may be a small time deviation, so in actual application, high-precision time synchronization technology is usually combined, and the application of specific synchronization technology is not described here.

[0061] Then, a graph structure among the visible light image, the infrared image, and the ultrasonic signal is constructed, and a spatial feature difference between each two modal data is extracted from the graph structure.

[0062] It needs to be explained that in the embodiment, spatial features of the visible light image, the infrared image, and the ultrasonic signal are first extracted. For the visible light image, edge detection is used to extract power equipment structure information such as device edges, shape features, and damage areas. For the infrared image, temperature distribution features and hot spot areas are extracted and matched with the geometric shape of the visible light image (SIFT can be used for matching). For the ultrasonic signal, STFT and wavelet transform are used to extract time-frequency features, and the features are mapped to two-dimensional or three-dimensional space in combination with sensor position information.

[0063] Secondly, the visible light image, the infrared image, and the ultrasonic signal are organized into a graph structure, and basic elements of the graph are defined. It is assumed that nodes of the constructed graph are feature points in the visible light image, high-temperature area center points in the infrared image, and positions of the ultrasonic sensors, and edges are connected to data points of different modalities according to Euclidean distance, SSIM, and feature vector similarity. If two nodes belong to the same region or their feature similarity is higher than a set threshold, an edge is established between them. The threshold is set through a large amount of labeled power equipment multi-modal data based on supervised learning or unsupervised learning method, which is not described here.

[0064] Meanwhile, since there are different types of correlations between multiple modal data, a heterogeneous graph can be used; specifically, for visible light images and infrared images, the correspondence of feature points of the two is calculated (such as SIFT matching), and a connection is established between the matching points; for visible light images and ultrasonic signals, if the sampling point corresponding to the ultrasonic sensor is located in a certain device region in the visible light image, the sensor point and the device feature point are connected; for infrared images and ultrasonic signals, if there is a spatial overlap between the infrared hotspot region and the ultrasonic abnormal detection region, a connection is established;

[0065] Then a graph neural network is designed for spatial feature difference analysis, and the graph neural network is forward propagated using weighted learning of multi-modal features, and the specific calculation formula is as follows: wherein, is the node feature of the i-th layer, is a trainable weight matrix, is an attention weight, is an activation function, is a set of neighbor nodes of the node; represents the node feature of the i-th layer; the node feature calculated by the graph neural network, the spatial feature difference function is defined to measure the spatial alignment error between different modal data, and the specific function calculation formula is as follows: wherein, represents the spatial feature difference between two modal data, and are two modal feature representations of the final layer of the graph neural network, represents the Euclidean distance, i.e., the L2 norm; Finally, the visible light image, infrared image and ultrasonic signal data are substituted into the graph neural network to construct a graph structure, and the spatial feature difference between the visible light image and the infrared image is obtained from the graph structure, i.e., the spatial feature difference between the visible light image and the infrared image; the spatial feature difference between the visible light image and the ultrasonic signal is obtained, i.e., the spatial feature difference between the visible light image and the ultrasonic signal; the spatial feature difference between the infrared image and the ultrasonic signal is obtained, i.e., the spatial feature difference between the infrared image and the ultrasonic signal.

[0066] S2: Perform logistic regression on the spatial feature difference and the temporal feature difference to obtain an alignable coefficient between each two modal data, and determine the to-be-aligned data combination of the power equipment defect detection based on the alignable coefficient.

[0067] S2: Perform logistic regression on the spatial feature difference and the temporal feature difference to obtain an alignable coefficient between each two modal data, and determine the to-be-aligned data combination of the power equipment defect detection based on the alignable coefficient.

[0068] ​It should be noted that in the embodiment, since the spatial feature difference value is determined according to the spatial coordinates, it is a dimensionless parameter, and since the time feature difference value is determined according to the time length, it has a time unit, therefore the time dimension difference value is de-dimensioned; the de-dimensioning method specifically uses the Z-score standardization method to standardize the normal distribution of the time feature difference, so that it is on the same scale as the spatial feature difference, and the specific Z-score standardization method formula is expressed as: , wherein, is the mean, is the standard deviation, is the original value of the time feature difference value, is the time feature difference value standardization result;

[0069] Specifically, in the embodiment, first, the spatial feature difference and the time feature difference between each two modal data are input into the logistic regression algorithm for logistic regression processing to obtain the alignable coefficient between each two modal data.

[0070] It should be noted that in the embodiment, the spatial feature difference and the time feature difference of the visible light image and the infrared image, the spatial feature difference and the time feature difference of the visible light image and the ultrasonic signal, and the spatial feature difference and the time feature difference of the infrared image and the ultrasonic signal are respectively substituted into the logistic regression algorithm, and the specific formula is expressed as: , , ; wherein, , and are the logistic regression calculation results, that is, represents the alignable coefficient between the visible light image and the infrared image, represents the alignable coefficient between the infrared image and the ultrasonic signal, represents the alignable coefficient between the infrared image and the ultrasonic signal; e is the natural base, , and are linear combination terms of the logistic regression model, which can be specifically set as: , wherein, , and are bias terms of the data corresponding to each modal; and are the time feature difference and the spatial feature difference between the visible light image and the infrared image, and are the regression coefficients of the corresponding parameters; and are the time feature difference and the spatial feature difference between the infrared image and the ultrasonic signal, and are regression coefficients of corresponding parameters respectively; and are time feature difference and spatial feature difference between the ultrasonic signal and the visible light image respectively, and are regression coefficients of corresponding parameters respectively.

[0071] Meanwhile, it also needs to be explained that when the alignable coefficient is larger, it means that the data alignment operation between the current corresponding modalities is more needed, and vice versa, it means that the data between the current corresponding modalities does not need to be aligned.

[0072] Then the alignable coefficient between every two modal data is compared with the preset alignable threshold; if the alignable coefficient between two modal data is not less than the preset alignable threshold, the modal weight of the two modal data is increased respectively; if the alignable coefficient between two modal data is less than the preset alignable threshold, the modal weight of the two modal data is not increased;

[0073] It needs to be explained that in the embodiment, the alignable threshold is obtained by historical alignment error distribution statistics and supervised learning optimization; the initial modal weight of all modal data is 0 point; at the same time, the modal weight of the two modal data with the alignable coefficient greater than or equal to the preset alignable threshold is increased by 1 point, and the modal weight of the two modal data with the alignable coefficient less than the preset alignable threshold remains unchanged;

[0074] Finally, after comparing all the alignable coefficients, the modal data with the largest modal weight is extracted as the to-be-aligned data combination for power equipment defect detection.

[0075] For example, the alignable coefficient of the visible light image and the infrared image is greater than or equal to the preset alignable threshold, so the modal weight of the visible light image and the infrared image is increased by 1 point respectively; if the alignable coefficient of the visible light image and the ultrasonic signal is less than the preset alignable threshold, the modal weight of the visible light image and the ultrasonic signal remains unchanged; if the alignable coefficient of the infrared image and the ultrasonic signal is greater than or equal to the preset alignable threshold, the modal weight of the infrared image and the ultrasonic signal is increased by 1 point respectively; therefore, the above judgment accumulation can obtain that the modal weight of the visible light image is recorded as 1 point, the modal weight of the infrared image is recorded as 2 points, and the modal weight of the ultrasonic signal is recorded as 1 point, so the infrared image is extracted as the to-be-aligned data combination for power equipment defect detection;

[0076] It should be noted that the above content represents a multi-level alignment strategy based on a ternary mode, and the modal weight of the three modal data can include multiple cases; if there are two same modal weights, and another modal weight is less than the two same modal weights, then the modal data corresponding to the two same modal weights are combined as the data to be aligned for data alignment operation; if the modal weight is all 1 or 2, then it means that the visible light image, infrared image and ultrasonic signal all need to be aligned; if the modal weight is all 0, then no alignment is needed, and each modal data is directly substituted into the time series model to determine the defect detection of the power equipment.

[0077] S3: performing data alignment on the data to be aligned combination to obtain aligned modal data, and performing defect detection analysis on the power equipment by using a time series model according to the aligned modal data, to obtain a power equipment defect detection result.

[0078] Specifically, in the embodiment, first, the time feature and the space feature of the modal data to be aligned in the data to be aligned combination are acquired, and the distribution difference between the time feature and the space feature of the remaining modal data other than the modal data to be aligned is acquired;

[0079] It should be noted that in the embodiment, first, the time feature and the space feature of the modal data to be aligned constituting the data to be aligned combination are acquired, and then the time feature and the space feature of the remaining two modal data are acquired, the distribution of the space feature and the time feature thereof is calculated by statistical contrast, the distribution consistency of the modal data features is evaluated, and the specific formula is expressed as: , wherein, is the distribution difference, the space and time features of the visible light image are , the space and time features of the infrared image are , and the space and time features of the ultrasonic signal are wherein, is the probability distribution of the feature;

[0080] Then, if the distribution difference is not less than a preset distribution threshold, data alignment is performed on the data to be aligned combination by using a pre-constructed alignment transformation function to obtain aligned modal data;

[0081] It should be noted that in the embodiment, if the distribution consistency of the modal data is high (i.e. If the distribution difference is less than the preset distribution threshold (i.e., the value is low), it means that the alignment loss between them is small, indicating that the modal data has been well aligned, and the default data alignment is adopted; meanwhile, the default data alignment refers to a preliminary alignment of the data of different modalities in time and space based on the existing technology, rules or predefined reference mode without additional complex alignment strategy or data correction, to ensure the consistency of the data, and the alignment mode of the default data alignment is not limited, which will not be described here; meanwhile, the setting of the distribution threshold is based on the historical data set by the technical personnel, which will not be described here.

[0082] If the distribution difference is greater than or equal to the preset distribution threshold, it means that the modal data to be aligned needs to be further adjusted and aligned with the remaining modal data, and the pre-constructed alignment transformation function is needed to align the data of the modal data to be aligned, and the pre-constructed alignment transformation function is obtained by training the contrast learning model. Contrast learning is to learn the alignment of data by minimizing the distance between similar modalities and maximizing the distance between different modalities. The training process is as follows: first, a contrast learning loss function is constructed , and the expression is , wherein and represent the feature representation of the visible light image and the infrared image respectively, is a label (1 represents alignment, and 0 represents non-alignment), represents the Euclidean distance between two features, and are the positive and negative thresholds of the alignment loss, represents the number of samples; then, according to the loss function above, the contrast learning model parameters are optimized by gradient descent method to reduce the feature distance between different modalities. During the training process, the same type of data (e.g., visible light image and infrared image of the same device) is aligned as much as possible, while the distance between different types of data (e.g., visible light image and ultrasonic signal) is pulled apart, and the like, which will not be described here. The key of contrast learning is to generate positive and negative sample pairs. The positive sample pair is from the same device of two modalities (e.g., visible light and infrared image), and the negative sample pair is from different devices or different modalities. During the training process, the model is trained using alternating positive and negative sample pairs, so that the alignment accuracy between modalities is maximized. After the training is completed, the optimized alignment transformation function is obtained, and then the data is aligned according to the alignment transformation function for the modalities to be aligned.

[0083] Finally, based on the aligned modal data, a time series model is used to perform defect detection analysis on the power equipment to obtain the defect detection results. Specifically, the time series model adopts the ARIMAX model, which is implemented as follows: the maximum likelihood estimation method is used to estimate the ARIMAX model parameters, and the estimated ARIMAX model parameters are input into the ARIMAX model for fitting to obtain the fitted ARIMAX model; the data features of the aligned modal data are extracted and input into the fitted ARIMAX model to perform defect detection analysis on the power equipment to obtain the defect detection results.

[0084] It should be noted that in this embodiment, defect alarm frequency data needs to be obtained. This defect alarm frequency data refers to historical alarm frequency data, which serves as the main variable in the time series. Then, the maximum likelihood estimation method is used to estimate the ARIMAX model parameters. , , , , The specific steps are as follows: Error Term Follows a normal distribution Then the likelihood function is: Taking the logarithm of the likelihood function yields the log-likelihood function: The parameter estimates are obtained by maximizing the log-likelihood function. , , , , The estimated ARIMAX model parameters are input into the ARIMAX model for fitting, resulting in the fitted ARIMAX model. The basic form of the model is as follows: ,in, The alarm frequency at the current time point corresponds to the severity of power equipment defects at the current time point; Lagging i Alarm frequency of the level For constant terms, For the first i Autoregressive parameters of order 1, It is the order of the autoregressive term. For the first j 1-order moving average parameters It is the order of the moving average term. It is a white noise term with a lag of j periods. It is a white noise term, representing random error. It is an exogenous variable coefficient, It is the lag order of the exogenous variable. is the exogenous variable of the lag k period; the exogenous variable part can integrate the influence of other related variables (such as visible light image features, infrared image features and ultrasonic signal features) on the severity of the power equipment defect; finally, the data features of the aligned modal data are extracted and input into the ARIMAX model, and the power equipment is detected for defects to obtain the power equipment defect detection result.

[0085] Meanwhile, it should be noted that the visible light image features, infrared image features and ultrasonic signal features include but are not limited to color features, texture features, shape features, temperature features, temperature distribution features, time sequence features and time-frequency domain features, and the specific feature acquisition method belongs to the prior art, which is not described here.

[0086] Specifically, in the present application, by collecting multi-modal data including visible light images, infrared images and ultrasonic signals, the spatial feature difference and the time feature difference between the visible light images, the infrared images and the ultrasonic signals are obtained, and the spatial feature difference and the time feature difference are substituted into the logistic regression algorithm to calculate the alignable coefficient between the multi-modal data, determine the data combination required for the alignment of the power equipment, and construct the distribution alignment standard. After feature calibration and fusion of data with different alignment standards by using contrast learning, a time sequence model is used to detect the defects of the power equipment. Based on the detection result, the defect alarm frequency can be optimized, the accuracy of data fusion can be improved, the robustness and generalization ability of data alignment can be improved, the accuracy and reliability of power equipment defect detection can be met, and the technical problems of lack of deep alignment mechanism in existing multi-modal fusion and fuzzy selection of multi-modal parameters are solved, which easily causes loss of key discriminative information.

[0087] Embodiment 2

[0088] Referring to Figure 2 As shown in the figure, the present application also provides a power equipment defect detection system based on multi-modal alignment, which is used in the power equipment defect detection method based on multi-modal alignment described above, and the system comprises:

[0089] The feature difference acquisition module 100 is used to acquire the spatial feature difference and the time feature difference between each two modal data in the collected visible light images, infrared images and ultrasonic signals;

[0090] The to-be-aligned data combination acquisition module 200 is used to perform logistic regression processing on the spatial feature difference and the time feature difference to obtain the alignable coefficient between each two modal data, and determine the to-be-aligned data combination for power equipment defect detection based on the alignable coefficient;

[0091] The power equipment defect detection module 300 is configured to perform data alignment on the to-be-aligned data combination, obtain aligned modal data, and perform defect detection analysis on the power equipment by using a time series model according to the aligned modal data, to obtain a power equipment defect detection result.

[0092] It should be noted that the modules in the system of Embodiment 2 correspond to the steps in the method of Embodiment 1, and the steps in the method of Embodiment 1 have been described in detail in Embodiment 1, and the content of the modules in the system will not be described in detail in Embodiment 2.

[0093] Embodiment 3

[0094] Referring to Figure 3 It should be noted that the processor 1001 is configured to execute the steps in the method embodiments described above according to instructions in the program code. Alternatively, the processor 1001 implements the functions of each module / unit in each system / device embodiment described above when executing the computer program.

[0095] It should be noted that the processor 1001 is configured to execute the steps in the method embodiments described above according to instructions in the program code. Alternatively, the processor 1001 implements the functions of each module / unit in each system / device embodiment described above when executing the computer program.

[0096] Specifically, in this embodiment, the computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the system memory 1005 and executed by the processor 1001 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the terminal device.

[0097] The terminal device can be a desktop computer, a notebook computer, a palm computer, and a cloud server, etc. The terminal device can include, but is not limited to, the processor 1001 and the system memory 1005. Those skilled in the art can understand that it does not constitute a limitation on the terminal device, and can include more or fewer components than the illustration, or combine certain components, or different components, for example, the terminal device can also include an input / output device 1003, a network access device 1002, a bus 1006, etc.

[0098] The processor 1001 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0099] The system memory 1005 can be an internal storage unit of the terminal device, for example, a hard disk or a memory of the terminal device. The system memory 1005 can also be a storage device 1004 of the terminal device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. Further, the system memory 1005 can include both the internal storage unit of the terminal device and the storage device 1004. The system memory 1005 is used to store computer programs and other programs and data required by the terminal device. The system memory 1005 can also be used to temporarily store data that has been output or will be output.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, system and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0101] Embodiment 4

[0102] The embodiment provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium. The computer program is executed by a processor to implement the steps of the method in any one of the foregoing embodiments.

[0103] The computer readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or apparatus or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), registers, a hard disk, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer readable storage medium of the foregoing is non-transitory only in the general sense that it does not have its data modified by the program executing on the computer system. The computer readable storage medium of the foregoing is non-transitory only in the general sense that it does not have its data modified by the program executing on the computer system.

[0104] An exemplary storage medium is coupled to the processor such that the processor can read information from, and can write information to, the storage medium. Of course, the storage medium can be part of the processor. Consistent with the teachings of the present disclosure, a storage medium can be implemented using any appropriate media, such as optical, magnetic or semiconductor storages. In the present embodiment, the computer readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device.

[0105] Embodiment 5

[0106] The present embodiment also provides a computer program product containing instructions which, when executed by a computer device cluster, cause the computer device cluster to perform the method as described in embodiment 1.

[0107] The above detailed description merely describes specific embodiments of the application, and the purpose of the above detailed description is to explain the purpose, technical solutions and beneficial effects of the application. It should be understood that the above detailed description is only a specific embodiment of the application, and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A method for power equipment defect detection based on multi-modal alignment, characterized in that, The method comprises: Respectively collect visible light image time features, infrared image time features and ultrasonic signal time features, and calculate the time feature difference between each two modal data to obtain the time feature difference between each two modal data; Construct a graph structure among the visible light image, the infrared image and the ultrasonic signal, and extract the spatial feature difference between each two modal data from the graph structure; Perform logical regression processing on the spatial feature difference and the time feature difference to obtain the alignable coefficient between each two modal data, and determine the to-be-aligned data combination for power equipment defect detection based on the alignable coefficient; Align the to-be-aligned data combination to obtain aligned modal data, and perform defect detection analysis on the power equipment by using a time series model according to the aligned modal data, to obtain a power equipment defect detection result.

2. The method of claim 1, wherein, The alignable coefficient between each two modal data is obtained by performing logical regression processing on the spatial feature difference and the time feature difference, and specifically: The spatial feature difference and the time feature difference between each two modal data are input into a logical regression algorithm for logical regression processing to obtain the alignable coefficient between each two modal data.

3. The method of claim 1, wherein, The to-be-aligned data combination for power equipment defect detection is determined based on the alignable coefficient, and specifically: The alignable coefficient between each two modal data is compared with a preset alignable threshold; If the alignable coefficient between two modal data is not less than the preset alignable threshold, the modal weight of the two modal data is increased respectively; If the alignable coefficient between two modal data is less than the preset alignable threshold, the modal weight of the two modal data is not increased; After comparing all the alignable coefficients, the modal data with the maximum modal weight is extracted as the to-be-aligned data combination for power equipment defect detection.

4. The method of claim 1, wherein, The to-be-aligned data combination is aligned to obtain aligned modal data, and specifically: The time feature and the spatial feature of the to-be-aligned modal data in the to-be-aligned data combination are obtained, and the distribution difference between the time feature and the spatial feature of the to-be-aligned modal data and the time feature and the spatial feature of the remaining modal data other than the to-be-aligned modal data is obtained; If the distribution difference is not less than a preset distribution threshold, the to-be-aligned data combination is aligned by using a pre-constructed alignment transformation function to obtain aligned modal data.

5. The method of claim 1, wherein, The time series model is used to perform defect detection analysis on the power equipment according to the aligned modal data, to obtain a power equipment defect detection result, wherein the time series model specifically adopts an ARIMAX model, and specifically: The ARIMAX model parameters are estimated by using a maximum likelihood estimation method, and the estimated ARIMAX model parameters are input into the ARIMAX model for fitting to obtain a fitted ARIMAX model; The data features of the aligned modal data are extracted, and the data features are input into the fitted ARIMAX model to perform defect detection analysis on the power equipment, to obtain a power equipment defect detection result.

6. A multi-modal alignment based power equipment defect detection system, comprising: The system is used in the power equipment defect detection method based on multi-modal alignment in any one of claims 1-5, and the system comprises: The feature difference acquisition module is configured to acquire spatial feature differences and temporal feature differences between each two modal data in the collected visible light image, infrared image and ultrasonic signal. The to-be-aligned data combination acquisition module is configured to perform logical regression processing on the spatial feature differences and the temporal feature differences to obtain an alignable coefficient between each two modal data, and determine a to-be-aligned data combination for power equipment defect detection based on the alignable coefficient. The power equipment defect detection module is configured to perform data alignment on the to-be-aligned data combination to obtain aligned modal data, and perform defect detection analysis on the power equipment by using a time series model according to the aligned modal data, to obtain a power equipment defect detection result.

7. The multi-modal alignment based power equipment defect detection system of claim 6, wherein, In the collected visible light image, infrared image and ultrasonic signal, spatial feature differences and temporal feature differences between each two modal data are acquired, specifically as follows: Visible light image temporal features, infrared image temporal features and ultrasonic signal temporal features are collected respectively, and time feature differences between each two modal data are calculated by difference calculation to obtain temporal feature differences between each two modal data. A graph structure between the visible light image, the infrared image and the ultrasonic signal is constructed, and spatial feature differences between each two modal data are extracted from the graph structure.

8. The multi-modal alignment based power equipment defect detection system of claim 6, wherein, Logical regression processing is performed on the spatial feature differences and the temporal feature differences to obtain an alignable coefficient between each two modal data, specifically as follows: The spatial feature differences and the temporal feature differences between each two modal data are input into a logical regression algorithm for logical regression processing to obtain an alignable coefficient between each two modal data.

9. A computer device comprising a system memory storing a computer program and a processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

Images