Power equipment defect detection method, system and equipment based on multi-modal alignment

Abstract

The invention discloses an electrical equipment defect detection method, system and equipment based on multi-modal alignment, and particularly relates to the technical field of electrical equipment detection. Obtaining a spatial feature difference and a time feature difference between every two pieces of modal data; performing logistic regression processing on the spatial feature difference and the time feature difference to obtain an alignable coefficient between each two modal data, and determining a to-be-aligned data combination for electrical equipment defect detection based on the alignable coefficients; and performing data alignment on the to-be-aligned data combination to obtain aligned modal data, and performing defect detection analysis on the power equipment by using the time sequence model according to the aligned modal data to obtain a defect detection result of the power equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of power equipment detection, and in particular to a method, system and device for detecting defects in power equipment based on multimodal alignment. Background Art

[0002] Power equipment inspection is a critical component in ensuring the safe operation of power grids. In recent years, detection technologies based on computer vision and deep learning have been widely adopted. However, existing methods primarily rely on single-modal data (such as infrared images, visible light images, or ultrasonic signals), which are susceptible to interference under complex operating conditions, resulting in reduced defect detection accuracy. Therefore, the integration of multimodal data and effective alignment of data features to achieve precise and intelligent detection has become an important research direction in this field.

[0003] At the same time, the existing technology mainly has the following deficiencies: single-modal recognition cannot capture the complementary features of the equipment's operating status, the false alarm rate increases sharply in scenarios such as foggy days and equipment occlusion, and the existing multimodal fusion lacks a deep alignment mechanism. Especially in extreme weather environments, the selection of multimodal parameters for analysis during feature interaction is relatively vague, which can easily lead to the loss of key discriminant information.

[0004] Based on this, the present invention aims to provide a method, system and device for detecting defects in power equipment based on multimodal alignment to solve the above-mentioned related problems. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the existing multimodal fusion lacks a deep alignment mechanism, and the selection of multimodal parameters for analysis is relatively vague, which can easily cause the loss of key discriminant information. The purpose is to provide a method, system and equipment for power equipment defect detection based on multimodal alignment. By collecting multimodal data including visible light images, infrared images and ultrasonic signals, the spatial feature differences and temporal feature differences between visible light images, infrared images and ultrasonic signals are obtained. At the same time, the spatial feature differences and temporal feature differences are substituted into the logistic regression algorithm to calculate the alignability coefficients between the multimodal data, determine the data combination required for alignment of the power equipment, and construct a distribution alignment standard. By using contrastive learning, the data with different alignment standards are feature calibrated and fused, and then the time series model is used to detect power equipment defects. Based on the detection results, the frequency of defect alarms can be optimized, the accuracy of data fusion can be improved, and the robustness and generalization ability of data alignment can be improved to meet the accuracy and reliability of power equipment defect detection.

[0006] The present invention is achieved through the following technical solutions: A method for detecting defects in power equipment based on multimodal alignment, the method comprising: In the collected visible light images, infrared images and ultrasonic signals, the spatial feature differences and temporal feature differences between each two modal data are obtained; Logistic regression is performed on the spatial feature differences and temporal feature differences to obtain the alignment coefficient between each two modal data. The data combination to be aligned for power equipment defect detection is determined based on the alignment coefficient. Data alignment is performed on the data combination to be aligned to obtain aligned modal data. Based on the aligned modal data, a time series model is used to perform defect detection analysis on the power equipment to obtain the power equipment defect detection results.

[0007] Furthermore, in the collected visible light image, infrared image and ultrasonic signal, the spatial feature difference and temporal feature difference between each two modal data are obtained, specifically: The time features of visible light images, infrared images and ultrasonic signals are collected respectively, and the time features between each two modal data are calculated to obtain the time feature difference between each two modal data; A graph structure is constructed between visible light images, infrared images and ultrasonic signals, and the spatial feature differences between each two modal data are extracted from the graph structure.

[0008] Furthermore, logistic regression is performed on the spatial feature differences and temporal feature differences to obtain the alignment coefficient between each two modal data, specifically: The spatial feature differences and temporal feature differences between each two modal data are input into the logistic regression algorithm for logistic regression processing to obtain the alignment coefficient between each two modal data.

[0009] Furthermore, the data combination to be aligned for power equipment defect detection is determined based on the alignable coefficient, specifically: Compare the alignability coefficient between each two modal data with a preset alignability threshold; If the alignability coefficient between two modal data is not less than the preset alignability threshold, the modal weights of the two modal data are increased respectively; If the alignment coefficient between two modal data is less than the preset alignment threshold, the modal weights of the two modal data are not increased; After comparing all the alignable coefficients, the modal data with the largest modal weight is extracted as the data combination to be aligned for power equipment defect detection.

[0010] Furthermore, data alignment is performed on the data combination to be aligned to obtain aligned modal data, specifically: Obtaining distribution differences between the time features and spatial features of the modal data to be aligned in the data combination to be aligned and the time features and spatial features of the remaining modal data other than the modal data to be aligned; If the distribution difference is not less than the preset distribution threshold, the pre-built alignment transformation function is used to align the data group to obtain the aligned modal data.

[0011] Furthermore, based on the aligned modal data, a time series model is used to perform defect detection analysis on the power equipment to obtain the power equipment defect detection results. The time series model specifically adopts an ARIMAX model, which is: The ARIMAX model parameters are estimated using the maximum likelihood estimation method, and the estimated ARIMAX model parameters are input into the ARIMAX model for fitting to obtain the fitted ARIMAX model; The data features of the aligned modal data are extracted and input into the fitted ARIMAX model to perform defect detection analysis on the power equipment to obtain the power equipment defect detection results.

[0012] The present invention further provides a power equipment defect detection system based on multimodal alignment, which is used in any of the above-mentioned power equipment defect detection methods based on multimodal alignment, and the system includes: A feature difference acquisition module is used to obtain the spatial feature difference and temporal feature difference between each two modal data in the collected visible light image, infrared image and ultrasonic signal; The module for acquiring the combination of data to be aligned is used to perform logistic regression processing on the spatial feature differences and the temporal feature differences to obtain the alignment coefficient between each pair of modal data, and to determine the combination of data to be aligned for power equipment defect detection based on the alignment coefficient; The power equipment defect detection module is used to perform data alignment on the data combination to be aligned to obtain aligned modal data, and to perform defect detection analysis on the power equipment using a time series model based on the aligned modal data to obtain power equipment defect detection results.

[0013] Furthermore, in the collected visible light image, infrared image and ultrasonic signal, the spatial feature difference and temporal feature difference between each two modal data are obtained, specifically: The time features of visible light images, infrared images and ultrasonic signals are collected respectively, and the time features between each two modal data are calculated to obtain the time feature difference between each two modal data; A graph structure is constructed between visible light images, infrared images and ultrasonic signals, and the spatial feature differences between each two modal data are extracted from the graph structure.

[0014] Furthermore, logistic regression is performed on the spatial feature differences and temporal feature differences to obtain the alignment coefficient between each two modal data, specifically: The spatial feature differences and temporal feature differences between each two modal data are input into the logistic regression algorithm for logistic regression processing to obtain the alignment coefficient between each two modal data.

[0015] The present invention also provides a computer device, comprising a system memory and a processor, wherein the system memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0016] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods described above are implemented.

[0017] The present invention also provides a computer program product comprising instructions, which, when executed by a computer device cluster, enables the computer device cluster to perform any of the above methods.

[0018] Compared with the prior art, the present invention has the following advantages and beneficial effects: In the present invention, by collecting multimodal data including visible light images, infrared images and ultrasonic signals, the spatial feature differences and temporal feature differences between the visible light images, infrared images and ultrasonic signals are obtained, and the spatial feature differences and temporal feature differences are substituted into the logistic regression algorithm to calculate the alignment coefficient between the multimodal data, the data combination required for alignment of the power equipment is determined, and a distribution alignment standard is constructed. By using contrastive learning, the data of different alignment standards are feature calibrated and fused, and then the defects of the power equipment are detected using a time series model. Based on the detection results, the frequency of defect alarms can be optimized, the accuracy of data fusion can be improved, and the robustness and generalization ability of data alignment can be enhanced to meet the accuracy and reliability of power equipment defect detection. This solves the technical problems that the existing multimodal fusion lacks a deep alignment mechanism, the selection of multimodal parameters for analysis is relatively vague, and key discrimination information is easily lost. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the examples. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be considered as limiting the scope. A person of ordinary skill in the art can also derive other relevant drawings based on these drawings without inventive effort. In the drawings: Figure 11 is a schematic diagram of a method flow of a power equipment defect detection method based on multimodal alignment in this embodiment; Figure 2 Schematic diagram of a module of a power equipment defect detection system based on multimodal alignment in this embodiment; Figure 3 This is a structural diagram of a computer device in this embodiment. DETAILED DESCRIPTION

[0020] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] In this disclosure, unless otherwise specified, the use of terms such as "first" and "second" to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases, based on the context of the description, they may also refer to different instances.

[0022] The terms used in the descriptions of various examples in this disclosure are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element may be one or more. In addition, the term "and / or" used in this disclosure encompasses any one and all possible combinations of the listed items.

[0023] Example 1 This embodiment provides a method for detecting defects in power equipment based on multimodal alignment. Figure 1 , Figure 1 A schematic flow chart of a method for detecting defects in power equipment based on multimodal alignment is shown, wherein the method includes: S1: Obtain the spatial feature difference and temporal feature difference between each two modal data in the collected visible light image, infrared image, and ultrasonic signal.

[0024] It should be noted that in this embodiment, the multiple modal data include visible light images, infrared images, and ultrasonic signals. The visible light image data is acquired using a high-resolution industrial camera or a visible light camera mounted on a drone. This camera typically features autofocus, optical zoom, and high dynamic range (HDR) imaging, ensuring clear images of power equipment can be captured under varying lighting conditions. Optionally, the visible light image acquisition process can be combined with multi-angle photography or structured light 3D reconstruction technology to obtain richer spatial information and improve the accuracy of equipment defect identification. Infrared image data is acquired using an infrared thermal imager or a drone-mounted infrared thermal imaging camera. Based on the principle of thermal radiation, this device can detect the temperature distribution on the surface of power equipment. Infrared cameras typically offer thermal imaging capabilities in different wavelength bands (shortwave, mediumwave, and longwave) and incorporate automatic temperature compensation and emissivity correction to improve temperature measurement accuracy. In practical applications, infrared data collection can be achieved through fixed-point inspections, mobile photography, or automated drone inspections to ensure coverage of the entire equipment area. Ultrasonic signal data is acquired through ultrasonic sensors or partial discharge detection equipment (such as ultrasonic detectors, array ultrasonic sensors, ultrasonic microphones, etc.). Ultrasonic sensors can detect defects such as mechanical vibrations, partial discharges, cracks, air gaps, etc. inside the equipment, usually using contact or non-contact measurement methods. Non-contact measurement relies on air-coupled ultrasound or laser ultrasonic technology, which can detect partial discharge signals at a long distance, while contact measurement uses piezoelectric sensors directly attached to the surface of the equipment for high-precision defect detection. In addition, ultrasonic signal acquisition is usually combined with time-frequency analysis (short-time Fourier transform STFT, wavelet transform, etc.) to extract features and improve defect recognition capabilities. Feature extraction uses conventional technical means in this field, which will not be elaborated here; At the same time, it should also be noted that the common spatial and temporal characteristics of visible light images, infrared images, and ultrasonic signal data are: they all have sampling or shooting after a certain time interval, that is, time characteristics, and at the same time have corresponding spatial characteristics, that is, point sampling or point shooting of various areas of the power equipment.

[0025] Specifically, in this embodiment, first, the time features of the visible light image, the time features of the infrared image, and the time features of the ultrasonic signal are collected respectively, and the time features between each two modal data are differenced to obtain the time feature difference between each two modal data; It should be noted that, in this embodiment, the visible light image time feature refers to the visible light image shooting timestamp, the infrared image time feature refers to the infrared image shooting timestamp, and the ultrasonic signal time feature refers to the ultrasonic signal sampling time; The difference between the visible light image timestamp and the infrared image shooting timestamp refers to the time interval between the two sensors (visible light camera and infrared thermal imager) shooting the same power equipment in the same monitoring task. The acquisition logic uses a unified time source for time synchronization. The visible light camera shooting timestamp is subtracted from the infrared camera shooting timestamp to obtain the difference between the visible light image and the infrared image shooting timestamp, that is, the difference in time characteristics between the visible light image and the infrared image. The sampling time difference between the visible light image and the ultrasonic signal refers to the time interval between the time when the visible light camera captures the image and the time when the ultrasonic sensor collects the signal in the same monitoring task. Its acquisition logic is to synchronize the time through a high-precision clock, and subtract the ultrasonic sensor sampling timestamp from the visible light camera shooting timestamp to obtain the sampling time difference between the visible light image and the ultrasonic signal, that is, to obtain the time characteristic difference between the visible light image and the ultrasonic signal. The sampling time difference between the infrared image and the ultrasonic signal refers to the time interval between the infrared image acquisition time and the ultrasonic signal sampling time during the same power equipment monitoring process. Its acquisition logic uses a unified clock source to subtract the ultrasonic sensor sampling time stamp from the infrared image capture time stamp to obtain the sampling time difference between the infrared image and the ultrasonic signal, that is, the time characteristic difference between the infrared image and the ultrasonic signal. At the same time, it should be noted that the above-mentioned time synchronization methods are all existing technologies. Those skilled in the art can understand that the data acquisition of different sensors is affected by factors such as hardware trigger delay, signal transmission delay, sensor sampling rate, etc., and there may be slight time deviations. Therefore, in practical applications, it is usually necessary to combine high-precision time synchronization technology. The application of specific synchronization technology will not be elaborated here.

[0026] Then, a graph structure is constructed between the visible light image, infrared image and ultrasonic signal, and the spatial feature difference between each two modal data is extracted from the graph structure.

[0027] It should be noted that in this embodiment, spatial features of visible light images, infrared images, and ultrasonic signals are first extracted. Edge detection is used for visible light images to extract structural information of the power equipment, such as equipment edges, shape features, and damaged areas. Temperature distribution features and hot spots are extracted from infrared images and matched with the geometric shapes of the visible light images (matching can be performed using scale-invariant feature transforms (SIFT)). Short-time Fourier transforms (STFTs) and wavelet transforms are used for ultrasonic signals to extract time-frequency features, which are then mapped into two-dimensional or three-dimensional space in combination with sensor position information. Secondly, the visible light image, infrared image, and ultrasonic signal are organized into a graph structure, and the basic elements of the graph are defined. The nodes of the graph are set as feature points in the visible light image, the center point of the high temperature area in the infrared image, and the location of the ultrasonic sensor. The edges are connected by Euclidean distance, structural similarity (SSIM), and feature vector similarity. If two nodes belong to the same area or their feature similarity is higher than a set threshold, an edge is established between them. The threshold is set by using supervised learning or unsupervised learning methods based on a large amount of labeled multimodal data of power equipment, which will not be elaborated here. At the same time, due to the existence of different types of associations between multiple modal data, heterogeneous graphs can be used. Specifically, for visible light images and infrared images, the correspondence between their feature points is calculated (such as SIFT matching) and connections are established between the matching points. For visible light images and ultrasonic signals, if the sampling point corresponding to the ultrasonic sensor is located in a device area in the visible light image, the sensor point is connected to the device feature point. For infrared images and ultrasonic signals, if there is spatial overlap between the infrared hotspot area and the ultrasonic anomaly detection area, a connection is established. Then, a graph neural network is designed to perform spatial feature difference analysis, and weighted learning of multimodal features is used to propagate the graph neural network forward. The specific calculation formula is as follows: ,in, For the The node characteristics of the layer, is the trainable weight matrix, is the attention weight, is the activation function, is the set of neighbor nodes of the node; Indicates the The node features of the layer are obtained by the graph neural network. The spatial feature difference function is defined to measure the spatial alignment error between different modal data. The specific function calculation formula is as follows: ,in, Represents the characteristic difference between two modal data in space, and They are the two modal feature representations of the final layer of the graph neural network, Represents the Euclidean distance, i.e. the L2 norm; Finally, the visible light image, infrared image and ultrasonic signal data are substituted into the graph neural network to construct a graph structure. From the graph structure, the spatial feature difference between the visible light image and the infrared image is obtained, that is, the spatial feature difference between the visible light image and the infrared image; the spatial feature difference between the visible light image and the ultrasonic signal is obtained, that is, the spatial feature difference between the visible light image and the ultrasonic signal; the spatial feature difference between the infrared image and the ultrasonic signal is obtained, that is, the spatial feature difference between the infrared image and the ultrasonic signal.

[0028] S2: Perform logistic regression processing on the spatial feature differences and temporal feature differences to obtain the alignment coefficient between each two modal data, and determine the data combination to be aligned for power equipment defect detection based on the alignment coefficient.

[0029] It should be noted that, in this embodiment, since the spatial feature difference is determined based on the spatial coordinates, it is a dimensionless parameter. Since the temporal feature drawing determines the difference based on the time length and has a time unit, the time dimension difference is de-dimensionalized. The de-dimensionalization method specifically uses the Z-score normalization method to standardize the temporal feature difference to keep it on the same scale as the spatial feature difference. The specific formula of the Z-score normalization method is expressed as: ,in, is the mean, is the standard deviation, is the original value of the time feature difference, Normalize the results for time feature difference; Specifically, in this embodiment, the spatial feature difference and the temporal feature difference between each two modal data are first input into a logistic regression algorithm for logistic regression processing to obtain the alignment coefficient between each two modal data; It should be noted that, in this embodiment, the differences in spatial and temporal features between the visible light image and the infrared image, the differences in spatial and temporal features between the visible light image and the ultrasonic signal, and the differences in spatial and temporal features between the infrared image and the ultrasonic signal are respectively substituted into the logistic regression algorithm, and the specific formula is expressed as follows: 、 、 ;in, 、 as well as are the logistic regression calculation results, namely represents the alignability coefficient between the visible light image and the infrared image, represents the alignability coefficient between the infrared image and the ultrasonic signal, Represents the alignment coefficient between infrared image and ultrasonic signal data; e is the natural base, 、 as well as is the linear combination term of the logistic regression model, which can be set as: ,in, 、 as well as are the bias items of the data corresponding to each mode; and are the temporal feature difference and spatial feature difference between visible light images and infrared images, and are the regression coefficients of the corresponding parameters respectively; and are the temporal feature difference and spatial feature difference between infrared images and ultrasonic signals, and are the regression coefficients of the corresponding parameters respectively; and are the temporal feature difference and spatial feature difference between ultrasonic signals and visible light images, respectively. and are the regression coefficients of the corresponding parameters.

[0030] At the same time, it should be noted that the larger the alignment coefficient is, the more data alignment operation is required between the current corresponding modes. Conversely, the smaller the coefficient is, the less data alignment operation is required between the current corresponding modes.

[0031] Then, the alignment coefficient between each two modal data is compared with the preset alignment threshold; if the alignment coefficient between the two modal data is not less than the preset alignment threshold, the modal weights of the two modal data are increased respectively; if the alignment coefficient between the two modal data is less than the preset alignment threshold, the modal weights of the two modal data are not increased; It should be noted that in this embodiment, the alignment threshold is obtained through historical alignment error distribution statistics and supervised learning optimization; the initial modal weight of all modal data is 0 points; at the same time, the modal weight of two modal data with an alignment coefficient greater than or equal to the preset alignment threshold is increased by 1 point, and the modal weight of two modal data with an alignment coefficient less than the preset alignment threshold remains unchanged; Finally, after comparing all the alignable coefficients, the modal data with the largest modal weight is extracted as the data combination to be aligned for power equipment defect detection.

[0032] Exemplarily, if the alignment coefficient of the visible light image and the infrared image is greater than or equal to the preset alignment threshold, the modal weights of the visible light image and the infrared image are respectively increased by 1 point; if at the same time the alignment coefficient of the visible light image and the ultrasonic signal is less than the preset alignment threshold, the modal weights of the visible light image and the ultrasonic signal remain unchanged; if at the same time the alignment coefficient of the infrared image and the ultrasonic signal is greater than or equal to the preset alignment threshold, the modal weights of the infrared image and the ultrasonic signal are respectively increased by 1 point; therefore, the above judgment accumulation can be obtained, and the modal weight of the visible light image is recorded as 1 point, the modal weight of the infrared image is recorded as 2 points, and the modal weight of the ultrasonic signal is recorded as 1 point, and the infrared image is extracted as the data combination to be aligned for power equipment defect detection; It should be noted that the above content expresses a multi-level alignment strategy based on the ternary pattern. The modal weights of the three modal data may include multiple situations. If there are two identical modal weights, and the other modal weight is smaller than the two identical modal weights, the modal data corresponding to the two identical modal weights are used as the data combination to be aligned for data alignment operation. If the modal weights are all 1 or 2, it means that the visible light image, infrared image and ultrasonic signal all need to be aligned. If the modal weights are all 0, no alignment is required, and each modal data is directly substituted into the time series model to determine the defect detection of power equipment.

[0033] S3: Performing data alignment on the data combination to be aligned to obtain aligned modal data, and performing defect detection analysis on the power equipment using a time series model based on the aligned modal data to obtain a power equipment defect detection result.

[0034] Specifically, in this embodiment, first, the distribution difference between the time features and spatial features of the modal data to be aligned in the data combination to be aligned and the time features and spatial features of the remaining modal data other than the modal data to be aligned is obtained; It should be noted that in this embodiment, the time features and spatial features of the modal data to be aligned that constitute the data combination to be aligned are first obtained, and then the time features and spatial features of the remaining two modal data are obtained. By statistically analyzing the contrast of their spatial features and time features, the distribution consistency of the modal data features is calculated to evaluate the distribution consistency of the modal data features. The specific formula is expressed as follows: , ,in, To represent the distribution difference, let the spatial and temporal characteristics of the visible light image be , the spatial and temporal characteristics of infrared images are , the spatial and temporal characteristics of the ultrasonic signal are ,in, is the probability distribution of the feature; Then, if the distribution difference is not less than the preset distribution threshold, the pre-built alignment transformation function is used to align the data set to obtain the aligned modal data; It should be noted that, in this embodiment, if the distribution consistency of the modal data is high (i.e. The value is low), that is, the distribution difference is less than the preset distribution threshold, then the alignment loss between them is small, indicating that these modal data are already well aligned, and the default data alignment is adopted; at the same time, the default data alignment refers to the preliminary alignment of data of different modalities in time and space based on existing technologies, rules or predefined benchmark methods without additional complex alignment strategies or data corrections to ensure data consistency. The specific alignment method of the default data alignment is not limited and will not be elaborated here; at the same time, the setting of the distribution threshold is based on the setting of technical personnel based on historical data and will not be elaborated here; If the distribution difference is greater than or equal to the preset distribution threshold, it means that the modal data to be aligned needs to be further adjusted and aligned with the other modal data. It is necessary to use the pre-built alignment transformation function to align the data group to be aligned. The pre-built alignment transformation function is obtained by training the contrastive learning model. Contrastive learning learns data alignment by minimizing the distance between similar modalities and maximizing the distance between different modalities. The training process is as follows: First, construct the contrastive learning loss function , the expression is: ,in, and Represent the feature representation of visible light image and infrared image respectively, is a label (1 for alignment, 0 for non-alignment), represents the Euclidean distance between two features, and is the positive and negative threshold of alignment loss, represents the number of samples; then, according to the above loss function, the gradient descent method is used to optimize the contrastive learning model parameters, reduce the feature distance between different modalities, and ensure that data of the same type (such as visible light images and infrared images of the same device) are aligned as much as possible during training, while data of different types (such as visible light images and ultrasonic signals) are kept apart, etc., which will not be elaborated here; the key to contrastive learning is to generate positive and negative sample pairs. The positive sample pair is two modalities from the same device (such as visible light and infrared images), and the negative sample pair is a combination of different devices or different modalities. During training, alternating positive and negative sample pairs are used to train the model to maximize the alignment accuracy between modalities; after training, the optimized alignment transformation function is obtained, and then for the modalities that need to be aligned, the data is aligned according to the alignment transformation function.

[0035] Finally, based on the aligned modal data, the time series model is used to perform defect detection analysis on the power equipment to obtain the power equipment defect detection results, wherein the time series model specifically adopts the ARIMAX model, specifically: the ARIMAX model parameters are estimated using the maximum likelihood estimation method, the estimated ARIMAX model parameters are input into the ARIMAX model for fitting, and the fitted ARIMAX model is obtained; the data features of the aligned modal data are extracted, and the data features are input into the fitted ARIMAX model to perform defect detection analysis on the power equipment to obtain the power equipment defect detection results.

[0036] It should be noted that, in this embodiment, it is necessary to obtain defect alarm frequency data, which refers to historical alarm frequency data, and the data is used as the main variable of the time series; then the maximum likelihood estimation method is used to estimate the ARIMAX model parameters ( 、 、 、 、 ), the specific steps are as follows: Error term Normal distribution , then the likelihood function is: ; Take the logarithm of the likelihood function to obtain the log-likelihood function: ; Obtain parameter estimates by maximizing the log-likelihood function 、 、 、 、 ; Input the estimated ARIMAX model parameters into the ARIMAX model to fit and obtain the fitted ARIMAX model. The basic form of the model is: ,in, The alarm frequency at the current time point corresponds to the severity of the power equipment defect at the current time point; Lag i The alarm frequency of the order, is a constant term, For the i The order autoregressive parameter, is the order of the autoregressive term, For the j The moving average parameter of order, is the order of the moving average term, is the white noise term with lag j, is a white noise term, representing random error, is an exogenous variable The coefficient of is the lag order of the exogenous variable, is an exogenous variable lagged by k periods; the exogenous variable part can integrate the influence of other related variables (such as visible light image features, infrared image features and ultrasonic signal features) on the severity of power equipment defects; finally, the data features of the aligned modal data are extracted and input into the ARIMAX model to perform defect detection analysis on the power equipment and obtain the power equipment defect detection results.

[0037] It should also be noted that visible light image features, infrared image features and ultrasonic signal features include but are not limited to color features, texture features, shape features, temperature features, temperature distribution features, time series features and time-frequency domain features. The specific method of obtaining features belongs to the existing technology and will not be elaborated here.

[0038] Specifically, in the present invention, by collecting multimodal data including visible light images, infrared images and ultrasonic signals, the spatial feature differences and temporal feature differences between the visible light images, infrared images and ultrasonic signals are obtained, and the spatial feature differences and temporal feature differences are substituted into the logistic regression algorithm to calculate the alignment coefficient between the multimodal data, the data combination required for alignment of the power equipment is determined, and a distribution alignment standard is constructed. By using comparative learning, the data of different alignment standards are feature calibrated and fused, and then the defects of the power equipment are detected using a time series model. Based on the detection results, the frequency of defect alarms can be optimized, the accuracy of data fusion can be improved, and the robustness and generalization ability of data alignment can be enhanced to meet the accuracy and reliability of power equipment defect detection. This solves the technical problems that the existing multimodal fusion lacks a deep alignment mechanism, the selection of multimodal parameters for analysis is relatively vague, and it is easy to cause the loss of key discrimination information.

[0039] Example 2 See also Figure 2 As shown, the present invention also provides a power equipment defect detection system based on multimodal alignment, which is used in any of the above-mentioned power equipment defect detection methods based on multimodal alignment, and the system includes: The feature difference acquisition module 100 is used to obtain the spatial feature difference and the temporal feature difference between each two modal data in the collected visible light image, infrared image and ultrasonic signal; The module 200 for acquiring the data combination to be aligned is used to perform logistic regression processing on the spatial feature differences and the temporal feature differences to obtain the alignment coefficient between each pair of modal data, and to determine the data combination to be aligned for power equipment defect detection based on the alignment coefficient; The power equipment defect detection module 300 is used to perform data alignment on the data combination to be aligned to obtain aligned modal data, and perform defect detection analysis on the power equipment using a time series model based on the aligned modal data to obtain power equipment defect detection results.

[0040] It should be noted that the modules in the system of Example 2 correspond to the steps in the method of Example 1. The steps in the method of Example 1 have been described in detail in Example 1. The contents of the modules in the system will not be described in detail in this Example 2.

[0041] Example 3

[0042] See also Figure 3 As shown, this embodiment further provides a computer device, including a system memory 1005 and a processor 1001, wherein the system memory 1005 stores a computer program, and the processor 1001 implements the steps of any of the above methods when executing the computer program.

[0043] It should be noted that the processor 1001 is configured to execute the steps of the above method embodiments according to the instructions in the program code. Alternatively, the processor 1001 implements the functions of the modules / units in the above system / device embodiments when executing the computer program.

[0044] Specifically, in this embodiment, the computer program may be divided into one or more modules / units, one or more modules / units being stored in the system memory 1005 and executed by the processor 1001 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0045] The terminal device may be a computing device such as a desktop computer, laptop, PDA, or cloud server. The terminal device may include, but is not limited to, a processor 1001 and a system memory 1005. Those skilled in the art will appreciate that this does not limit the terminal device and may include more or fewer components than shown, or a combination of certain components, or different components. For example, the terminal device may also include an input / output device 1003, a network access device 1002, a bus 1006, and the like.

[0046] The processor 1001 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0047] The system memory 1005 can be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. The system memory 1005 can also be the storage device 1004 of the terminal device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the terminal device. Furthermore, the system memory 1005 can also include both the internal storage unit of the terminal device and the storage device 1004. The system memory 1005 is used to store computer programs and other programs and data required by the terminal device. The system memory 1005 can also be used to temporarily store data that has been output or is about to be output.

[0048] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0049] Example 4

[0050] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of any one of the above methods are implemented.

[0051] Among them, the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, system or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk. Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM), a register, a hard disk, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above, or any other form of computer-readable storage medium known in the art.

[0052] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (ASIC). In an embodiment of the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device.

[0053] Example 5

[0054] This embodiment further provides a computer program product comprising instructions. When the instructions are executed by a computer device cluster, the computer device cluster executes the method described in Embodiment 1.

[0055] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting defects in power equipment based on multimodal alignment, characterized in that: Methods include: In the collected visible light images, infrared images and ultrasonic signals, the spatial feature differences and temporal feature differences between each two modal data are obtained; Logistic regression is performed on the spatial feature differences and temporal feature differences to obtain the alignment coefficient between each two modal data. The data combination to be aligned for power equipment defect detection is determined based on the alignment coefficient. Data alignment is performed on the data combination to be aligned to obtain aligned modal data. Based on the aligned modal data, a time series model is used to perform defect detection analysis on the power equipment to obtain the power equipment defect detection results.

2. The method for detecting defects in power equipment based on multimodal alignment according to claim 1, characterized in that: In the collected visible light images, infrared images, and ultrasonic signals, the spatial feature differences and temporal feature differences between each two modal data are obtained, specifically: The time features of visible light images, infrared images and ultrasonic signals are collected respectively, and the time features between each two modal data are calculated to obtain the time feature difference between each two modal data; A graph structure is constructed between visible light images, infrared images and ultrasonic signals, and the spatial feature differences between each two modal data are extracted from the graph structure.

3. The method for detecting defects in power equipment based on multimodal alignment according to claim 1, characterized in that: Logistic regression is performed on the spatial feature differences and temporal feature differences to obtain the alignment coefficient between each two modal data, specifically: The spatial feature differences and temporal feature differences between each two modal data are input into the logistic regression algorithm for logistic regression processing to obtain the alignment coefficient between each two modal data.

4. The method for detecting defects in power equipment based on multimodal alignment according to claim 1, characterized in that: The data combination to be aligned for power equipment defect detection is determined based on the alignable coefficient, specifically: Compare the alignability coefficient between each two modal data with a preset alignability threshold; If the alignability coefficient between two modal data is not less than the preset alignability threshold, the modal weights of the two modal data are increased respectively; If the alignment coefficient between two modal data is less than the preset alignment threshold, the modal weights of the two modal data are not increased; After comparing all the alignable coefficients, the modal data with the largest modal weight is extracted as the data combination to be aligned for power equipment defect detection.

5. The method for detecting defects in power equipment based on multimodal alignment according to claim 1, characterized in that: Perform data alignment on the data combination to be aligned to obtain the aligned modal data, specifically: Obtaining distribution differences between the time features and spatial features of the modal data to be aligned in the data combination to be aligned and the time features and spatial features of the remaining modal data other than the modal data to be aligned; If the distribution difference is not less than the preset distribution threshold, the pre-built alignment transformation function is used to align the data group to obtain the aligned modal data.

6. The method for detecting defects in power equipment based on multimodal alignment according to claim 1, characterized in that: According to the aligned modal data, the time series model is used to perform defect detection analysis on the power equipment to obtain the power equipment defect detection results. The time series model specifically adopts the ARIMAX model, which is as follows: The ARIMAX model parameters are estimated using the maximum likelihood estimation method, and the estimated ARIMAX model parameters are input into the ARIMAX model for fitting to obtain the fitted ARIMAX model; The data features of the aligned modal data are extracted and input into the fitted ARIMAX model to perform defect detection analysis on the power equipment to obtain the power equipment defect detection results.

7. A power equipment defect detection system based on multimodal alignment, characterized in that: The system is used in a method for detecting defects in electric power equipment based on multimodal alignment as described in any one of claims 1 to 6, and the system comprises: A feature difference acquisition module is used to obtain the spatial feature difference and temporal feature difference between each two modal data in the collected visible light image, infrared image and ultrasonic signal; The module for acquiring the combination of data to be aligned is used to perform logistic regression processing on the spatial feature differences and the temporal feature differences to obtain the alignment coefficient between each pair of modal data, and to determine the combination of data to be aligned for power equipment defect detection based on the alignment coefficient; The power equipment defect detection module is used to perform data alignment on the data combination to be aligned to obtain aligned modal data, and to perform defect detection analysis on the power equipment using a time series model based on the aligned modal data to obtain power equipment defect detection results.

8. The power equipment defect detection system based on multimodal alignment according to claim 7, characterized in that: In the collected visible light images, infrared images, and ultrasonic signals, the spatial feature differences and temporal feature differences between each two modal data are obtained, specifically: The time features of visible light images, infrared images and ultrasonic signals are collected respectively, and the time features between each two modal data are calculated to obtain the time feature difference between each two modal data; A graph structure is constructed between visible light images, infrared images and ultrasonic signals, and the spatial feature differences between each two modal data are extracted from the graph structure.

9. The power equipment defect detection system based on multimodal alignment according to claim 7, characterized in that: Logistic regression is performed on the spatial feature differences and temporal feature differences to obtain the alignment coefficient between each two modal data, specifically: The spatial feature differences and temporal feature differences between each two modal data are input into the logistic regression algorithm for logistic regression processing to obtain the alignment coefficient between each two modal data.

10. A computer device comprising a system memory and a processor, wherein the system memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images