Power transformer multi-modal image data space-time synchronization method and device

Abstract

Embodiments of the present disclosure provide a power transformer multi-modal image data space-time synchronization method and device. The method comprises: collecting multi-modal image data of a power transformer; determining one kind of modal image data from the multi-modal image data as first modal image data, and determining other modal image data as second modal image data; performing time synchronization on the first modal image data and the second modal image data according to respective time stamps of the first modal image data and the second modal image data; calculating a coordinate transformation matrix from the second modal image data to the first modal image data according to the first modal image data and the second modal image data at the same time after time synchronization, and performing space synchronization on the first modal image data and the second modal image data after time synchronization based on the coordinate transformation matrix. In this way, the space-time synchronization effect of multi-modal image data can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of power equipment monitoring and diagnosis technology, and in particular to a method and apparatus for spatiotemporal synchronization of multimodal image data of power transformers. Background Technology

[0002] In the condition monitoring of power transformers, using multi-modal image data such as infrared, visible light, and ultraviolet light can more comprehensively reflect the transformer's operating status. Infrared image data can display temperature distribution, used to monitor abnormal temperature rise, poor contact, and other problems; visible light image data can identify external defects, facilitating intuitive inspection; and ultraviolet image data can be used to detect partial discharge. However, because these image data are heterogeneous, direct analysis of them will result in temporal and spatial inconsistencies, affecting the accuracy of fault diagnosis. Therefore, spatiotemporal synchronization of multi-modal image data is necessary before analysis, and a high-precision, low-latency spatiotemporal synchronization method is urgently needed to support real-time monitoring and diagnosis of power transformer conditions. Summary of the Invention

[0003] In a first aspect, embodiments of this disclosure provide a method for spatiotemporal synchronization of multimodal image data of power transformers, the method comprising:

[0004] Acquire multimodal image data of power transformers;

[0005] One modality of image data is identified from the multimodal image data as the first modality image data, and the other modal image data in the multimodal image data are identified as the second modality image data;

[0006] Based on the timestamps corresponding to the first modal image data and the second modal image data, time synchronization is performed on the first modal image data and the second modal image data;

[0007] Based on the first modal image data and the second modal image data at the same time after time synchronization, calculate the coordinate transformation matrix from the second modal image data to the first modal image data, and then perform spatial synchronization on the first modal image data and the second modal image data after time synchronization based on this matrix.

[0008] In some possible implementations of the first aspect, the multimodal image data includes: infrared image data, visible light image data, and ultraviolet image data; acquiring multimodal image data of power transformers includes:

[0009] The infrared, visible light, and ultraviolet cameras deployed around the power transformer are clock-calibrated. The calibrated infrared, visible light, and ultraviolet cameras are then used to capture images of the power transformer, obtaining infrared, visible light, and ultraviolet image data of the power transformer.

[0010] In some possible implementations of the first aspect, visible light image data is the first modality image data, and infrared and ultraviolet image data are the second modality image data; time synchronization of the first modality image data and the second modality image data is performed according to their respective timestamps, including:

[0011] If the sampling frequencies of the infrared camera, visible light camera, and ultraviolet camera are the same, then the infrared image data, visible light image data, and ultraviolet image data are aligned on the time axis according to their respective timestamps.

[0012] In some possible implementations of the first aspect, visible light image data is the first modality image data, and infrared and ultraviolet image data are the second modality image data; time synchronization of the first modality image data and the second modality image data is performed according to their respective timestamps, including:

[0013] If the sampling frequencies of the infrared camera, visible light camera, and ultraviolet camera are different, then based on the timestamps corresponding to the infrared image data, visible light image data, and ultraviolet image data, an interpolation algorithm is used to compensate for the low-frequency image data in the infrared image data, visible light image data, and ultraviolet image data, so that they are aligned with the high-frequency data on the time axis.

[0014] In some possible implementations of the first aspect, multiple structural feature points on the surface of the power transformer are pre-selected as calibration points; based on the first modal image data and the second modal image data at the same time after time synchronization, a coordinate transformation matrix from the second modal image data to the first modal image data is calculated, including:

[0015] Based on the first modal image data and the second modal image data at the same time after time synchronization, determine the coordinates of the calibration points corresponding to the first modal image data and the second modal image data at the same time after time synchronization.

[0016] Based on the calibration point coordinates corresponding to the first and second modal image data at the same time after time synchronization, calculate the coordinate transformation matrix from the second modal image data to the first modal image data.

[0017] In some possible implementations of the first aspect, based on the first modal image data and the second modal image data at the same time after time synchronization, a coordinate transformation matrix from the second modal image data to the first modal image data is calculated, including:

[0018] Line detection is performed on the first modal image data and the second modal image data at the same time after time synchronization, and feature extraction is performed on the detected lines to obtain the line features corresponding to the first modal image data and the second modal image data at the same time after time synchronization.

[0019] Line matching is performed based on the line features corresponding to the first modal image data and the second modal image data at the same time after time synchronization, to determine the line pairs corresponding to the first modal image data and the second modal image data at the same time after time synchronization.

[0020] Calculate the coordinate transformation matrix from the second modal image data to the first modal image data based on the line pairs.

[0021] In some possible implementations of the first aspect, based on the first modal image data and the second modal image data at the same time after time synchronization, a coordinate transformation matrix from the second modal image data to the first modal image data is calculated, including:

[0022] After time synchronization, the first and second modal image data at the same time are denoised and normalized, and then multi-scale decomposition is performed. By downsampling layer by layer, a resolution hierarchy from coarse to fine is generated, thereby constructing the multi-resolution pyramids corresponding to the first and second modal image data respectively. The first and second modal image data are located at the bottom layer of their respective multi-resolution pyramids.

[0023] Multi-resolution rigid registration is performed based on the multi-resolution pyramids corresponding to the first and second modal image data to determine the coordinate transformation matrix from the second modal image data to the first modal image data. Specifically, starting from the top layer of the multi-resolution pyramid, mutual information is used as a similarity measure, and the rigid affine transformation matrix is ​​iteratively optimized using a gradient descent algorithm. The final rigid affine transformation matrix of each layer is used as the initial rigid affine transformation matrix of the next layer and passed down level by level. Finally, the final rigid affine transformation matrix of the bottom layer is used as the coordinate transformation matrix from the second modal image data to the first modal image data.

[0024] Among some possible implementations of the first aspect, the method also includes:

[0025] The first and second modal image data after spatiotemporal synchronization are fused to obtain multimodal fused image data. The multimodal fused image data is then analyzed to diagnose whether there is a fault in the power transformer.

[0026] In some possible implementations of the first aspect, the first modal image data and the second modal image data after spatiotemporal synchronization are fused to obtain multimodal fused image data, including:

[0027] Multimodal fused image data is obtained by fusing the first and second modal image data after spatiotemporal synchronization using discrete wavelet transform based on principal component averaging. Specifically, the first and second modal image data after spatiotemporal synchronization are decomposed into one low-frequency sub-band and three high-frequency sub-bands using discrete wavelet transform, where the low-frequency sub-band is LL and the high-frequency sub-bands are LH, HL, and HH, respectively. Then, principal component fusion is performed on the LL part of both, and average fusion is performed on the LL, HL, and HH parts of both. Inverse discrete wavelet transform is performed on the fusion results of the LL part, HL part, and HH part to obtain the multimodal fused image data.

[0028] Secondly, embodiments of this disclosure provide a spatiotemporal synchronization device for multimodal image data of a power transformer, the device comprising:

[0029] The acquisition module is used to acquire multimodal image data of power transformers;

[0030] The determination module is used to determine one type of image data from the multimodal image data as the first modal image data, and to determine the other modal image data from the multimodal image data as the second modal image data;

[0031] The time synchronization module is used to synchronize the first modal image data and the second modal image data in time according to the timestamps corresponding to the first modal image data and the second modal image data, respectively.

[0032] The spatial synchronization module is used to calculate the coordinate transformation matrix from the second modal image data to the first modal image data based on the first modal image data and the second modal image data at the same time after time synchronization, and to perform spatial synchronization on the first modal image data and the second modal image data after time synchronization based on this matrix.

[0033] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0034] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0035] Compared with the prior art, this disclosure has at least the following technical effects:

[0036] Improve diagnostic accuracy: Through synchronous processing of time and space, multimodal image data can be accurately registered and aligned, ensuring the consistency of basic data for fault analysis, thereby improving diagnostic accuracy.

[0037] Enhanced monitoring comprehensiveness: After simultaneous fusion of multimodal image data, temperature, appearance and discharge information can be analyzed at the same time, providing comprehensive transformer condition monitoring.

[0038] Reduced false positives and false negatives: Spatial and temporal registration and alignment reduce errors caused by inconsistencies in image data, significantly reducing the probability of false positives and false negatives of faults.

[0039] Improved real-time performance: This disclosure enables rapid synchronous processing of multimodal image data, making it suitable for real-time monitoring and intelligent diagnosis.

[0040] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0041] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0042] Figure 1 A flowchart of a spatiotemporal synchronization method for multimodal image data of a power transformer provided by an embodiment of this disclosure is shown;

[0043] Figure 2 A structural diagram of a spatiotemporal synchronization device for multimodal image data of a power transformer provided in an embodiment of this disclosure is shown.

[0044] Figure 3 A structural diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0046] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0047] To address the problems in the background art, embodiments of this disclosure provide a method, apparatus, device, and storage medium for spatiotemporal synchronization of multimodal image data of power transformers, aiming to improve the spatiotemporal synchronization effect of multimodal image data of power transformers, thereby providing accurate input for intelligent fault diagnosis of power transformers.

[0048] The following detailed description, with reference to the accompanying drawings and specific embodiments, illustrates a method, apparatus, device, and storage medium for spatiotemporal synchronization of multimodal image data of a power transformer provided by the present disclosure.

[0049] Figure 1 A flowchart illustrating a spatiotemporal synchronization method for multimodal image data of a power transformer provided by an embodiment of this disclosure is shown, such as... Figure 1 As shown, the spatiotemporal synchronization method 100 for multimodal image data of power transformers may include the following steps:

[0050] S110 collects multimodal image data of power transformers.

[0051] In some embodiments, the multimodal image data may include infrared image data, visible light image data, and ultraviolet image data. Accordingly, the infrared, visible light, and ultraviolet cameras deployed around the power transformer can be clock-calibrated to ensure that the clocks of each camera are synchronized with a unified system clock. Then, the clock-calibrated infrared, visible light, and ultraviolet cameras are used to capture images of the power transformer, obtaining infrared, visible light, and ultraviolet image data of the power transformer. The various image data include image frames and their corresponding timestamps and initial position information, etc., which are not limited herein.

[0052] S120, determine one modal image data from the multimodal image data as the first modal image data, and designate the other modal image data from the multimodal image data as the second modal image data.

[0053] In some embodiments, the multimodal image data described above may include infrared image data, visible light image data, and ultraviolet image data. Here, visible light image data can be identified as the first modal image data, and infrared image data and ultraviolet image data can be identified as the second modal image data.

[0054] S130, based on the timestamps corresponding to the first modal image data and the second modal image data, time synchronization is performed on the first modal image data and the second modal image data.

[0055] In some embodiments, referring to the above description of determining visible light image data as first modal image data and infrared image data and ultraviolet image data as second modal image data, if the sampling frequencies of the infrared camera, visible light camera, and ultraviolet camera are the same, the infrared image data, visible light image data, and ultraviolet image data can be aligned on the time axis according to their respective timestamps to achieve time synchronization; if the sampling frequencies of the infrared camera, visible light camera, and ultraviolet camera are different, the low-frequency image data in the infrared image data, visible light image data, and ultraviolet image data can be compensated using interpolation algorithms (such as linear interpolation, spline interpolation, etc.) according to their respective timestamps to keep them aligned with the high-frequency data on the time axis.

[0056] S140: Based on the first modal image data and the second modal image data at the same time after time synchronization, calculate the coordinate transformation matrix from the second modal image data to the first modal image data, and perform spatial synchronization on the first modal image data and the second modal image data after time synchronization based on this matrix.

[0057] In some embodiments, multiple structural feature points on the surface of the power transformer are pre-selected as calibration points. To this end, the coordinates of the calibration points corresponding to the first and second modal image data at the same time after time synchronization can be determined based on the first and second modal image data at the same time after time synchronization. Then, based on the coordinates of the calibration points corresponding to the first and second modal image data at the same time after time synchronization, the coordinate transformation matrix from the second modal image data to the first modal image data can be calculated.

[0058] As an example, the calculation process of the above coordinate transformation matrix can be shown below:

[0059] The coordinates of the calibration point corresponding to the first modality image data (such as visible light image data) are designated as: P VIS ={P VIS,1 ,P VIS,2 ,,P VIS,N}, where P VIS,i =(x VIS,i ,y VIS,i );

[0060] The coordinates of the calibration point corresponding to the second modality image data (such as infrared image data) are designated as: PIR ={P IR,1 ,P IR,2 ,,P IR,N}, where P IR,i =(x IR,i ,y IR,i );

[0061] A 2×3 transformation matrix H1 is used to represent the first coordinate transformation matrix from the second modality image data to the first modality image data:

[0062]

[0063] The transformation relationship is as follows:

[0064]

[0065] Right now:

[0066]

[0067] Solve for H1 using the least squares method:

[0068]

[0069] In other embodiments, it is not necessary to pre-select multiple structural feature points on the surface of the power transformer as calibration points. Instead, line detection can be performed on the first and second modal image data at the same time after time synchronization, and feature extraction can be performed on the detected lines to obtain the line features corresponding to each of the first and second modal image data at the same time after time synchronization. Then, line matching is performed based on the line features corresponding to each of the first and second modal image data at the same time after time synchronization to determine the line pairs corresponding to the first and second modal image data at the same time after time synchronization. Finally, based on the line pairs, the coordinate transformation matrix from the second modal image data to the first modal image data is calculated.

[0070] As an example, the calculation process of the above coordinate transformation matrix can be shown below:

[0071] The coordinates of the points in the first modality image data (such as visible light image data) are labeled as P. VIS ;

[0072] The coordinates of the points in the second modality image data (such as infrared image data) are labeled as P. IR ;

[0073] The first line pair corresponding to the second modality image data and the first modality image data is denoted as: L1 = {(P IR ,P VIS )};

[0074] The coordinate transformation matrix from the second modal image data to the first modal image data is denoted as H1;

[0075] The formula for calculating H1 is:

[0076]

[0077] In some embodiments, the first and second modal image data at the same time after time synchronization can be denoised and normalized to obtain grayscale image data or image data of the same format. Then, multi-scale decomposition is performed, generating a coarse-to-fine resolution hierarchy through layer-by-layer downsampling, thereby constructing multi-resolution pyramids corresponding to the first and second modal image data. The first and second modal image data are located at the bottom layer of their respective multi-resolution pyramids. Multi-resolution rigid registration is performed based on the multi-resolution pyramids corresponding to the first and second modal image data to determine the coordinate transformation matrix from the second modal image data to the first modal image data. Specifically, starting from the top of the multi-resolution pyramid, mutual information (MI) is used as a similarity metric, i.e., an evaluation index for image data registration quality. The rigid affine transformation matrix is ​​iteratively optimized using the gradient descent algorithm. The final rigid affine transformation matrix of each layer is used as the initial rigid affine transformation matrix of the next layer and passed down level by level. Finally, the final rigid affine transformation matrix of the bottom layer is used as the coordinate transformation matrix from the second modality image data to the first modality image data.

[0078] As an example, some details of the above processing procedure can be elaborated as follows:

[0079] Step 1: Constructing a multi-resolution pyramid

[0080] The first and second modal image data are converted into multiple resolution levels, and a multi-resolution pyramid for registration is constructed. This multi-resolution pyramid has four layers. Specifically, the original input image data is located at the bottom layer of the multi-resolution pyramid. If the size of this image data is N×N (original resolution), then the size of the upper layer will be N / 2×N / 2 (medium resolution), the next upper layer will be N / 4×N / 4 (low to medium resolution), and the top layer will be N / 8×N / 8 (lowest resolution).

[0081] Step 2: Multi-resolution rigid registration

[0082] By optimizing the rigid affine transformation matrix, the second modality image data is spatially registered and aligned with the first modality image data as much as possible.

[0083] (1) In order to avoid getting trapped in local optima and improve speed, pyramid resolution registration is adopted. That is, for the multi-resolution pyramid established in the first step, coarse registration is started from the top layer, which is the lowest resolution layer, and the final rigid affine transformation matrix of the current layer is used as the initial parameter of the next layer for iteration.

[0084] (2) Select a similarity metric. For the similarity metric between heterogeneous image data, to ensure registration even when the contrast is inconsistent, mutual information (MI) is selected. AB As a similarity measure, that is, a similarity measurement function:

[0085] MI AB =H A +H B -H AB ;

[0086] Among them, H A H is the joint entropy of the first modality image data. B H is the joint entropy of the second modality image data. AB The joint entropy is the data between the first modality image data and the second modality image data.

[0087] (3) Select an optimization algorithm. Since the similarity metric function is a non-convex function, the gradient descent algorithm is selected for optimization until the similarity between the second modality image data and the first modality image data reaches its maximum.

[0088] (4) Rigid affine transformation: Rigid affine transformation in 2D image data includes rotation and translation. The rotation parameter is only the rotation angle θ (around the center point), and the translation parameter includes t. x t y The rigid affine transformation matrix is:

[0089]

[0090] (5) B-spline image data interpolation: After the image data undergoes rigid affine transformation, the pixel positions of the original image data have been shifted or rotated. At this time, many new positions no longer fall on integer pixel coordinates. Therefore, the pixel values ​​of these non-integer positions are re-estimated by the B-spline interpolation method, and pixel values ​​under new coordinates are generated.

[0091] In some embodiments, spatial synchronization can be achieved by transforming the time-synchronized second modal image data to the coordinate system of the time-synchronized first modal image data according to the coordinate transformation matrix.

[0092] Furthermore, after S140, the first modal image data and the second modal image data after spatiotemporal synchronization can be fused to obtain multimodal fused image data, and the multimodal fused image data can be analyzed to diagnose whether there is a fault in the power transformer.

[0093] As an example, the above multimodal image data fusion process can be described as follows:

[0094] Multimodal fused image data is obtained by fusing the first and second modal image data after spatiotemporal synchronization using discrete wavelet transform based on principal component averaging. Specifically, the first and second modal image data after spatiotemporal synchronization are decomposed into one low-frequency sub-band and three high-frequency sub-bands using discrete wavelet transform, where the low-frequency sub-band is LL and the high-frequency sub-bands are LH, HL, and HH, respectively. Then, principal component fusion is performed on the LL component of both data, and average fusion is performed on the LL, HL, and HH components of both data respectively. Finally, inverse discrete wavelet transform is performed on the fusion results of the LL, HL, and HH components to obtain the multimodal fused image data.

[0095] To make it easier to understand, the above steps can be further elaborated as follows:

[0096] (1) Using discrete wavelet transform (db2 wavelet), the first modal image data I1 and the second modal image data I2 after spatiotemporal synchronization are decomposed into one low-frequency sub-band (LL) and three high-frequency sub-bands (LH, HL, HH), that is:

[0097] DWT(I1)→{LL1,LH1,HL1,HH1};

[0098] DWT(I2)→{LL2,LH2,HL2,HH2};

[0099] The low-frequency subband (LL) contains the main brightness and structural information of the image data; the high-frequency subband (LH, HL, HH) contains the edge, texture and other detailed information of the image data.

[0100] (2) Principal component fusion is performed on the low-frequency part LL.

[0101] Construct matrix X.

[0102]

[0103] Calculate the covariance matrix cov(X) of matrix X, and find the eigenvector w = [w1, w2] corresponding to the largest eigenvalue. T The principal component fusion result is:

[0104] LL F(i,j)=w1·LL1(i,j)+w2·LL2(i,j);

[0105] This weighting method automatically assigns weights based on the energy of image features, thereby increasing the amount of information fused.

[0106] (3) Average fusion of the high-frequency components LH, HL, and HH.

[0107]

[0108] The same applies to HL and HH.

[0109] (4) Use inverse discrete wavelet transform to synthesize the final multimodal fused image data.

[0110] I F =IDWT(LL F ,LH F HL F ,HH F );

[0111] (5) The Structural Similarity Index Metric (SSIM) is calculated as an evaluation index for the quality of image data fusion. For the first modality image data I1 and the second modality image data I2, the calculation formula is as follows:

[0112]

[0113] Where μ1 and μ2 are the mean values ​​(brightness information) of image data I1 and I2, respectively, and σ x 2 σ y 2 These are the variances (contrast information) of image data I1 and I2, respectively, σ xy The covariance (structural information) of image data I1 and I2 is given. C1 and C2 are constants and set to very small values ​​to prevent the denominator from being 0.

[0114] As an example, this analysis of multimodal fusion image data is used to diagnose whether a power transformer has a fault. This is mainly based on the structural state in the visible light image data, the temperature in the infrared image data, and the discharge phenomenon in the ultraviolet image data. The analysis comprehensively judges whether the transformer has an anomaly and provides fault location information.

[0115] In summary, this disclosure achieves at least the following technical effects:

[0116] Improve diagnostic accuracy: By synchronizing time and space, multimodal image data can be accurately aligned, ensuring the consistency of basic data for fault analysis, thereby improving diagnostic accuracy.

[0117] Enhanced monitoring comprehensiveness: After simultaneous fusion of multimodal image data, temperature, appearance and discharge information can be analyzed at the same time, providing comprehensive transformer condition monitoring.

[0118] Reduced false positives and false negatives: Spatial and temporal alignment reduces errors caused by inconsistencies in image data, significantly lowering the probability of false positives and false negatives for faults.

[0119] Improved real-time performance: This disclosure enables rapid synchronous processing of multimodal image data, making it suitable for real-time monitoring and intelligent diagnosis.

[0120] At the same time, this disclosure has at least the following potential applications:

[0121] This disclosure is applicable to condition monitoring and intelligent diagnostic systems for power transformers, and is particularly suitable for scenarios requiring synchronous analysis of multimodal image data. Through spatiotemporal synchronization methods, various image data can be integrated into a unified information source, providing strong technical support for fault diagnosis, health assessment, and preventative maintenance of power equipment.

[0122] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0123] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0124] Figure 2 A structural diagram of a spatiotemporal synchronization device for multimodal image data of a power transformer, as provided in an embodiment of this disclosure, is shown. Figure 2 As shown, the spatiotemporal synchronization device 200 for multimodal image data of power transformers may include:

[0125] The acquisition module 210 is used to acquire multimodal image data of power transformers.

[0126] The determining module 220 is used to determine one type of image data from the multimodal image data as the first modal image data, and to determine the other modal image data from the multimodal image data as the second modal image data.

[0127] The time synchronization module 230 is used to synchronize the first modal image data and the second modal image data in time according to the timestamps corresponding to the first modal image data and the second modal image data, respectively.

[0128] The spatial synchronization module 240 is used to calculate the coordinate transformation matrix from the second modal image data to the first modal image data based on the first modal image data and the second modal image data at the same time after time synchronization, and to perform spatial synchronization on the first modal image data and the second modal image data after time synchronization based on this matrix.

[0129] Understandable, Figure 2 Each module / unit in the multimodal image data spatiotemporal synchronization device 200 for power transformers shown has the ability to realize Figure 1 The functions of each step in the spatiotemporal synchronization method 100 for multimodal image data of power transformers shown are explained, and their corresponding technical effects are achieved. For the sake of brevity, these will not be elaborated here.

[0130] Figure 3 A structural diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Electronic device 300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 300 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0131] like Figure 3 As shown, the electronic device 300 may include a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0132] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0133] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer program product, including a computer program tangibly contained in a computer-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0134] The various embodiments described above can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), payload programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] It should be noted that this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute method 100 and achieve the corresponding technical effects achieved by the embodiments of this disclosure in executing the method. For the sake of brevity, these will not be elaborated here.

[0138] In addition, this disclosure also provides a computer program product including a computer program that implements method 100 when executed by a processor.

[0139] To provide interaction with a user, the embodiments described above can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0140] The embodiments described above can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with the implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0141] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0142] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0143] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for spatiotemporal synchronization of multimodal image data of power transformers, characterized in that, The method includes: Acquire multimodal image data of power transformers; One modality of image data is identified from the multimodal image data as the first modality image data, and the other modal image data in the multimodal image data are identified as the second modality image data; Based on the timestamps corresponding to the first modal image data and the second modal image data, time synchronization is performed on the first modal image data and the second modal image data; Based on the first modal image data and the second modal image data at the same time after time synchronization, calculate the coordinate transformation matrix from the second modal image data to the first modal image data, and perform spatial synchronization on the first modal image data and the second modal image data after time synchronization based on this matrix; The step of calculating the coordinate transformation matrix from the second modal image data to the first modal image data based on the first modal image data and the second modal image data at the same time after time synchronization includes: Line detection is performed on the first modal image data and the second modal image data at the same time after time synchronization, and feature extraction is performed on the detected lines to obtain the line features corresponding to the first modal image data and the second modal image data at the same time after time synchronization. Line matching is performed based on the line features corresponding to the first modal image data and the second modal image data at the same time after time synchronization, to determine the line pairs corresponding to the first modal image data and the second modal image data at the same time after time synchronization. Based on the line pairs, calculate the coordinate transformation matrix from the second modality image data to the first modality image data; or, After time synchronization, the first and second modal image data at the same time are denoised and normalized, and then multi-scale decomposition is performed. By downsampling layer by layer, a resolution hierarchy from coarse to fine is generated, thereby constructing the multi-resolution pyramids corresponding to the first and second modal image data respectively. The first and second modal image data are located at the bottom layer of their respective multi-resolution pyramids. Multi-resolution rigid registration is performed based on the multi-resolution pyramids corresponding to the first and second modal image data to determine the coordinate transformation matrix from the second modal image data to the first modal image data. Specifically, starting from the top layer of the multi-resolution pyramid, mutual information is used as a similarity measure, and the rigid affine transformation matrix is ​​iteratively optimized using a gradient descent algorithm. The final rigid affine transformation matrix of each layer is used as the initial rigid affine transformation matrix of the next layer and passed down level by level. Finally, the final rigid affine transformation matrix of the bottom layer is used as the coordinate transformation matrix from the second modal image data to the first modal image data.

2. The method according to claim 1, characterized in that, The multimodal image data includes: infrared image data, visible light image data, and ultraviolet image data; the acquired multimodal image data of the power transformer includes: The infrared, visible light, and ultraviolet cameras deployed around the power transformer are clock-calibrated. The calibrated infrared, visible light, and ultraviolet cameras are then used to capture images of the power transformer, obtaining infrared, visible light, and ultraviolet image data of the power transformer.

3. The method according to claim 2, characterized in that, The visible light image data is the first modality image data, and the infrared image data and the ultraviolet image data are the second modality image data; the step of time synchronization of the first modality image data and the second modality image data according to their respective timestamps includes: If the sampling frequencies of the infrared camera, visible light camera, and ultraviolet camera are the same, then the infrared image data, visible light image data, and ultraviolet image data are aligned on the time axis according to their respective timestamps.

4. The method according to claim 2, characterized in that, The visible light image data is the first modality image data, and the infrared image data and the ultraviolet image data are the second modality image data; the step of time synchronization of the first modality image data and the second modality image data according to their respective timestamps includes: If the sampling frequencies of the infrared camera, visible light camera, and ultraviolet camera are different, then based on the timestamps corresponding to the infrared image data, visible light image data, and ultraviolet image data, an interpolation algorithm is used to compensate for the low-frequency image data in the infrared image data, visible light image data, and ultraviolet image data, so that they are aligned with the high-frequency data on the time axis.

5. The method according to claim 1, characterized in that, The method further includes: The first and second modal image data after spatiotemporal synchronization are fused to obtain multimodal fused image data. The multimodal fused image data is then analyzed to diagnose whether there is a fault in the power transformer.

6. The method according to claim 5, characterized in that, The process of fusing the first modal image data and the second modal image data after spatiotemporal synchronization to obtain multimodal fused image data includes: Multimodal fused image data is obtained by fusing the first and second modal image data after spatiotemporal synchronization using discrete wavelet transform based on principal component averaging. Specifically, the first and second modal image data after spatiotemporal synchronization are decomposed into one low-frequency sub-band and three high-frequency sub-bands using discrete wavelet transform, where the low-frequency sub-band is LL and the high-frequency sub-bands are LH, HL, and HH, respectively. Then, principal component fusion is performed on the LL part of both, and average fusion is performed on the LL, HL, and HH parts of both. Inverse discrete wavelet transform is performed on the fusion results of the LL part, HL part, and HH part to obtain the multimodal fused image data.

7. A spatiotemporal synchronization device for multimodal image data of a power transformer, characterized in that, The apparatus is used to perform the method according to any one of claims 1-6, comprising: The acquisition module is used to acquire multimodal image data of power transformers; The determination module is used to determine one type of image data from the multimodal image data as the first modal image data, and to determine the other modal image data in the multimodal image data as the second modal image data; The time synchronization module is used to synchronize the first modal image data and the second modal image data in time according to the timestamps corresponding to the first modal image data and the second modal image data, respectively. The spatial synchronization module is used to calculate the coordinate transformation matrix from the second modal image data to the first modal image data based on the first modal image data and the second modal image data at the same time after time synchronization, and to perform spatial synchronization on the first modal image data and the second modal image data after time synchronization based on this matrix.

Images