Substation twin heating state virtual-real mapping method based on deep learning feature point matching network
Through a feature point matching network based on deep learning, SuperPoint and SuperGlue networks are used to extract and match thermal infrared image feature points, and combined with RANSAC and SinkHorn algorithms for image registration, the automation and accuracy problems of heat generation status detection of traditional substation equipment are solved. It realizes high-precision real-time monitoring and management of the heating status of substation equipment, and improves the efficiency and safety of equipment management.
Patent Information
- Application Number
- CN202411205685.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2025-01-14
AI Technical Summary
The heat status detection of traditional substation equipment relies on manual inspection, which is time-consuming and prone to human errors, lacks automation and intelligent management, and the existing virtual and real behavior synchronization mapping technology is not accurate enough to render the heat status, affecting the safety and stability of the equipment.
Adopt a feature point matching network based on deep learning, and update the texture map of the digital twin model in real time through thermal infrared sensor data, real-time real-life behavior state synchronization and dynamic mapping, using SuperPoint and SuperGlue networks to extract feature points and match them, combining RANSAC and SinkHorn The algorithm performs image registration and dynamically updates texture maps.
It realizes high-precision real-time monitoring and management of the heating status of substation equipment, improves the efficiency and safety of equipment management, reduces manual intervention, reduces operating costs, and improves the reliability and real-timeness of data processing.
Smart Images
Figure CN119313801A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of virtual-real mapping of twins of substation equipment, and specifically to a virtual-real mapping method of heating state of twins of substation equipment based on a deep learning feature point matching network. Background Art
[0002] In the traditional detection of heating status of substation equipment, manual inspection is not only time-consuming and laborious, but also easy to cause economic losses. Therefore, it is urgent to carry out automated and intelligent management of substation equipment, and the construction of digital twins is the key to realize intelligent management. Digital twins can be adjusted in real time according to parameters to more accurately reflect the heating status of real entities. It is particularly important to create a high-fidelity virtual-real mapping twin model. Virtual-real behavior synchronization mapping technology is a technology that synchronously maps physical entities into virtual environments. This technology combines sensor data to capture, analyze and map the operating status of physical equipment in real time, and synchronizes the status in the twin model. Nowadays, the heating rendering of twin equipment is to extract the temperature range of thermal infrared images according to the algorithm for visualization. Therefore, the research on the rendering method of the heating status of the digital twin of substation equipment using virtual-real behavior synchronization mapping technology has certain engineering value, plays an irreplaceable role in maintaining the safety, stability and equipment protection of substation equipment, and is of great significance to the intelligent management of power systems. Summary of the invention
[0003] The present application provides a method for virtual-real mapping of the heating state of a twin body of a substation equipment based on a deep learning feature point matching network, which updates the texture map of the twin model in real time according to external real thermal infrared sensor data to achieve state synchronization and dynamic mapping of virtual and real behaviors.
[0004] The technical solutions adopted in this application are as follows:
[0005] A virtual-real mapping method of heating state of twin body of substation equipment based on deep learning feature point matching network mainly includes the following steps:
[0006] Step 1: Data collection and data processing;
[0007] The historical infrared temperature measurement images of the thermal infrared sensors of each substation equipment in the target substation are obtained to form an original mapping data set; the original data set is constructed into a time series data set according to the time series characteristics of the actual sensor for data preprocessing (image denoising, image enhancement), forming a thermal infrared image data set with time series characteristics;
[0008] Step 2: Build a twin model;
[0009] Using Cinema4D modeling software, build components (insulating sleeves, wiring) according to the basic structural characteristics of each device; combine the spatial position of the components in the whole, assemble constraints; render the material of the substation equipment according to its material structure; assemble the component geometry models into a digital twin model of standard substation equipment;
[0010] Step 3: Build texture reference map;
[0011] Set angle limits and island margins, perform UV unfolding on all surfaces of the twin model to obtain a UV texture map, and project the three-dimensional coordinates onto a two-dimensional plane; select the initial (first moment) registration image, manually select points, and map the corresponding parts of the thermal infrared image to the corresponding positions of the UV texture map by point registration;
[0012] Step 4: Construction of virtual and real synchronized texture atlas;
[0013] Step 5: Perform image feature matching;
[0014] Step 6: Perform image registration;
[0015] Based on the matching point pairs obtained in step 5, the RANSAC algorithm is used to estimate the transformation matrix H between the images. t , register the thermal infrared image at time t to the reference UV texture map, Among them, cv2.findHomography(.) is a function used to calculate the homography matrix between two images, and RANSAC is an iterative algorithm used to filter out incorrect matching points and increase the accuracy of the homography matrix. is the matching point in the reference image, is the matching point in the image at time t; using the transformation matrix H t The thermal infrared image at time t is registered to align it with the reference UV texture map, and the registered thermal infrared texture map at time t is obtained:
[0016] I t ′=cv2.warpPerspective(I t ,H t ,(w,h)), where (w,h) are the width and height of the image, and cv2.warpPerspective represents the perspective transformation function that aligns the input image with the reference image;
[0017] Step 7: Generate texture atlas;
[0018] Generate the texture map of the corresponding component from the image at each moment after registration to form a virtual-real synchronous mapping texture map set Among them, T tRepresents the texture image at the tth moment;
[0019] Step 8: Use the TextureFromImage function to convert the image data into texture data, apply it to the corresponding geometric components of the twin for display, and dynamically update it.
[0020] Furthermore, the specific method of step 4 is:
[0021] Step 4.1: Extract image feature points;
[0022] A convolutional neural network (CNN) is used to extract the original thermal infrared image (I0, I1, I2...I n ) extracts the feature map F, F = CNN (I); the feature map F is passed through a convolutional layer to generate a feature point confidence map, and the confidence value is calculated for each pixel position (x, y) and category c: (x, y) represents the pixel position, where conv() represents the convolution operation; the confidence map is used for the softmax function and normalized: in is the normalized probability of position (x, y) in category c; non-maximum suppression NMS is applied to the normalized confidence map to retain the local maximum and determine the location of the key point:
[0023] Step 4.2: Another convolution branch is used to calculate the descriptor used to represent the local image area of the key point. After a convolution layer, the descriptor feature map D is obtained. For each pixel position (x, y) and the descriptor dimension d, the value of the descriptor is calculated. Then for the descriptor D x,y Normalize Provide a basis for image feature matching.
[0024] Further, the method of step 5 is:
[0025] Step 5.1: Extract feature points and descriptors of image A and image B using the method in step 4: in is the position of the i-th feature point in image A, is the corresponding descriptor, and is the location of the feature points in image B and its descriptor, N A 、N B Indicates the total number of feature points;
[0026] Step 5.2: Input the descriptor of the feature point as the node feature of the neural network: Use the attention mechanism to update node features and first calculate the self-attention weight: where q i =W q d i is the query vector, k j =W k d j is the key vector, W q and W k is a trainable weight matrix, d i d j Descriptor representing the node;
[0027] Step 5.3: Update the node features as follows: W v Represents the weighted transformation matrix, which applies weights to each node feature; after multiple layers of iteration of the graph neural network GNN, the matching probability between each pair of feature points is obtained, which is expressed as the matching score matrix S, and the SinkHorn algorithm is used for normalization and optimization to obtain the final matching matrix M = SinkHorn (S); finally, the matching matrix M is used to select the optimal matching pair. For each feature point i in image A, the feature point j with the highest matching probability is found in image B, that is, Get matching point pairs Among them, M ij represents the ijth element in the matching matrix M, and m(i) represents the matching point of the i-th feature point in the image at the initial moment in the image at the t moment.
[0028] This method accurately achieves state synchronization and dynamic mapping of virtual and real behaviors by updating the texture map of the twin model in real time. The main steps include: data collection and processing, twin model construction, texture reference map construction, virtual and real synchronized texture atlas construction, and map synchronization dynamic mapping. Specifically, the thermal infrared image data of the substation equipment is first collected and processed through the thermal infrared sensor data, and then the digital twin of the substation equipment is built using the Cinema4D modeling software, and the texture reference map is created by setting UV unfolding and manual registration. Then, the SuperPoint network and SuperGlue network are used to extract and match feature points to achieve accurate registration of the thermal infrared image and the twin model. Finally, the dynamic update and display of the texture map is realized through C# code, so as to realize real-time monitoring and management of the heating state of the substation equipment.
[0029] The beneficial effects of the present application are reflected in the use of a deep learning-based image feature point matching network to automate the virtual-to-real mapping process. In the detection of the heating state of traditional substation equipment, manual image updates are usually required, which is not only time-consuming but also prone to human errors. By introducing the SuperPoint and SuperGlue networks, the present invention realizes automated image feature point extraction and matching, can update the texture map of the twin model in real time, and realize dynamic monitoring of the heating state of the substation equipment. This method ensures high-precision rendering and timely reflection of the heating state of the substation equipment, thereby improving the efficiency and safety of equipment management. Ultimately, by reducing manual intervention, operating costs are reduced, while improving the reliability and real-time performance of data processing, providing strong technical support for the intelligent management of substation equipment. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 A flowchart of a method for virtual-real mapping of heating states of twin bodies of substation equipment based on a feature point matching network is provided for an embodiment of the present application;
[0031] Figure 2 A flowchart for constructing a virtual-real synchronized texture atlas provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following embodiments are not intended to represent all implementation methods consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.
[0033] See also Figure 1 , which is a flowchart of a method for virtual-real mapping of heating states of twin bodies of substation equipment based on a feature point matching network, for an embodiment of the present application; see Figure 2 , which is a flow chart for constructing a virtual-real synchronized texture atlas provided in an embodiment of the present application.
[0034] Embodiment 1
[0035] The virtual-real mapping method of the heating state of the twin body of the substation equipment based on the feature point matching network provided in this application mainly includes the following steps:
[0036] Step 1: Obtain historical thermal infrared temperature measurement images of thermal infrared sensors in the target substation area. Preprocess the original image data, mainly including image denoising and enhancement, to construct a thermal infrared image dataset with time series characteristics.
[0037] Step 2: Use Cinema4D modeling software to build sub-component models of the equipment. Assemble the components into a complete digital twin according to their spatial positions, and render the materials according to the material properties to generate a standard digital twin model of the substation equipment.
[0038] Step 3: UV unwrap all surfaces of the twin model and project the three-dimensional coordinates onto a two-dimensional plane. Select the initial (first moment) thermal infrared image, and manually select points to map the corresponding parts of the thermal infrared image to the corresponding positions of the UV texture map to create a texture reference map.
[0039] Step 4: Use the SuperPoint network to extract feature points and descriptors from each thermal infrared texture map. Use the SuperGlue network to match these feature points to obtain a matching probability matrix, which is normalized and optimized using the Sinkhorn algorithm to generate the final matching matrix. Select the best matching pair, accurately align the thermal infrared texture map at each moment with the UV texture of the geometric model, and crop to generate the texture map of the corresponding component.
[0040] Step 5: Write a C# code script to read the thermal infrared images of each device component at each moment from the local computer. Use the TextureFromImage function to convert the image data into texture data and apply it to the geometric components of the twin to achieve real-time dynamic update and display.
[0041] Step 6: Conduct a comprehensive test on the entire virtual-reality mapping system to evaluate the synchronization accuracy of texture mapping and the dynamic update effect. Based on the test results, make necessary optimizations on the data processing, model building, and texture mapping steps to improve the stability and accuracy of the system.
[0042] Embodiment 2:
[0043] The present application is described in detail below in conjunction with specific implementation methods.
[0044] This application uses the historical data of thermal infrared image acquisition of the insulating bushing of the transformer in a substation by a drone equipped with a thermal infrared sensor for three days as an example, and includes the following steps:
[0045] Step 1: Use the thermal infrared sensor carried by the drone to continuously collect thermal infrared images of the transformer in the substation and record historical data within three days. These data include thermal infrared images of the transformer from multiple perspectives (a total of 229 images), and the images contain the heating status information of the transformer insulation bushing at different time points (this embodiment performs dynamic mapping for the transformer insulation bushing, and there are slight differences in the implementation of other application scenarios).
[0046] Step 2: Preprocess the collected thermal infrared images, including:
[0047] Image denoising: Apply denoising algorithms (median filtering or Gaussian filtering) to remove noise from thermal images; Image enhancement: Use image enhancement technology (histogram equalization) to improve the contrast and clarity of thermal infrared images; Data normalization: Normalize thermal infrared image data to the range of 0 to 1 for subsequent processing.
[0048] Step 3: Use Cinema4D modeling software to build the digital twin of the substation equipment.
[0049] Component modeling: Build component models of the equipment based on the actual structural features of the substation transformer (such as insulating bushings, radiators, oil pillows, etc.); component assembly: assemble the components according to their spatial positions to generate a complete digital twin of the substation transformer; material rendering: perform material rendering on the equipment to create a transformer model with realistic texture.
[0050] Step 4: Build the base texture map:
[0051] UV unfolding: UV unfold all surfaces of the transformer twin, project the three-dimensional coordinates onto a two-dimensional plane, and each pixel point of the UV area of the insulating sleeve on the UV unfolding map has a coordinate position of (u, v, w), corresponding to the same point on the three-dimensional model;
[0052] Manual registration: Select the thermal infrared image of the transformer at the initial moment and the UV display of the transformer twin, import them into ArcGIS, select the coordinate control point range of the insulating sleeve on the two images by manual point selection (select as many as possible according to the shape of the display, at least six pairs), and Figure 1 The corresponding control point coordinates are aligned to obtain the reference texture map of the initial transformer insulation bushing position.
[0053] Step 5: Use the SuperPoint and SuperGlue algorithms to construct the thermal infrared texture atlas of the transformer insulation bushing: First, use the SuperPoint network to extract feature points and descriptors from the thermal infrared image at each moment. t (Thermal infrared image of the insulating sleeve at time t), generate the corresponding feature points and descriptors Composition feature set Where N t is the number of feature points in the image at time t; further, the SuperGlue network is used to match the feature points to obtain matching point pairs: Based on the matching point pairs, the RANSAC algorithm is used to estimate the transformation matrix between the images. The thermal infrared image at time t is registered to the reference thermal infrared texture map, and the texture map I after registration at time t is obtained. t ′=cv2.warpPerspective(I t ,H t ,(w,h)), the texture map at each moment is clipped according to the UV area, and the texture maps at all moments are collected to form the virtual-real synchronous mapping texture map set T of the transformer insulation bushing t =generateTextureMap(G,I t ′), where G is the corresponding set component.
[0054] Step 5: Write a C# code script to read the thermal infrared texture map of the transformer insulation bushing at each moment locally (it can actually be read from the sensor in real time), create a material ball to update and render the transformer bushing area, and dynamically display the rendering status in real time.
[0055] The entire virtual-to-real mapping process was fully tested to evaluate the synchronization accuracy and dynamic update effect of texture mapping. It can be observed from the visual effect that the registered texture can be correctly aligned to the specified area of the UV grid, and the texture mapping at subsequent moments can be superimposed with the reference texture, and the alignment effect is good. After multiple implementations, the real-time update rate of the map is 2.37s, the reprojection error of the map registration is about 4.67 pixels, and the matching accuracy rate reaches 72.6%, showing relatively good performance in the matching of thermal infrared images.
[0056] In summary, the experimental verification results of the virtual-real synchronous state rendering of the twin model based on the thermal infrared texture map of the transformer insulation bushing based on the SuperPoint and SuperGlue algorithms strongly prove the application value of the present invention in the field of substation equipment monitoring and status assessment. This method not only achieves high precision and efficiency of image registration in theory, but also shows significant effects and advantages in practical applications. These advantages make this application an efficient and reliable tool in substation equipment monitoring and fault diagnosis, which helps to improve the accuracy and real-time performance of automated equipment maintenance, thereby contributing to the intelligent management and operation and maintenance of substation equipment.
Claims
1. A virtual-real mapping method of heating state of twin body of substation equipment based on deep learning feature point matching network mainly includes the following steps: Step 1: Data collection and data processing; Obtain the historical infrared temperature measurement images of the thermal infrared sensors of each substation equipment in the target substation to form an original mapping data set; The original data set is constructed into a time series data set according to the time series characteristics of the actual sensor for data preprocessing, thus forming a thermal infrared image data set with time series characteristics; Step 2: Build a twin model; Use Cinema4D modeling software to build components based on the basic structural features of each device; Combine the spatial position of the components in the whole and assemble the constraints; render the material of the substation equipment according to its material structure; assemble the component geometry models into a digital twin model of standard substation equipment; Step 3: Build texture reference map; Set angle limits and island margins, perform UV unfolding on all surfaces of the twin model to obtain a UV texture map, and project the 3D coordinates onto a 2D plane; Select the initial registration image, manually select points and map the corresponding parts of the thermal infrared image to the corresponding positions of the UV texture image by point registration; Step 4: Construction of virtual and real synchronized texture atlas; Step 5: Perform image feature matching; Step 6: Perform image registration; Based on the matching point pairs obtained in step 5, the RANSAC algorithm is used to estimate the transformation matrix H between the images. t , register the thermal infrared image at time t to the reference UV texture map, Among them, cv2.findHomography(.) is a function used to calculate the homography matrix between two images, and RANSAC is an iterative algorithm used to filter out incorrect matching points and increase the accuracy of the homography matrix. is the matching point in the reference image, is the matching point in the image at time t; using the transformation matrix H t The thermal infrared image at time t is registered to align it with the reference UV texture map, and the registered thermal infrared texture map at time t is obtained: I′ t =cv2.warpPerspective(I t ,H t ,(w,h)), where (w,h) are the width and height of the image, and cv2.warpPerspective represents the perspective transformation function that aligns the input image with the reference image; Step 7: Generate texture atlas; Generate the texture map of the corresponding component from the image at each moment after registration to form a virtual-real synchronous mapping texture map set Among them, T t Represents the texture image at the tth moment; Step 8: Use the TextureFromImage function to convert the image data into texture data, apply it to the corresponding geometric components of the twin for display, and dynamically update it.
2. A virtual-real mapping method of heating state of a substation twin body based on a deep learning feature point matching network as claimed in claim 1, characterized in that: The specific method of step 4 is: Step 4.1: Extract image feature points; A convolutional neural network (CNN) is used to extract the original thermal infrared image (I0, I1, I2...I n ) extracts the feature map F, F = CNN (I); the feature map F is passed through a convolutional layer to generate a feature point confidence map, and the confidence value is calculated for each pixel position (x, y) and category c: (x, y) represents the pixel position, where conv() represents the convolution operation; the confidence map is used for the softmax function and normalized: in is the normalized probability of position (x, y) in category c; non-maximum suppression NMS is applied to the normalized confidence map to retain the local maximum and determine the location of the key point: Step 4.2: Another convolution branch is used to calculate the descriptor used to represent the local image area of the key point. After a convolution layer, the descriptor feature map D is obtained. For each pixel position (x, y) and the descriptor dimension d, the value of the descriptor is calculated. Then for the descriptor D x,y Normalize Provide a basis for image feature matching.
3. A virtual-real mapping method of heating state of a substation twin body based on a deep learning feature point matching network as claimed in claim 1, characterized in that: The method of step 5 is: Step 5.1: Extract feature points and descriptors of image A and image B using the method in step 4: in is the position of the i-th feature point in image A, is the corresponding descriptor, and is the location of the feature points in image B and its descriptor, N A 、N B Indicates the total number of feature points; Step 5.2: Input the descriptor of the feature point as the node feature of the neural network: Use the attention mechanism to update node features and first calculate the self-attention weight: where q i =W q d i is the query vector, k j =W k d j is the key vector, W q and W k is a trainable weight matrix, d i ,d j Descriptor representing the node; Step 5.3: Update the node features as follows: W v Represents the weighted transformation matrix, which applies weights to each node feature; after multiple layers of iteration of the graph neural network GNN, the matching probability between each pair of feature points is obtained, which is expressed as the matching score matrix S, and the SinkHorn algorithm is used for normalization and optimization to obtain the final matching matrix M = SinkHorn (S); finally, the matching matrix M is used to select the optimal matching pair. For each feature point i in image A, the feature point j with the highest matching probability is found in image B, that is, Get matching point pairs Among them, M ij represents the ijth element in the matching matrix M, and m(i) represents the matching point of the i-th feature point in the image at the initial moment in the image at the t moment.