A 3D Visualization Method for Multi-Source Heterogeneous Data of Power Equipment

Abstract

The present invention discloses a three-dimensional visualization method for multi-source heterogeneous data of power equipment, including reconstructing a three-dimensional mesh model of the equipment through the spatial positions of the camera and the equipment; registering and transforming the data by using the spatial correlation relationship of multi-band images; and performing three-dimensional visualization of the transformed heterogeneous and homogeneous data by using the texture mapping method and the three-dimensional mesh model of the equipment. The present invention uses the three-dimensional reconstruction method to recover the spatial position of the equipment and the three-dimensional mesh model of the equipment from the visible light image of the equipment, uses the homogeneous and heterogeneous image registration methods to respectively establish the spatial correlation relationship between the homogeneous and heterogeneous state data, selects the optimal matching data, and assigns the registered multi-source heterogeneous data visualization texture to the three-dimensional mesh model of the equipment to complete the three-dimensional visualization of the equipment state data. This method improves the efficiency of equipment state evaluation, the processing efficiency of monitoring data, the intuitiveness and integrity of monitoring data, and the reliability of equipment state simulation results.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D visualization, and particularly to a 3D visualization method for multi-source heterogeneous data of power equipment. Background Art

[0002] In recent years, with the implementation of the national digital economy development strategy, the operation of the power grid has gradually shifted towards digital and intelligent construction. As a key technology in the process of power grid digital and intelligent construction, digital twin, based on multi-source sensors as the hardware foundation, combines advanced technologies such as artificial intelligence, virtual reality, and the Internet of Things, aiming to achieve all-round state perception and long-term safe and stable operation of the power grid.

[0003] At the equipment level, the application of digital twin technology can conduct intelligent monitoring, fault early warning, and diagnosis of the state of power equipment. For example, after mapping the real state of the equipment to the digital twin model, simulation calculations and data analysis can be carried out using the digital model, and the prediction or evaluation results of the equipment state can be given and visualized in 3D. However, the amount of multi-source heterogeneous data such as visible light, infrared, ultraviolet, and audio generated by equipment state monitoring is large, and it is time-consuming, laborious, and inefficient to manually map the data between the physical and virtual models. Summary of the Invention

[0004] The purpose of this part is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this part, as well as in the abstract and title of the specification of this application, to avoid obscuring the purpose of this part, the abstract, and the title. However, such simplifications or omissions shall not be used to limit the scope of the present invention.

[0005] In view of the above existing problems, a 3D visualization method for multi-source heterogeneous data of power equipment according to the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is that the data mapping between the physical and virtual models is time-consuming, laborious, and inefficient.

[0007] To solve the above technical problem, the present invention provides the following technical solution: A 3D visualization method for multi-source heterogeneous data of power equipment, including,

[0008] Reconstructing a 3D mesh model of the equipment by taking multi-view pictures of the power equipment with a visible light camera;

[0009] Registering and transforming the data using the spatial correlation relationship of multi-band images;

[0010] Performing 3D visualization of the transformed heterogeneous and homogeneous data using the texture mapping method and the 3D mesh model of the equipment.

[0011] As a preferred solution of the three-dimensional visualization method for multi-source heterogeneous data of power equipment according to the present invention, wherein: the multi-view pictures of power equipment captured by a visible light camera, and reconstructing the three-dimensional mesh model of the equipment includes:

[0012] Using monocular visible light three-dimensional reconstruction technology to automatically match images from the visible light image sequence, calculate the camera shooting pose and the equipment spatial position, and reconstruct the three-dimensional mesh model of the equipment. The three-dimensional mesh model of the equipment is the model basis for the three-dimensional visualization of equipment status data.

[0013] As a preferred solution of the three-dimensional visualization method for multi-source heterogeneous data of power equipment according to the present invention, wherein: calculating the camera shooting pose and the equipment spatial position includes:

[0014] Perform feature matching on the input visible light image sequence;

[0015] Solve the camera pose and perform adjustment and optimization to restore the shooting position of the camera and the sparse point cloud model of the equipment;

[0016] Use the solved camera pose as the input image and perform further traversal matching on it to obtain image matching points, and then restore the dense point cloud, surface mesh and texture model of the scene.

[0017] As a preferred solution of the three-dimensional visualization method for multi-source heterogeneous data of power equipment according to the present invention, wherein: using the solved camera pose as the input and performing further traversal matching on it to obtain image matching points includes:

[0018] Use the RANSAC method to perform traversal matching on each group of feature points in the input image sequence to obtain the two-dimensional matching point relationship between the input images, and obtain the three-dimensional depth and spatial coordinates of the two-dimensional feature points through the rotation and translation matrix between the cameras and the triangulation result to determine the three-dimensional space points.

[0019] As a preferred solution of the three-dimensional visualization method for multi-source heterogeneous data of power equipment according to the present invention, wherein: restoring the dense point cloud, surface mesh and texture model of the scene includes:

[0020] Perform traversal matching on the region of each feature point neighborhood in the input image and the region on the epipolar line of the matching image to obtain the dense point cloud model of the equipment. Use the weak texture support region classification method to classify all the point clouds in the dense point cloud according to their spatial positions, and optimize the meshing weights of the point clouds in the weak texture support region according to the classification results to construct a complete surface mesh and texture model to obtain the three-dimensional mesh model of the equipment and the texture mapping relationship between it and the visible light image.

[0021] As a preferred solution of the three-dimensional visualization method for multi-source heterogeneous data of power equipment according to the present invention, wherein: the registration and transformation of data by using the spatial correlation relationship of multi-band images includes:

[0022] Based on visible light image data, other heterogeneous data is traversally registered with it in the two-dimensional space by using heterologous image registration. The other heterogeneous data includes infrared temperature measurement data and ultraviolet detection data, and the spatial transformation relationship between the visible light image and other band images is obtained. The other band images include infrared images and ultraviolet images. Combining with the data decision method, a one-to-one matching relationship between heterogeneous data is obtained.

[0023] As a preferred solution of the three-dimensional visualization method for multi-source heterogeneous data of power equipment according to the present invention, wherein: the heterologous image registration includes:

[0024] Extract the feature points on the equipment contour in the heterologous image, calculate the main direction of the feature points, then extract the improved SIFT feature descriptors at multiple scales of the heterologous multi-images, obtain the initial matching points and perform iterative screening on them to obtain a matching result without incorrect matching points, and solve the perspective transformation relationship matrix between the other band images and the visible light image from the matching point pairs, and interpolate and transform the original detection data corresponding to the other band images to the same scale and perspective as the visible light image to establish the spatial correlation relationship between heterogeneous data.

[0025] As a preferred solution of the three-dimensional visualization method for multi-source heterogeneous data of power equipment according to the present invention, wherein: the data decision method includes:

[0026] The data decision method is based on the index ratio sum and is used to select the optimal single image to establish a one-to-one matching relationship when the registration of a single image with multiple homologous images is successful.

[0027] As a preferred solution of the three-dimensional visualization method for multi-source heterogeneous data of power equipment according to the present invention, wherein: the decision-making process of the data decision method is as follows:

[0028] Calculate the registration error value between a single image and the set of homologous or heterologous images corresponding to it, and this value is used to reflect the alignment accuracy after heterologous image registration;

[0029] Calculate the information entropy value of the corresponding grayscale image after the set of homologous or heterologous images undergoes perspective transformation, and this value is used to reflect the clarity of the single image and the richness of texture information;

[0030] Calculate the ratio of the number of pixel points of the coordinates of all pixel points of the set of homologous or heterologous images located within the pixel coordinate index range of the single image after perspective transformation to the total number, and this value reflects the proportion of valid data after registration of any image in the set of homologous or heterologous images with the single image;

[0031] Calculate the sum of the weighted ratios of the metrics for each image in the set of homologous or heterologous images according to the weights of the registration error value, the information entropy value, and the proportion of valid data, and use the image with the largest sum of the weighted ratios of the metrics in the set of homologous or heterologous images as the optimal matching image for the single image.

[0032] As a preferred embodiment of the three-dimensional visualization method for multi-source heterogeneous data of power equipment according to the present invention, wherein: the three-dimensional visualization of the transformed heterologous and homologous data by using the texture mapping method and the device three-dimensional grid model includes:

[0033] Utilize the texture mapping relationship between the device three-dimensional grid model and the visible light image, the spatial association relationship between heterologous images, and combine the UV coordinate texture mapping method to automatically map the transformed heterologous and homologous data into the device three-dimensional grid model, perform multi-source texture mapping on the device three-dimensional grid model, and endow the device three-dimensional grid model with visual texture information to obtain a three-dimensional visualized multi-source texture model.

[0034] The beneficial effects of the present invention: Use the monocular visible light three-dimensional reconstruction method to restore the spatial position where the device is located and the device three-dimensional grid model from the device visible light image, and then use the homologous and heterologous image registration methods to establish the spatial association relationship between the homologous and heterologous state data respectively, and select the optimal matching data to solve the data decision problem in the one-to-many matching relationship existing between multi-source heterogeneous data, and based on the visible light three-dimensional grid model, endow the grid model with the visualized texture of the registered multi-source heterogeneous data to complete the three-dimensional visualization of the device state data. This method improves the efficiency of device state evaluation, the processing efficiency of monitoring data, the intuitiveness and integrity of monitoring data, and the reliability of the device state simulation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. Among them:

[0036] Figure 1 It is a schematic diagram of the method framework for the three-dimensional visualization method of multi-source heterogeneous data of power equipment;

[0037] Figure 2 It is a schematic diagram of simulated ultraviolet detection;

[0038] Figure 3 It is a schematic diagram of the experimental results of GIS equipment;

[0039] Figure 4 Schematic diagram of the test results of the casing equipment;

[0040] Figure 5 Schematic diagram of the test results of the suspension insulator equipment. Specific implementation manners

[0041] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will describe the specific implementation manners of the present invention in detail with reference to the accompanying drawings of the specification. Obviously, the described embodiments are a part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0043] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation manner of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0044] The present invention is described in detail with reference to the schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of explanation, the cross-sectional views showing the device structure will be enlarged locally out of the general proportion, and the schematic diagrams are only examples and should not limit the protection scope of the present invention herein. In addition, in actual production, three-dimensional spatial dimensions including length, width, and depth should be included.

[0045] At the same time, in the description of the present invention, it should be noted that the orientation or positional relationships indicated by terms such as "upper, lower, inner, and outer" are based on the orientation or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention. In addition, the terms "first, second, or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0046] Unless otherwise clearly specified and defined in the present invention, the terms "installation, connection, and coupling" shall be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may also be a mechanical connection, an electrical connection, or a direct connection, or it may be indirectly connected through an intermediate medium, or it may be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] Embodiment 1

[0048] Referring to Figure 1 , this embodiment provides a three-dimensional visualization method for multi-source heterogeneous data of power equipment,

[0049] including,

[0050] 1) Reconstructing the three-dimensional mesh model of the equipment from the multi-view pictures of the power equipment taken by the visible light camera:

[0051] Automatically matching the images from the visible light image sequence by using the monocular visible light three-dimensional reconstruction technology, calculating the pose of the camera shooting and the spatial position of the equipment, and reconstructing the three-dimensional mesh model of the equipment. The three-dimensional mesh model of the equipment is the model basis for the three-dimensional visualization of the equipment status data.

[0052] The specific steps are as follows:

[0053] Performing feature matching on the input visible light image sequence;

[0054] Solving and adjusting and optimizing the camera pose to restore the shooting position of the camera and the sparse point cloud model of the equipment;

[0055] Using the RANSAC method to traverse and match each group of feature points in the input image sequence, obtaining the two-dimensional matching point relationship between the input images, obtaining the three-dimensional depth and spatial coordinates of the two-dimensional feature points through the rotation and translation matrix between the cameras and the triangulation result to determine the three-dimensional space points, traversing and matching the area of each feature point domain in the input image with the area on the epipolar line of the matching image to obtain the dense point cloud model of the equipment, classifying all the point clouds in the dense point cloud according to their spatial positions by using the weak texture support area classification method, and optimizing according to the classification result and the point cloud meshing weight in the weak texture support area to construct a complete surface mesh and texture model to obtain the three-dimensional mesh model of the equipment and its texture mapping relationship with the visible light image;

[0056] 2) Registering and transforming the data by using the spatial correlation relationship of the multi-band images:

[0057] Based on visible light image data, traverse and register other heterogeneous data with it in two-dimensional space using heterologous image registration. The other heterogeneous data includes infrared temperature measurement data and ultraviolet detection data, obtain the spatial transformation relationship between the visible light image and images of other bands. The images of other bands include infrared images and ultraviolet images, and combine data decision methods to obtain one-to-one matching relationships between heterogeneous data.

[0058] The steps of heterologous image registration are as follows:

[0059] Extract feature points on the device contour in the heterologous images, calculate the main directions of the feature points, then extract improved SIFT feature descriptors at multiple heterologous image scales, obtain initial matching points and perform iterative screening on them to get a matching result without incorrect matching points, solve the perspective transformation relationship matrix between the images of other bands and the visible light image from the matching point pairs, and interpolate and transform the original detection data corresponding to the images of other bands to the same scale and perspective as the visible light image to establish a spatial association relationship between heterogeneous data.

[0060] The data decision method is based on the sum of index ratios and is used to select the optimal single image to establish a one-to-one matching relationship when the registration of a single image with multiple homologous images is successful.

[0061] The decision-making process of the data decision method is as follows:

[0062] Calculate the registration error value between a single image and the set of homologous or heterologous images corresponding to it. This value is used to reflect the alignment accuracy after heterologous image registration;

[0063] Calculate the information entropy value of the corresponding grayscale image after the set of homologous or heterologous images undergoes perspective transformation. This value is used to reflect the clarity of the single image and the richness of texture information;

[0064] Calculate the ratio of the number of pixel points whose coordinates in the set of homologous or heterologous images after perspective transformation are within the pixel coordinate index range of the single image to the total number. This value reflects the proportion of valid data after the registration of any image in the set of homologous or heterologous images with the single image;

[0065] Calculate the weighted ratio sum of indicators for each image in the set of homologous or heterologous images according to the weights of the registration error value, information entropy value, and proportion of valid data, and take the image with the largest weighted ratio sum of indicators in the set of homologous or heterologous images as the optimal matching image of the single image.

[0066] 3) Use texture mapping methods and the device three-dimensional grid model to perform three-dimensional visualization on the transformed heterologous and homologous data:

[0067] Using the texture mapping relationship between the three-dimensional mesh model of the device and the visible light image, the spatial correlation relationship between heterogeneous images, and combining the UV coordinate texture mapping method, the transformed heterogeneous and homologous data are automatically mapped into the three-dimensional mesh model of the device, and multi-source texture mapping is performed on the three-dimensional mesh model of the device to endow the three-dimensional mesh model of the device with visual texture information, so as to obtain a multi-source texture model with three-dimensional visualization.

[0068] This method uses a monocular visible light three-dimensional reconstruction method to recover the spatial position of the device and the three-dimensional mesh model of the device from the visible light image of the device, and then uses the homologous and heterogeneous image registration methods to establish the spatial correlation relationship between the homologous and heterogeneous state data respectively, and selects the optimal matching data to solve the data decision problem in the one-to-many matching relationship existing between multi-source heterogeneous data, and based on the three-dimensional mesh model of the device, endows the registered multi-source heterogeneous data visualization texture for the mesh model, completes the three-dimensional visualization of the device state data, and this method improves the efficiency of device state evaluation, the processing efficiency of monitoring data, the intuitiveness and integrity of monitoring data, and the reliability of device state simulation results.

[0069] Example 2

[0070] Refer to Figures 2 - 5 , this embodiment provides a three-dimensional visualization method for heterogeneous data of power equipment. To verify the applicability and effectiveness of this method, GIS, bushing and suspension insulator are selected as experimental objects, and visible light images and infrared temperature measurement data of 3 types of power transmission and transformation equipment are collected for three-dimensional visualization display experiments.

[0071] The main hardware and software parameters of the experimental platform are as follows:

[0072] Power equipment: GIS, suspension insulator equipment not in operation in the laboratory, local thermoelectric simulation equipment for bushing;

[0073] Visible light monocular camera: Model is SAMSUNG SM-G9980, resolution is 4000×3000;

[0074] Infrared thermal imager: Model is FLIR T1040, resolution is 1024×768;

[0075] Computing host: CPU is Intel(R) Core(TM) i9-10900X@3.70GHz, GPU is NVIDIA GeForce RTX 208-Ti O11G;

[0076] Application: It is written based on the open-source C++ language 3D reconstruction libraries openMVG and openMVS, the open-source Matlab language heterologous image registration library CAO-C2F, and the open-source software MashLab. This application only requires the user to input multi-source heterogeneous data, and the program will automatically run according to default parameters or custom parameters. Finally, it outputs a 3D mesh model file of a single device and multiple source texture map images and conducts visual display.

[0077] I. GIS Experimental Results

[0078] Experimental tests were carried out on 11 GIS visible light images, 35 infrared temperature measurement images at two rounds with a temperature measurement interval of 24 hours, and 11 ultraviolet detection simulation images. The simulation method of the ultraviolet detection images is to overlay a blue device simulated discharge distribution map on the visible light images. The simulated ultraviolet image detection effect is as Figure 2 shown.

[0079] First, use the visible light images to reconstruct the 3D mesh model of the device and assign the visible light image texture to it to obtain the GIS visible light 3D texture model; then register the first-round infrared temperature measurement and visible light images to obtain the correlation between the infrared image and the visible light image, and assign the infrared temperature data to the 3D mesh model of the device to obtain the temperature measurement texture model; then perform homologous image registration on the second-round and first-round infrared temperature measurement images to indirectly obtain the texture mapping relationship between the second-round infrared image and the 3D mesh model of the device, and finally obtain the second-round infrared temperature measurement 3D model; finally, register the visible light and ultraviolet images to indirectly obtain the texture mapping relationship between the ultraviolet image and the 3D mesh model of the device to obtain the simulated ultraviolet detection texture model. The total running time of the program is 26 min 28 s, and the average running time for processing a single image is 27.86 s. The test results are as Figure 3 shown. The irrelevant background has been removed from the model diagram for easy observation and analysis.

[0080] It can be seen from the results that the fully automated processing method proposed in this paper uses heterologous image registration technology to establish the spatial correlation between heterogeneous data; at the same time, it uses homologous image registration technology to establish the matching relationship of device state data at different times. After reconstructing the visible light 3D model by the method in this paper, the infrared temperature measurement and ultraviolet simulation data can be three-dimensionally visualized with the same fineness as the visible light model, and it also provides an intuitive comparison of the device state at different times, greatly improving the intuitiveness of the data and reducing the workload of staff evaluating the device state from a large amount of two-dimensional data, verifying the applicability and effectiveness of the 3D visualization method in this paper for multi-source heterogeneous data of large power equipment.

[0081] II. Bushing Experimental Results

[0082] Twenty visible light images of the bushing, forty infrared temperature measurement images at two different times with an 8-hour temperature measurement interval, and twenty ultraviolet detection simulation images were collected for experimental testing. The ultraviolet detection image simulation method is the same as that of GIS. The first round of infrared temperature measurement has two local simulated hot spots, and the second round has four local simulated hot spots. The temperature difference between the hot spots and the environment is about 1.5°C. The test process is the same as that of the GIS device. The total running time of the program is 34 minutes and 43 seconds, and the average processing time for a single image is 26.04 seconds. The test results are as Figure 4 shown.

[0083] Similar to the GIS experimental results, the method in this paper successfully synthesized the visible light images of the bushing, infrared temperature measurement, and ultraviolet detection information into the same 3D model for 3D visualization. Different simulated hot spots in the two rounds of temperature measurement were accurately and clearly restored to the device model, verifying that the method in this paper can effectively restore the abnormal state information of the device. This information can not only be provided to the staff for condition assessment but also be used for later device state simulation to predict the future state of the device, upgrading the traditional point or surface measurement method to a future volume measurement diagnosis method, further verifying the applicability and effectiveness of the 3D visualization method in this paper for multi-source heterogeneous data of small power equipment.

[0084] III. Experimental Results of Insulators

[0085] Fourteen visible light images of suspension insulators, thirty infrared temperature measurement images at two different times with a 3-hour temperature measurement interval, and fourteen ultraviolet detection simulation images were collected for experimental testing. The test process is the same as that of GIS and bushing devices. The total running time of the program is 23 minutes and 50 seconds, and the average processing time for a single image is 22.93 seconds. The test results are as Figure 5 shown.

[0086] As can be seen from the results, the method in this paper can recover the fine grid model and multi-source heterogeneous texture model of suspension insulators from a large amount of visible light, infrared temperature measurement, and ultraviolet image information. Considering the time-consuming of single-image processing of GIS, bushings, and insulators, the data synthesis time of the method in this paper is stable and efficient, and it can quickly perform spatial synthesis on a large amount of multi-source heterogeneous data, solving the problem of easy fatigue in manual labor such as manual texturing. Considering that the current condition monitoring of power equipment on transmission lines mainly detects equipment images or videos and multi-band images through digital cameras and drones, the method in this paper can reconstruct the device model from its detected images or videos and synthesize multi-band information into the device model, verifying the applicability and effectiveness of the 3D visualization method in this paper for multi-source heterogeneous data of transmission equipment.

[0087] The fully automated 3D visualization method framework proposed in this paper is effective and general for the 3D visualization of multi-source heterogeneous data of power equipment. This method can stably and quickly recover the fine 3D mesh model information of the equipment from a large amount of multi-source heterogeneous data of power equipment, and establish the spatial correlation relationship between multi-source heterogeneous data, so that heterogeneous data with different resolutions can be mapped into a consistent model for three-dimensional display. The method in this paper synthesizes the 2D state data of power equipment in space and displays it in 3D form, which not only improves the integrity and three-dimensionality of viewing the multi-source heterogeneous state data of the equipment, but also provides a method reference for equipment state simulation analysis and prediction based on multi-source heterogeneous data in digital twin technology.

[0088] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The methods can be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with the computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, for this purpose the program can run on a dedicated integrated circuit programmed for this purpose.

[0089] In addition, the operations of the processes described herein can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by the context. The processes described herein (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed commonly on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions executable by one or more processors.

[0090] Further, the method can be implemented in any type of computing platform operatively connected to a suitable one, including but not limited to personal computers, minicomputers, mainframes, workstations, network or distributed computing environments, separate or integrated computer platforms, or communicating with charged particle tools or other imaging devices, etc. Aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into the computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it can be read by a programmable computer and can be used to configure and operate the computer to perform the processes described herein when the storage medium or device is read by the computer. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above-described steps in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself. The computer program is capable of applying to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the transformed data represents physical and tangible objects, including a specific visual depiction of the physical and tangible objects generated on a display.

[0091] As used in this application, the terms "component", "module", "system", etc. are intended to refer to a computer-related entity, which can be hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to: a process running on a processor, a processor, an object, an executable file, a thread in execution, a program, and / or a computer. As an example, an application running on a computing device and the computing device can both be components. One or more components can exist in a process and / or thread in execution, and the components can be located in one computer and / or distributed between two or more computers. In addition, these components can execute from various computer-readable media having various data structures thereon. These components can communicate in a local and / or remote process manner through signals such as in accordance with one or more data packets (e.g., data from one component that interacts with another component in a local system, a distributed system, and / or communicates with other systems via a network such as the Internet in a signal manner).

[0092] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A three-dimensional visualization method for multi-source heterogeneous data of power equipment, characterized in that: including capturing multi - perspective pictures of power equipment by a visible - light camera and reconstructing a 3D mesh model of the equipment registering and transforming data using the spatial correlation relationship of multi - band images performing 3D visualization of the transformed heterogeneous and homologous data using the texture mapping method and the 3D mesh model of the equipment The capturing multi - perspective pictures of power equipment by a visible - light camera and reconstructing a 3D mesh model of the equipment includes: using monocular visible - light 3D reconstruction technology to automatically match images from a visible - light image sequence, calculating the camera pose and the equipment spatial position, and reconstructing a 3D mesh model of the equipment. The 3D mesh model of the equipment is the model basis for 3D visualization of equipment status data The calculating the camera pose and the equipment spatial position includes: performing feature matching on the input visible - light image sequence solving and optimizing the camera pose to restore the camera shooting position and the equipment sparse point - cloud model using the solved camera pose as the input image and further traversing and matching it to obtain image matching points, and then restoring the dense point - cloud, surface mesh, and texture model of the scene The using the solved camera pose as the input and further traversing and matching it to obtain image matching points includes: using the RANSAC method to traverse and match each group of feature points in the input image sequence, obtaining the two - dimensional matching point relationship between the input images, and obtaining the three - dimensional depth and spatial coordinates of the two - dimensional feature points through the rotation - translation matrix between cameras and the triangulation result to determine the three - dimensional space points The restoring the dense point - cloud, surface mesh, and texture model of the scene includes: traversing and matching the region of each feature point neighborhood in the input image with the region on the epipolar line of the matching image to obtain the dense point - cloud model of the equipment, classifying all the point - clouds in the dense point - cloud according to their spatial positions using the weak - texture - supported region classification method, and optimizing the meshing weights of the point - clouds in the weak - texture - supported region according to the classification result and constructing a complete surface mesh and texture model to obtain the 3D mesh model of the equipment and the texture mapping relationship between it and the visible - light image The registering and transforming data using the spatial correlation relationship of multi - band images includes: based on visible - light image data, using heterogeneous image registration to traverse and register other heterogeneous data with it in two - dimensional space. The other heterogeneous data includes infrared temperature - measurement data and ultraviolet detection data, obtaining the spatial transformation relationship between the visible - light image and other band images. The other band images include infrared images and ultraviolet images, and combining with a data - decision method to obtain the one - to - one matching relationship between the heterogeneous data The heterogeneous image registration includes: Extract the feature points on the device contour in the heterologous images, calculate the main direction of the feature points, then extract the improved SIFT feature descriptors at multiple scales of the heterologous multi-images, obtain the initial matching points and perform iterative screening on them to get the matching result without incorrect matching points, solve the perspective transformation relationship matrix between the other band images and the visible light image from the matching point pairs, and interpolate and transform the original detection data corresponding to the other band images to the same scale and perspective as the visible light image to establish the spatial association relationship between the heterogeneous data; The data decision-making method includes: The data decision-making method is based on the index proportion sum and is used to select the optimal single image to establish a one-to-one matching relationship when the registration of a single image and multiple homologous images is successful; The decision-making process of the data decision-making method is as follows: Calculate the registration error value between the single image and the set of homologous or heterologous images corresponding to it, and this value is used to reflect the alignment accuracy after the registration of the heterologous images; Calculate the information entropy value of the corresponding grayscale image after the perspective transformation of the set of homologous or heterologous images, and this value is used to reflect the clarity of the single image and the richness of the texture information; Calculate the ratio of the number of pixel points whose coordinates of all pixel points in the set of homologous or heterologous images are within the pixel coordinate index range of the single image after the perspective transformation to the total number, and this value reflects the proportion of the effective data after the registration of any image in the set of homologous or heterologous images and the single image; Calculate the index weighted proportion sum of each image in the set of homologous or heterologous images according to the weights of the registration error value, the information entropy value and the proportion of effective data, and take the image with the largest index weighted proportion sum in the set of homologous or heterologous images as the optimal matching image of the single image.

2. The three-dimensional visualization method for multi-source heterogeneous data of power equipment according to claim 1, characterized in that: The three-dimensional visualization of the transformed heterologous and homologous data by using the texture mapping method and the device three-dimensional grid model includes: Using the device three-dimensional grid model and its texture mapping relationship with the visible light image, and the spatial association relationship between the heterologous images, combined with the UV coordinate texture mapping method, automatically map the transformed heterologous and homologous data into the device three-dimensional grid model, perform multi-source texture mapping on the device three-dimensional grid model, and endow the device three-dimensional grid model with visual texture information to obtain a multi-source texture model with three-dimensional visualization.

Images