A method for automatically updating three-dimensional space of heterogeneous data of power equipment

Abstract

The present invention discloses a method for automatic updating of three-dimensional space of heterogeneous data of electric equipment, including: using a monocular visible light three-dimensional reconstruction strategy to recover the spatial position of the electric equipment and a fine three-dimensional grid model of the equipment from the visible light image of the electric equipment; establishing spatial correlation relationships between homologous and heterogeneous state data based on homologous and heterogeneous image registration strategies; using a data decision strategy to select the best matching data according to the spatial correlation relationship, optimizing the data decision problem when there is a one-to-many matching relationship between heterogeneous data, and realizing the update of the image detection data of the electric equipment. The present invention can ensure the consistency of multi-source heterogeneous data in spatial data mapping by establishing the correlation relationship between heterogeneous data and obtaining the best matching image by using the best data decision, so that low-resolution data has a high-resolution three-dimensional display effect.

Description

Technical Field

[0001] The present invention relates to the technical field of power equipment image data detection and updating, and in particular to a method for automatic updating of three-dimensional space of heterogeneous data of power equipment. Background Art

[0002] At the equipment level, the application of digital twin technology can be used to intelligently monitor the status of power equipment, warn of faults and diagnose them. For example, after mapping the real status of the equipment to the digital twin model, the digital model can be used for simulation calculation and data analysis to give predictions or evaluation results of the equipment status and perform three-dimensional display. However, the amount of multi-source heterogeneous data such as visible light, infrared, ultraviolet and audio generated by equipment status monitoring is large, and manual data mapping between physical and virtual models is time-consuming, labor-intensive and inefficient. Therefore, the study of fully automatic spatial synthesis and three-dimensional display methods of multi-source heterogeneous data of power equipment is of great significance for the following tasks: improving the efficiency of equipment status assessment; improving the efficiency of monitoring data processing; improving the intuitiveness and integrity of monitoring data; and improving the reliability of equipment status simulation analysis results.

[0003] However, visible light detection images can be directly used in the equipment mesh model reconstruction process because they have high-resolution equipment texture information. However, the resolution of infrared or ultraviolet detection data is low and a large amount of equipment texture information is lost. Therefore, it is impossible to directly use three-dimensional reconstruction technology to restore the multi-band three-dimensional model of the equipment for stereoscopic display. Therefore, how to obtain the spatial transformation relationship between visible light images and other band images and the one-to-one matching relationship between heterogeneous data is a problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is that the resolution of infrared or ultraviolet detection data is low and a large amount of texture information of the device is lost, so it is impossible to directly use three-dimensional reconstruction technology to restore the multi-band three-dimensional model of the device for stereoscopic display.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: using a monocular visible light three-dimensional reconstruction strategy to recover the spatial position of the power equipment and a fine three-dimensional grid model of the equipment from the visible light image of the power equipment; based on homologous and heterologous image registration strategies, establishing spatial correlation relationships between homologous and heterologous state data respectively; using a data decision strategy to select the optimal matching data based on the spatial correlation relationship, optimizing the data decision problem when there is a one-to-many matching relationship between heterogeneous data, and realizing the update of the image detection data of the power equipment.

[0008] As a preferred solution of the method for automatically updating three-dimensional space of heterogeneous data of power equipment described in the present invention, the optimal data decision strategy includes defining a set of homologous or heterogeneous images corresponding to a single image V1 as {I1, I2, ..., I n}; Calculate the {I1,I2,…,I n} and the registration error ε of the single image V1, {I1,I2,…,I n}The information entropy value E of the corresponding grayscale image after perspective transformation, {I1,I2,…,I n}The ratio S of the number of pixels whose coordinates are located in the V1 pixel coordinate index range after perspective transformation to the total number of pixels, {I1,I2,…,I n The weighted ratio and R of the indicators for each image in}.

[0009] As a preferred solution of the method for automatically updating three-dimensional space of heterogeneous data of electric power equipment described in the present invention, the calculation formula of the registration error ε includes:

[0010]

[0011] in, represents the original matching point pixel coordinates in V1, Indicates I i The coordinates of the matching points in the image are transformed by perspective, N c Indicates the number of matching point pairs.

[0012] As a preferred solution of the method for automatically updating three-dimensional space of heterogeneous data of power equipment described in the present invention, the calculation formula of the information entropy value E includes:

[0013] E = -∑p(i)log(p(i))

[0014] Where p(i) represents the frequency of pixels with grayscale value i in the grayscale image, and the range of i is [0,255].

[0015] As a preferred solution of the method for automatically updating three-dimensional space of heterogeneous data of electric power equipment described in the present invention, wherein: the {I1, I2, ..., I n The calculation formula for the ratio S of the number of pixels whose coordinates are located in the V1 pixel coordinate index range to the total number after perspective transformation includes:

[0016]

[0017] Among them, Counter() represents the counting function, and M and N represent the horizontal and vertical resolutions of V1 respectively.

[0018] As a preferred solution of the method for automatically updating three-dimensional space of heterogeneous data of power equipment described in the present invention, the calculation formula of the weighted ratio of the index of each image and R includes:

[0019]

[0020] Among them, w ε 、w e 、w s are the weights of registration error, information entropy value, and effective data ratio, respectively.

[0021] As a preferred solution of the method for automatic three-dimensional spatial updating of heterogeneous data of power equipment described in the present invention, the homologous and heterologous image registration strategies include: using heterologous image registration strategies to automatically establish spatial correlation relationships among visible light, ultraviolet, and infrared multi-band monitoring images of the same equipment, and transforming the multi-band images to the same scale and viewing angle as the visible light images according to the spatial correlation relationships, and using the transformed data for model texture mapping in the three-dimensional display link; using homologous registration strategies to match equipment status monitoring data captured at different time periods or different inspection rounds within a month or day, and obtaining matching relationships between status data at different times.

[0022] As a preferred solution of the method for automatic updating of three-dimensional space of heterogeneous data of power equipment described in the present invention, the heterogeneous image registration strategy includes extracting feature points on the equipment contour in the heterogeneous image in the curvature scale space, constructing auxiliary feature points according to the positions of the curvature minimum points on the left and right sides of each feature point, and constructing the feature direction vector of the feature point; extracting improved SIFT feature descriptors under the scale of heterogeneous multi-images respectively, using the bilateral matching strategy to obtain initial matching points, and using the RANSAC strategy to iteratively screen the matching points to obtain matching results without erroneous matching points; using the perspective transformation model, solving the perspective transformation relationship matrix between the infrared or ultraviolet image and the visible light image from the matching point pairs, and interpolating the original detection data corresponding to the infrared or ultraviolet image to the same scale and viewing angle as the visible light image to establish the spatial correlation relationship between the heterogeneous data.

[0023] As a preferred solution of the method for automatically updating three-dimensional space of heterogeneous data of power equipment described in the present invention, the construction of the characteristic direction vector of the characteristic point includes:

[0024] v fL =(x fL -x f ,y fL -y f )

[0025] v fR =(x fR -x f ,y fR -y f )

[0026]

[0027] Among them, (x f ,y f ) is the coordinate of the feature point, (x fL ,y fL ) is the coordinate of the auxiliary feature point on the left, (x fR ,y fR ) is the coordinate of the auxiliary feature point on the right side, (x fm ,y fm ) are the eigenvector coordinates.

[0028] As a preferred solution of the method for automatically updating three-dimensional space of heterogeneous data of power equipment described in the present invention, the calculation formula of the main direction of the feature point includes:

[0029]

[0030] Among them, φ(P f ) is the main direction of the feature point contour angle feature.

[0031] The beneficial effects of the present invention are as follows: by establishing an association relationship between heterogeneous data and using optimal data decisions to obtain the optimal matching image, the consistency of multi-source heterogeneous data during spatial data mapping can be guaranteed, so that low-resolution data can have a high-resolution three-dimensional stereoscopic display effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0033] Figure 1A basic flow chart of a method for automatically updating three-dimensional space of heterogeneous data of electric power equipment provided by an embodiment of the present invention;

[0034] Figure 2 A result diagram of a heterogeneous image data decision experiment with different weights for a method for automatically updating three-dimensional space of heterogeneous data of electric power equipment provided by an embodiment of the present invention;

[0035] Figure 3 A diagram of the result of a decision experiment on homologous image data with different weights for a method for automatically updating three-dimensional space of heterogeneous data of electric power equipment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0038] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0039] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0040] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0041] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0042] Example 1

[0043] Reference Figure 1 , as an embodiment of the present invention, provides a method for automatically updating three-dimensional space of heterogeneous data of power equipment, comprising:

[0044] S1: Use the monocular visible light 3D reconstruction strategy to restore the spatial position of the power equipment and the detailed 3D mesh model of the equipment from the visible light image of the power equipment;

[0045] It should be noted that monocular visible light 3D reconstruction technology can recover the fine 3D mesh model of power equipment from visible light detection images with high resolution and rich texture details. Among them, the Structure from Motion (SfM) technology part is based on the theoretical basis of epipolar geometry and triangulation principles. It performs local (or global) feature matching on the input ordered (or unordered) image sequence, and then solves the camera pose and performs incremental or global bundle adjustment optimization, finally recovering the precise shooting pose of the detection camera and the sparse point cloud model of the equipment.

[0046] S2: Based on the homologous and heterologous image registration strategies, the spatial correlation between homologous and heterologous state data is established respectively;

[0047] It should be noted that visible light detection images can be directly used in the equipment mesh model reconstruction process because of their high-resolution equipment texture information. However, the resolution of infrared or ultraviolet detection data is low and a large amount of equipment texture information is lost. Therefore, it is impossible to directly use three-dimensional reconstruction technology to restore the multi-band three-dimensional model of the equipment for stereoscopic display. To address this problem, the present invention is based on visible light images and uses heterogeneous image registration technology to traverse and register other heterogeneous data with them in two-dimensional space to obtain the spatial transformation relationship between visible light images and other band images, and then uses the proposed data decision method to obtain a one-to-one matching relationship between heterogeneous data.

[0048] Among them, the process of heterogeneous image registration is:

[0049] (1) Feature points on the device contour in the heterogeneous image are extracted in the curvature scale space, and auxiliary feature points are constructed based on the positions of the curvature minimum points on the left and right sides of each feature point. Then, the feature direction vector of the feature point is constructed according to the following formula:

[0050] v fL =(x fL -x f ,y fL -y f )

[0051] v fR =(x fR -x f ,y fR -y f )

[0052]

[0053] Among them, (x f ,y f ) is the coordinate of the feature point, (x fL ,y fL ) is the coordinate of the auxiliary feature point on the left, (x fR ,y fR ) is the coordinate of the auxiliary feature point on the right side, (x fm ,y fm ) are the eigenvector coordinates.

[0054] The main direction of the feature point is calculated by the following formula:

[0055]

[0056] Among them, φ(P f ) is the main direction of the feature point contour angle feature.

[0057] (2) The improved SIFT feature descriptors of heterogeneous multi-image scales are extracted respectively, and the initial matching points are obtained using the bilateral matching method. Since there are a large number of erroneous matching points in the initial matching points, the RANSAC method is further used to iteratively screen the matching points to obtain matching results without erroneous matching points.

[0058] (3) Using the perspective transformation model, the perspective transformation relationship matrix between the infrared or ultraviolet image and the visible light image is solved from the matching point pairs, and the original detection data (temperature dot matrix, ultraviolet spectrum, etc.) corresponding to the infrared or ultraviolet image is interpolated and transformed to the same scale and perspective as the visible light image to establish the spatial correlation relationship between heterogeneous data.

[0059] S3: According to the spatial correlation relationship, the data decision strategy is used to select the best matching data, optimize the data decision problem when there is a one-to-many matching relationship between heterogeneous data, and realize the update of power equipment image detection data;

[0060] It should be noted that after the registration of heterogeneous image data is completed, there is a situation where a visible light image is successfully registered with multiple infrared or ultraviolet images; similarly, after the homologous image data registration is completed using the SIFT method, there is also a situation where a single image is successfully registered with multiple homologous images. For this reason, the present invention proposes a data decision method based on the ratio and of indicators to select the optimal single image for establishing a one-to-one image matching relationship.

[0061] Specifically, suppose that a set of homologous or heterologous images corresponding to a single image V1 is {I1,I2,…,I n}, the decision-making process is as follows:

[0062] (1) Calculate {I1,I2,…,I n The registration error ε between} and the single image V1 is used to reflect the alignment accuracy after heterogeneous image registration:

[0063]

[0064] in, represents the original matching point pixel coordinates in V1, Indicates I i The coordinates of the matching points in the image are transformed by perspective, N c Indicates the number of matching point pairs.

[0065] (2) Calculate {I1,I2,…,I n The information entropy value E of the grayscale image after perspective transformation is used to reflect the clarity of a single image and the richness of texture information:

[0066] E = -∑p(i)log(p(i))

[0067] Where p(i) represents the frequency of pixels with grayscale value i in the grayscale image, and the range of i is [0,255].

[0068] (3) Calculate {I1,I2,…,I n The ratio S of the number of pixels whose coordinates are located in the V1 pixel coordinate index range after perspective transformation to the total number of pixels, which reflects the I i The percentage of valid data after alignment with V1:

[0069]

[0070] Among them, Counter() represents the counting function, and M and N represent the horizontal and vertical resolutions of V1 respectively.

[0071] (4) Calculate {I1,I2,…,I n The weighted ratio and R of the indicators for each image in}:

[0072]

[0073] Among them, w ε 、w e 、w s are the weights of registration error, information entropy value, and effective data ratio, respectively.

[0074] This value is used to reflect {I1,I2,…,I n The comprehensive matching quality of each image in the image set {I1,I2,…,I n} is used as the optimal matching image of the single input image V1, and the spatial transformation relationship between the matching images is obtained.

[0075] Example 2

[0076] Reference Figures 2-3 This is another embodiment of the present invention. Different from the first embodiment, this embodiment provides a verification test of a method for automatic updating of three-dimensional space of heterogeneous data of power equipment. In order to verify and illustrate the technical effect adopted in this method, this embodiment selects GIS, bushings and suspension insulators as experimental objects, collects visible light images and infrared temperature measurement data of three types of power transmission and transformation equipment, and conducts data space synthesis and three-dimensional display experiments. The method of the present invention is used for testing, and the real effect of the method is verified by scientific argumentation.

[0077] The main hardware and software parameters of the experimental platform are as follows:

[0078] (1) Power equipment: GIS, suspension insulator equipment, and bushing local hot spot simulation equipment that are not in operation in the laboratory;

[0079] (2) Visible light monocular camera: model: SAMSUNG SM-G9980, resolution: 4000-3000;

[0080] (3) Infrared thermal imager: model FLIR T1040, resolution 1024768;

[0081] (4) Computing host: CPU is Intel(R) Core(TM) i9-10900X@3.70GHz, GPU is NVIDIAGeForce RTX 2080Ti O11G;

[0082] (5) Application: This application is written based on the open source C++ 3D reconstruction libraries openMVG and openMVS, the open source Matlab language heterogeneous image registration library CAO-C2F, and the open source software MashLab. The application only requires the user to input multi-source heterogeneous data, and the program will automatically run according to default parameters or customized parameters, and finally output a single 3D mesh model file and multiple multi-source texture map images for visual display.

[0083] In order to test the influence of the indicator weights on the decision results in the optimal data decision process, this embodiment performs matching experiments with different weights on 20 casing visible light images and 40 infrared temperature measurement images collected at different times. The experimental results of heterogeneous and homogeneous image data decision are shown in Figure 2. Figure 2 and Figure 3 shown.

[0084] Depend on Figure 2 As can be seen from the results, the first column is the reference image, and the second to fourth columns are the decision matching images corresponding to different weights of the reference image. The matching results corresponding to different registration error weights are quite different. For the decision matching of heterogeneous image data, when a larger registration error weight is set, the final matching image has the highest degree of alignment. Considering that the accuracy of multi-band image detection data, especially the accuracy of infrared temperature measurement data, is crucial for subsequent equipment status assessment and fault diagnosis, even if its pixel area is slightly smaller than that of matching images with other weights, its matching result is also desirable.

[0085] Different from heterogeneous image registration, homologous image registration methods are relatively mature. Figure 3The results show that the first column is the reference image, and the second to fourth columns are the decision matching images corresponding to different weights of the reference image. The matching images corresponding to different weights all show a high degree of alignment. When a larger information entropy weight is set, the final matching image has the best visual perception and the largest effective area. When the image is used for texture mapping, a more complete texture mapping model can be obtained.

[0086] According to the above analysis, for heterogeneous image data decision making, this embodiment uses the weight w ε 、w e 、w s are set to 0.6, 0.1, and 0.3 respectively; for the decision of homologous image data, this embodiment sets the weight w ε 、w e 、w s Set to 0.1, 0.6, and 0.3 respectively.

[0087] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for automatically updating three-dimensional space of heterogeneous data of power equipment, characterized in that: include: Using a monocular visible light 3D reconstruction strategy to recover the spatial position of the power equipment and a detailed 3D mesh model of the equipment from the visible light image of the power equipment; Based on the homologous and heterologous image registration strategies, the spatial correlation between homologous and heterologous state data is established respectively; According to the spatial correlation relationship, a data decision strategy is adopted to select the best matching data, optimize the data decision problem when there is a one-to-many matching relationship between heterogeneous data, and realize the update of the power equipment image detection data; the optimal data decision strategy includes: Define a set of homologous or heterologous images corresponding to a single image V1 as {I1,I2,…,I n }; Calculate the {I1,I2,…,I n } and the registration error ε of the single image V1, {I1,I2,…,I n }The information entropy value E of the corresponding grayscale image after perspective transformation, {I1,I2,…,I n }The ratio S of the number of pixels whose coordinates are located in the V1 pixel coordinate index range after perspective transformation to the total number of pixels, {I1,I2,…,I n The weighted ratio of the indicators for each image in} and R; The calculation formula of the registration error ε includes: in,( , ) represents the original matching point pixel coordinates in V1, ( , ) indicates I i The coordinates of the matching points in the image are transformed by perspective, N c Indicates the number of matching point pairs; The calculation formula of the information entropy value E includes: E = -∑p(i)log(p(i)) Where p(i) represents the frequency of the pixel with gray value i in the gray image, and the range of i is [0,255]; The {I1,I2,…,I n The calculation formula for the ratio S of the number of pixels whose coordinates are located in the V1 pixel coordinate index range to the total number after perspective transformation includes: Among them, Counter() represents the counting function, M and N represent the horizontal and vertical resolutions of V1 respectively; The calculation formula of the weighted ratio of the index of each image and R includes: Among them, w ε 、w e 、w s are the weights of registration error, information entropy value, and effective data ratio, respectively.

2. The method for automatically updating three-dimensional space of heterogeneous data of electric power equipment according to claim 1, characterized in that: The homologous and heterologous image registration strategies include: A heterogeneous image registration strategy is used to automatically establish the spatial correlation relationship of visible light, ultraviolet light, and infrared multi-band monitoring images of the same device, and the multi-band images are transformed to the same scale and viewing angle as the visible light images according to the spatial correlation relationship, and the transformed data is used for model texture mapping in the three-dimensional display link; The homologous registration strategy is used to match the equipment status monitoring data captured at different time periods or different inspection rounds within a month or a day, and the matching relationship between the status data at different times is obtained.

3. The method for automatically updating three-dimensional space of heterogeneous data of electric power equipment according to claim 1 or 2, characterized in that: The heterogeneous image registration strategy includes: Extract feature points on the device contour in the heterogeneous image in the curvature scale space, construct auxiliary feature points according to the positions of the curvature minimum points on the left and right sides of each feature point, and construct the feature direction vector of the feature point; The improved SIFT feature descriptors of heterogeneous multi-image scales are extracted respectively, the initial matching points are obtained using the bilateral matching strategy, and the matching points are iteratively screened using the RANSAC strategy to obtain the matching results without erroneous matching points. The perspective transformation model is adopted to solve the perspective transformation relationship matrix between the infrared or ultraviolet image and the visible light image from the matching point pairs, and the original detection data corresponding to the infrared or ultraviolet image is interpolated and transformed to the same scale and viewing angle as the visible light image to establish the spatial correlation relationship between heterogeneous data.

4. The method for automatically updating three-dimensional space of heterogeneous data of electric power equipment according to claim 3, characterized in that: The construction of the feature direction vector of the feature point includes: v fL (x) fL -x f ,and fL -and f ) v fR (x) fR -x f ,and fR -and f ) Among them, (x f ,y f ) is the coordinate of the feature point, (x fL ,y fL ) is the coordinate of the auxiliary feature point on the left, (x fR ,y fR ) is the coordinate of the auxiliary feature point on the right side, (x fm ,y fm ) are the eigenvector coordinates.

5. The method for automatically updating three-dimensional space of heterogeneous data of electric power equipment according to claim 4, characterized in that: The calculation formula of the main direction of the feature point includes: Among them, φ(P f ) is the main direction of the feature point contour angle feature.

Images