A method and apparatus for 3D visualization of power equipment by fusing infrared temperature measurement images

Abstract

The present application discloses a method and device for three-dimensional visualization of electric power equipment fused with infrared temperature measurement images, the method comprising: segmenting the infrared electric power equipment image in the infrared temperature measurement image based on the section of the three-dimensional texture image of the electric power equipment; extracting key points in the three-dimensional texture image of the electric power equipment and the infrared electric power equipment image using the SIFT algorithm, performing local image matching based on the calculated Euclidean distance between the key points in the infrared key point image and the three-dimensional texture key point image, and obtaining the infrared local image of the electric power equipment; converting the infrared local image of the electric power equipment into a three-dimensional infrared image based on a preset rotation and translation matrix; performing image fusion of the three-dimensional infrared image with the three-dimensional texture image of the electric power equipment based on edge information and fusion weights, and obtaining a three-dimensional infrared texture image; and visualizing the three-dimensional infrared texture image. The present application can alleviate the technical problems of the prior art that the visualization is single, the intuitiveness is poor, and it is unable to provide more abundant information on electric power equipment.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional image display, and in particular to a three-dimensional visualization method and device for electric power equipment integrating infrared temperature measurement images. Background Art

[0002] With the continuous progress and development of science and technology, 3D modeling technology is widely used in substations. 3D models display the size, location, structure and other information of substation equipment in a visual way. With the application and upgrading of 3D technology, it is necessary to provide more abundant information display for substation staff on 3D models, and enhance the intuitiveness and visualization ability of abstract spatial information, so as to assist operation and maintenance personnel to make quick and accurate judgments in their work.

[0003] The existing visualization of power equipment is either to obtain infrared thermal imaging temperature measurement images, and then manually process these unstructured data to display them in two dimensions; or to directly perform three-dimensional modeling and independently display the temperature distribution of the equipment. Obviously, the current visualization of power equipment can no longer meet the development needs of the power system. Summary of the invention

[0004] The present application provides a three-dimensional visualization method and device for power equipment integrated with infrared temperature measurement images, which is used to alleviate the technical problem that the existing two-dimensional infrared temperature measurement image display or independent three-dimensional modeling display technology has a single visualization method and poor intuitiveness, and cannot provide more abundant power equipment information.

[0005] In view of this, the first aspect of the present application provides a three-dimensional visualization method for power equipment by integrating infrared temperature measurement images, comprising:

[0006] Segmenting the corresponding infrared power equipment image in the infrared temperature measurement image based on the preset section of the three-dimensional texture image of the power equipment;

[0007] Using SIFT algorithm to extract key points from the three-dimensional texture image of the electric power equipment and the infrared image of the electric power equipment, respectively, to obtain an infrared key point image and a three-dimensional texture key point image;

[0008] Performing local image matching based on the calculated Euclidean distance between the corresponding key points of the infrared key point image and the three-dimensional texture key point image to obtain a plurality of infrared local images of power equipment;

[0009] The infrared local image of the power equipment is processed into three dimensions according to a preset rotation and translation matrix to obtain a three-dimensional infrared image, wherein the preset rotation and translation matrix is ​​determined according to world coordinates, pixel coordinates and camera parameters;

[0010] The three-dimensional infrared image is fused with the three-dimensional texture image of the electric power equipment based on edge information and fusion weights to obtain a three-dimensional infrared texture image;

[0011] The three-dimensional infrared texture image is processed into a scene three-dimensional model map to achieve visual display.

[0012] Preferably, the method of segmenting the corresponding infrared power equipment image in the infrared temperature measurement image based on the preset section of the three-dimensional texture image of the power equipment includes:

[0013] Acquire pixel data of the infrared temperature measurement image, and determine the candidate area according to a rectangular pixel array formed by the pixel data;

[0014] Using a preset screening range to screen the pixel points of the candidate area to obtain a preliminary infrared device image;

[0015] Taking the preset section of the three-dimensional texture image of the electric power equipment as a reference standard, edge and background segmentation is performed on the preliminary infrared equipment image to obtain an infrared electric power equipment image.

[0016] Preferably, the adopting of SIFT algorithm to extract key points from the three-dimensional texture image of the electric equipment and the infrared electric equipment image respectively to obtain an infrared key point image and a three-dimensional texture key point image comprises:

[0017] Using different preset blur parameters to blur the three-dimensional texture image of the electric equipment and the infrared electric equipment image respectively, to obtain an infrared blurred image and a three-dimensional texture blurred image;

[0018] The SIFT algorithm is used to extract key points from the infrared blurred image and the three-dimensional texture blurred image respectively to obtain an infrared key point image and a three-dimensional texture key point image.

[0019] Preferably, the method of using SIFT algorithm to extract key points from the infrared blurred image and the three-dimensional texture blurred image to obtain an infrared key point image and a three-dimensional texture key point image respectively, further comprises:

[0020] Determine the position information and scale information of the infrared key point image and the three-dimensional texture key point image by means of preset quadratic function fitting;

[0021] The direction information and the gradient information of each key point in the infrared key point image and the three-dimensional texture key point image are calculated respectively.

[0022] Preferably, the three-dimensional processing of the infrared local image of the power equipment according to the preset rotation and translation matrix to obtain a three-dimensional infrared image also includes:

[0023] The world coordinate system is converted into the camera coordinate system by using the first preset conversion parameter, and the camera coordinate system is converted into the pixel coordinate system by using the second preset conversion parameter, so as to obtain a coordinate conversion relationship between the pixel coordinate system and the world coordinate system;

[0024] Based on the coordinate conversion relationship, a preset rotation and translation matrix is ​​calculated according to the world coordinates related to the pixel, the pixel coordinates and the camera parameters.

[0025] Preferably, the fusing the three-dimensional infrared image with the three-dimensional texture image of the electric power equipment based on edge information and fusion weights to obtain the three-dimensional infrared texture image includes:

[0026] Using a Gaussian filter to extract edge information of the three-dimensional infrared image and the three-dimensional texture image of the electric power equipment, respectively, to obtain three-dimensional infrared edge information and three-dimensional texture edge information;

[0027] Decomposing sub-feature information from the three-dimensional infrared image and the three-dimensional texture image of the electric power equipment respectively to obtain three-dimensional infrared sub-feature information and three-dimensional texture sub-feature information;

[0028] Calculating the fusion weight between the three-dimensional infrared sub-feature information and the three-dimensional texture sub-feature information by a preset weight calculation method;

[0029] The three-dimensional infrared image is fused with the three-dimensional texture image of the electric power equipment based on the three-dimensional infrared edge information, the three-dimensional texture edge information and the fusion weight to obtain a three-dimensional infrared texture image.

[0030] The second aspect of the present application provides a three-dimensional visualization device for electric power equipment integrating infrared temperature measurement images, comprising:

[0031] An image segmentation module, used to segment the corresponding infrared power equipment image in the infrared temperature measurement image based on a preset section of the three-dimensional texture image of the power equipment;

[0032] A key point extraction module, used to extract key points from the three-dimensional texture image of the electric power equipment and the infrared electric power equipment image respectively by using a SIFT algorithm to obtain an infrared key point image and a three-dimensional texture key point image;

[0033] An image matching module, used for performing local image matching based on the calculated Euclidean distance between the infrared key point image and the corresponding key points of the three-dimensional texture key point image, to obtain a plurality of infrared local images of power equipment;

[0034] A three-dimensional conversion module, used for performing three-dimensional processing on the infrared local image of the power equipment according to a preset rotation and translation matrix to obtain a three-dimensional infrared image, wherein the preset rotation and translation matrix is ​​determined according to world coordinates, pixel coordinates and camera parameters;

[0035] An image fusion module, used for fusing the three-dimensional infrared image with the three-dimensional texture image of the electric power equipment based on edge information and fusion weights to obtain a three-dimensional infrared texture image;

[0036] The three-dimensional display module is used to process the three-dimensional infrared texture image into a three-dimensional model of the scene to achieve visual display.

[0037] Preferably, the image segmentation module comprises:

[0038] An image preprocessing submodule, used to obtain pixel data of the infrared temperature measurement image, and determine the candidate area according to a rectangular pixel array formed by the pixel data;

[0039] A pixel point screening submodule is used to screen the pixel points of the candidate area using a preset screening range to obtain a preliminary infrared device image;

[0040] The background segmentation submodule is used to perform edge background segmentation on the preliminary infrared equipment image with the preset section of the three-dimensional texture image of the power equipment as a reference standard to obtain an infrared power equipment image.

[0041] Preferably, it also includes:

[0042] A first feature information module is used to determine the position information and scale information of the infrared key point image and the three-dimensional texture key point image by fitting a preset quadratic function;

[0043] The second feature information module is used to calculate the direction information and gradient information of each key point in the infrared key point image and the three-dimensional texture key point image respectively.

[0044] Preferably, the image fusion module includes:

[0045] An edge information extraction submodule is used to respectively extract edge information of the three-dimensional infrared image and the three-dimensional texture image of the electric power equipment by using a Gaussian filter to obtain three-dimensional infrared edge information and three-dimensional texture edge information;

[0046] A feature information extraction submodule, used to decompose sub-feature information from the three-dimensional infrared image and the three-dimensional texture image of the electric power equipment, respectively, to obtain three-dimensional infrared sub-feature information and three-dimensional texture sub-feature information;

[0047] A fusion weight calculation submodule, used to calculate the fusion weight between the three-dimensional infrared sub-feature information and the three-dimensional texture sub-feature information by a preset weight calculation method;

[0048] The three-dimensional image fusion submodule is used to fuse the three-dimensional infrared image with the three-dimensional texture image of the power equipment based on the three-dimensional infrared edge information, the three-dimensional texture edge information and the fusion weight to obtain a three-dimensional infrared texture image.

[0049] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0050] In the present application, a three-dimensional visualization method for power equipment fused with infrared temperature measurement images is provided, including: segmenting the corresponding infrared power equipment image in the infrared temperature measurement image based on the preset section of the three-dimensional texture image of the power equipment; using the SIFT algorithm to extract key points of the three-dimensional texture image of the power equipment and the infrared power equipment image respectively, to obtain an infrared key point image and a three-dimensional texture key point image; performing local image matching based on the calculated Euclidean distance between the corresponding key points of the infrared key point image and the three-dimensional texture key point image, to obtain multiple infrared local images of power equipment; performing three-dimensional processing on the infrared local image of the power equipment according to a preset rotation and translation matrix to obtain a three-dimensional infrared image, and the preset rotation and translation matrix is ​​determined according to world coordinates, pixel coordinates and camera parameters; performing image fusion on the three-dimensional infrared image and the three-dimensional texture image of the power equipment based on edge information and fusion weights to obtain a three-dimensional infrared texture image; performing scene three-dimensional model mapping processing on the three-dimensional infrared texture image to achieve visual display.

[0051] The three-dimensional visualization method of power equipment fused with infrared temperature measurement images provided by the present application can extract infrared equipment from the infrared temperature measurement images based on the three-dimensional image of the equipment, and can roughly separate the target power equipment image; and through the key point matching between the three-dimensional texture map and the two-dimensional infrared image, multiple different faces of the equipment can be cut out from the two-dimensional infrared key point image, and the cut different faces are processed in three dimensions to obtain a three-dimensional infrared image, and finally the three-dimensional infrared image is fused with the three-dimensional texture image, and the obtained three-dimensional infrared texture image can not only maintain the temperature distribution of the infrared temperature measurement image, but also realize the three-dimensional texture expression of the equipment, and can provide operators with richer power equipment information. Therefore, the present application can alleviate the technical problem that the visualization of the existing two-dimensional infrared temperature measurement image display or independent three-dimensional modeling display technology is too simple and less intuitive, and cannot provide richer power equipment information. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic flow chart of a method for three-dimensional visualization of electric power equipment by integrating infrared temperature measurement images provided in an embodiment of the present application;

[0053] Figure 2 Another schematic flow chart of a method for three-dimensional visualization of electric power equipment by integrating infrared temperature measurement images provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of a three-dimensional visualization device for electric power equipment integrating infrared temperature measurement images provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of the conversion relationship between the world coordinate system, camera coordinate system and pixel coordinate system provided in an embodiment of the present application;

[0056] Figure 5 Schematic diagram of detailed information of two three-dimensional image fusion processes provided in the embodiments of the present application. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] For easier understanding, see Figure 1 , Embodiment 1 of a method for three-dimensional visualization of electric power equipment by integrating infrared temperature measurement images provided by the present application includes:

[0059] Step 101 : segmenting a corresponding infrared power equipment image in an infrared temperature measurement image based on a preset section of a three-dimensional texture image of the power equipment.

[0060] The three-dimensional texture image of the power equipment includes multiple different faces of the power equipment, from which different sections are extracted, that is, a certain face of the power equipment, which is used for the segmentation of the equipment in the two-dimensional infrared temperature measurement image, and the power equipment in the infrared temperature measurement image can be roughly separated. It can be understood that the infrared temperature measurement image is an image obtained after a series of pre-processing in the early stage, such as denoising and other processing.

[0061] Step 102: Use the SIFT algorithm to extract key points from the three-dimensional texture image of the power equipment and the infrared power equipment image, respectively, to obtain an infrared key point image and a three-dimensional texture key point image.

[0062] The SIFT algorithm, or scale-invariant feature transform algorithm, is a description used for image processing. It retains the detail features in the image by using function filtering when constructing the scale space. The filtering gradually reduces the detail features, thereby simulating the feature expression in a large-scale situation. The detail features in this embodiment are key points, that is, key points extracted into different scale spaces using the SIFT algorithm, also called feature points; the images composed of these key points include two-dimensional infrared key point images and three-dimensional texture key point images. It can be understood that the key points will not change due to the influence of light or noise. Even the images of substation power equipment under different weather conditions can extract accurate and reliable key point features.

[0063] Step 103: Perform local image matching based on the calculated Euclidean distance between the corresponding key points of the infrared key point image and the three-dimensional texture key point image to obtain multiple infrared local images of power equipment.

[0064] The process of local image matching is essentially the process of key point matching. The matching between a key point in a two-dimensional infrared key point image and a key point at a corresponding position in a three-dimensional texture key point image is calculated. It is generally described by calculating the distance between the two key points; and a target key point can select more than one matching key point. Traversing all key points to complete the matching operation can differentiate multiple different power equipment infrared local images of the same device in the infrared key point image of the power equipment. These images are two-dimensional and can express red text temperature measurement.

[0065] Step 104 , three-dimensionalizing the infrared local image of the power equipment according to a preset rotation and translation matrix to obtain a three-dimensional infrared image. The preset rotation and translation matrix is ​​determined according to world coordinates, pixel coordinates and camera parameters.

[0066] The preset rotation and translation matrix can reflect the relationship between the pixel coordinate system and the world coordinate system, so that the pixel image on the two-dimensional plane can be directly converted into a three-dimensional image in the world coordinate system. A three-dimensional infrared image refers to a three-dimensional expression of an infrared temperature measurement image of a certain power equipment.

[0067] Step 105 : Fusing the three-dimensional infrared image with the three-dimensional texture image of the electric power equipment based on the edge information and the fusion weight to obtain a three-dimensional infrared texture image.

[0068] Edge information helps to fuse the boundaries of the two three-dimensional images, and the fusion weight can clarify the specific fusion method. The obtained three-dimensional infrared texture image can reflect both the infrared temperature measurement distribution of the power equipment and the texture information of the equipment. It is more intuitive and has stronger expressive ability.

[0069] Step 106: Process the three-dimensional infrared texture image into a three-dimensional model of the scene to achieve visual display.

[0070] The three-dimensional model of the actual scene may include the construction of the entire scene and the display of a variety of different devices. The three-dimensional infrared texture image of each device is obtained through the above method and then pasted onto the corresponding three-dimensional model. The power equipment in the entire scene can be comprehensively visualized, which is more in line with actual development needs and provides operators with rich visual information.

[0071] The three-dimensional visualization method of power equipment fused with infrared temperature measurement images provided in the embodiment of the present application can roughly separate the target power equipment image by performing infrared equipment extraction on the infrared temperature measurement image based on the three-dimensional image of the equipment; and by matching the key points between the three-dimensional texture map and the two-dimensional infrared map, multiple different faces of the equipment can be cut out from the two-dimensional infrared key point image, and the three-dimensional infrared image can be obtained by performing three-dimensional processing on the cut different faces. Finally, the three-dimensional infrared image and the three-dimensional texture image are fused, and the obtained three-dimensional infrared texture image can not only maintain the temperature distribution of the infrared temperature measurement image, but also realize the three-dimensional texture expression of the equipment, and can provide operators with richer power equipment information. Therefore, the embodiment of the present application can alleviate the technical problem that the visualization of the existing two-dimensional infrared temperature measurement image display or independent three-dimensional modeling display technology is too simple and less intuitive, and cannot provide richer power equipment information.

[0072] For easier understanding, see Figure 2 The present application provides a second embodiment of a three-dimensional visualization method for power equipment integrating infrared temperature measurement images, including:

[0073] Step 201: Obtain pixel data of an infrared temperature measurement image, and determine a candidate area according to a rectangular pixel array formed by the pixel data.

[0074] The infrared temperature measurement image can be filtered before specific segmentation to effectively remove noise interference. Generally, there is a lot of Gaussian noise in the image. Therefore, a Gaussian filter can be used to perform Gaussian filtering on the infrared temperature measurement image. The specific two-dimensional Gaussian filter function is expressed as:

[0075] F(x,y)=f(x,y)·G(x,y)

[0076]

[0077] Among them, e is the rotational symmetry coefficient of the image, f(x,y) is the infrared temperature measurement image before filtering, G(x,y) is the Gaussian function, and F(x,y) is the infrared temperature measurement image after Gaussian filtering. The infrared temperature measurement image after Gaussian filtering not only has a higher signal-to-noise ratio, but also can repair the edge details of the infrared temperature measurement image and make the blurred edge image clear; moreover, the edge of the pixel array of the infrared image will be more obvious.

[0078] After locating all pixel arrays in the infrared temperature measurement image, the corresponding pixel data is obtained, and a regular rectangular pixel array is constructed based on the obtained pixel data as a candidate area for the segmentation task.

[0079] Step 202: Filter the pixels in the candidate area using a preset filtering range to obtain a preliminary infrared device image.

[0080] Since the grayscale of the entire image is almost the same, it is also necessary to perform pixel screening operations on the candidate area. The length, width and area of ​​the candidate area are used as analysis objects, and compared with the preset screening range of the array to screen out the pixel array that meets the preset screening range. The specific screening can be expressed as:

[0081]

[0082] Among them, R i It is the preliminary infrared device pixel area after screening, H is the length, W is the width, t1 and t2 are the preset screening ranges.

[0083] Step 203: Using the preset section of the three-dimensional texture image of the electric power equipment as a reference standard, edge and background segmentation is performed on the preliminary infrared equipment image to obtain an infrared electric power equipment image.

[0084] The preset cross-section of the three-dimensional texture image of the power equipment can be understood as a two-dimensional image of the power equipment. By using the two-dimensional equipment image in the same direction for comparison and taking the preset cross-section as the reference standard, the preliminary infrared equipment image can be accurately segmented and redundant background pixels can be removed to obtain an accurate infrared power equipment image.

[0085] Step 204: Use different preset blur parameters to blur the three-dimensional texture image of the electric equipment and the infrared electric equipment image respectively to obtain an infrared blurred image and a three-dimensional texture blurred image.

[0086] Select the corresponding scale space and use the blur degree of the image to simulate the imaging process of a person on the retina when the distance from the object changes from far to near. The closer the distance to the object, the blurrier the image. Using different preset blur parameters can get different degrees of blur images. Both two-dimensional infrared temperature measurement images and three-dimensional texture graphics can be processed in this way.

[0087] Step 205: Use the SIFT algorithm to extract key points from the infrared blurred image and the three-dimensional texture blurred image respectively, to obtain an infrared key point image and a three-dimensional texture key point image.

[0088] The three-dimensional texture image of the power equipment and the infrared power equipment image are extracted in the same way as the key points and images; in order not to repeat the specific process, the infrared power equipment image is taken as an example here, and the local extreme point of the differential Gaussian space is used as the key point in this embodiment, and the key point is extracted by comparing the point cloud data of two adjacent layers of the differential Gaussian in the same group:

[0089]

[0090] Among them, G(x, y, z, σ) is the differential Gaussian function, I(x, y, z) is the pixel coordinate of the infrared power equipment image, L(x, y, z, σ) is the Gaussian function of the variable scale, and k is the Gaussian fuzzy scale of different sizes. The detection point is compared with the cover point at the 8 adjacent points of the scale and the 9×2 (18) points corresponding to the upper and lower adjacent scales, a total of 26 points. If the point is larger or smaller than these 26 field points, then the monitoring point is the feature point of the infrared temperature measurement data, that is, the key point.

[0091] Step 206: Determine the position information and scale information of the infrared key point image and the three-dimensional texture key point image by fitting a preset quadratic function.

[0092] In order to describe the key points more accurately and prepare data for subsequent calculations, it is necessary to determine some necessary information of the key points, such as location information and scale information. In addition, determining the location and describing the scale of the key points can also delete unstable edge points and feature points with low contrast. The differential operator in the scale space is curve fitted, and the Taylor expansion of the differential operator is used to accurately locate the key points in the scale space.

[0093] Step 207: Calculate the direction information and gradient information of each key point in the infrared key point image and the three-dimensional texture key point image respectively.

[0094] The direction of each key point is determined as follows:

[0095] θ(x,y,z)=arctan(L y / L x )

[0096]

[0097] Among them, θ(x,y,z) represents the direction information of the point, v(x,y,z) represents the gradient information of the point, and L x , L y , Lz are the pixel coordinate differences after the scale change, and can be expressed as:

[0098] L x =L(x+1,y,z)-L(x-1,y,z)

[0099] L y =L(x,y+1,z)-L(x,y-1,z)

[0100] L z =L(x,y,z+1)-L(x,y,z-1)

[0101] The above method can be used to obtain some necessary information of key points: position information, scale information, direction information and gradient information.

[0102] Step 208: Perform local image matching based on the calculated Euclidean distance between the corresponding key points of the infrared key point image and the three-dimensional texture key point image to obtain multiple infrared local images of power equipment.

[0103] The matching process is as follows: first determine the target key point p1, then use the kd tree search algorithm based on BBF (BestBinFirst) to find the nearest neighbor point q1 and the second nearest neighbor point q2 of the target key point p1 in the benchmark 3D texture key point image; then calculate the Euclidean distance between the target key point p1 and the nearest neighbor point q1 and the second nearest neighbor point q2, and record the two distances as D11 and D12 respectively. The calculation process of the Euclidean distance is:

[0104]

[0105] Among them, p1 and p2 are two key points, p 1i 、p 2i are the i-th pair of adjacent key points respectively.

[0106] If the distance ratio D11 / D12 is less than the preset ratio TH, the current matching point is determined to be a qualified matching point of the target key point, otherwise it is determined that the target key point has no matching point; all key points are traversed to determine whether each key point has a matching point.

[0107] Step 209: convert the world coordinate system into the camera coordinate system by using the first preset conversion parameter, and convert the camera coordinate system into the pixel coordinate system by using the second preset conversion parameter, so as to obtain a coordinate conversion relationship between the pixel coordinate system and the world coordinate system.

[0108] Converting infrared images into three-dimensional images can achieve better image fusion and display of three-dimensional temperature measurement data. Converting infrared images into three-dimensional images requires the use of three spatial coordinate systems and the conversion between them, namely the world coordinate system (UVW), the camera coordinate system (XYZ), and the pixel coordinate system (xy).

[0109] See also Figure 4 , the conversion process from the world coordinate system to the camera coordinate system is:

[0110]

[0111] The conversion process from the camera coordinate system to the pixel coordinate system is:

[0112]

[0113] The coordinate transformation relationship between the pixel coordinate system and the world coordinate system is obtained by iteratively solving the DLT (Direct Linear Transform) algorithm combined with the least squares method as follows:

[0114]

[0115] Among them, R is the spatial posture relationship matrix of the object relative to the camera, T is the spatial position relationship matrix of the object relative to the camera, and f x 、f y are the x-coordinate and y-coordinate of the pixel coordinate system, c x 、c y are the x and y coordinates of the camera coordinate system respectively.

[0116] Step 210: Based on the coordinate transformation relationship, a preset rotation and translation matrix is ​​calculated according to the pixel-related world coordinates, pixel coordinates and camera parameters.

[0117] According to the coordinate transformation relationship obtained above, the preset rotation and translation matrix can be calculated according to the obtained world coordinates, pixel coordinate positions and camera parameters related to the pixel points:

[0118]

[0119] Among them, r 00 ~r 22 are the elements of the preset rotation and translation matrix, φ, ψ, and γ are the rotation angles in the x, y, and z directions in the rotation matrix respectively.

[0120] Step 211: Process the infrared local image of the power equipment into three dimensions according to a preset rotation and translation matrix to obtain a three-dimensional infrared image.

[0121] The obtained preset rotation and translation matrix can be used to perform three-dimensional transformation processing on the pixels of multiple infrared local images of the same power equipment to obtain a three-dimensional infrared image, that is, a three-dimensional device image of infrared temperature measurement of the power equipment can be displayed.

[0122] Step 212: Use a Gaussian filter to extract edge information of the three-dimensional infrared image and the three-dimensional texture image of the electric power equipment, respectively, to obtain three-dimensional infrared edge information and three-dimensional texture edge information.

[0123] The Gaussian filter combined with the guided filter can extract the edge information of the three-dimensional infrared image and the three-dimensional texture image of the power equipment. The specific filtering formula is:

[0124]

[0125] Where I is the filter channel, p and q are the number of filter samples and the number of selected points of filter channel I, respectively. is the Gaussian variance σ after the corresponding transformation s is the standard deviation of the corresponding Gaussian function, K P is the normalization coefficient, which can be expressed as:

[0126]

[0127] See also Figure 5 After the edge information is obtained, we can obtain the small-scale texture detail information D(I), the large-scale edge information E(I) and the underlying strategic scale image information A(I), and A(I) = G(I). The principle is to solve the average value of infinite linear functions at a certain point, and then use this result to replace the value of the complex function at the corresponding position.

[0128] Step 213: Decompose sub-feature information from the three-dimensional infrared image and the three-dimensional texture image of the electric power equipment respectively to obtain three-dimensional infrared sub-feature information and three-dimensional texture sub-feature information.

[0129] The process of decomposing sub-feature information is the process of extracting features of different properties of the image, and the fusion weight is also determined based on the sub-feature information. The values ​​of the pixel positions corresponding to these sub-feature information can increase the fusion weight, texture details and edge information of the point.

[0130] Step 214: Calculate the fusion weight between the three-dimensional infrared sub-feature information and the three-dimensional texture sub-feature information by using a preset weight calculation method.

[0131] The fusion weight calculation process is:

[0132]

[0133] Among them, t is the three-dimensional texture sub-feature information, υ is the fused pixel point, and c i is the infrared feature i, which can be expressed as:

[0134]

[0135] Among them, u i,j is the matching weight of the 3D texture sub-feature corresponding to the infrared sub-feature i, and K is the number of 3D texture sub-features

[0136] Step 215 : Based on the 3D infrared edge information, the 3D texture edge information and the fusion weight, the 3D infrared image is fused with the 3D texture image of the electric power equipment to obtain a 3D infrared texture image.

[0137] Infrared thermal imaging temperature measurement images can effectively reflect the two-dimensional temperature information of substation equipment, and color texture images can effectively reflect the texture, mapping and other information of substation equipment. By fusing the two images, the three-dimensional infrared texture image can provide operators with richer power equipment information, which is convenient for operators to intuitively analyze.

[0138] Step 216: Process the three-dimensional infrared texture image into a three-dimensional model of the scene to achieve visual display.

[0139] The visualization process is a three-dimensional display process that adds scene attributes to the power equipment. The displayed scene may include specific infrared temperature measurement power equipment information, three-dimensional model information that expresses texture, and some environmental three-dimensional modeling information, which is more in line with actual scene display needs.

[0140] The three-dimensional visualization method of power equipment fused with infrared temperature measurement images provided in the embodiment of the present application can roughly separate the target power equipment image by performing infrared equipment extraction on the infrared temperature measurement image based on the three-dimensional image of the equipment; and by matching the key points between the three-dimensional texture map and the two-dimensional infrared map, multiple different faces of the equipment can be cut out from the two-dimensional infrared key point image, and the three-dimensional infrared image can be obtained by performing three-dimensional processing on the cut different faces. Finally, the three-dimensional infrared image and the three-dimensional texture image are fused, and the obtained three-dimensional infrared texture image can not only maintain the temperature distribution of the infrared temperature measurement image, but also realize the three-dimensional texture expression of the equipment, and can provide operators with richer power equipment information. Therefore, the embodiment of the present application can alleviate the technical problem that the visualization of the existing two-dimensional infrared temperature measurement image display or independent three-dimensional modeling display technology is too simple and less intuitive, and cannot provide richer power equipment information.

[0141] For easier understanding, see Figure 3The present application also provides an embodiment of a three-dimensional visualization device for electric power equipment integrating infrared temperature measurement images, including:

[0142] An image segmentation module 301 is used to segment the corresponding infrared power equipment image in the infrared temperature measurement image based on a preset section of the three-dimensional texture image of the power equipment;

[0143] The key point extraction module 302 is used to extract key points from the three-dimensional texture image of the power equipment and the infrared power equipment image respectively using the SIFT algorithm to obtain an infrared key point image and a three-dimensional texture key point image;

[0144] An image matching module 303 is used to perform local image matching based on the calculated Euclidean distance between the corresponding key points of the infrared key point image and the three-dimensional texture key point image to obtain multiple infrared local images of power equipment;

[0145] A three-dimensional conversion module 304 is used to perform three-dimensional processing on the infrared local image of the power equipment according to a preset rotation and translation matrix to obtain a three-dimensional infrared image. The preset rotation and translation matrix is ​​determined according to world coordinates, pixel coordinates and camera parameters;

[0146] An image fusion module 305 is used to fuse the three-dimensional infrared image with the three-dimensional texture image of the power equipment based on edge information and fusion weights to obtain a three-dimensional infrared texture image;

[0147] The three-dimensional display module 306 is used to process the three-dimensional infrared texture image into a three-dimensional model of the scene to achieve visual display.

[0148] Furthermore, the image segmentation module 301 includes:

[0149] The image preprocessing submodule 3011 is used to obtain pixel data of the infrared temperature measurement image and determine the candidate area according to the rectangular pixel array formed by the pixel data;

[0150] The pixel point screening submodule 3012 is used to screen the pixel points of the candidate area using a preset screening range to obtain a preliminary infrared device image;

[0151] The background segmentation submodule 3013 is used to perform edge background segmentation on the preliminary infrared equipment image with the preset section of the three-dimensional texture image of the power equipment as a reference standard to obtain the infrared power equipment image.

[0152] Furthermore, it also includes:

[0153] The first feature information module 307 is used to determine the position information and scale information of the infrared key point image and the three-dimensional texture key point image by fitting a preset quadratic function;

[0154] The second feature information module 308 is used to calculate the direction information and gradient information of each key point in the infrared key point image and the three-dimensional texture key point image respectively.

[0155] Furthermore, the image fusion module 305 includes:

[0156] The edge information extraction submodule 3051 is used to respectively extract edge information of the three-dimensional infrared image and the three-dimensional texture image of the power equipment using a Gaussian filter to obtain three-dimensional infrared edge information and three-dimensional texture edge information;

[0157] The feature information extraction submodule 3052 is used to decompose sub-feature information from the three-dimensional infrared image and the three-dimensional texture image of the power equipment to obtain three-dimensional infrared sub-feature information and three-dimensional texture sub-feature information;

[0158] A fusion weight calculation submodule 3053 is used to calculate the fusion weight between the three-dimensional infrared sub-feature information and the three-dimensional texture sub-feature information by a preset weight calculation method;

[0159] The 3D image fusion submodule 3054 is used to fuse the 3D infrared image with the 3D texture image of the power equipment based on the 3D infrared edge information, the 3D texture edge information and the fusion weight to obtain the 3D infrared texture image.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0164] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A three-dimensional visualization method for power equipment integrating infrared temperature measurement images, characterized in that: include: The corresponding infrared power equipment image is segmented in the infrared temperature measurement image based on the preset section of the three-dimensional texture image of the power equipment. The specific process is as follows: Acquire pixel data of the infrared temperature measurement image, and determine the candidate area according to a rectangular pixel array formed by the pixel data; Using a preset screening range to screen the pixel points of the candidate area to obtain a preliminary infrared device image; Using a preset section of the three-dimensional texture image of the electric power equipment as a reference standard, edge and background segmentation is performed on the preliminary infrared equipment image to obtain an infrared electric power equipment image; The SIFT algorithm is used to extract key points from the three-dimensional texture image of the power equipment and the infrared power equipment image, respectively, to obtain an infrared key point image and a three-dimensional texture key point image. The specific process is as follows: Using different preset blur parameters to blur the three-dimensional texture image of the electric equipment and the infrared electric equipment image respectively, to obtain an infrared blurred image and a three-dimensional texture blurred image; Using SIFT algorithm to extract key points from the infrared blurred image and the three-dimensional texture blurred image respectively, to obtain an infrared key point image and a three-dimensional texture key point image; Determine the position information and scale information of the infrared key point image and the three-dimensional texture key point image by means of preset quadratic function fitting; Calculating the direction information and gradient information of each key point in the infrared key point image and the three-dimensional texture key point image respectively; Performing local image matching based on the calculated Euclidean distance between the corresponding key points of the infrared key point image and the three-dimensional texture key point image to obtain a plurality of infrared local images of power equipment; The infrared local image of the power equipment is processed into three dimensions according to a preset rotation and translation matrix to obtain a three-dimensional infrared image, wherein the preset rotation and translation matrix is ​​determined according to world coordinates, pixel coordinates and camera parameters; The three-dimensional infrared image is fused with the three-dimensional texture image of the electric power equipment based on edge information and fusion weights to obtain a three-dimensional infrared texture image; The three-dimensional infrared texture image is processed into a scene three-dimensional model map to achieve visual display.

2. The three-dimensional visualization method of electric power equipment by integrating infrared temperature measurement images according to claim 1 is characterized in that: The infrared local image of the power equipment is processed into three dimensions according to a preset rotation and translation matrix to obtain a three-dimensional infrared image, and the method further includes: The world coordinate system is converted into the camera coordinate system by using the first preset conversion parameter, and the camera coordinate system is converted into the pixel coordinate system by using the second preset conversion parameter, so as to obtain a coordinate conversion relationship between the pixel coordinate system and the world coordinate system; Based on the coordinate conversion relationship, a preset rotation and translation matrix is ​​calculated according to the world coordinates related to the pixel, the pixel coordinates and the camera parameters.

3. The three-dimensional visualization method of electric power equipment by integrating infrared temperature measurement images according to claim 1 is characterized in that: The method of fusing the three-dimensional infrared image with the three-dimensional texture image of the electric power equipment based on edge information and fusion weights to obtain a three-dimensional infrared texture image includes: Using a Gaussian filter to extract edge information of the three-dimensional infrared image and the three-dimensional texture image of the electric power equipment, respectively, to obtain three-dimensional infrared edge information and three-dimensional texture edge information; Decomposing sub-feature information from the three-dimensional infrared image and the three-dimensional texture image of the electric power equipment respectively to obtain three-dimensional infrared sub-feature information and three-dimensional texture sub-feature information; Calculating the fusion weight between the three-dimensional infrared sub-feature information and the three-dimensional texture sub-feature information by a preset weight calculation method; The three-dimensional infrared image is fused with the three-dimensional texture image of the electric power equipment based on the three-dimensional infrared edge information, the three-dimensional texture edge information and the fusion weight to obtain a three-dimensional infrared texture image.

4. A three-dimensional visualization device for electric power equipment integrating infrared temperature measurement images, characterized in that: include: An image segmentation module is used to segment the corresponding infrared power equipment image in the infrared temperature measurement image based on a preset section of the three-dimensional texture image of the power equipment. The image segmentation module includes: An image preprocessing submodule, used to obtain pixel data of the infrared temperature measurement image, and determine the candidate area according to a rectangular pixel array formed by the pixel data; A pixel point screening submodule is used to screen the pixel points of the candidate area using a preset screening range to obtain a preliminary infrared device image; A background segmentation submodule, for performing edge background segmentation on the preliminary infrared equipment image with a preset section of the three-dimensional texture image of the power equipment as a reference standard to obtain an infrared power equipment image; The key point extraction module is used to extract key points from the three-dimensional texture image of the power equipment and the infrared power equipment image respectively using the SIFT algorithm to obtain an infrared key point image and a three-dimensional texture key point image. The specific process is as follows: Using different preset blur parameters to blur the three-dimensional texture image of the electric equipment and the infrared electric equipment image respectively, to obtain an infrared blurred image and a three-dimensional texture blurred image; Using SIFT algorithm to extract key points from the infrared blurred image and the three-dimensional texture blurred image respectively, to obtain an infrared key point image and a three-dimensional texture key point image; Determine the position information and scale information of the infrared key point image and the three-dimensional texture key point image by means of preset quadratic function fitting; Calculating the direction information and gradient information of each key point in the infrared key point image and the three-dimensional texture key point image respectively; An image matching module, used for performing local image matching based on the calculated Euclidean distance between the infrared key point image and the corresponding key points of the three-dimensional texture key point image, to obtain a plurality of infrared local images of power equipment; A three-dimensional conversion module, used for performing three-dimensional processing on the infrared local image of the power equipment according to a preset rotation and translation matrix to obtain a three-dimensional infrared image, wherein the preset rotation and translation matrix is ​​determined according to world coordinates, pixel coordinates and camera parameters; An image fusion module, used for fusing the three-dimensional infrared image with the three-dimensional texture image of the electric power equipment based on edge information and fusion weights to obtain a three-dimensional infrared texture image; The three-dimensional display module is used to process the three-dimensional infrared texture image into a three-dimensional model of the scene to achieve visual display.

5. The three-dimensional visualization device for electric power equipment integrating infrared temperature measurement images according to claim 4 is characterized in that: Also includes: A first feature information module is used to determine the position information and scale information of the infrared key point image and the three-dimensional texture key point image by fitting a preset quadratic function; The second feature information module is used to calculate the direction information and gradient information of each key point in the infrared key point image and the three-dimensional texture key point image respectively.

6. The three-dimensional visualization device for electric power equipment integrating infrared temperature measurement images according to claim 4 is characterized in that: The image fusion module comprises: An edge information extraction submodule is used to respectively extract edge information of the three-dimensional infrared image and the three-dimensional texture image of the electric power equipment by using a Gaussian filter to obtain three-dimensional infrared edge information and three-dimensional texture edge information; A feature information extraction submodule, used to decompose sub-feature information from the three-dimensional infrared image and the three-dimensional texture image of the electric power equipment, respectively, to obtain three-dimensional infrared sub-feature information and three-dimensional texture sub-feature information; A fusion weight calculation submodule, used to calculate the fusion weight between the three-dimensional infrared sub-feature information and the three-dimensional texture sub-feature information by a preset weight calculation method; The three-dimensional image fusion submodule is used to fuse the three-dimensional infrared image with the three-dimensional texture image of the power equipment based on the three-dimensional infrared edge information, the three-dimensional texture edge information and the fusion weight to obtain a three-dimensional infrared texture image.

Images