Substation model construction method based on laser point cloud and visual data

Abstract

The invention discloses a transformer substation model construction method based on laser point cloud and visual data, and relates to the technical field of transformer substation model construction.The transformer substation model construction method based on the laser point cloud and the visual data comprises the following steps that an equipment model library is constructed based on equipment data in a transformer substation; obtaining a substation field depth prediction map by using monocular depth estimation; calibrating the substation field depth prediction map based on the point cloud data; obtaining three-dimensional model data of the substation equipment based on the calibrated substation field depth prediction map; according to the transformer substation model construction method based on the laser point cloud and the visual data, by means of monocular depth estimation and automatic processing of the point cloud data, dependence on manual modeling is reduced, and the modeling process is more efficient; the equipment model library is constructed based on the equipment data, so that high equipment precision in the model library is ensured; and the depth prediction map is calibrated by using the point cloud data, so that the accuracy of the three-dimensional model is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of substation model construction, and in particular relates to a substation model construction method based on laser point cloud and visual data. Background Art

[0002] Substation model construction usually relies on 3D reconstruction technology. Specifically, 3D reconstruction provides basic data and preliminary geometric information for substation model construction. 3D reconstruction mainly restores several 2D images to obtain the 3D structure associated with the target object. This link has been widely used in various fields such as 3D measurement and virtual reality.

[0003] In the early stage of research, the 3D reconstruction method is suitable for working under completely static conditions. In this link, first, the camera should be calibrated in advance, and second, the various parameters involved should be solved before calibration. After a long period of development, the 3D reconstruction method has become more and more perfect. Scholar Faugeras has developed an innovative method that can achieve 3D reconstruction of images that have not been calibrated, but the calculation is relatively complicated.

[0004] After that, Davis et al. proposed a dual-camera range scanner design, specifically choosing a calibration that minimizes complexity and cost, and using only statically arranged components for the design, greatly reducing the costs associated with calibration. However, this design does not have real-time performance in a practical sense because it is difficult to ensure the absolute repeatability of moving parts. Mulligan et al. analyzed and came up with a three-eye stereo vision method, but it still does not have real-time performance in a practical sense. Subsequently, other scholars have proposed stereo vision based on a larger number of lenses, which has not really solved the real-time problem, but has higher accuracy in reconstructing the geometric structure of objects.

[0005] The above multi-vision-based reconstruction methods all use the principle of optical triangulation, which can effectively improve the measurement accuracy. However, the reconstruction process is relatively complex, and it also places strict requirements on the hardware, and the camera calibration is too complicated.

[0006] To effectively solve many problems in the 3D reconstruction process, such as the low accuracy of the differential method, Liang et al. have studied a scientific and innovative 3D reconstruction method based on the contour of the target object. This method directly estimates the differential structure of the surface and produces a relatively complete 3D model, which is similar to the volume method. In addition, this method directly ignores the spatial order of the viewpoints, which improves the reconstruction effect. Whether it is a monocular, binocular or multi-eye stereo vision method, it is based on image-based 3D reconstruction. The more lenses there are, the higher the requirements for experimental instruments and equipment and the higher the cost. For economic considerations, most of the 3D reconstruction methods that scholars have developed are now based on monocular cameras. This method is mainly based on the corresponding points in the two images to solve the coordinates of the associated spatial points to achieve good 3D reconstruction results.

[0007] However, 3D reconstruction technology that relies solely on images cannot meet the requirements of high precision, rapidity, automation and intelligence for the overall 3D modeling of substations. In addition to image-based 3D reconstruction technology, there is also point cloud-based 3D reconstruction technology. Currently, most of the automatic modeling methods for substations are aimed at relatively simple 3D scenes and cannot meet the needs of high-precision models of substations. Currently, the overall 3D modeling of substations mainly relies on manual modeling and mapping of data point clouds, which is labor-intensive and difficult to achieve fast and refined modeling. The overall modeling time is long and the cost is high, which seriously restricts the promotion and advanced application of substation-based 3D models. Summary of the invention

[0008] The purpose of the present invention is to provide a substation model construction method based on laser point cloud and visual data with a simple structure and reasonable design in order to solve the above problems.

[0009] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0010] A method for constructing a substation model based on laser point cloud and visual data, specifically comprising the following steps:

[0011] Build equipment model library based on equipment data in substation;

[0012] Use monocular depth estimation to obtain the substation site depth prediction map;

[0013] Calibrate the substation site depth prediction map based on point cloud data;

[0014] The three-dimensional model data of the substation equipment is obtained based on the calibrated substation site depth prediction map.

[0015] Further, construct a device library model;

[0016] Specifically, based on the substation equipment information data, the equipment in the substation is regarded as being composed of a plurality of components of standard geometric shapes, wherein each component has a fixed geometric shape, and the geometric component data of the standard shape after disassembly is obtained based on the component data;

[0017] The model is built using geometric component data of standard shapes, and the model data is stored in a template library, and a substation equipment model is generated based on the model data.

[0018] Further, a substation site depth prediction map is obtained;

[0019] Use deep learning-based monocular depth estimation to complete scene modeling;

[0020] Specifically, a convolutional neural network is trained to automatically learn the mapping relationship between the depth map and the color image, and a depth prediction map is directly generated.

[0021] Further, obtaining point cloud data;

[0022] Based on the mobile video acquisition device and the lidar equipment, point cloud data is obtained, including the actual spatial position and geometric shape information data of the equipment.

[0023] Furthermore, the substation site depth prediction map is calibrated based on the point cloud data;

[0024] Specifically include:

[0025] a. Convert point cloud radar coordinates to camera coordinates;

[0026] The conversion relationship from the laser radar coordinate system to the camera coordinate system is as follows:

[0027]

[0028] Among them, P L (x L ,y L , z L ) represents the XYZ coordinates of the point in the laser radar coordinate system, P c (x c ,y c , z c ) represents the XYZ coordinates of the point in the camera coordinate system.

[0029] b. The depth prediction map generates a three-dimensional scene model in the camera coordinate system;

[0030] The camera imaging model, that is, the transformation relationship from the camera coordinate system to the pixel coordinate system is as follows:

[0031]

[0032] Where f is the focal length of the camera, d x ,d y is the quantized value from the image coordinate system to the pixel coordinate system, u 0 、v 0 is the coordinate of the optical center in the pixel coordinate system, (u, v) represents the coordinate point (x c ,y c , z c ) to pixel coordinates.

[0033] z c That is the predicted depth corresponding to (u, v) in the depth prediction map.

[0034] By z c , u, v are substituted into the camera imaging model to calculate (x c ,y c , z c ).

[0035] c. Determine the scaling ratio μ between the depth prediction map and the point cloud data;

[0036] For the depth prediction map, according to the camera imaging model, first generate a 3D scene model in the camera coordinate system, calculate the FPFH 3D key point features, calculate the 3D point cloud converted to the FPFH 3D key point features in the camera coordinate system, perform feature point matching, and obtain a series of feature matching pairs. Calculate the ratio of the depth difference of each pair of feature matching pairs to the corresponding depth prediction of the FPFH 3D key point pair, and take the average value as the scaling ratio μ;

[0037] d. Adjust the depth prediction map according to the scaling ratio;

[0038] The new prediction graph J is:

[0039] J=μ*I

[0040] e. According to the principle of closest distance, the depth prediction of each pixel is searched for the nearest point cloud data, and the depth prediction of each pixel is adjusted;

[0041] Assume that the camera coordinates corresponding to each pixel are (x J ,y J , z J ), the radar is converted to the camera coordinates as (x L ,y L , z L ), the distance between them is

[0042]

[0043] For each pixel’s depth prediction, find the nearest point cloud data point and adjust its depth prediction value based on the point cloud data point. Assume that the camera coordinates corresponding to the nearest point cloud are (x J ,y J , z J ), adjust the depth prediction of this point to z J .

[0044] Further, the three-dimensional model data of the substation equipment is obtained based on the calibrated substation site depth prediction map; specifically including:

[0045] a. Use the random sampling consensus (RANSAC) method to extract standard geometric shapes;

[0046] b. Target detection determines the pixel location of the device;

[0047] The device detected by the target detection algorithm is the upper left 2D position and the 2D position of the upper right corner c represents the category, i represents the i-th device target;

[0048] c. Build a 3D model of the device based on the device model library and target pixel position extraction;

[0049] According to the projection direction of the target frame, the process of selecting the scene area where the segmented device is located is as follows:

[0050] The upper left corner of the device target detection box is relative to Z c Vertical angle of the axis Angle with horizontal for

[0051]

[0052]

[0053] The lower right corner of the device's target detection box is relative to Z c Vertical angle of the axis Angle with horizontal for

[0054]

[0055]

[0056] A point P in the world coordinate system iW (x iW ,y iW , z iW ) The XYZ coordinates of the camera coordinate system after coordinate conversion are P ic (x ic ,y ic, z ic ), the point relative to Z c The vertical angle and horizontal angle of the axis are

[0057]

[0058]

[0059] Vertical angle of view of the target With horizontal viewing angle β i The ranges are and

[0060] Select the frame to split and satisfy the vertical viewing angle Vertical viewing angle of the target With horizontal viewing angle β i Horizontal viewing angle of the target The point cloud, that is, the scene area where the device is located is

[0061]

[0062] Based on the model library of the device, the standard geometric shape of the device in the scene area where the device is located is extracted by the combination box selection segmentation, so as to determine the three-dimensional model of the device.

[0063] The present invention has at least the following beneficial effects: by utilizing monocular depth estimation and automated processing of point cloud data, the present invention reduces reliance on manual modeling and makes the modeling process more efficient; constructs an equipment model library based on equipment data, ensuring that the equipment in the model library is of high precision and can better reflect the characteristics of actual substation equipment; utilizes point cloud data to calibrate the depth prediction map, thereby improving the accuracy of the on-site depth map, thereby generating a more accurate three-dimensional model; relies on algorithms to automatically complete modeling and mapping, reducing the need for manual intervention, thereby reducing the overall modeling cost, and achieving accuracy, rapidity, automation and intelligence in the overall three-dimensional modeling of the substation, thereby improving modeling efficiency and reducing modeling costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flow chart of a method for constructing a substation model based on laser point cloud and visual data of the present invention; DETAILED DESCRIPTION

[0065] The present application is further described in detail below in conjunction with the accompanying drawings. It is necessary to point out here that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technical personnel in this field can make some non-essential improvements and adjustments to the present application based on the above application content.

[0066] The purpose of the present invention is to propose a substation model construction method based on laser point cloud and visual data to achieve accuracy, rapidity, automation and intelligence of the overall three-dimensional modeling of the substation.

[0067] like Figure 1 As shown, a method for constructing a substation model based on laser point cloud and visual data specifically includes the following steps:

[0068] Build equipment model library based on equipment data in substation;

[0069] Based on the equipment model library, the depth prediction map of the substation site is obtained by using monocular depth estimation;

[0070] Get point cloud data;

[0071] Calibrate the substation site depth prediction map based on point cloud data;

[0072] The three-dimensional model data of the substation equipment is obtained based on the calibrated substation site depth prediction map.

[0073] Build a device library model;

[0074] Specifically, based on the substation equipment information data, the equipment in the substation is regarded as being composed of a plurality of components of standard geometric shapes, wherein each component has a fixed geometric shape, and the geometric component data of the standard shape after disassembly is obtained based on the component data;

[0075] Modeling is performed using geometric component data of standard shapes, and the model data is stored in a template library, and a substation equipment model is generated based on the model data;

[0076] A careful analysis of the characteristics of substation equipment shows that the main body of substation equipment is rigid equipment with a fixed geometric shape. Although there are many details, each part can be disassembled into a standard geometric shape. Therefore, the equipment can be abstracted into a combination of geometric components of various standard shapes. These components can be modeled and used as standard components in the template library.

[0077] Obtain the substation site depth prediction map;

[0078] Use deep learning-based monocular depth estimation to complete scene modeling;

[0079] Specifically, the mapping relationship between the depth map and the color image is automatically learned by training a convolutional neural network, and a depth prediction map is directly generated. That is, a convolutional neural network (CNN) is used to train the model so that it can directly predict the depth map from the color image, which can effectively automate the depth estimation task without manually annotating the depth information of each image.

[0080] The difference between the deep learning-based monocular depth estimation method and the traditional monocular depth estimation method is that the deep learning-based method does not need to rely on additional assumptions;

[0081] Representative algorithms for monocular depth estimation based on deep learning include DCNN, DRON, FastDepth, AdaBins, NewCRFs, Depth Anything V2, Metric3Dv2, etc.

[0082] Among them, the monocular depth estimation method based on deep learning is a method that estimates the depth information of each pixel in an image through machine learning technology. This technology uses images taken by a single camera (i.e. "monocular") to predict the distance from each point in the scene to the camera, without relying on multiple cameras or other depth sensors; depth estimation refers to inferring the distance from each pixel to the camera from the image. Depth information enables us to convert a two-dimensional image into a point in three-dimensional space.

[0083] Get point cloud data;

[0084] Based on the mobile video acquisition device and the lidar equipment, point cloud data is obtained, including the actual spatial position and geometric shape information data of the equipment.

[0085] Calibrate the substation site depth prediction map based on point cloud data;

[0086] Specifically include:

[0087] a. Convert point cloud radar coordinates to camera coordinates;

[0088] The conversion relationship from the laser radar coordinate system to the camera coordinate system is as follows:

[0089]

[0090] Among them, P L (x L ,y L , z L ) represents the XYZ coordinates of the point in the laser radar coordinate system, P c (x c ,y c , z c ) represents the XYZ coordinates of the point in the camera coordinate system.

[0091] b. The depth prediction map generates a three-dimensional scene model in the camera coordinate system;

[0092] The camera imaging model, that is, the transformation relationship from the camera coordinate system to the pixel coordinate system is as follows:

[0093]

[0094] Where f is the focal length of the camera, d x ,d y is the quantized value from the image coordinate system to the pixel coordinate system, u 0 、v 0 is the coordinate of the optical center in the pixel coordinate system, (u, v) represents the coordinate point (x c ,y c , z c ) to pixel coordinates.

[0095] z c That is the predicted depth corresponding to (u, v) in the depth prediction map.

[0096] By z c , u, v are substituted into the camera imaging model to calculate (x c ,y c , z c ).

[0097] c. Determine the scaling ratio μ between the depth prediction map and the point cloud data;

[0098] For the depth prediction map, according to the camera imaging model, first generate a 3D scene model in the camera coordinate system, calculate the FPFH 3D key point features, calculate the 3D point cloud converted to the FPFH 3D key point features in the camera coordinate system, perform feature point matching, and obtain a series of feature matching pairs. Calculate the ratio of the depth difference of each pair of feature matching pairs to the corresponding depth prediction of the FPFH 3D key point pair, and take the average value as the scaling ratio μ;

[0099] d. Adjust the depth prediction map according to the scaling ratio;

[0100] The new prediction graph J is:

[0101] J=μ*I

[0102] e. According to the principle of closest distance, the depth prediction of each pixel is searched for the nearest point cloud data, and the depth prediction of each pixel is adjusted;

[0103] Assume that the camera coordinates corresponding to each pixel are (x J ,y J ,z J ), the radar is converted to the camera coordinates as (x L ,y L ,z L ), the distance between them is

[0104]

[0105] For each pixel’s depth prediction, find the nearest point cloud data point and adjust its depth prediction value based on the point cloud data point. Assume that the camera coordinates corresponding to the nearest point cloud are (x J ,y J ,z J ), adjust the depth prediction of this point to z J .

[0106] The three-dimensional model data of the substation equipment is obtained based on the calibrated substation site depth prediction map; specifically including:

[0107] a. Use the random sampling consensus (RANSAC) method to extract standard geometric shapes;

[0108] b. Target detection determines the pixel location of the device;

[0109] The device detected by the target detection algorithm is the upper left 2D position and the 2D position of the upper right corner c represents the category, i represents the i-th device target;

[0110] c. Build a 3D model of the device based on the device model library and target pixel position extraction;

[0111] According to the projection direction of the target frame, the process of selecting the scene area where the segmented device is located is as follows:

[0112] The upper left corner of the device target detection box is relative to Z c Vertical angle of the axis Angle with horizontal for

[0113]

[0114]

[0115] The lower right corner of the device's target detection box is relative to Z c Vertical angle of the axis Angle with horizontal for

[0116]

[0117]

[0118] A point P in the world coordinate system iW (x iW ,y iW ,z iW ) The XYZ coordinates of the camera coordinate system after coordinate conversion are P ic (x ic ,y ic ,zic ), the point relative to Z c The vertical angle and horizontal angle of the axis are

[0119]

[0120]

[0121] Vertical angle of view of the target With horizontal viewing angle β i The ranges are and

[0122] Select the frame to split and satisfy the vertical viewing angle Vertical viewing angle of the target With horizontal viewing angle β i Horizontal viewing angle of the target The point cloud, that is, the scene area where the device is located is

[0123]

[0124] Based on the model library of the device, the standard geometric shape of the device in the scene area where the device is located is extracted by the combination box selection segmentation, so as to determine the three-dimensional model of the device.

[0125] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for constructing a substation model based on laser point cloud and visual data, characterized in that: The specific steps include: Build equipment model library based on equipment data in substation; Use monocular depth estimation to obtain the substation site depth prediction map; Calibrate the substation site depth prediction map based on point cloud data; The three-dimensional model data of the substation equipment is obtained based on the calibrated substation site depth prediction map.

2. A method for constructing a substation model based on laser point cloud and visual data according to claim 1, characterized in that: Build a device library model; Specifically, the method includes: based on the substation equipment information data, the substation equipment is regarded as being composed of a plurality of components of standard geometric shapes, wherein each component has a fixed geometric shape, and based on the component data, the geometric component data of the standard shape after disassembly is obtained; The model is built using geometric component data of standard shapes, and the model data is stored in a template library, and a substation equipment model is generated based on the model data.

3. A method for constructing a substation model based on laser point cloud and visual data according to claim 2, characterized in that: Obtain the substation site depth prediction map; Use deep learning-based monocular depth estimation to complete scene modeling; Specifically, the mapping relationship between depth map and color image is automatically learned by training convolutional neural network, and depth prediction map is directly generated.

4. A method for constructing a substation model based on laser point cloud and visual data according to claim 3, characterized in that: Get point cloud data; Based on the mobile video acquisition device and the lidar equipment, point cloud data is obtained, including the actual spatial position and geometric shape information data of the equipment.

5. A method for constructing a substation model based on laser point cloud and visual data according to claim 4, characterized in that: Calibrate the substation site depth prediction map based on point cloud data; Specifically include: a. Convert point cloud radar coordinates to camera coordinates; The conversion relationship from the laser radar coordinate system to the camera coordinate system is as follows: Among them, P L (x L ,y L ,z L ) represents the XYZ coordinates of the point in the laser radar coordinate system, P c (x c ,y c ,z c ) represents the XYZ axis coordinates of the point in the camera coordinate system; b. The depth prediction map generates a three-dimensional scene model in the camera coordinate system; The camera imaging model, that is, the transformation relationship from the camera coordinate system to the pixel coordinate system is as follows: Where f is the focal length of the camera, d x ,d y is the quantized value from the image coordinate system to the pixel coordinate system, u0 and v0 are the coordinates of the optical center in the pixel coordinate system, and (u, v) represents the coordinate point (x c ,y c ,z c ) to the coordinates in the pixel coordinate system; z c That is, the predicted depth corresponding to (u, v) in the depth prediction map; By z c , u, v into the camera imaging model to calculate (x c ,y c ,z c ); c. Determine the scaling ratio μ between the depth prediction map and the point cloud data; For the depth prediction map, according to the camera imaging model, first generate a 3D scene model in the camera coordinate system, calculate the FPFH 3D key point features, calculate the 3D point cloud converted to the FPFH 3D key point features in the camera coordinate system, perform feature point matching, and obtain a series of feature matching pairs. Calculate the ratio of the depth difference of each pair of feature matching pairs to the corresponding depth prediction of the FPFH 3D key point pair, and take the average value as the scaling ratio μ; d. Adjust the depth prediction map according to the scaling ratio; The new prediction graph J is: J=μ*I e. According to the principle of closest distance, the depth prediction of each pixel is searched for the nearest point cloud data, and the depth prediction of each pixel is adjusted; Assume that the camera coordinates corresponding to each pixel are (x J ,y J ,z J ), the radar is converted to the camera coordinates (x L ,y L ,z L ), the distance between them is For each pixel’s depth prediction, find the nearest point cloud data point and adjust its depth prediction value based on the point cloud data point. Suppose the camera coordinates corresponding to the nearest point cloud are (x J ,y J ,z J ), adjust the depth prediction of this point to z J .

6. A method for constructing a substation model based on laser point cloud and visual data according to claim 5, characterized in that: Obtain 3D model data of substation equipment based on calibrated substation site depth prediction map; Specifically include: a. Use the random sampling consensus (RANSAC) method to extract standard geometric shapes; b. Target detection determines the pixel location of the device; The device detected by the target detection algorithm is the upper left 2D position and the 2D position of the upper right corner c represents the category, i represents the i-th device target; c. Build a 3D model of the device based on the device model library and target pixel position extraction; According to the projection direction of the target frame, the process of selecting the scene area where the segmented device is located is as follows: The upper left corner of the device target detection box is relative to Z c Vertical angle of the axis Angle with horizontal for The lower right corner of the device's target detection box is relative to Z c Vertical angle of the axis Angle with horizontal for A point P in the world coordinate system iW (x iW ,y iW ,z iW ) The XYZ coordinates of the camera coordinate system after coordinate conversion are P ic (x ic ,y ic ,z ic ), the point relative to Z c The vertical angle and horizontal angle of the axis are Vertical angle of view of the target With horizontal viewing angle β i The ranges are and Select the frame to split and satisfy the vertical viewing angle Vertical viewing angle of the target With horizontal viewing angle β i Horizontal viewing angle of the target The point cloud, that is, the scene area where the device is located is Based on the model library of the device, the standard geometric shape of the device in the scene area where the device is located is extracted by the combination box selection segmentation, so as to determine the three-dimensional model of the device.

Images