An adaptive segmentation system and method for substation scenarios

Abstract

The present invention discloses an adaptive segmentation system and method for substation scenarios. The adaptive segmentation system includes a substation inspection robot, a server, a data preprocessing module, an error judgment module, a clustering module, a pose specification conversion module, and a data filling module; the method of the present invention processes point cloud data, simplifies the difficulty of online calibration to a great extent through Euclidean distance, enhances real-time performance and robustness, and provides new ideas and methods for online self-calibration in substation environments. In combination with the point cloud data in substation scenarios, a flexible scene constraint is proposed to cope with the change of the external camera parameters. In a two-stage three-dimensional point cloud segmentation framework, the traditional Euclidean algorithm and deep learning are combined to reduce the amount of point cloud data, accelerate the segmentation speed and accuracy of substation scenarios, shorten the system operation time, and improve real-time performance and robustness. And pose conversion is performed to fill the data, making subsequent robot navigation more accurate.

Description

Technical Field

[0001] The present invention relates to the fields of point cloud segmentation and adaptive calibration, and specifically relates to a method for point cloud segmentation of electrical equipment and adaptive calibration of a camera in a substation scenario. Background Art

[0002] In today's society, as an important infrastructure of the country, whether the equipment of the smart grid can safely carry out power production and transmission and maintain the efficiency and stability of daily power supply is of great significance. Based on extracting the outer contour of substation equipment, and using the extracted equipment to segment and screen the equipment by borrowing three-phase lines. At the same time, according to the position and attitude information obtained by PNP, it is possible to clearly know the pose of the robot in the substation equipment and the parallel relationship with the three-phase lines, and further perform on-line calibration of the camera for scene constraints. According to the known equipment model and its spatial position relationship, re-estimate the existing pose of the robot, and establish a model reconstruction of the existing substation, fill the model of the overhead equipment, so as to truly reflect the specific framework and distribution of the substation equipment, and facilitate the obstacle avoidance function of the robot under the hollow equipment during navigation. At the same time, provide more accurate data according to the update and reconstruction of the substation in real time.

[0003] In a substation, the types of electrical equipment are relatively scattered, so the adaptive segmentation and model reconstruction of existing equipment are key technologies for subsequent navigation, and their recognition comes from camera point cloud data. Three-dimensional data of the outer shape of substation equipment is obtained through a depth camera. Due to the hollow characteristics of substation equipment, when using an RGB-D camera to obtain point cloud data, the near-infrared laser inside the depth camera projects onto the hollow equipment, and only the main equipment and lines return information. This results in inaccurate depth information, and the equipment is classified and recognized using the known outer shape characteristics of the substation equipment. At the same time, realizing the adaptive segmentation and recognition of substation equipment plays a fundamental supporting role in the large-scale positioning accuracy of the substation.

[0004] Visual ranging uses the pictures obtained by a camera to obtain depth information. If classified according to the measurement principle, it can be divided into two methods: known object and known motion state. Assume that we perform ranging when the object information is known, and use the data obtained by the camera to further infer the depth information. Assume that in the initial calibration, the camera focal length is F, a target object with a length of L is placed in front of the camera, and the pixel length of the object after the camera takes a picture is P. We can calculate the distance D through the following formula:

[0005] D = (F × L) / P

[0006] In this special environment of a substation, using visual ranging for non-contact ranging and providing relatively accurate poses for inspection robots can effectively solve the problem of accurate ranging of black objects and thin strips on overhead equipment.

[0007] Performing regular inspections on the equipment in a substation is an important guarantee for ensuring the daily operation of the substation. Using manual inspections to monitor the status, pose, and faults of equipment is still the mainstream method in current substations. However, there are still some inherent difficulties and problems that cannot be avoided during manual inspections:

[0008] (1) Manual inspections are highly subjective, mainly relying on experience for operations. The work efficiency is low, and there are also varying degrees of differences in the detection quality and data. Moreover, when the substation is operating and faulty, certain radiation will be generated, which will also cause certain damage to the human body, being unfavorable to human health and posing a high risk.

[0009] (2) Substations cover a large area, and most of the equipment is widely distributed. The manual operation intensity is high, data management is decentralized, and due to distance reasons, real-time transmission of information cannot be achieved, resulting in low accuracy.

[0010] Therefore, precise positioning, adaptive segmentation, and filling are the top priorities for gradually improving the functions of inspection robots and giving full play to their advantages. It is also beneficial for subsequent substation model construction and precise navigation, which can not only reduce the work intensity of employees and manpower, but also comprehensively improve the real-time nature of substation data and centralized data management.

[0011] The internal and external parameters of the camera during initialization offline calibration are prerequisites for the reconstruction algorithm. Zhang Zhengyou calibration method calibrates at least two different views of a planar calibration pattern. However, during the operation of the substation, camera jitter and cumulative errors cause parameter changes, resulting in a decline in accuracy. Ordinary offline calibration cannot meet the accuracy requirements, and in severe cases, it may cause the robot to collide, leading to substation accidents. Self-calibration selects the three-phase lines of the transformer in the substation for constraint to meet online camera calibration. And the point cloud collected by the camera is adaptively segmented to form each independent subset, and each subset can be understood as substation equipment with actual physical significance. Point cloud segmentation is to ensure filling of the subsets during robot navigation to further meet the obstacle avoidance requirements.

[0012] Point cloud segmentation is the essence of processing point cloud data and reflects the greatest advantage of 3D images over 2D images. The earliest solution was to perform point cloud segmentation based on the Euclidean algorithm. This method is efficient enough based on color, but the segmentation results are not ideal in the textureless environment and under lighting conditions in the substation environment. In the aspect of image semantic segmentation, some scholars have tried to project 3D point clouds into 2D images and then use mature supervised learning methods (convolutional neural networks) to assign corresponding semantic labels to pixels. However, this approach ignores the geometric information of 3D point clouds, resulting in poor algorithm performance. However, the camera and lidar fusion algorithm can effectively solve these problems. For the detection in the 2D space of robot navigation, it will fail in some special cases, while it is the most convenient solution in the 3D point cloud data space. However, when dealing with a large number of images and point cloud inputs, 3D point clouds generally run slowly and the real-time performance decreases. Summary of the Invention

[0013] In order to gradually improve the technology and functions of the substation inspection robot in the smart grid to ensure the normal operation of the substation, and also for the subsequent better realization of the model reconstruction classification of the substation and the precise navigation of the robot, the present invention solves the problems of self-calibration using the Euclidean distance and adaptive segmentation of point cloud data in the substation scene, and provides a method for improving the accuracy and substation 3D point cloud segmentation.

[0014] In order to achieve the goal under the conditions of real-time and rapidity, we equip the inspection robot with an RGB-D camera and an IMU, and mainly use the camera to obtain the point cloud data of the substation. Since the distances between different transformers and the three-phase distances of the same transformer in the substation vary greatly, the scene constraint calibration can be carried out through the known three-phase distances under the same transformer to further optimize the external parameter matrix, and the existing point cloud is adaptively segmented using this threshold. Further, to ensure rapidity, first, the original point cloud can be preprocessed to remove textureless point clouds such as the ground and walls, and then the effective point cloud is further clustered and optimized to determine the point cloud clusters. Then, the point cloud is trained to divide the substation equipment, and the point cloud clusters are filled with data, thereby realizing the segmentation of the point cloud map. The key idea of the present invention is to use the parallel line distances of the three phases of the transformer and other equipment to constrain the external parameters of the camera for self-calibration, and to segment the entire substation scene with this, and the clustered point cloud is designated as a labeled cluster, that is, each electrical equipment in the substation. Because we found that each electrical equipment in the substation scene is relatively independent in 3D space, so under the condition of known parallel lines, the point cloud after removing textureless areas such as the ground is relatively independent. Therefore, the present invention believes that it is feasible and effective to perform self-calibration by determining the parallel line threshold, filtering out the point cloud without textureless areas, and then performing clustering division and filling the point cloud to represent each electrical equipment, which is also beneficial to the subsequent real-time, rapid and accurate realization of the navigation of the inspection robot and the reconstruction of the substation model.

[0015] An adaptive segmentation system for substation scenarios, comprising a substation inspection robot, a server, a data preprocessing module, an error judgment module, a clustering module, a pose specification conversion module, and a data filling module.

[0016] An RGB-D camera, a Robot Operating System (ROS), and a data transmission module are provided on the substation inspection robot; the substation inspection robot scans each electrical equipment in the substation through the equipped RGB-D camera to obtain three-dimensional point cloud data; the RGB-D camera is calibrated offline through the Robot Operating System (ROS), the real three-dimensional space points are calculated using the internal parameters of the RGB-D camera, and then they are converted to the PCD format. Finally, the processed substation scene point cloud data is transmitted to the server through the data transmission module.

[0017] The error judgment module, the data preprocessing module, the clustering module, and the pose specification conversion module are set on the server, and the obtained substation scene point cloud data is processed through the server.

[0018] The error judgment module is used to judge whether the difference between the projection and reprojection of the real three-dimensional space points on the image plane exceeds a set threshold. When the difference is greater than the set threshold, self-calibration operation is performed, and the substation scene point cloud data is obtained again through the substation inspection robot.

[0019] The data preprocessing module is used to perform a filtering operation on the substation scene point cloud data judged by the error judgment module to remove the ground textureless point cloud.

[0020] The clustering module uses the Euclidean distance to perform point cloud clustering on the preprocessed substation scene point cloud data according to the adaptive density.

[0021] The pose specification conversion module is used to perform pose specification conversion on the clustering result, reduce the deviation caused by coordinate transformation, and fit the three-dimensional contour corresponding to each class of point cloud.

[0022] The data filling module is used to fill the data of the three-dimensional contour fitted by the pose specification conversion module.

[0023] Further, the self-calibration operation of the error judgment module is as follows:

[0024] First, select a set of parallel lines with known intervals from each other and any straight line perpendicular to and intersecting the parallel lines in the scene as the calibration target. In the actual scene, a three-phase transformer can be selected. Straight lines a, b, and c respectively represent three parallel lines, and straight line d intersects them, with the intersection points being A, B, and C. a', b', c', and d' are the projections of straight lines a, b, c, and d on the image plane respectively, and O is the camera position. Establish a coordinate system. Assume that the origin is at the intersection point B of the parallel line b and the straight line d, the positive direction of the X-axis is horizontally to the right, the positive direction of the Y-axis points straight ahead along the parallel line b, and the Z-axis is perpendicular to the ground upwards. Through the intervals between adjacent parallel lines of a, b, and c, the coordinates of the intersection points of the intersecting straight lines in space, and the camera focal length known in the offline calibration, obtain the slope and general equation of the corresponding vanishing line, and find the coordinates of the projection straight line of the intersecting straight line on the image plane and the vanishing point coordinates, that is, the coordinates of the vanishing point corresponding to the intersecting straight line on the image plane. Finally, complete the calibration by obtaining the rotation angle, declination angle, pitch angle of the camera, and the three-dimensional position of the camera center from the intervals between adjacent two straight lines of the three parallel lines, the intersection point coordinates of the intersecting straight line and the three parallel lines in space and on the image plane, and the projection and vanishing point coordinates of the intersecting straight line.

[0025] Furthermore, the specific operations of the data preprocessing module are as follows:

[0026] First, perform a preliminary filtering operation on the overall substation scene. Since the textureless environment of the ground is a plane and the depth information in RGB-D is easily recognizable, assume that the lowest point cloud data on the ground belongs to the ground, and express the ground through a simple mathematical model. Due to the visual error caused by the long-distance acquisition of the camera, divide the scene into N parts along the advancing direction of the inspection robot, extract the lowest point cloud data for each part and fit the ground model. By comparing with the known height of the RGB-D camera and the set threshold, determine whether each point belongs to the ground point. Continuously calculate the points belonging to the plane in the N parts in a loop, and iteratively update the plane of the entire substation, and then remove the ground point cloud data from the substation scene point cloud data.

[0027] Furthermore, the specific operations of the clustering module are as follows:

[0028] In the data after removing the ground point cloud, first calculate the Euclidean distance between pairwise feature points according to the following formula, and calculate the average value e of the Euclidean distances of the feature points.

[0029]

[0030] Set the average value e as the initial radius, and calculate the number of points within the neighborhood with the origin of the image physical coordinate system as the center and the initial radius of e. Calculate the number of points within the neighborhood by gradually increasing the radius e. As the radius e increases, the number of points within the neighborhood will also increase, and the increase in the amount of point data gradually decreases and finally stabilizes. Select the radius e at the inflection point during the increase in the number of points within the neighborhood.1 Perform further calculations: Arbitrarily select a point p in the point cloud data 1 Take it as the center and perform a search operation to find points within the radius e 1 in the neighborhood, and let them be a class K. Then arbitrarily select a point p within K 2 Take it as the center and repeat the search operation. Search all points in class K until no new points are added to class K, then stop the search. Then further select any point D outside class K 1 Take it as the center and continue the above operations. Finally, cluster the point cloud without the ground into several parts and output multiple sets of connected regions.

[0031] Furthermore, the specific operations of the pose specification conversion module are as follows:

[0032] Further perform pose conversion on the clustered results. Since the camera rotates and translates during the scene scanning, and the devices are distributed at various positions in the scene, the coordinate changes of the targets are different. Considering such factors, first perform operations of rotating 90°, 180°, and 270° on the samples of each class after clustering to create another 3 samples. Due to the irregularity of the devices, there is a certain possibility that the contours of all four samples exist in the real world. Therefore, input these four types of samples into RANSAC together, and use RANSAC to fit the three-dimensional contours corresponding to each class of point cloud.

[0033] The functions achieved by the present invention are as follows: Through a two-stage preprocessing of point cloud data, use the Euclidean distance to perform scene constraint calibration, and then perform an adaptive segmentation framework. First, perform an operation to remove textureless points on the original point cloud data of the substation, further quickly cluster to obtain point cloud clusters, then perform pose changes and data filling on the point cloud clusters, and divide the segmented point cloud clusters to achieve substation equipment through supervised learning. The specific implementation steps of the method are as follows:

[0034] Step 1: Obtain the substation scene point cloud data through a substation inspection robot and transmit it to the server;

[0035] The substation inspection robot end scans each electrical equipment in the substation through the equipped RGB-D camera to obtain three-dimensional point cloud data. The three-dimensional point cloud data consists of spatial three-dimensional coordinate values, depth, and color information. First, use the Kalibr tool in the Robot Operating System (ROS) to perform offline calibration on the RGB-D camera, calculate the real three-dimensional space points using the internal parameters of the RGB-D camera, and then convert it to the PCD format due to the discrete and sparse structure of the point cloud data. Finally, transmit the processed substation scene point cloud data to the server through the data transmission module.

[0036] Step 2: Perform error judgment through the error judgment module. If it meets the error range, proceed to Step 3; if not, perform self-calibration of the external parameters of the scene constraint and return to Step 1;

[0037] Electrical equipment in the substation scenario is independent of each other in three-dimensional space. Therefore, the electrical equipment in the substation scenario is significantly separated in three-dimensional Euclidean space. Due to the external camera parameter transformation and calculation errors, the difference between the projection and reprojection of real three-dimensional space points on the image plane, that is, the reprojection pixel difference, cannot be exactly 0. When the reprojection error is less than or equal to 0.8 pixels, step 3 is performed. If the reprojection error is greater than 0.8 pixels, self-calibration operation is performed and step 1 is returned.

[0038] The self-calibration steps are as follows: First, select a group of parallel lines with known intervals from each other and any straight line perpendicular to and intersecting the parallel lines in the scene as the calibration target. In the actual scene, a three-phase transformer can be selected. Straight lines a, b, and c respectively represent three parallel lines, and straight line d intersects them at intersections A, B, and C. a', b', c', and d' are the projections of straight lines a, b, c, and d on the image plane, and O is the camera position. Establish a coordinate system. Assume that the origin is at the intersection B of parallel line b and straight line d. The positive direction of the X-axis is horizontally to the right, the positive direction of the Y-axis points straight ahead along parallel line b, and the Z-axis is above the ground vertically. Through the distances between adjacent parallel lines of a, b, and c, the coordinates of the intersection points of the intersecting straight lines in space, and the known camera focal length in offline calibration, the slope and general equation of the corresponding vanishing line are obtained, and the coordinates of the projection line of the intersecting straight lines on the image plane and the vanishing point are calculated, that is, the coordinates of the vanishing point corresponding to the intersecting straight lines on the image plane. Finally, the rotation angle, declination angle, pitch angle of the camera and the three-dimensional position of the camera center are obtained from the distances between adjacent two straight lines of the three parallel lines, the intersection coordinates of the intersecting straight line and the three parallel lines in space and on the image plane, the projection of the intersecting straight line and the vanishing point coordinates to complete the calibration.

[0039] Step 3: The data preprocessing module performs data preprocessing on the substation scene point cloud data passed through the error judgment, and removes the point cloud at the textureless ground;

[0040] First, perform a preliminary filtering operation on the overall substation scene. Since the textureless environment of the ground is a plane, the depth information in RGB-D is easily recognizable. Assume that the lowest point cloud data on the ground belongs to the ground, and express the ground through a simple mathematical model. Due to the visual error caused by the long-distance acquisition of the camera, the scene is divided into N parts along the forward direction of the inspection robot. The lowest point cloud data of each part is extracted and the ground model is fitted. By comparing with the known height of the RGB-D camera and the set threshold, it is judged whether each point belongs to the ground point. And continuously calculate the points belonging to the plane in the N parts and perform iterative update on the plane of the entire substation, and then remove the ground point cloud data from the substation scene point cloud data.

[0041] Step 4: The clustering module performs point cloud clustering according to the Euclidean distance and adaptive density;

[0042] In the substation scene point cloud data after removing the ground point cloud, first calculate the Euclidean distance between pairwise feature points according to the following formula, and calculate the average value e of the Euclidean distances of the feature points.

[0043]

[0044] Set the average value e as the initial radius, and calculate the number of points within the neighborhood with the origin of the image physical coordinate system as the center and the initial radius of e. Calculate the number of points within the neighborhood by gradually increasing the radius e. As the radius e increases, the number of points within the neighborhood will also increase, and the increase in the amount of point data gradually decreases and finally stabilizes. Select the radius e at the inflection point during the increase in the number of points within the neighborhood. 1 Perform further calculations: Arbitrarily select a point p in the point cloud data. 1 Search with p as the center to find the points within the neighborhood with a radius of e. 1 Let them be a class K, and arbitrarily select a point p within K. 2 Repeat the search operation with p as the center, and search all the points in class K until no new points are added to class K, then stop the search. Then further select any point D outside class K. 1 Search with D as the center and continue the above operation. Finally, cluster the point cloud without the ground into several parts and output multiple sets of connected regions.

[0045] Step 5: Perform pose normalization transformation on the clustering result through the pose normalization transformation module to reduce the deviation caused by coordinate transformation, and fit the three-dimensional contour corresponding to each class of point cloud.

[0046] Perform pose transformation on the clustering result. Since there are rotations and translations during the camera scanning in the scene, and at the same time the devices are distributed at various positions in the scene, the coordinate changes of the targets are different from each other. Considering such factors, first perform operations of rotating 90°, 180°, and 270° on each class of samples after clustering to create another 3 samples. Due to the irregularity of the devices, the contours of all four samples have a certain possibility in the real world. Therefore, input these four types of samples into RANSAC together, and fit the three-dimensional contour corresponding to each class of point cloud through RANSAC.

[0047] Step 6: After filling the data of the three-dimensional contour through the data filling module, send the result back to the substation inspection robot.

[0048] Further perform a filling operation on the three-dimensional contour box, treat the hollow devices in the substation as solid objects, and then in the local navigation planning, regard the filled hollow devices in the substation as a whole obstacle for path planning. Finally, the server transmits the processing result to the substation inspection robot to achieve the obstacle avoidance function.

[0049] Advantages of the present invention:

[0050] Innovation point 1: After scanning the substation scene by a camera and processing the point cloud data, the online calibration difficulty is greatly simplified through the Euclidean distance. Different from traditional camera calibration, online calibration can strengthen real-time performance and robustness under the condition of calibration without sufficient data support and camera jitter, providing a new idea and method for online self-calibration in the substation environment. At the same time, it also lays a foundation for substation model reconstruction and navigation.

[0051] Innovation point 2: Through a data preprocessing framework, combining the traditional Euclidean distance and advanced artificial neural networks reduces the subsequent point cloud data volume, speeds up the substation scene segmentation speed, greatly shortens the running time of the entire system, and strengthens real-time performance and robustness.

[0052] Innovation point 3: Considering the Euclidean distance of equipment in the substation scene, an adaptive clustering and segmentation is proposed, and further pose transformation is carried out, which can effectively alleviate the coordinate deviation in three-dimensional space, improve the network performance. It is beneficial to solve the extraction of the contour of empty equipment in the substation, reduce the backend calculation amount, speed up the system processing time, and can effectively improve the subsequent navigation and obstacle avoidance accuracy.

[0053] The method of the present invention combines the point cloud data in the substation scene to propose a flexible scene constraint to cope with the change of the external camera parameters. In the two-stage three-dimensional point cloud segmentation framework, it combines the traditional Euclidean algorithm and deep learning to reduce the point cloud data volume, speed up the substation scene segmentation speed and accuracy, shorten the system running time, improve real-time performance and robustness. And pose transformation is carried out to fill the data, making the subsequent robot navigation more accurate. Brief description of the drawings

[0054] Figure 1 It is the implementation flowchart of the present invention.

[0055] Figure 2 It is the schematic diagram of the calibration target projection image and the vanishing line in the present invention.

[0056] Figure 3 It is the schematic diagram of the calibration target coordinate system relationship in the present invention. Detailed implementation manners

[0057] The present invention will be described in detail below with reference to the drawings, and the purpose and effect of the present invention will become more obvious.

[0058] For a better understanding of the present invention, we will define or explain the following rules:

[0059] 1. Point cloud data

[0060] Point cloud is a data set of points in space. Point cloud data refers to the data obtained by the scanner recorded in the form of points. Each point contains three-dimensional coordinates (XYZ), and some may contain color information (RGB) or reflection intensity information (Intensity). RGB-D camera data generally uses 3D format. Each pixel of its collected image has four attributes: red (R), green (G), blue (B) and depth (D). We can locate any pixel by coordinates and obtain four attributes respectively. The acquisition of intensity information is the echo intensity collected by the laser scanner receiving device. This intensity information is related to the surface material, roughness, incident angle direction of the target, as well as the emission energy of the instrument and the laser wavelength. Because each point cloud has spatial coordinate information, they all have measurement capabilities. Secondly, the point cloud itself is independent of the viewing angle and can be rotated arbitrarily. Point clouds observed from different angles and directions can be directly fused when transferred to the same coordinate system. Two points form a line, three points form a surface, and four points form a body. Through these point clouds, the coordinate information of a point in a three-dimensional scene can be determined, and the length, area, volume, angle and other information can be further calculated. Point cloud data is often also used to create digital elevation models or for three-dimensional modeling.

[0061] 2. Euclidean distance / metric

[0062] Euclidean distance is also called Euclidean distance. It is usually defined as a measure of the real distance between two points in m-dimensional space, or the natural length of a vector (i.e., the distance from the point to the origin), while the Euclidean distance in two-dimensional and three-dimensional space is expressed as the actual distance between two points. In the scenario of calculating similarity (such as face recognition), Euclidean distance is a relatively intuitive and common similarity algorithm. The smaller the Euclidean distance, the greater the similarity; the larger the Euclidean distance, the smaller the similarity. The so-called Euclidean distance transform refers to the conversion of the value of the pixel in the foreground of a binary image (we assume that white is the foreground color and black is the background color) into the distance from the point to the nearest background point. Euclidean distance transform has a wide range of applications in digital image processing, especially for image skeleton extraction, it is a good reference.

[0063] 3. Vanishing point self-calibration

[0064] When we use cameras to implement interactive scenes such as 3D reconstruction or virtual reality, we need to calibrate the camera, or call it calibration. Common tasks such as 3D reconstruction, target detection, scene mapping, object reconstruction, or self-localization all require scene calibration. Simply capturing images is not enough. Explicit camera calibration means that the calibration process ends with a set of physical parameters, obtaining a detailed model that is as close to a complete description of the real system as possible.

[0065] The camera converts the 3D scene into a 2D image through perspective projection transformation. In projective transformation, parallel lines intersect at a point called the vanishing point, and the vanishing line is the line composed of vanishing points in the horizontal (numerical) direction. The line connecting the vanishing point and the optical center is parallel to the parallel lines of the plane forming the vanishing point. Therefore, for parallel lines that are perpendicular to each other in space, the lines connecting the corresponding vanishing points and the camera optical center are perpendicular to each other.

[0066] For the purpose of giving full play to the advantages of substation inspection robots and maintaining the efficient operation of inspection work, and for better completion in the follow-up, self-calibration, model reconstruction, and precise navigation and obstacle avoidance under the constraints of the substation scene camera. The present invention solves the problem of adaptively segmenting the substation scene using RGB-D point cloud data, and provides a method for segmenting the point cloud of the substation scene. The implementation flowchart is as Figure 1 shown.

[0067] Taking the 500KV substation scene as an example, the specific implementation steps are as follows:

[0068] Step (1):

[0069] The substation inspection robot scans each electrical equipment in the 500KV substation through the equipped RGB-D camera to obtain three-dimensional point cloud data. The three-dimensional point cloud data consists of spatial three-dimensional coordinate values, depth, and color information. First, we use the Kalibr tool in the Robot Operating System (ROS) to perform offline calibration on the RGB-D camera, and use the internal parameters of the RGB-D camera to calculate real three-dimensional space points. Since the point cloud data is in a discrete and coefficient structure, it is then converted to the PCD format. After that, the substation inspection robot transmits the processed 500KV substation scene point cloud data to the corresponding server side for subsequent processing of a large amount of point cloud data.

[0070] Step (2):

[0071] The electrical equipment in the 500KV substation scene is independent of each other in three-dimensional space. Therefore, the electrical equipment in the 500KV substation scene is significantly separated in three-dimensional Euclidean space. Due to the external camera transformation and calculation errors, the difference between the projection and reprojection of real three-dimensional space points on the image plane, that is, the reprojection pixel difference, cannot be exactly 0. When the reprojection error is less than or equal to 0.8 pixels, step (3) is performed. If the reprojection error is greater than 0.8 pixels, self-calibration operation is performed and step (1) is returned.

[0072] The self-calibration steps are as follows: First, select a group of parallel lines with known intervals from each other and any line perpendicular to the parallel lines intersecting as the calibration target in the scene. In the actual scene, a three-phase transformer can be selected. As Figure 2As shown, the straight lines a, b, and c respectively represent three parallel lines. The straight line d intersects with them, and the intersection points are A, B, and C. The projections of the straight lines a, b, c, and d on the image plane are a’, b’, c’, and d’ respectively, and O is the camera position. As Figure 3 Establish a coordinate system. Assume that the origin is at the intersection point B of the parallel line b and the straight line d. The positive direction of the X-axis is horizontally to the right, the positive direction of the Y-axis points straight ahead along the parallel line b, and the Z-axis is perpendicular to the ground upward. Through the spacing between adjacent parallel lines a, b, and c, the coordinates of the intersection points of the intersecting straight lines in space, and the camera focal length known in the off-line calibration, obtain the slope and general form equation of the corresponding vanishing line, and find the projection straight line of the intersecting straight line on the image plane and the vanishing point coordinates, that is, the coordinates of the vanishing point corresponding to the intersecting straight line on the image plane. Finally, the rotation angle, declination, pitch angle of the camera and the three-dimensional position of the camera center are obtained from the spacing between two adjacent straight lines of the three parallel lines, the intersection point coordinates of the intersecting straight line and the three parallel lines in space and on the image plane, and the projection and vanishing point coordinates of the intersecting straight line to complete the calibration.

[0073] Step (3):

[0074] Perform a filtering operation on the acquired 500KV substation three-dimensional point cloud data to remove the ground textureless point cloud.

[0075] First, perform a preliminary filtering operation on the overall scene of the 500KV substation. Since the textureless environment of the ground is a plane, the depth information in RGB-D is easily recognizable. Assume that the lowest point cloud data on the ground belongs to the ground, and express the ground through a simple mathematical model. Due to the visual error caused by the long-distance acquisition of the camera, divide the scene into N parts along the advancing direction of the inspection robot, extract the lowest point cloud data for each part and fit the ground model. By comparing with the known height of the RGB-D camera and the set threshold, determine whether each point belongs to the ground point. Continuously calculate the points belonging to the plane in the N parts in a loop, and iteratively update the plane of the entire 500KV substation, and then remove the ground point cloud data from the 500KV substation scene point cloud data.

[0076] Step (4):

[0077] When classifying each electrical equipment, it is not necessary to consider all the point clouds in the scene. Since removing the ground points makes the equipment in the scene independent of each other in the three-dimensional Euclidean space, the number of points involved is greatly reduced. In the 500KV substation scene point cloud data after removing the ground point cloud, first calculate the Euclidean distance between two pairwise feature points according to the following formula, and calculate the average value e of the Euclidean distance of the feature points.

[0078]

[0079] Set the mean value e as the initial radius, and calculate the number of points within the neighborhood with the origin of the image physical coordinate system as the center and the initial radius e. Calculate the number of points within the neighborhood by gradually increasing the radius e. As the radius e increases, the number of points within the neighborhood also increases, and the increase in the point data gradually decreases and finally stabilizes. Select the radius e at the inflection point during the increase in the number of points within the neighborhood. 1 Perform the following calculation: Arbitrarily select a point p from the point cloud data. 1 As the center, perform a search operation to find the points within the neighborhood with a radius of e. 1 Let them be a class K, and arbitrarily select a point p within K. 2 As the center, repeat the search operation, and perform the search for all points in class K until no new points are added to class K, then stop the search. Then further select any point D outside class K. 1 As the center, continue the above operations. Finally, cluster the point cloud without the ground into several parts and output multiple connected regions.

[0080] Step (5):

[0081] Further perform pose transformation on the clustering results. Since there are rotations and translations during the camera scanning in the scene, and at the same time, the devices are distributed at various positions in the scene, the coordinate changes of the targets are different. Considering such factors, first perform operations of rotating 90°, 180°, and 270° on each class of samples after clustering to create another 3 samples. Due to the irregularity of the devices, there is a certain possibility that the contours of all four samples exist in the real world. Therefore, input these four types of samples into RANSAC together. In this way, to a certain extent, it reduces the negative impact brought by coordinate transformation and deviation, making the network pay more attention to the relative pose of points, and fitting out the corresponding three-dimensional contours of each class of point cloud through RANSAC.

[0082] Step (6):

[0083] Further perform a filling operation on the three-dimensional contour box. Treat the hollow devices in the 500KV substation as solid objects, and then in the local navigation planning, regard the filled hollow devices in the substation as a whole obstacle for path planning. Finally, the server transmits the processing result to the substation inspection robot to achieve the obstacle avoidance function.

Claims

1. An adaptive segmentation system for substation scenarios, characterized in that, it includes a substation inspection robot, a server, a data preprocessing module, an error judgment module, a clustering module, a pose normalization conversion module, and a data filling module; The substation inspection robot is equipped with an RGB-D camera, a robot operating system, and a data transmission module; the substation inspection robot scans various electrical equipment in the substation through the equipped RGB-D camera to obtain three-dimensional point cloud data; the RGB-D camera is calibrated offline through the robot operating system, the true three-dimensional space points are calculated using the internal parameters of the RGB-D camera, then it is converted to the PCD format, and finally the processed substation scene point cloud data is transmitted to the server through the data transmission module; The error judgment module, the data preprocessing module, the clustering module, and the pose normalization conversion module are set on the server, and the server processes the obtained substation scene point cloud data; The error judgment module is used to judge whether the difference between the projection and reprojection of the true three-dimensional space points on the image plane exceeds a set threshold. When the difference is greater than the set threshold, self-calibration operations are performed, and the substation inspection robot is used to re-obtain the substation scene point cloud data; The data preprocessing module is used to perform filtering operations on the substation scene point cloud data judged by the error judgment module to remove the ground textureless point cloud; The clustering module uses the Euclidean distance to perform point cloud clustering on the preprocessed substation scene point cloud data according to the adaptive density; The pose normalization conversion module is used to perform pose normalization conversion on the clustering results, reduce the deviation caused by coordinate transformation, and fit the three-dimensional contour corresponding to each class of point cloud; The data filling module is used to fill the data of the three-dimensional contour fitted by the pose normalization conversion module; The specific operation of the clustering module is as follows: In the data after removing the ground point cloud, first calculate the Euclidean distance between two-by-two feature points according to the following formula, and calculate the average value e of the Euclidean distance of the feature points; Set the mean value e as the initial radius, and calculate the number of points within the neighborhood with the origin of the image physical coordinate system as the center and the initial radius e; calculate the number of points within the neighborhood by gradually increasing the radius e. As the radius e increases, the number of points within the neighborhood will also increase, and the increase in the amount of point data gradually decreases and finally stabilizes; select the radius e at the inflection point during the increase in the number of points within the neighborhood. 1 Perform further calculations: randomly select a point p from the point cloud data. 1 As the center, perform a search operation to find the points within the neighborhood with a radius of e. 1 Let them be a class K, and randomly select a point p within K. 2 As the center, repeat the search operation, and perform the search for all points in class K until no new points are added to class K, then stop the search; then further select any point D outside class K. 1 As the center, continue the above operations; finally, cluster the point cloud without the ground into several parts and output multiple connected regions.

2. The adaptive segmentation system for substation scenarios according to claim 1, characterized in that, The self-calibration operation of the error judgment module is as follows: First, select a set of parallel lines with known intervals from each other and an arbitrary line perpendicular to the parallel lines in the scene as the calibration target. In the actual scene, a three-phase transformer can be selected; lines a, b, and c respectively represent three parallel lines, line d intersects them, and the intersection points are A, B, and C. a', b', c', and d' are the projections of lines a, b, c, and d on the image plane, and O is the camera position; establish a coordinate system. Assume that the origin is at the intersection point B of the parallel line b and the line d, the positive direction of the X-axis is horizontally to the right, the positive direction of the Y-axis points straight ahead along the parallel line b, and the Z-axis is perpendicular to the ground upwards; through the distances between adjacent parallel lines of a, b, and c and the coordinates of the intersection points of the intersecting lines in space, as well as the known camera focal length in the off-line calibration, obtain the slope and general equation of the corresponding vanishing line, and find the projection line of the intersecting line on the image plane and the vanishing point coordinates, that is, the coordinates of the vanishing point corresponding to the intersecting line on the image plane; finally, obtain the rotation angle, yaw angle, pitch angle of the camera and the three-dimensional position of the camera center from the distances between adjacent two lines of the three parallel lines, the intersection point coordinates of the intersecting line and the three parallel lines in space and on the image plane, the projection of the intersecting line and the vanishing point coordinates to complete the calibration.

3. An adaptive segmentation system for a substation scene according to claim 2, characterized in that the specific operations of the data preprocessing module are as follows: First, perform a preliminary filtering operation on the overall substation scene. Since the textureless environment of the ground is a plane, the depth information in RGB-D is easily recognizable. Assume that the lowest point cloud data on the ground belongs to the ground, and express the ground through a simple mathematical model; due to the visual error caused by the long-distance acquisition of the camera, divide the scene into N parts along the forward direction of the inspection robot, extract the lowest point cloud data for each part and fit the ground model; by comparing with the known height of the RGB-D camera and the set threshold, determine whether each point belongs to the ground point; and continuously calculate the points belonging to the plane in the N parts in a loop, and iteratively update the plane of the entire substation, and then remove the ground point cloud data from the substation scene point cloud data.

4. An adaptive segmentation system for a substation scene according to claim 3, characterized in that the specific operations of the pose specification conversion module are as follows: Further perform pose conversion on the clustered results. Since there are rotations and translations during the camera scanning in the scene, and at the same time the devices are distributed at various positions in the scene, the coordinate changes of the targets are different from each other; considering such factors, first perform operations of rotating 90°, 180°, and 270° on each sample after clustering to create another 3 samples. Since the irregularity of the devices, there is a certain possibility that the contours of the total four samples exist in the real world; therefore, input these four types of samples into RANSAC together, and fit the three-dimensional contour corresponding to each type of point cloud through RANSAC.

5. An adaptive segmentation method for a substation scene, characterized in that the specific steps are as follows: Step 1: Obtain the substation scene point cloud data through the substation inspection robot and transmit it to the server; The substation inspection robot scans each electrical equipment in the substation through the equipped RGB-D camera to obtain three-dimensional point cloud data; the three-dimensional point cloud data consists of spatial three-dimensional coordinate values, depth, and color information; first, use the Kalibr tool in the robot operating system to perform offline calibration on the RGB-D camera, and use the internal parameters of the RGB-D camera to calculate real three-dimensional space points. Since the point cloud data is in a discrete and coefficient structure, it is then converted to the PCD format; finally, the processed substation scene point cloud data is transmitted to the server through the data transmission module; Step 2: Perform error judgment through the error judgment module. If it meets the error range, proceed to Step 3; if not, return to Step 1 for self-calibration of the external parameters with scene constraints; The electrical equipment in the substation scene is independent of each other in three-dimensional space, so the electrical equipment in the substation scene is significantly separated in three-dimensional Euclidean space; due to the camera external parameter transformation and calculation errors, the difference between the projection and reprojection of real three-dimensional space points on the image plane, that is, the reprojection pixel difference, cannot be exactly 0; when the reprojection error is less than or equal to 0.8 pixels, proceed to Step 3; if the reprojection error is greater than 0.8 pixels, perform self-calibration operations and return to Step 1; The self-calibration steps are as follows: First, select a set of parallel lines with known intervals from each other and any straight line perpendicular to and intersecting the parallel lines in the scene as the calibration target. In the actual scene, a three-phase transformer can be selected; Lines a, b, and c respectively represent three parallel lines, and line d intersects them at points A, B, and C. The projections of lines a, b, c, and d on the image plane are a', b', c', and d' respectively, and O is the camera position; establish a coordinate system. Assume that the origin is at the intersection point B of the parallel line b and the line d, the positive direction of the X-axis is horizontally to the right, the positive direction of the Y-axis points straight ahead along the parallel line b, and the Z-axis is above the ground perpendicular to it; through the spacing between adjacent parallel lines a, b, and c and the coordinates of the intersection points of the intersecting lines in space, as well as the known camera focal length in the offline calibration, obtain the slope and general equation of the corresponding vanishing line, and find the coordinates of the projection line of the intersecting line on the image plane and the vanishing point, that is, the coordinates of the vanishing point corresponding to the intersecting line on the image plane; finally, obtain the rotation angle, declination angle, pitch angle of the camera and the three-dimensional position of the camera center from the spacing between adjacent two lines of the three parallel lines, the intersection coordinates of the intersecting line and the three parallel lines in space and on the image plane, the projection and vanishing point coordinates of the intersecting line to complete the calibration; Step 3: Perform data preprocessing on the substation scene point cloud data that has passed the error judgment through the data preprocessing module to remove the point cloud at the textureless ground; First, perform a preliminary filtering operation on the overall substation scene. Since the textureless environment of the ground is a plane, the depth information in RGB-D is easily recognizable. Assume that the lowest point cloud data on the ground belongs to the ground, and express the ground through a simple mathematical model. Due to the visual error caused by the long-distance acquisition of the camera, divide the scene into N parts along the advancing direction of the inspection robot, extract the lowest point cloud data for each part, and fit the ground model. By comparing with the height of the known RGB-D camera and the set threshold, determine whether each point belongs to the ground point. Continuously loop and calculate the points belonging to the plane in the N parts, and iteratively update the plane of the entire substation. Then, remove the ground point cloud data from the substation scene point cloud data. Step 4: Use the Euclidean distance to perform point cloud clustering according to the adaptive density through the clustering module. In the substation scene point cloud data after removing the ground point cloud, first calculate the Euclidean distance between pairwise feature points according to the following formula, and calculate the average value e of the Euclidean distance of the feature points. Set the mean value e as the initial radius, and calculate the number of points within the neighborhood with the origin of the image physical coordinate system as the center and the initial radius e; calculate the number of points within the neighborhood by gradually increasing the radius e. As the radius e increases, the number of points within the neighborhood will also increase, and the increase in the point data gradually decreases and finally stabilizes; select the radius e at the inflection point during the increase in the number of points within the neighborhood 1 Perform further calculations: randomly select a point p from the point cloud data 1 As the center, perform a search operation to find the points within the neighborhood with a radius of e 1 Let them be a class K, and randomly select a point p within K 2 As the center, repeat the search operation, and perform the search for all points in class K until no new points are added to class K, then stop the search; then further select any point D outside class K 1 As the center, continue the above operations; finally, cluster the point cloud without the ground into several parts and output multiple sets of connected regions; Step 5: Perform pose normalization transformation on the clustering result through the pose normalization transformation module to reduce the deviation caused by coordinate transformation, and fit the three-dimensional contour corresponding to each type of point cloud. Perform pose transformation on the clustering result. Since there are rotations and translations during the camera scanning in the scene, and the equipment is distributed at various positions in the scene, the coordinate changes of the targets are different from each other. Considering these factors, first perform operations of rotating 90°, 180°, and 270° on each sample of each cluster after clustering to create another 3 samples. Due to the irregularity of the equipment, there is a certain possibility that the contours of the total four samples exist in the real world. Therefore, input these four types of samples into RANSAC together, and fit the three-dimensional contour corresponding to each type of point cloud through RANSAC. Step 6: Fill in the data of the three-dimensional contour through the data filling module and then send the result back to the substation inspection robot. Further perform a filling operation on the three-dimensional contour box, treat the hollow equipment in the substation as a solid object, and then, in the local navigation planning, regard the filled hollow equipment in the substation as an overall obstacle for path planning. Finally, the server transmits the processing result to the substation inspection robot to achieve the obstacle avoidance function.

Images