A method and system for inspection based on laser radar and thermal imaging

Abstract

The application relates to a kind of based on laser radar and inspection method of thermal imaging, comprising: inspection device reaches detection point according to preset route;Inspection device obtains point cloud data at detection point;Point cloud data is projected to depth image;According to the depth distance of pixel point in depth image, the depth distance difference value accumulation value of pixel point and multiple adjacent pixel points, the roughness of pixel point in depth image is calculated;The depth distance is inversely proportional to roughness, and the depth distance difference value accumulation value is proportional to roughness;Roughness threshold is set;The point cloud corresponding to the pixel point with roughness threshold less than roughness threshold is divided into boundary point;Boundary point is clustered, and a plurality of point cloud clusters are obtained;In the plurality of point cloud clusters, the point cloud cluster corresponding to insulator is identified;According to the point cloud cluster corresponding to insulator, the first position of insulator is calculated;According to the first position of the insulator, the temperature of the insulator is obtained by the inspection device.

Description

Technical Field

[0001] This invention relates to an inspection method and system based on lidar and thermal imaging, belonging to the field of power equipment inspection. Background Technology

[0002] In power equipment, high-voltage transmission lines are subject to regular inspections and maintenance due to factors such as natural environmental damage, human-caused disruptions, unexpected operational situations, and equipment aging, ensuring safe operation. In particular, regular inspections are necessary to check for poor contact that could lead to power line faults, focusing on the insulators and nearby circuitry of high-voltage transmission lines. Inspections begin by determining the location of the insulators, then images are acquired using drone-based infrared thermal imaging equipment to assess their heating status.

[0003] Existing technologies use image recognition algorithms to identify and locate insulators, but in practical applications, they are affected by weather, lighting, rain, snow, and dust storms, causing deviations in target identification and making it impossible to obtain the insulator's status. Furthermore, GPS positioning can only determine the approximate location of power line inspection drones, with a positional deviation of 1-2 meters, making it impossible to accurately determine the drone's position and attitude. LiDAR can create point cloud maps for power line inspection, allowing aircraft to precisely navigate to the test points, but the accuracy of insulator identification and location in existing inspection methods needs further improvement.

[0004] The paper "Point Cloud Boundary Point Extraction Based on Depth Image" by Liu Hao et al. discloses a method to determine object boundary points based on the 3D distance difference between sampled points and neighborhood points in a depth image, thereby identifying objects in point cloud data. However, this method has a time complexity of O(n^3), making it difficult to apply in practice. Furthermore, LiDAR acquires a point cloud by rotating it in one direction; while this method assumes a curved surface and calculates distances within the surface's neighborhood, which does not conform to the characteristics of mechanical LiDAR point cloud acquisition, and its accuracy needs further improvement. Summary of the Invention

[0005] To overcome the problems existing in the prior art, this invention designs an inspection method and system based on lidar and thermal imaging. It designs a roughness calculation formula, projects point cloud data onto a depth image, and calculates the roughness based on the depth distance of pixels in the depth image and the cumulative value of the difference between the depth distance of pixels and the depth distance of neighboring pixels, so as to distinguish the boundary points corresponding to the object, thereby identifying the insulator and obtaining the insulator position; the identification accuracy is higher.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An inspection method based on lidar and thermal imaging includes the following steps:

[0008] The inspection device arrives at the inspection point according to the preset route;

[0009] The inspection device acquires point cloud data at the detection points;

[0010] Identify the insulators in the point cloud data to obtain the first position of the insulators:

[0011] Point cloud data is projected onto a depth image; the roughness of pixels in the depth image is calculated based on the depth distance of pixels and the cumulative difference of depth distances between a pixel and its multiple neighboring pixels; the depth distance is inversely proportional to the roughness, and the cumulative difference of depth distances is directly proportional to the roughness; a roughness threshold is set; the point clouds corresponding to pixels with a roughness threshold less than the roughness threshold are classified as boundary points; the boundary points are clustered to obtain multiple point cloud clusters; among the multiple point cloud clusters, the point cloud cluster corresponding to the insulator is identified; the first position of the insulator is calculated based on the point cloud cluster corresponding to the insulator.

[0012] Based on the first position of the insulator, the inspection device obtains the temperature of the insulator.

[0013] Furthermore, the inspection device is equipped with a lidar for acquiring point cloud data and a depth camera for acquiring depth images.

[0014] Furthermore, it also includes:

[0015] The inspection device acquires depth images at the detection points;

[0016] Identify the insulator in the depth image to obtain the second position of the insulator;

[0017] Calculate the positional deviation between the depth camera and the lidar;

[0018] The position deviation value is added to the second position of the insulator to obtain the third position of the insulator; it is determined whether the overlap between the first position and the third position of the insulator is less than the overlap threshold. If it is less than the overlap threshold, the inspection device obtains the temperature of the insulator based on the first position of the insulator.

[0019] Furthermore, it also includes:

[0020] The depth image is segmented into several sub-images;

[0021] Calculate the roughness of each pixel in each sub-image;

[0022] The pixels in each sub-image are sorted according to their roughness.

[0023] Furthermore, the point cloud cluster corresponding to the identified insulator specifically includes:

[0024] Among the multiple point cloud clusters, find the point cloud cluster whose length meets the first preset condition and whose relative positional relationship with the neighboring point cloud clusters meets the second preset condition, and consider that the point cloud cluster corresponds to the insulator.

[0025] Furthermore, the roughness calculation is expressed by the formula:

[0026]

[0027] In the formula, r i Represents pixel p in a depth image i ' depth distance; r j Represents pixel p in a depth image j The depth distance of '; S represents the distance from pixel p. i 'A set of pixels in the same row; |S| represents the number of pixels in set S.'

[0028] Furthermore, the route setting method is as follows:

[0029] Track points are collected at each target location, and latitude, longitude, and altitude information are recorded to form trajectory data; the positions of detection points are marked in the trajectory data to obtain the route.

[0030] Compared with the prior art, the present invention has the following features and beneficial effects:

[0031] 1. The present invention designs a roughness calculation formula, projects point cloud data onto a depth image, and calculates roughness based on the depth distance of pixels in the depth image and the cumulative value of the difference between the depth distance of pixels and the depth distance of neighboring pixels, so as to distinguish the boundary points corresponding to the object, thereby identifying the insulator and obtaining the insulator position; the identification accuracy is higher and the amount of calculation is small.

[0032] 2. This invention determines the accuracy of the first position obtained from point cloud data by using a second position obtained from a depth image to further improve the accuracy of insulator positioning by the inspection drone. Attached Figure Description

[0033] Figure 1 This is a flowchart of the invention;

[0034] Figure 2 , 3 This is a schematic diagram of point cloud data;

[0035] Figure 4 This is a schematic diagram of the relative positions of point cloud clusters;

[0036] Figure 5 This is a schematic diagram showing the locations of the depth camera and lidar. Detailed Implementation

[0037] The present invention will now be described in more detail with reference to the embodiments.

[0038] Example 1

[0039] like Figure 1 As shown, an inspection method based on lidar and thermal imaging includes the following steps:

[0040] Route setting: Track and collect points at each target location, record latitude, longitude and altitude information to form trajectory data; mark the positions of detection points in the trajectory data to obtain the route;

[0041] The inspection device arrives at the inspection point according to the preset route;

[0042] The inspection device acquires point cloud data at the detection points. For example... Figure 2 , 3 As shown;

[0043] Identify the insulators in the point cloud data to obtain the first position of the insulators;

[0044] Projecting point cloud data onto a depth image;

[0045] The roughness of a pixel in a depth image is calculated based on the depth distance of the pixel and the sum of the depth distance differences between the pixel and multiple neighboring pixels. The depth distance is inversely proportional to the roughness, and the sum of the depth distance differences is directly proportional to the roughness.

[0046] Set a roughness threshold; divide the point cloud corresponding to pixels with a roughness threshold less than the roughness threshold into boundary points; perform clustering processing on the boundary points to obtain multiple point cloud clusters; identify the point cloud cluster corresponding to the insulator among the multiple point cloud clusters; calculate the first position of the insulator based on the point cloud cluster corresponding to the insulator.

[0047] Based on the first position of the insulator, the inspection device obtains the temperature of the insulator.

[0048] Example 2

[0049] Install a 3D LiDAR directly in front of the drone's flight path, marking this position as point 0. The area directly in front of the LiDAR is the X-axis, the right side is the Y-axis when facing directly forward, and the top is the Z-axis. Mount a thermal imaging camera at position d directly above the LiDAR's point 0, ensuring that the horizontal position (X-axis) directly in front of the LiDAR is perpendicular to the camera's surface.

[0050] Calibrate the depth camera: Use a black and white checkerboard to calibrate the camera's intrinsic and extrinsic parameters, calculate the corner coordinates by inverse calculation, and then perform error calculation between the corner coordinates and the identified footpad coordinates to obtain the deviation value, which is used to correct the camera's ranging and obstacle orientation.

[0051] The lidar was calibrated using a black and white checkerboard with edge lines between the black and white areas, with the same distance between them.

[0052] Insulators in the point cloud data are identified to obtain the first position of the insulator {d1,x1,y1}; insulators in the depth image are identified to obtain the second position of the insulator {d2,x2,y2}; where d1 and d2 are the distance values ​​between the obstacle and the lidar, and x1, x2, y1, x2 are the horizontal and vertical coordinates.

[0053] like Figure 5 As shown, since the camera is installed at position d directly above the lidar, the second position coordinates are shifted down by d. The overlap between the shifted {d2,x2,y2} and {d1,x1,y1} is determined, thereby indicating whether the obstacle coordinates under the depth camera coincide with the obstacle coordinates of the lidar. If the overlap is less than the overlap threshold, the insulator position is determined to be {d1,x1,y1}; otherwise, the depth camera and lidar are recalibrated or the first insulator position is recalculated.

[0054] Example 3

[0055] The specific steps for identifying insulators in point cloud data are as follows:

[0056] Let P t ={p1,p2,...,p n} represents the point cloud data obtained by the lidar at time t, where p i It is P t The i-th valid point.

[0057] Point cloud data P t Projected onto depth image P t In this embodiment, the horizontal and vertical angular resolutions of the lidar are 0.2° and 2°, respectively, therefore the resolution of the projected depth image is 1800×16. Valid point p i Each and only uniquely derived from the depth image P t 'pixel p in i 'express.

[0058] Let S be the distance between pixel p in the depth image. i 'The set of other consecutive pixels in the same row, and pixel p i 'Located in the middle of set S, meaning that half of the points in set S are located at p' i 'On both sides'.

[0059] Evaluation of pixel p i The roughness in set S is expressed by the formula:

[0060]

[0061] In the formula, r i Represents pixel p in a depth image i The depth distance of ', i.e., the effective point p i The Euclidean distance from the corresponding object to the lidar; r j Represents pixel p in a depth image j The depth distance; S represents the pixel p. i 'The set of other consecutive pixels in the same row; |S| represents the number of pixels in the set S. In this embodiment, |S| = 10.

[0062] To more accurately determine the position of insulators in point cloud data, the depth image is segmented into multiple equal sub-images. Specifically, if the horizontal field of view of a lidar is 360°, the depth image is segmented into six equal sub-images, each containing a 60° horizontal field of view.

[0063] The pixels in each row of the sub-image are sorted from largest to smallest by roughness c. The smaller the depth distance, the larger the roughness c; the larger the cumulative difference in depth distance with other pixels, the larger the roughness c; therefore, pixels ranked higher are more likely to correspond to objects. For example, in the image, pixels A and B both correspond to the outline of an insulator, while pixel C near pixel B corresponds to the background or another object. Therefore, pixels A and B have relatively small differences in roughness, while pixels B and C have relatively large differences in roughness.

[0064] Use threshold c th To distinguish point clouds, the value greater than c th The pixels corresponding to the valid points are divided into boundary points, and those smaller than c are... th The pixels are divided into planar points corresponding to the valid points.

[0065] Clustering of boundary points yields several point cloud clusters;

[0066] Identify the point cloud clusters corresponding to the insulators:

[0067] The height (height of the cluster center in this embodiment), length, and positional relationships between point cloud clusters are calculated. The methods for calculating the height, width, length, and positional relationships between multiple point cloud clusters are existing technologies and will not be elaborated upon here.

[0068] like Figure 4As shown, if the length of point cloud cluster D meets the preset condition (i.e., greater than or equal to length a), and satisfies the following positional relationships: point cloud cluster C exists on one side above point cloud cluster D; point cloud clusters A and B exist on both sides below point cloud cluster D respectively; point cloud cluster D is perpendicular to point cloud clusters A and C respectively; the point cloud lengths of point cloud clusters C and A are both greater than a; point cloud cluster A can extend into point cloud cluster B on the other side of D, and the point cloud length of point cloud cluster B is greater than a. Point cloud cluster C is identified as a metal tower, point cloud cluster A as a cable, point cloud cluster B as a cable point cloud cluster, and D as an insulator.

[0069] Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. An inspection method based on lidar and thermal imaging, characterized in that, Includes the following steps: The inspection device arrives at the inspection point according to the preset route; The inspection device acquires point cloud data at the detection points; Identify the insulators in the point cloud data to obtain the first position of the insulators: Point cloud data is projected onto a depth image; the roughness of pixels in the depth image is calculated based on the depth distance of pixels in the depth image and the cumulative difference of depth distances between a pixel and multiple neighboring pixels; the depth distance is inversely proportional to the roughness, and the cumulative difference of depth distances is directly proportional to the roughness; Use threshold c th To distinguish point clouds, the value greater than c th The pixels corresponding to the valid points are divided into boundary points, and those smaller than c are... th The pixels are divided into planar points corresponding to the valid points; Clustering is performed on the boundary points to obtain multiple point cloud clusters; among the multiple point cloud clusters, a point cloud cluster whose length meets the first preset condition and whose relative positional relationship with the neighboring point cloud clusters meets the second preset condition is found, and the point cloud cluster corresponding to the insulator is identified. If the length of point cloud cluster D meets the preset conditions and satisfies the following positional relationships: there is point cloud cluster C on one side above point cloud cluster D, and point cloud clusters A and B are respectively on the two sides below point cloud cluster D; point cloud cluster D is perpendicular to point cloud clusters A and C respectively; the point cloud lengths of point cloud clusters C and A are both greater than a; point cloud cluster A can extend to point cloud cluster B on the other side of D, and the point cloud length of point cloud cluster B is greater than a, then point cloud cluster D is identified as an insulator. Calculate the first position of the insulator based on the point cloud cluster corresponding to the insulator; Based on the first position of the insulator, the inspection device obtains the temperature of the insulator.

2. The inspection method based on lidar and thermal imaging according to claim 1, characterized in that, The inspection device is equipped with a lidar for acquiring point cloud data and a depth camera for acquiring depth images.

3. The inspection method based on lidar and thermal imaging according to claim 2, characterized in that, Also includes: The inspection device acquires depth images at the detection points; Identify the insulator in the depth image to obtain the second position of the insulator; Calculate the positional deviation between the depth camera and the lidar; The position deviation value is added to the second position of the insulator to obtain the third position of the insulator; it is determined whether the overlap between the first position and the third position of the insulator is less than the overlap threshold. If it is less than the overlap threshold, the inspection device obtains the temperature of the insulator based on the first position of the insulator.

4. The inspection method based on lidar and thermal imaging according to claim 1, characterized in that, Also includes: The depth image is segmented into several sub-images; Calculate the roughness of each pixel in each sub-image.

5. The inspection method based on lidar and thermal imaging according to claim 1, characterized in that, The roughness calculation is expressed by the formula: In the formula, r i Represents pixel p in a depth image i ' depth distance; r j Represents pixel p in a depth image j The depth distance of '; S represents the distance from pixel p. i 'A set of pixels in the same row; |S| represents the number of pixels in set S.' 6. The inspection method based on lidar and thermal imaging according to claim 1, characterized in that, The route setting method is as follows: Track points are collected at each target location, and latitude, longitude, and altitude information are recorded to form trajectory data; the positions of detection points are marked in the trajectory data to obtain the route.

7. An inspection system based on lidar and thermal imaging, characterized in that, include: Route setting unit, used to set the route; The inspection device route is used to reach the inspection point according to the preset route. And acquiring point cloud data at the detection points; The identification unit is used to identify insulators in the point cloud data, obtain the first position of the insulator, and obtain the temperature of the insulator based on the first position of the insulator. The identification of insulators in point cloud data includes the following steps: projecting the point cloud data onto a depth image; calculating the roughness of pixels in the depth image based on the depth distance of pixels in the depth image and the accumulated difference of depth distances between a pixel and multiple neighboring pixels; wherein the depth distance is inversely proportional to the roughness, and the accumulated difference of depth distances is directly proportional to the roughness; and using a threshold c. th To distinguish point clouds, the value greater than c th The pixels corresponding to the valid points are divided into boundary points, and those smaller than c are... th The pixels are divided into planar points corresponding to the valid points; Clustering is performed on the boundary points to obtain multiple point cloud clusters; among the multiple point cloud clusters, a point cloud cluster whose length meets the first preset condition and whose relative positional relationship with the neighboring point cloud clusters meets the second preset condition is found, and the point cloud cluster corresponding to the insulator is identified. If the length of point cloud cluster D meets the preset conditions and satisfies the following positional relationships: point cloud cluster C exists on one side above point cloud cluster D, and point cloud clusters A and B exist on the two sides below point cloud cluster D respectively; point cloud cluster D is perpendicular to point cloud clusters A and C respectively; the point cloud lengths of point cloud clusters C and A are both greater than a; point cloud cluster A can extend to point cloud cluster B on the other side of D, and the point cloud length of point cloud cluster B is greater than a, then point cloud cluster D is identified as an insulator; calculate the first position of the insulator based on the point cloud cluster corresponding to the insulator.

8. The inspection system based on lidar and thermal imaging according to claim 7, characterized in that, The inspection device is equipped with a lidar for acquiring point cloud data and a depth camera for acquiring depth images.

9. The inspection system based on lidar and thermal imaging according to claim 8, characterized in that, Also includes: The inspection device acquires depth images at the detection points; Identify the insulator in the depth image to obtain the second position of the insulator; Calculate the positional deviation between the depth camera and the lidar; The position deviation value is added to the second position of the insulator to obtain the third position of the insulator; it is determined whether the overlap between the first position and the third position of the insulator is less than the overlap threshold. If it is less than the overlap threshold, the inspection device obtains the temperature of the insulator based on the first position of the insulator.

Images