System and method for aerial image-based infrastructure inspection

Abstract

A system attached to an aerial vehicle, the system comprising a LiDAR scanner, a camera, and a computing unit, wherein the computing unit is adapted to: identify, from scan data generated by the LiDAR scanner, a non-zero reflection corresponding to a location of a linear object within a free space; determine, based on the non-zero reflection, a distance and angle between the aerial vehicle and the linear object; and control the camera to track and image the linear object using the distance and angle.

Description

SYSTEM AND METHOD FOR AERIAL IMAGE-BASED INFRASTRUCTURE INSPECTIONField

[0001] The present invention generally relates to a system and method for aerial image-based infrastructure inspection.Background

[0002] Image-based inspection, which uses images of infrastructure objects taken at an inspection site, is increasingly being adopted to replace a range of existing types of infrastructure inspection, such as manual visual inspection. Images of geographically distributed, linear infrastructure objects, such as power transmission lines of electricity distribution networks, can be captured by aerial cameras mounted on helicopters or unmanned aerial vehicles (UAVs). The aerial images of the power lines captured by the aerial cameras can then be analysed to detect defects in the power lines.

[0003] Existing methods and systems for aerial image-based infrastructure inspection suffer several shortcomings. Failures stem from anomalies, not the overall system. Detecting them requires inspecting the entire system. The challenge is reliably and efficiently imaging an object in free space to assess the condition of infrastructure assets such as electrical overhead line conductors and hardware connected to conductors, cables, towers, and poles associated with electrical utility assets located in the field.

[0004] Aerial-image based inspection of power lines has previously been proposed using aerial cameras mounted in helicopter pan-tilt stabilised turrets. However, this type of aerial camera stabilisation is incompatible with high-speed camera movement necessary to continuously track dynamically moving target linear objects in real time as the helicopter follows above and alongside. Turret-based camera systems are instead designed to track single fixed objects on the ground, and as such it is difficult for the camera operator to manually track the camera along the length of a power line as the helicopter flies above and alongside the power lines. The same disadvantages apply to equivalent solutions mounted on UAVs.

[0005] Furthermore, the quality and resolution of aerial images acquired by existing aerial camera systems are typically insufficient for reliable and accurate detection of small defects or anomalies in thin, linear infrastructure objects such as power lines. This is due to: (i) difficultly in accuratelytracking thin power lines against objects such as vegetation and ground terrain; (ii) difficulty in accurately tracking the location of the airborne cameras when successive aerial images are captured; (iii) blurring of images due to motion of the aerial platform and vibration of the aerial cameras during capture of the aerial images; (iv) variability in the power lines being imaged; and (v) capturing imagery efficiently and economically over large distances. All these effects result in inadequate quality, low-resolution aerial images which limits the usefulness of the acquisition of aerial images from existing systems and methods.

[0006] In view of this background, there is a need for an improved system and method for aerial image-based infrastructure inspection.Summary

[0007] According to the present invention, there is provided a system attached to an aerial vehicle, the system comprising a LiDAR scanner, a camera, and a computing unit, wherein the computing unit is adapted to: identify, from scan data generated by the LiDAR scanner, a non-zero reflection corresponding to a location of a linear object within a free space; determine, based on the non-zero reflection, a distance and angle between the aerial vehicle and the linear object; and control the camera to track and image the linear object using the distance and angle.

[0008] The scan data may correspond to a scan of a two-dimensional plane.

[0009] Alternatively, the scan data may comprise three-dimensional scan data, and the computing unit may be further adapted to derive a synthetic two-dimensional slice from the three- dimensional scan data to identify the non-zero reflection.

[0010] The LiDAR scanner and the camera may have distinct coordinate systems, and the computing unit may be further adapted to align the distinct coordinate systems using a software calibration.

[0011] Alternatively, the LiDAR scanner and the camera may be co-located on a platform that provides a physical common datum.

[0012] The platform may comprise a vibration-damping platform.

[0013] The computing unit may be further adapted to distinguish the non-zero reflection corresponding to the linear object from reflections from background objects by disregarding reflections having a measured distance beyond a predetermined threshold.

[0014] Alternatively, the computing unit may be further adapted to distinguish the non-zero reflection corresponding to the linear object from reflections from background objects by selecting a reflection or group of reflections having a minimum measured distance from the aerial vehicle.

[0015] In operation, the aerial vehicle may be moved relative to the linear object such that a portion of the linear object's longitudinal extent is continuously maintained within a field of view of the LiDAR scanner and the camera.

[0016] The computing unit may be further adapted to receive flight data of the aerial vehicle comprising position, velocity, orientation, and combinations thereof.

[0017] The computing unit may be further adapted to control acceleration and deceleration of the camera based on a jerk limited control algorithm.

[0018] The images captured by the camera may have a resolution comprising a pixel size of less than around 1 mm.

[0019] The linear object may extend generally horizontally or vertically.

[0020] The linear object may comprise a linear object of an infrastructure asset.

[0021] The infrastructure asset may comprise an electricity distribution network, and the linear object comprises a power line or a power pole.

[0022] The system may further comprise multiple cameras to track and image multiple linear objects simultaneously.

[0023] The aerial vehicle may comprise a helicopter, a UAV, or a fixed-wing aircraft.

[0024] The present invention also provides an aerial vehicle comprising the system described above.

[0025] The present invention further provides a method, comprising:identifying, with a computing unit from scan data generated by a LiDAR scanner on an aerial vehicle, a non-zero reflection corresponding to a location of a linear object within a free space; determining, based on the non-zero reflection, a distance and angle between the aerial vehicle and the linear object; and controlling a camera to track and image the linear object using the distance and angle.

[0026] The scan data may correspond to a scan of a two-dimensional plane.

[0027] Alternatively, the scan data may comprise three-dimensional scan data, and the method may further comprise deriving a synthetic two-dimensional slice from the three-dimensional scan data to identify the non-zero reflection.

[0028] The LiDAR scanner and the camera may have distinct coordinate systems, and the method may further comprise aligning the distinct coordinate systems using a software calibration.

[0029] The LiDAR scanner and the camera may be co-located on a platform that provides a physical common datum.

[0030] The platform may comprise a vibration-damping platform.

[0031] The method may further comprise distinguishing the non-zero reflection corresponding to the linear object from reflections from background objects by disregarding reflections having a measured distance beyond a predetermined threshold.

[0032] Alternatively, the method may further comprise distinguishing the non-zero reflection corresponding to the linear object from reflections from background objects by selecting a reflection or group of reflections having a minimum measured distance from the aerial vehicle.

[0033] The identifying, determining, and controlling steps may be performed while moving the aerial vehicle relative to the linear object such that a portion of the linear object's longitudinal extent is continuously maintained within a field of view of the LiDAR scanner and the camera.Brief Description of Drawings

[0034] Non-limiting and non-exhaustive examples are described with reference to the following figures:Figure 1 is a simplified block diagram of an example system attached to an aerial vehicle;Figure 2 illustrates an example helicopter equipped with the system to image power lines of an electricity distribution network;Figure 3 is a flowchart of an example method performed by the system;Figure 4 depicts an example visualisation generated by the system comprising images of power lines above a map showing a flight path of the helicopter next to a satellite image of the power lines; andFigure 5 depicts example defects detected in example high-resolution aerial images of power lines captured by the system.Detailed Description

[0035] Referring to Figure 1 , an example system 100 attached to an aerial vehicle may comprise a LiDAR scanner 110, a camera 120, and a computing unit 130. In some examples, the aerial vehicle 200 may comprise a helicopter, a UAV, or a fixed-wing aircraft. Figure 2 depicts an example system 100 attached to a helicopter 200.

[0036] In some implementations, the LiDAR scanner 110 and camera 120 may be adapted to share a common datum that comprises a two-dimensional plane in free space 300 substantially perpendicular to a direction of flight (or flight path) 210 of the aerial vehicle 200. The two- dimensional plane may therefore comprise a common plane of reference for the LiDAR scanner 110 and camera 120.

[0037] In other implementations, the LiDAR scanner 110 and the camera 120 may have distinct coordinate systems, and the computing unit 130 may be further adapted to align the distinct coordinate systems using a software calibration.

[0038] The LiDAR scanner 110 may be further adapted to generate scan data corresponding to the free space 300. The scan data may comprise a two- or three-dimensional representation of the free space 300. In some examples, the LiDAR scanner 110 may comprise a commercially available two- or three-dimensional LiDAR scanner. In some examples, multiple LiDAR scanners 120 may be used to generate the scan data.

[0039] The camera 120 may be further adapted to have adjustable focus and to be movable in the two-dimensional plane. In some examples, the camera 110 may comprise a commercially available machine vision camera equipped with a commercially available adjustable focus lens. In some examples, the camera 110 may comprise a commercially available pan-tilt-zoom camera. In other examples, the camera 110 may be provided on a servo-driven camera platform that is movable in the two-dimensional plane.

[0040] The computing unit 130 may be adapted to receive and process the scan data in real time to detect a non-zero reflection in the two-dimensional plane that indicates the presence of a linear object 310 in the free space 300. In examples where the scan data is three-dimensional, the two- dimensional plane may comprise a synthetic two-dimensional slice derived from the three- dimensional scan data.

[0041] The computing unit 130 may be further adapted to determine a distance and angle in the two-dimensional plane between the aerial vehicle 200 and the linear object 310 based on the scan data corresponding to the non-zero reflection. Based on the distance and angle, the computing unit 130 may be further adapted to control focus and movement of the camera 120 in the two-dimensional plane to automatically track and continuously image the linear object 310 in the free space 300 as the aerial vehicle 200 moves in the direction of flight 210.

[0042] It has been realised that detection of non-zero reflections in two-dimensional planes is sufficient to allow distances and angles between the aerial vehicle 200 and the linear object 310 to be determined to control focus and movement of the camera 120 in real time. The two- dimensional planes may be substantially vertical or horizontal relative the direction of flight 210 of the aerial vehicle 200. For example, in examples where the linear object 310 is expected to extend generally horizontally in the free space 300, the two-dimensional planes may be substantially vertical, and vice versa. In some examples, the computing unit 130 may comprise commercially available processors and commercially available storage devices.

[0043] In some examples, the LiDAR scanner 110 and camera 120 may be co-located on a vibration-damping platform that is adapted to absorb vibration from the aerial vehicle 200. In some examples, the computing unit 130 may be further adapted to control acceleration and deceleration of the camera 120 based on a jerk limited control algorithm. In these examples, blurring of the captured images may be avoided or minimised by one or both of the vibrationdamping platform and the jerk limited control algorithm. In some examples, the captured images may have a resolution comprising a pixel size of less than around 1 mm.

[0044] In some examples, the computing unit 130 may be further adapted to receive and process flight data of the aerial vehicle 200 comprising position, velocity, orientation, and combinations thereof. The flight data may be received from a flight control system of the aerial vehicle 200, a global navigation satellite system (GNSS), such as the global positioning system (GPS), and combinations thereof. In some examples, a GNSS receiver may be provided as a component of the system 100. In some examples, the computing unit 130 may be further adapted to control flight of the aerial vehicle 200 relative to the linear object 310 based on the flight data and the scan data.

[0045] In some examples, the linear object 310 may comprise a linear object or structure of an infrastructure asset. In these examples, the infrastructure asset may comprise an electricity distribution network, and the linear object 310 may comprise a power line or a power pole.

[0046] To facilitate the inspection of the linear object 310, the aerial vehicle 200 may typically be operated in a manner that allows for sequential imaging along the object's length. This operational context involves moving the aerial vehicle 200 relative to the linear object 310 such that a portion of the object's longitudinal extent is continuously maintained within the field of view of the LiDAR scanner 110 and the camera 120. This ensures that the computing unit 130 continuously receives scan data corresponding to the object 310 for uninterrupted tracking. For example, in the case of a substantially straight power line, this may be achieved by flying the aerial vehicle 200 on a flight path substantially parallel to the power line.

[0047] In some examples, the system 100 may be scaled up to comprise multiple cameras 120 to selectively and individually track and image multiple linear objects 310 simultaneously. In these examples, the multiple linear objects 310 may be detected in the free space 300 based on multiple non-zero reflections in a single two-dimensional plane of the scan data. In some examples, the multiple cameras 120 may be co-located with a single LiDAR scanner 110 on the vibration-damping platform, and the multiple cameras 120 may be mounted individually on the vibration-damping platform by multiple servo-driven camera platforms.

[0048] In some implementations, to distinguish one or more linear objects from background clutter, such as vegetation or ground terrain, the computing unit 130 may be configured to filter non-zero reflections based on their measured distance. The system 100 may define a foreground region of interest within a predetermined range from the aerial vehicle, a range selected to encompass the expected positions of the multiple linear objects 310. The computing unit 130 may then apply a distance threshold, or range gate, to the scan data. Non-zero reflections having a measured distance that falls outside this threshold may be classified as background clutter andcomputationally disregarded. This filtering process allows the computing unit to isolate a set of reflections originating from the foreground, which correspond to the multiple linear objects 310, thereby ensuring their accurate identification.

[0049] In other implementations, the computing unit 130 may be configured to identify multiple linear objects by prioritizing a set of the closest non-zero reflections. As the LiDAR 110 scans the free space, it may receive multiple non-zero reflections from the group of linear objects 310 and more distant background objects. The computing unit 130 may analyse the scan data to determine the distance of each reflection and select a group of reflections that are all within a certain proximity to the aerial vehicle 200. By selecting this set of first-returned or closest reflections, the system 100 may effectively isolate the group of foreground objects from the background, as the multiple linear objects 310 are presumed to be closer to the aerial vehicle 200 than any significant background clutter.

[0050] Referring to Figure 3, an example method 400 performed using the system 100 starts at step 410 by receiving and processing scan data from a LiDAR scanner in real time to detect a non-zero reflection indicating the presence of a linear object in a two-dimensional plane in free space perpendicular to a direction of flight of an aerial vehicle.

[0051] Next, at step 420, a distance and angle in the two-dimensional plane between the aerial vehicle and the linear object may be determined based on the scan data corresponding to the non-zero reflection.

[0052] The method 400 may end at step 430 by controlling focus and movement of a camera in the two-dimensional plane based on the distance and angle to automatically track and continuously image the linear object in the free space as the aerial vehicle moves in the direction of flight.

[0053] The following Example is intended to illustrate the invention. It is not intended to limit the scope of the invention.Example

[0054] Referring again to Figure 2, an example system 100 comprising one LiDAR scanner 110, five focus- and tilt-controlled cameras 120, and a computing unit 130 was attached to a helicopter 200. The LiDAR scanner 110 and the five cameras 120 were co-located on a vibration-damping platform as discussed above, and the five cameras 120 were mounted individually to thevibration-damping platform by five servo-driven camera tilt platforms as also discussed above. The LiDAR scanner 110 and the five cameras 120 were adapted to share a common datum comprising a two-dimensional radial plane oriented generally vertical so as to be generally orthogonal to the expected generally horizontal extent of five power lines 310 of an electricity distribution network (note, only one power line 310 is shown for clarity).

[0055] The helicopter 200 was flown on a flight path 210 substantially parallel to free space 300 on one side of the helicopter 200 containing the five power lines 310. In this example, a control display was provided in the helicopter 200 to allow an operator of the system 100 to view individual detected power lines 310 and selectively and individually designate them to be tracked and imaged by the five cameras 120 simultaneously. As described in detail above, the five power lines 310 were then automatically tracked and continuously imaged simultaneously by selectively and individually controlling focal distances and tilt angles of the five cameras 120 in a single flight of the helicopter 200 above and alongside the five power lines 310 based on the computing unit 130 detecting them as non-zero reflections in the two-dimensional radial plane of the scan data generated by the LiDAR scanner 110. The scan data, captured images, and corresponding flight data of the helicopter 200 were stored by the computing unit 130 in solid state storage devices for subsequent processing and analysis.

[0056] Figure 4 depicts an example visualisation in which successive high-resolution image frames of the five power lines 310 captured by the five cameras 120 in this example are simultaneously displayed above a Geographic Information System (GIS) map of the electricity network showing the GPS flight path 210 of the helicopter 200. A selected image frame of one of the power lines 310 captured by one of the cameras 120 designated “Camera 2” is shown as an enlarged image next to the GIS map.

[0057] The high-resolution images captured by the system 100 in this example were processed and analysed using computer vision techniques, such as trained machine learning or deep learning models, to detect defects or anomalies in the power lines 310. Figure 5 depicts example defects detected in the power lines 310 using the captured images.

[0058] The invention is not limited to the example that has just been given. Those skilled in the art will appreciate that the example may be reproduced without difficulty, and with similar success, by substituting or varying any of the generically or specifically described elements or sequence of method steps, mentioned anywhere in this specification forthose actually used in the preceding example.

[0059] Embodiments of the present invention provide a system and method that are both generally and specifically useful for aerial image-based infrastructure inspection.

[0060] An advantage of embodiments of the system and method of the present invention is the computational efficiency gained by identifying the linear object based on the presence of a nonzero reflection. Instead of capturing and processing a detailed three-dimensional point cloud to analyse the shape and features of all objects in the LiDAR's field of view - a task that requires substantial computing power - the system simplifies the detection problem. By reducing the identification task to a query for the presence of a non-zero reflection, the computing unit avoids the need for complex object recognition or scene analysis algorithms. This reduction in computational complexity allows the system to identify and track the linear object rapidly using more compact and efficient hardware.

[0061] Unless the context requires otherwise, the word "comprising" means "including but not limited to," and the word "comprises" has a corresponding meaning.

[0062] Any reference to prior art is not an admission that the prior art is common general knowledge.

[0063] The invention is not limited to the examples given above. Instead, the scope of the invention supported by the above examples is defined by the claims that follow.

Claims

Claims1. A system attached to an aerial vehicle, the system comprising a LiDAR scanner, a camera, and a computing unit, wherein the computing unit is adapted to: identify, from scan data generated by the LiDAR scanner, a non-zero reflection corresponding to a location of a linear object within a free space; determine, based on the non-zero reflection, a distance and angle between the aerial vehicle and the linear object; and control the camera to track and image the linear object using the distance and angle.

2. The system of claim 1 , wherein the scan data corresponds to a scan of a two-dimensional plane.

3. The system of claim 1, wherein the scan data comprises three-dimensional scan data, and the computing unit is further adapted to derive a synthetic two-dimensional slice from the three-dimensional scan data to identify the non-zero reflection.

4. The system of claim 1, wherein the LiDAR scanner and the camera have distinct coordinate systems, and the computing unit may be further adapted to align the distinct coordinate systems using a software calibration.

5. The system of claim 1 , wherein the LiDAR scanner and the camera are co-located on a platform that provides a physical common datum.

6. The system of claim 5, wherein the platform comprises a vibration-damping platform.

7. The system of claim 1, wherein the computing unit is further adapted to distinguish the non-zero reflection corresponding to the linear object from reflections from background objects by disregarding reflections having a measured distance beyond a predetermined threshold.

8. The system of claim 1, wherein the computing unit is further adapted to distinguish the non-zero reflection corresponding to the linear object from reflections from background objects by selecting a reflection or group of reflections having a minimum measured distance from the aerial vehicle.

9. The system of claim 1 , wherein, in operation, the aerial vehicle is moved relative to the linear object such that a portion of the linear object's longitudinal extent is continuously maintained within a field of view of the LiDAR scanner and the camera.

10. The system of claim 1 , wherein the computing unit is further adapted to receive flight data of the aerial vehicle comprising position, velocity, orientation, and combinations thereof.

11. The system of claim 1, wherein the computing unit is further adapted to control acceleration and deceleration of the camera based on a jerk limited control algorithm.

12. The system of claim 1, wherein the images captured by the camera have a resolution comprising a pixel size of less than around 1 mm.

13. The system of claim 1 , wherein the linear object extends generally horizontally or vertically.

14. The system of claim 13, wherein the linear object comprises a linear object of an infrastructure asset.

15. The system of claim 14, wherein the infrastructure asset comprises an electricity distribution network, and the linear object comprises a power line or a power pole.

16. The system of claim 1, further comprising multiple cameras to track and image multiple linear objects simultaneously.

17. The system of claim 1 , wherein the aerial vehicle comprises a helicopter, a UAV, or a fixed-wing aircraft.

18. An aerial vehicle comprising the system of claim 1.

19. A method, comprising: identifying, with a computing unit from scan data generated by a LiDAR scanner on an aerial vehicle, a non-zero reflection corresponding to a location of a linear object within a free space; determining, based on the non-zero reflection, a distance and angle between the aerial vehicle and the linear object; and controlling a camera to track and image the linear object using the distance and angle.

20. The method of claim 19, wherein the scan data corresponds to a scan of a two- dimensional plane.

21. The method of claim 19, wherein the scan data comprises three-dimensional scan data, and the method further comprises deriving a synthetic two-dimensional slice from the three- dimensional scan data to identify the non-zero reflection.

22. The method of claim 19, wherein the LiDAR scanner and the camera have distinct coordinate systems, and the method further comprises aligning the distinct coordinate systems using a software calibration.

23. The method of claim 19, wherein the LiDAR scanner and the camera are co-located on a platform that provides a physical common datum.

24. The method of claim 23, wherein the platform comprises a vibration-damping platform.

25. The method of claim 19, further comprising distinguishing the non-zero reflection corresponding to the linear object from reflections from background objects by disregarding reflections having a measured distance beyond a predetermined threshold.

26. The method of claim 19, further comprising distinguishing the non-zero reflection corresponding to the linear object from reflections from background objects by selecting a reflection or group of reflections having a minimum measured distance from the aerial vehicle.

27. The method of claim 19, wherein the identifying, determining, and controlling steps are performed while moving the aerial vehicle relative to the linear object such that a portion of the linear object's longitudinal extent is continuously maintained within a field of view of the LiDAR scanner and the camera.

Images