A method and system for fusing transmission line images and point cloud data

Abstract

The present invention discloses a method and system for fusing transmission line images with point cloud data, the method comprising: collecting an optical image and point cloud data of a target transmission line; performing image processing on the point cloud data to obtain a point cloud image; in response to a user operation, selecting image feature points from the optical image, and selecting point cloud feature points corresponding to the image feature points from the point cloud image, wherein the point cloud feature points correspond to the spatial coordinates of the point cloud data; marking the image feature points and the corresponding point cloud feature points and establishing a mapping relationship to form a reference data pair; constructing a coordinate transformation relationship between the optical image and the point cloud data based on the reference data pair, and obtaining a correspondence between the coordinates of each pixel in the optical image and the spatial coordinates of the point cloud in the point cloud data based on the coordinate transformation relationship; the method can fuse the point cloud data and feature data to obtain data with spatial position information.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for fusing transmission line images and point cloud data. Background Art

[0002] The safe operation of transmission lines is crucial to the stable transmission of electric energy. These lines carry large capacities and high voltages, and are subject to long-term weather conditions such as wind, frost, rain, and snow. They are also distributed over a wide area and face complex terrain. Failures such as corrosion, wear, and power outages can lead to complex, cascading failures within the power grid, resulting in widespread outages and power outages. Furthermore, tree obstructions present a significant risk along transmission corridors, making regular inspections of transmission equipment crucial. Traditional inspection methods, influenced by the inspector's individual condition and measurement angle, often result in significant measurement errors, low efficiency, and limited accuracy. These methods are no longer sufficient to meet the needs of modernizing the expanding distribution network.

[0003] The visible light 2D images captured by drones require reconstruction to create a 3D power scene, which can deviate from the actual scene. The technical advantage of LiDAR point cloud mapping lies in its ability to accurately restore the spatial information of power scenes and measure distances. However, LiDAR point cloud mapping suffers from a lack of color information, resulting in poor visualization and difficulty in accurately classifying power objects using LiDAR point cloud data.

[0004] Therefore, when inspecting power energy targets, visible light image data can be fused with LiDAR point cloud data. For example, patent document CN113111751A discloses a 3D target detection method that adaptively fuses visible light and point cloud data. This method takes a camera image and a raw point cloud as input and adaptively fuses point cloud features with point cloud image features using a dual-stream region candidate network. However, this method employs a complex data fusion algorithm and suffers from low accuracy. Summary of the Invention

[0005] The present invention provides a method and system for fusing transmission line images with point cloud data, which can conveniently determine the conversion relationship between two-dimensional image data and point cloud data, allowing users to use intuitive two-dimensional images to accurately measure the distance of transmission lines. The method is simple, accurate, and reliable.

[0006] A method for fusing a transmission line image with point cloud data, comprising:

[0007] Collect optical images and point cloud data of target transmission lines;

[0008] Performing image processing on the point cloud data to obtain a point cloud image;

[0009] In response to a user operation, selecting an image feature point from the optical image, and determining a point cloud feature point corresponding to the image feature point from the point cloud image, wherein the point cloud feature point corresponds to a spatial coordinate of the point cloud data;

[0010] Marking the image feature points and the corresponding point cloud feature points and establishing a mapping relationship to form a reference data pair;

[0011] According to the reference data pair, a coordinate transformation relationship between the optical image and the point cloud data is constructed, and based on the coordinate transformation relationship, a corresponding relationship between the coordinates of each pixel in the optical image and the spatial coordinates of the point cloud in the point cloud data is obtained.

[0012] Furthermore, optical images and point cloud data of the target transmission line are collected, including:

[0013] Acquire an optical image of the target transmission line at a first viewing angle using a camera in a first time period and a first position;

[0014] The point cloud data of the target transmission line at a second perspective is acquired by a point cloud acquisition device in a second time period and a second position, wherein the first time period is the same as or different from the second time period, the first position is the same as or different from the second position, and the first perspective is the same as or different from the second perspective.

[0015] Furthermore, in response to a user operation, selecting image feature points from the optical image includes:

[0016] Using pixel coordinates corresponding to vertices of utility poles and / or towers as the image feature points;

[0017] A plurality of pixel coordinates are selected from pixels corresponding to the ground in order from left to right as the image feature points.

[0018] Furthermore, in response to a user operation, selecting an image feature point from the optical image, and determining a point cloud feature point corresponding to the image feature point from the point cloud image, comprises:

[0019] In response to a click operation by the user, determining a first click coordinate corresponding to the click operation in the optical image and a second click coordinate corresponding to the click operation in the point cloud image;

[0020] Determine a first candidate area according to the first click coordinates, and determine a second candidate area according to the second click coordinates;

[0021] Determine a first sharp point from the first candidate area, and determine a second sharp point from the second candidate area;

[0022] The first sharp point is used as the image feature point, and the second sharp point is used as the point cloud feature point.

[0023] Furthermore, the first candidate area is an area within a preset radius around the first click coordinate as the center;

[0024] The second candidate area is an area within a preset radius around the second click coordinate as the center;

[0025] Determining a first sharp point from the first candidate area and determining a second sharp point from the second candidate area includes:

[0026] Calculate the average normal angle between each pixel and its k-neighborhood within the first candidate area, and take the pixel with the largest average normal angle as the first sharp point;

[0027] The average normal angle between each pixel and its k-neighborhood in the second candidate area is calculated, and the pixel with the largest average normal angle is taken as the second sharp point.

[0028] Furthermore, the average normal angle between each pixel and its k-neighborhood is calculated using the following formula:

[0029]

[0030] Among them, α j is the angle between the normal vector of the pixel to be calculated and the normal vectors of other pixels in its k-neighborhood, is the average normal vector angle of the pixel to be calculated;

[0031] The k-neighborhood is an area consisting of k pixels with the smallest Euclidean distance between the pixel to be calculated and its surrounding pixels.

[0032] Furthermore, constructing a coordinate transformation relationship between the optical image and the point cloud data based on the reference data pair includes:

[0033] Establishing a transformation model using the intrinsic parameters of the camera device;

[0034] Inputting multiple sets of reference data into the transformation model to calculate the translation vector from the camera coordinate system to the point cloud three-dimensional space coordinate system;

[0035] A coordinate transformation relationship between the optical image and the point cloud data is obtained according to the translation vector and the internal parameters of the shooting device.

[0036] Furthermore, the transformation model is as follows:

[0037]

[0038] The rotation matrix R is as follows:

[0039]

[0040] Among them, d x with d y Respectively represent the physical size of each pixel on the horizontal axis x and vertical axis y of the optical image, (u0, v0) is the pixel coordinate of the intersection of the optical axis of the camera and the image plane, f is the focal length of the camera, R is the rotation matrix, T is the translation vector of the camera in the point cloud coordinate system, (u, v) is the two-dimensional pixel coordinate of the image feature point in the reference data, (X w ,Y W ,Z W ) is the three-dimensional spatial coordinate of the feature point of the point cloud in the reference data, Indicates the rotation angles of the camera coordinate axis around the y-axis, x-axis, and z-axis of the point cloud coordinate system.

[0041] Furthermore, the coordinate transformation relationship is as follows:

[0042]

[0043] Among them, (x, y) is the pixel coordinate of the target point, (u0, v0) is the pixel coordinate of the intersection of the optical axis of the camera and the optical image plane, f is the focal length of the camera, (X s ,Y S ,Z S ) is the coordinate of the center of the camera in the point cloud coordinate system, (X A ,Y A ,Z A ) represents the three-dimensional coordinates of the target point, a i ,b i ,c i is the rotation matrix, 1≤i≤3, i is an integer.

[0044] A system for fusing transmission line images and point cloud data, comprising:

[0045] a photographing device, for acquiring an optical image of a target transmission line;

[0046] A point cloud acquisition device, configured to scan a target transmission line and obtain point cloud data of the target transmission line;

[0047] The computing device includes a memory and a processor, wherein the memory stores a plurality of instructions, and the processor is configured to read the instructions and execute the above method.

[0048] The method and system for fusing transmission line images and point cloud data provided by the present invention have at least the following beneficial effects:

[0049] (1) Based on the correspondence between the image feature points in the optical image and the point cloud feature points in the point cloud data, the conversion relationship between the optical image and the point cloud data is automatically calculated to fuse the point cloud data and the feature data to obtain data with spatial location information. Users can use the data with intuitive spatial location information to perform tasks such as clear distance measurement, significantly reducing operation and maintenance costs, and achieving wide coverage, all-weather, high-precision remote safety monitoring of transmission line channels;

[0050] (2) In scenarios where the resolution of the captured optical image is high or a large amount of point cloud data is selected, the user moves the cursor to the candidate area of the target object and determines the sharp point by calculating the average normal angle, which effectively improves the accuracy of the marking. In addition, the number of times the user uses operations such as zooming in and out is effectively reduced, which improves the convenience of the user's marking work;

[0051] (3) The optical image and point cloud data of the target transmission line can be data collected from different perspectives. The user manually constructs the mapping relationship between the pixels in the optical image and the point data in the point cloud data, and sets labels respectively, so that the corresponding point cloud feature points and image feature points can be found based on the labels to provide a reference data pair for the automatic labeling process;

[0052] (4) By determining the transformation relationship between image feature points and point cloud feature points, the point cloud data and image data of the target transmission line can be captured separately from different perspectives and / or different time periods. Compared with the solution in the related art that requires the same perspective and / or the same time period to collect data on the target transmission line, the applicable scenarios of data collection are effectively improved;

[0053] (5) The rotation matrix is determined by the internal parameters of the shooting device, which effectively improves the accuracy of the transformation relationship between the determined image feature points and the point cloud feature points, and thus improves the accuracy of the mapping relationship between the point cloud feature points and the image feature points. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of an application scenario of the method for fusing transmission line images and point cloud data provided by the present invention.

[0055] Figure 2 A schematic diagram of another application scenario of the method for fusing transmission line images and point cloud data provided by the present invention.

[0056] Figure 3 The present invention provides a flowchart of an embodiment of the method for fusing transmission line images and point cloud data.

[0057] Figure 4This is a schematic diagram of an optical image and a point cloud image shown in an embodiment of the method for fusing transmission line images and point cloud data provided by the present invention.

[0058] Figure 5 This is a schematic diagram of adding markers in an embodiment of the method for fusing transmission line images and point cloud data provided by the present invention.

[0059] Figure 6 This is a schematic diagram of candidate areas and sharp points shown in an embodiment of the method for fusing transmission line images and point cloud data provided by the present invention.

[0060] Figure 7 The present invention provides a flowchart of an embodiment of selecting image feature points and point cloud feature points in the method for fusing transmission line images and point cloud data.

[0061] Figure 8 The present invention provides a flowchart of an embodiment of the method for fusing transmission line images and point cloud data to construct a coordinate transformation relationship between optical images and point cloud data.

[0062] Figure 9 This is a structural schematic diagram of an embodiment of the device for fusing transmission line images and point cloud data provided by the present invention.

[0063] Figure 10 This is a structural diagram of an embodiment of the system for fusing transmission line images and point cloud data provided by the present invention.

[0064] Figure 11 The figure is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0066] In order to facilitate the understanding of this application, some concepts involved in this application are first explained.

[0067] Point cloud data refers to a collection of vectors in a three-dimensional coordinate system. Scanned data is recorded as points, each containing three-dimensional coordinates. Some points may also contain color information (such as red, green, and blue) or reflection intensity information.

[0068] Related technologies use methods such as airborne lidar to collect point cloud data for mapping, which contains the location information of multiple objects. However, point cloud data is not as intuitive as image data. If image data can be used directly to measure the distance to the target object, it can effectively improve the convenience of transmission line inspection.

[0069] In order to achieve accurate measurement directly using transmission line images, it is necessary to make each pixel in the image have spatial coordinate information. In certain embodiments of the present application, two types of data (optical image and point cloud data) for the same target object are fused based on multiple sets of reference data. Because the fusion algorithm combines the internal parameters of the shooting device (such as focal length, sensor size, distortion parameters, etc.), the accuracy of the fusion result is effectively improved. For example, the fused transmission line image can be used to measure the clearance distance of the transmission line, etc., with an accuracy of sub-meter level.

[0070] The technical solutions of the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0071] Figure 1 This is a schematic diagram of an application scenario in an embodiment of the present invention. It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, to help those skilled in the art understand the technical content of the present application, but does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0072] See also Figure 1 According to this embodiment, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0073] Users can use terminal devices 101, 102, and 103 to interact with other mobile devices and server 105 via network 104 to receive or send information, such as requests for transmission line image data / point cloud data, clearance distance requests, transformation relationship calculation requests, transformation relationship update requests, image pixel coordinates, group reference data pairs, etc. Terminal devices 101, 102, and 103 can be installed with various communication client applications, such as image processing applications, monitoring applications, web browser applications, database applications, search applications, instant messaging tools, email clients, social platform software, and the like. It should be noted that the transmission line image and point cloud feature point selection method can be executed on the terminal device, and the transmission line image and point cloud feature point fusion method can be executed on the terminal device or on the server.

[0074] Terminal devices 101, 102, and 103 include but are not limited to surveying and mapping equipment, robots, drones, cameras, tablet computers, desktop computers, and the like.

[0075] The server 105 may receive requests for image data / point cloud data, clearance distance, and transformation relationship calculations of power transmission lines, and may also send image data / point cloud data, clearance distances, transformation relationships, and group reference data to the terminal devices 101, 102, and 103. For example, the server 105 may be a backend management server, a server cluster, or the like.

[0076] It should be noted that the number of mobile devices, networks, and cloud servers is only illustrative, and any number of mobile devices, networks, and cloud servers may be provided as needed.

[0077] Figure 2 It is a schematic diagram of another application scenario of the method provided by the present invention.

[0078] Figure 2 The scene of transmission line inspection is shown. Inspection robots (such as drones 20) and shooting devices 10 can be used to monitor transmission lines. The drone 20 can be provided with a sensor system 21, which can collect point cloud data for the transmission line. The shooting device 10 can obtain image data of the transmission line. Among them, the shooting device 10 can be installed on an electric tower, and the shooting device 10 can be facing the direction of another electric tower. The shooting device 10 can be powered by a power supply device such as a solar cell. It should be noted that the shooting device 10 can send the collected image data to a remote device or server in a wired or wireless manner. The shooting device 10 can be installed at a location far away from the power lines to reduce installation risks. The installation method and position of the shooting device 10 are shown for example only and should not be understood as a limitation of this application.

[0079] Figure 2 The aircraft 20 shown may be an aircraft with remote control flight function and automatic flight function. The aircraft 20 may include a sensor system 21 and a power mechanism 22. In addition, the aircraft 20 may also include a communication system.

[0080] The sensing system 21 may include one or more sensors to sense at least one of the following: surrounding obstacles, spatial position, speed, or acceleration of the aircraft 20. Sensor types include, but are not limited to, ranging sensors, position sensors, motion sensors, inertial sensors, or image sensors. The sensing data provided by the sensing system 21 may be used to control the spatial position, speed, and / or acceleration of the aircraft 20. The sensing system 21 is used to collect relevant information about the aircraft 20. Different types of sensors may sense different types of signals or signals from different sources. For example, sensors include inertial sensors, GPS sensors, distance sensors, or visual / image sensors (such as cameras).

[0081] The power mechanism 22 may include one or more rotating bodies, propellers, blades, engines, motors, wheels, bearings, magnets, nozzles, etc. The aircraft 20 may include one or more power mechanisms 22. The types of all power mechanisms 22 may be the same or different. The power mechanism 22 may be installed at any suitable position of the aircraft 20, such as the top, bottom, front, rear, side, or any combination thereof. For example, the power mechanism 22 may enable the aircraft 20 to take off from a surface at a certain angle (such as obliquely or vertically) relative to the ground plane, or to land on a surface. The power mechanism 22 may enable the aircraft 20 to fly or remain stationary in the air at a certain speed, altitude, etc.

[0082] The aircraft 20 can communicate with remote devices via the communication system. The remote devices include, but are not limited to, control terminals, flight control center servers, etc. For example, the communication system can transmit at least a portion of the point cloud data collected by the sensor system 21 to the remote device.

[0083] It should be noted that the inspection robot can also be an inspection vehicle. The inspection vehicle is equipped with at least one laser radar (LIDAR) on its roof and / or side. The LIDAR's detection area can be fixed, for example, a certain LIDAR can be used only to detect a preset area. The LIDAR's detection area can also be adjustable, for example, a LIDAR on the vehicle body can scan multiple detection areas by adjusting its posture, or by adjusting the LIDAR's own field of view.

[0084] In addition, the vehicle may also be equipped with a camera 10. The camera 10 can capture the environment ahead of the vehicle at a predetermined viewing angle. For example, the camera 10 may be a monocular camera, a multi-camera camera, or the like.

[0085] In addition, the present application may also use monitoring equipment at a fixed position (such as a camera 10 on a monitoring pole) to monitor the transmission line.

[0086] During the monitoring of power transmission lines, it was discovered that image data is intuitive but lacks spatial coordinate information. Point cloud data does have spatial coordinate information, but its intuitiveness is not as good as that of image data. Certain embodiments of the present application fuse image data and point cloud data to generate image data with spatial coordinates, allowing users to obtain accurate spatial location information using intuitive transmission line images.

[0087] refer to Figure 3 In some embodiments, a method for fusing a transmission line image with point cloud data is provided, comprising:

[0088] S1. Collect optical images and point cloud data of the target transmission line;

[0089] S2. Performing image processing on the point cloud data to obtain a point cloud image;

[0090] S3. In response to a user operation, selecting an image feature point from the optical image, and selecting a point cloud feature point corresponding to the image feature point from the point cloud image, wherein the point cloud feature point corresponds to a spatial coordinate of the point cloud data;

[0091] S4. Mark the image feature points and the corresponding point cloud feature points and establish a mapping relationship to form a reference data pair;

[0092] S5. Construct a coordinate transformation relationship between the optical image and the point cloud data according to the reference data pair, and obtain a correspondence between the coordinates of each pixel in the optical image and the spatial coordinates of the point cloud in the point cloud data based on the coordinate transformation relationship.

[0093] Specifically, in step S1, the optical image of the target transmission line may be the collected data, as shown in FIG. Figure 2 The optical image captured by the camera 10, such as the optical image, can be data captured in real time by a monocular camera. The optical image of the target transmission line can be read data, such as historically captured data stored in a database. However, the position of the target object at the time of capture has not changed relative to the position of the target object at the time of monitoring.

[0094] The point cloud data of the target transmission line can be the collected data, such as Figure 2 The point cloud data collected by the laser radar on the drone 20 can be real-time data. The point cloud data of the target transmission line can also be read data, such as point cloud data downloaded from a commercial database.

[0095] It should be noted that the optical image includes but is not limited to: optical images in various light bands such as visible light image data, infrared image data, ultraviolet image data, and X-ray image data.

[0096] In some embodiments, acquiring the optical image and point cloud data of the target transmission line includes at least one of the following:

[0097] Image data of a target power transmission line at a first viewing angle is acquired by a photographing device in a first time period and a first position.

[0098] Point cloud data of the target transmission line at a second perspective is acquired by radar in a second time period and a second position, wherein the first time period and the second time period are the same or different, the first position and the second position are the same or different, and the first perspective and the second perspective are the same or different.

[0099] Please refer to Figure 4 and Figure 5, it can be seen that the optical image and point cloud data are obtained from different perspectives. In addition, the optical image and point cloud data can be acquired at different time periods. The embodiment of the present application can determine the corresponding relationship between the point cloud data and the optical image to achieve multi-temporal data fusion, so that the objects in the two-dimensional image have coordinate information in the point cloud coordinate system, and spatial measurement can be performed based on the two-dimensional image.

[0100] Furthermore, in step S2, the point cloud data is image-processed to obtain a point cloud image, and the coordinates of each pixel in the point cloud image correspond to the point cloud space coordinates of the point cloud data.

[0101] In the process of converting point cloud data into an image, there will be a correspondence between the point cloud data and a pixel in the image. When the user clicks a pixel, the point in the point cloud selected by the user can be determined based on the correspondence, and then the coordinate information of the click target in the point cloud coordinate system can be obtained.

[0102] Furthermore, in step S3, in response to a user operation, image feature points are selected from the optical image, and point cloud feature points corresponding to the image feature points are selected from the point cloud image, wherein the point cloud feature points correspond to spatial coordinates of point cloud data.

[0103] In this embodiment, user operations include, but are not limited to, clicking, double-clicking, long pressing, continuous clicking, scrolling, and inputting operation instructions, etc., which can achieve human-computer interaction. The operation instructions may include target object identification information, etc., so that the computer can determine image feature points or point cloud feature points.

[0104] Image feature points can be represented using pixel coordinates. For example, using a vertex of the target transmission line image or display screen as the coordinate system origin (0, 0) yields the pixel coordinates (x1, y1). Point cloud feature points can be represented using the spatial coordinates (x2, y2, z2) of the point cloud space. For example, the highest vertex of a power tower can be used as a feature point. In the point cloud image, the pixel coordinates corresponding to the highest vertex represent the image feature point. The point coordinates corresponding to the highest vertex in the point cloud represent the point cloud feature point.

[0105] In some embodiments, in response to a user operation, image feature points are selected from the optical image, including at least one of the following.

[0106] For example, pixel coordinates corresponding to the vertices of a power pole and / or a tower are used as image feature points. Feature points of a tower and a pole can be distributed at the vertices of the cross arms of the tower.

[0107] For example, multiple pixel coordinates are selected from pixels corresponding to the ground in order from left to right as image feature points. Feature points on the ground can be evenly selected from left to right. It is understood that to ensure the accuracy of the mapping relationship, as many feature points as possible that are evenly distributed in the scene should be selected.

[0108] In some embodiments, the target transmission line image has a large image size, such as a high resolution. How to quickly and easily select the target object (such as the top of the horizontal pole of the power tower) from the image becomes a bottleneck problem that restricts the accuracy and efficiency of the fusion work. Figure 4 The vertex of a first-level horizontal bar on a tower may correspond to multiple pixels. To accurately mark a vertex, the user can first zoom in on the image, then move the mouse to the corresponding pixel and add the marker. To mark the vertex on the other side, the user may need to zoom out, then move the mouse to the vertex on the other side, zoom in, and add the marker. Manual operation during this process cannot guarantee accuracy and error rate.

[0109] In order to at least partially solve the above problem, the above-mentioned selecting a first pixel point from the first image and selecting a second pixel point from the second image in response to a user operation may include the following operations.

[0110] refer to Figure 7 , selecting image feature points from the optical image, and selecting point cloud feature points corresponding to the image feature points from the point cloud image, comprising:

[0111] S31, responding to a click operation of a user, determining a first click coordinate corresponding to the click operation in the optical image and a second click coordinate corresponding to the click operation in the point cloud image;

[0112] S32. Determine a first candidate area based on the first click coordinates, and determine a second candidate area based on the second click coordinates;

[0113] S33, determining a first sharp point from the first candidate area, and determining a second sharp point from the second candidate area;

[0114] S34: Use the first sharp point as the image feature point, and use the second sharp point as the point cloud feature point.

[0115] In some embodiments, the first candidate area is an area within a preset radius around the first click coordinate as the center;

[0116] The second candidate area is an area within a preset radius around the second click coordinate as the center;

[0117] Determining a first sharp point from the first candidate area and determining a second sharp point from the second candidate area includes:

[0118] Calculate the average normal angle between each pixel and its k-neighborhood within the first candidate area, and take the pixel with the largest average normal angle as the first sharp point;

[0119] The average normal angle between each pixel and its k-neighborhood in the second candidate area is calculated, and the pixel with the largest average normal angle is taken as the second sharp point.

[0120] The average normal angle between each pixel and its k-neighborhood is calculated using the following formula:

[0121]

[0122] Among them, α j is the angle between the normal vector of the pixel to be calculated and the normal vectors of other pixels in its k-neighborhood, is the average normal vector angle of the pixel to be calculated;

[0123] The k-neighborhood is an area consisting of k pixels with the smallest Euclidean distance between the pixel to be calculated and its surrounding pixels.

[0124] In this way, the current point Q can be determined based on the average normal angle i Whether it is a sharp point. This allows the computer to assist the user in determining the precise target pixel.

[0125] Figure 6 Schematic diagram of candidate areas and sharp points shown in one embodiment of the present application.

[0126] See also Figure 6 , is aimed at Figure 5 In order to accurately mark the vertices, the related technology requires manually moving the mouse to the vertex and then clicking on the data to add the mark. On the one hand, the user's mouse movement accuracy is extremely high, and the mouse may drift away from the accurate position when clicking the mouse. In this embodiment, when the user moves the mouse to a roughly accurate candidate area (such as Figure 6 After the point cloud is captured (as indicated by the dashed circle in the middle), the method described above can be used to automatically find sharp point pixels. This not only improves the accuracy of feature point selection but also effectively enhances operational convenience. For example, a single pixel deviation in the image can result in a point cloud coordinate system offset of several to tens of centimeters (the offset is related to the distance between the target object and the camera).

[0127] It should be noted that in the scenario of power transmission line monitoring, since the transmission lines are long, such as several kilometers to thousands of kilometers, the amount of point cloud data that needs to be stored is huge, and in order to ensure the accuracy of the data, point cloud data with higher resolution needs to be used. In the embodiment of the present application, the optical image and the point cloud data are fused, and the fused point cloud data can also include geometric information, so that the low-resolution point cloud data can be interpolated to achieve the use of lower-resolution point cloud data, and still obtain ranging results similar to those of high-resolution point cloud data. For example, the vertical pole of the electric pole corresponding to two points in the point cloud data, the lower-resolution point cloud data has lost the point corresponding to the horizontal pole, then the height information of the horizontal pole can be obtained by simulation calculation based on the geometric information in the image and the data of the two points in the point cloud data, and the point cloud data can be updated.

[0128] Furthermore, in step S4, the image feature points and the corresponding point cloud feature points are marked and a mapping relationship is established to form a reference data pair.

[0129] Figure 5 This is a schematic diagram of adding marks according to an embodiment of the present application.

[0130] See also Figure 5 , the first marker of the image feature point is P1, and the second marker of the point cloud feature point is P1, and the two markers are the same. In addition, the first marker and the second marker can be different, but a mapping relationship must exist between the two markers so that the point cloud feature point corresponding to the second marker can be found based on the first marker, or the image feature point corresponding to the first marker can be found based on the second marker. For example, the marker P1 is set at the position of the same feature point in the two-dimensional image and the three-dimensional point cloud.

[0131] Repeating the above operation can achieve multiple sets of benchmark data pairs, and the number of benchmark data pairs can be determined according to user needs, algorithm accuracy, etc. For example, the more benchmark data pairs there are, the more accurate the conversion relationship can be.

[0132] For example, the selected feature points should be unique and evenly distributed throughout the scene. In addition, 10 sets of benchmark data pairs can be obtained, with the coordinate information of the corresponding image feature points and point cloud feature points in the 2D image and 3D point cloud respectively.

[0133] Further, refer to Figure 8 In step S5, a coordinate transformation relationship between the optical image and the point cloud data is constructed based on the reference data pair, specifically including:

[0134] S51, establishing a transformation model using the internal parameters of the shooting device;

[0135] S52, inputting multiple sets of reference data into the transformation model, and calculating the translation vector from the camera coordinate system to the point cloud three-dimensional space coordinate system;

[0136] S53: Obtain a coordinate transformation relationship between the optical image and the point cloud data according to the translation vector and the internal parameters of the shooting device.

[0137] Specifically, the transformation model is related to the focal length of the shooting device that shoots the transmission line image, the coordinates of the shooting device in the point cloud coordinate system, the rotation matrix of the shooting device in the point cloud coordinate system, and the translation vector of the shooting device in the point cloud coordinate system.

[0138] Specifically, the transformation model is as follows:

[0139]

[0140] The rotation matrix R is as follows:

[0141]

[0142] Among them, d x with d y Respectively represent the physical size of each pixel on the horizontal axis x and vertical axis y of the optical image, (u0, v0) is the pixel coordinate of the intersection of the optical axis of the camera and the image plane, f represents the focal length of the camera, the above parameters are the internal parameters of the camera, R represents the rotation matrix, T represents the translation vector of the camera in the point cloud coordinate system, (u, v) is the two-dimensional pixel coordinate of the image feature point in the reference data, (X w ,Y W ,Z W ) is the three-dimensional spatial coordinate of the feature point of the point cloud in the reference data, Indicates the rotation angles of the camera coordinate axis around the y-axis, x-axis, and z-axis of the point cloud coordinate system.

[0143] The coordinate transformation relationship is as follows:

[0144]

[0145] Among them, (x, y) is the pixel coordinate of the target point, (u0, v0) is the pixel coordinate of the intersection of the optical axis of the camera and the optical image plane, f is the focal length of the camera, (X s ,Y S ,Z S ) is the coordinate of the center of the camera in the point cloud coordinate system, (X A ,Y A ,Z A ) represents the three-dimensional coordinates of the target point, a i ,b i ,ci is the rotation matrix, 1≤i≤3, i is an integer.

[0146] Specifically, the target point must exist in both the optical image returned by the camera and the point cloud data scanned by the point cloud acquisition device, and the target object must not change position or form within the scene. For example, the focal length of the camera lens is 3.8 milliseconds, the sensor size can be 0.00094*0.00094 meters, and the image size can be 5280*2992 pixels. Furthermore, image distortion can be taken into account and corrected.

[0147] refer to Figure 9 In some embodiments, a device for fusing transmission line images and point cloud data is further provided, comprising:

[0148] Acquisition module 201, for acquiring optical images and point cloud data of the target transmission line;

[0149] An imaging processing module 202 is used to perform imaging processing on the point cloud data to obtain a point cloud image;

[0150] A selection module 203 is configured to select an image feature point from the optical image in response to a user operation, and determine a point cloud feature point corresponding to the image feature point from the point cloud image, wherein the point cloud feature point corresponds to a spatial coordinate of the point cloud data;

[0151] A data pair forming module 204 is used to mark the image feature points and the corresponding point cloud feature points and establish a mapping relationship to form a reference data pair;

[0152] The conversion module 205 is used to construct a coordinate conversion relationship between the optical image and the point cloud data according to the reference data pair, and obtain a correspondence between the coordinates of each pixel in the optical image and the spatial coordinates of the point cloud in the point cloud data based on the coordinate conversion relationship.

[0153] The acquisition module 201 acquires optical images and point cloud data of the target transmission line, including:

[0154] Acquire an optical image of the target transmission line at a first viewing angle using a camera in a first time period and a first position;

[0155] The point cloud data of the target transmission line at a second perspective is acquired by a point cloud acquisition device in a second time period and a second position, wherein the first time period is the same as or different from the second time period, the first position is the same as or different from the second position, and the first perspective is the same as or different from the second perspective.

[0156] Furthermore, the selection module 203 is further configured to:

[0157] Using pixel coordinates corresponding to vertices of utility poles and / or towers as the image feature points;

[0158] A plurality of pixel coordinates are selected from pixels corresponding to the ground in order from left to right as the image feature points.

[0159] Furthermore, the selection module 203 is further configured to:

[0160] In response to a click operation by the user, determining a first click coordinate corresponding to the click operation in the optical image and a second click coordinate corresponding to the click operation in the point cloud image;

[0161] Determine a first candidate area according to the first click coordinates, and determine a second candidate area according to the second click coordinates;

[0162] Determine a first sharp point from the first candidate area, and determine a second sharp point from the second candidate area;

[0163] The first sharp point is used as the image feature point, and the second sharp point is used as the point cloud feature point.

[0164] Furthermore, the first candidate area is an area within a preset radius around the first click coordinate as the center;

[0165] The second candidate area is an area within a preset radius around the second click coordinate as the center.

[0166] Furthermore, the selection module 203 is further configured to:

[0167] Calculate the average normal angle between each pixel and its k-neighborhood within the first candidate area, and take the pixel with the largest average normal angle as the first sharp point;

[0168] The average normal angle between each pixel and its k-neighborhood in the second candidate area is calculated, and the pixel with the largest average normal angle is taken as the second sharp point.

[0169] The average normal angle between each pixel and its k-neighborhood is calculated using formula (1), which will not be repeated here.

[0170] Furthermore, the conversion module 205 is further configured to:

[0171] Establishing a transformation model using the intrinsic parameters of the camera device;

[0172] Inputting multiple sets of reference data into the transformation model to calculate the translation vector from the camera coordinate system to the point cloud three-dimensional space coordinate system;

[0173] A coordinate transformation relationship between the optical image and the point cloud data is obtained according to the translation vector and the internal parameters of the shooting device.

[0174] The transformation model is shown in formula (2) and formula (3), and the coordinate transformation relationship is shown in formula (4), which will not be repeated here.

[0175] The method and device for fusing transmission line images and point cloud data provided in the above embodiments have at least the following beneficial effects:

[0176] (1) Based on the correspondence between the image feature points in the optical image and the point cloud feature points in the point cloud data, the conversion relationship between the optical image and the point cloud data is automatically calculated to fuse the point cloud data and the feature data to obtain data with spatial location information. Users can use the data with intuitive spatial location information to perform tasks such as clear distance measurement, significantly reducing operation and maintenance costs, and achieving wide coverage, all-weather, high-precision remote safety monitoring of transmission line channels;

[0177] (2) In scenarios where the resolution of the captured optical image is high or a large amount of point cloud data is selected, the user moves the cursor to the candidate area of the target object and determines the sharp point by calculating the average normal angle, which effectively improves the accuracy of the marking. In addition, the number of times the user uses operations such as zooming in and out is effectively reduced, which improves the convenience of the user's marking work;

[0178] (3) The optical image and point cloud data of the target transmission line can be data collected from different perspectives. The user manually constructs the mapping relationship between the pixels in the optical image and the point data in the point cloud data, and sets labels respectively, so that the corresponding point cloud feature points and image feature points can be found based on the labels to provide a reference data pair for the automatic labeling process;

[0179] (4) By determining the transformation relationship between image feature points and point cloud feature points, the point cloud data and image data of the target transmission line can be captured separately from different perspectives and / or different time periods. Compared with the solution in the related art that requires the same perspective and / or the same time period to collect data on the target transmission line, the applicable scenarios of data collection are effectively improved;

[0180] (5) The rotation matrix is determined by the internal parameters of the shooting device, which effectively improves the accuracy of the transformation relationship between the determined image feature points and the point cloud feature points, and thus improves the accuracy of the mapping relationship between the point cloud feature points and the image feature points.

[0181] refer to Figure 10 In some embodiments, a system 1100 for fusing transmission line images and point cloud data is further provided, including:

[0182] A photographing device 1110 is used to obtain an optical image of a target transmission line;

[0183] The point cloud acquisition device 1120 is used to scan the target transmission line to obtain point cloud data of the target transmission line;

[0184] The computing device 1130 includes a memory and a processor. The memory stores a plurality of instructions, and the processor is configured to read the instructions and execute the above method.

[0185] The camera 1110 is positioned in space in a specific posture to acquire image data of the target transmission line. For example, the camera 1110 can be a camera fixed on a power tower or a camera installed on a monitoring pole.

[0186] The point cloud acquisition device 1120 is used to scan the target transmission line to obtain point cloud data of the target transmission line. For example, the point cloud acquisition device 1120 can be a variety of exploration equipment, such as a car equipped with a laser radar, a drone, a robot, etc.

[0187] The computing device 1130 includes a memory on which executable codes are stored. When the executable codes are executed by a processor, the processor is caused to execute the selection method as described above, or the fusion method as described above.

[0188] Another aspect of the present application provides an electronic device.

[0189] Figure 11 It is a structural diagram of an electronic device shown in an embodiment of the present application.

[0190] See also Figure 11 The electronic device 1200 may include a memory 1210 and a processor 1220. In addition, the electronic device 1200 may be provided with at least one of a random number generating circuit, a random number detecting circuit, or a radar.

[0191] The processor 1220 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0192] The memory 1210 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. ROM may store static data or instructions required by the processor 1220 or other modules of the computer. The permanent storage may be a readable and writable storage device. The permanent storage may be a non-volatile storage device that retains stored instructions and data even after the computer loses power. In some embodiments, the permanent storage device uses a large-capacity storage device (e.g., a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (e.g., a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 1210 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 1210 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0193] The memory 1210 stores executable codes. When the executable codes are processed by the processor 1220 , the processor 1220 may execute part or all of the above-mentioned methods.

[0194] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0195] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0196] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A method for fusing transmission line images and point cloud data, characterized in that: include: Collect optical images and point cloud data of target transmission lines; Performing image processing on the point cloud data to obtain a point cloud image; In response to a user operation, selecting an image feature point from the optical image, and selecting a point cloud feature point corresponding to the image feature point from the point cloud image, wherein the point cloud feature point corresponds to a spatial coordinate of the point cloud data; Marking the image feature points and the corresponding point cloud feature points and establishing a mapping relationship to form a reference data pair; constructing a coordinate transformation relationship between the optical image and the point cloud data according to the reference data pair, and obtaining a correspondence between the coordinates of each pixel in the optical image and the spatial coordinates of the point cloud in the point cloud data based on the coordinate transformation relationship; In response to a user operation, selecting an image feature point from the optical image and selecting a point cloud feature point corresponding to the image feature point from the point cloud image, including: In response to a click operation by the user, determining a first click coordinate corresponding to the click operation in the optical image and a second click coordinate corresponding to the click operation in the point cloud image; Determine a first candidate area according to the first click coordinates, and determine a second candidate area according to the second click coordinates; Determine a first sharp point from the first candidate area, and determine a second sharp point from the second candidate area; Using the first sharp point as the image feature point and the second sharp point as the point cloud feature point; The first candidate area is an area within a preset radius with the first click coordinate as the center; The second candidate area is an area within a preset radius around the second click coordinate as the center; Determining a first sharp point from the first candidate area and determining a second sharp point from the second candidate area includes: Calculate the average normal angle between each pixel and its k-neighborhood within the first candidate area, and take the pixel with the largest average normal angle as the first sharp point; The average normal angle between each pixel and its k-neighborhood in the second candidate area is calculated, and the pixel with the largest average normal angle is taken as the second sharp point.

2. The method according to claim 1, characterized in that Collect optical images and point cloud data of the target transmission line, including: Acquire an optical image of the target transmission line at a first viewing angle using a camera in a first time period and a first position; The point cloud data of the target transmission line at a second perspective is acquired by a point cloud acquisition device in a second time period and a second position, wherein the first time period is the same as or different from the second time period, the first position is the same as or different from the second position, and the first perspective is the same as or different from the second perspective.

3. The method according to claim 1, characterized in that In response to a user operation, selecting image feature points from the optical image includes: Using pixel coordinates corresponding to vertices of utility poles and / or towers as the image feature points; A plurality of pixel coordinates are selected from pixels corresponding to the ground in order from left to right as the image feature points.

4. The method according to claim 1, wherein The average normal angle between each pixel and its k-neighbors is calculated using the following formula: Among them, α j is the angle between the normal vector of the pixel to be calculated and the normal vectors of other pixels in its k-neighborhood, is the average normal vector angle of the pixel to be calculated; The k-neighborhood is an area consisting of k pixels with the smallest Euclidean distance between the pixel to be calculated and its surrounding pixels.

5. The method according to claim 2, characterized in that Constructing a coordinate transformation relationship between the optical image and the point cloud data according to the reference data pair, including: Establishing a transformation model using the intrinsic parameters of the camera device; Inputting the plurality of sets of the reference data into the transformation model, and calculating the translation vector from the camera coordinate system to the point cloud three-dimensional space coordinate system; A coordinate transformation relationship between the optical image and the point cloud data is obtained according to the translation vector and the internal parameters of the shooting device.

6. The method according to claim 5, characterized in that The transformation model is as follows: The rotation matrix R is as follows: Among them, d x with d y Respectively represent the physical size of each pixel on the horizontal axis x and vertical axis y of the optical image, (u0, v0) is the pixel coordinate of the intersection of the optical axis of the camera and the image plane, f is the focal length of the camera, R is the rotation matrix, T is the translation vector of the camera in the point cloud coordinate system, (u, v) is the two-dimensional pixel coordinate of the image feature point in the reference data, (X w ,Y W ,Z W ) is the three-dimensional spatial coordinate of the feature point of the point cloud in the reference data, Indicates the rotation angles of the camera coordinate axis around the y-axis, x-axis, and z-axis of the point cloud coordinate system.

7. The method according to claim 5, characterized in that The coordinate transformation relationship is as follows: Among them, (x, y) is the pixel coordinate of the target point, (u0, v0) is the pixel coordinate of the intersection of the optical axis of the camera and the optical image plane, f is the focal length of the camera, (X s ,Y S ,Z S is the coordinate of the center of the camera in the point cloud coordinate system, (X A ,Y A ,Z A ) represents the three-dimensional coordinates of the target point, a i ,b i , c i is the rotation matrix, 1≤i≤3, and i is an integer.

8. A system for fusing transmission line images and point cloud data, characterized in that: include: a photographing device, for acquiring an optical image of a target transmission line; A point cloud acquisition device, configured to scan a target transmission line and obtain point cloud data of the target transmission line; A computing device includes a memory and a processor, wherein the memory stores a plurality of instructions, and the processor is configured to read the instructions and execute the method according to any one of claims 1 to 7.

Images