Electric power corridor laser point cloud semantic segmentation method and system acquired by unmanned aerial vehicle

Abstract

The invention belongs to the technical field of point cloud semantic segmentation, and discloses a power corridor laser point cloud semantic segmentation method and system collected by an unmanned aerial vehicle, and the method comprises the steps: collecting laser point cloud and image data; reconstructing point cloud data based on the image data to obtain an image point cloud, performing spatial registration on the laser point cloud and the image point cloud, and projecting the registered laser point cloud onto the image; operating the image data to obtain a depth image and a normal image, and performing fusion operation on the depth image, the normal image, the image and the projected laser point cloud to obtain multimode fusion data; and carrying out two-dimensional semantic segmentation on the multi-mode fusion data, and carrying out back projection operation on a segmentation result to obtain a three-dimensional point cloud semantic segmentation result. According to the method, the visible light image of the power corridor and the laser point cloud data are fused, high-precision semantic segmentation of the laser point cloud is realized by using the appearance information contained in the visible light image, and the accuracy, efficiency and fineness of semantic segmentation of the laser point cloud of the power corridor are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud semantic segmentation, and in particular to a method and system for semantic segmentation of laser point clouds collected by unmanned aerial vehicles (UAVs). Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, my country's power grid construction has been in a period of rapid development, and the coverage of transmission lines has become wider and wider. Data show that in 2023, the total length of my country's transmission lines above 220 kV will reach 920,457 kilometers. Ensuring the safe operation of transmission lines is related to national energy security, people's life and health, and social and economic development. The power corridor is a general term for adjacent towers in high-voltage transmission lines and their transmission wires, personnel, green plants, buildings and other targets. Safety accidents in power corridors occur frequently, and one of the most common accidents is the discharge of transmission wires caused by insufficient safety distance between ground targets such as personnel and trees and transmission wires.

[0004] In order to calculate the safe distance between ground targets and transmission lines, drones can be used to collect 3D point cloud data of power corridors, and the 3D point clouds corresponding to ground targets and transmission lines can be distinguished through 3D point cloud semantic segmentation algorithms, and then the spatial distance between the two types of targets can be analyzed. Since it is difficult to directly establish the correlation between the 3D point cloud collected by drones and the visible light image, the existing semantic segmentation of power corridors mainly processes laser point clouds. Semantic segmentation methods mainly include 3D point cloud semantic segmentation based on direct methods and 3D point cloud semantic segmentation based on projection. 3D point cloud semantic segmentation based on direct methods directly processes 3D point clouds, including PointNet using multi-layer perceptrons to extract 3D point cloud features, and PointNet++ further using multi-scale sampling mechanisms to aggregate global and local features. However, the above methods fail to take into account the inconsistent sparsity of 3D point clouds of targets at different distances in outdoor scenes, so they do not perform well in power corridor scenes. Projection-based methods use efficient projection methods, such as spherical projection or bird's-eye projection, to project 3D point clouds onto a 2D image plane for processing. This method is convenient to use a relatively more mature 2D convolutional neural network structure, and has higher accuracy and execution efficiency. For the power corridor scene, the images collected by the drone contain more appearance information (such as color information and texture information), so using image information can obtain more accurate and detailed semantic segmentation results. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a method and system for semantic segmentation of laser point clouds in power corridors collected by drones, which fuses the visible light images of the power corridors with the laser point cloud data, and uses the appearance information contained in the visible light images to achieve high-precision semantic segmentation of the laser point clouds, so as to improve the accuracy, efficiency and precision of the semantic segmentation of laser point clouds in the power corridors.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for semantic segmentation of a power corridor laser point cloud collected by a drone, comprising the following steps:

[0008] Collect laser point cloud data and image data of power corridors;

[0009] Reconstruct three-dimensional point cloud data based on image data to obtain image point cloud, use semantic histogram distance to assist laser point cloud and image point cloud in three-dimensional spatial registration, project the registered laser point cloud onto the image, and obtain the projected laser point cloud;

[0010] Performing depth image estimation and normal image estimation on the image data to obtain a depth image and a normal image respectively, and fusing the depth image and the normal image with the image and the projected laser point cloud to obtain multi-mode fusion data;

[0011] Perform two-dimensional semantic segmentation on the multi-modal fusion data, and perform back-projection operation on the two-dimensional semantic segmentation result to obtain the final three-dimensional point cloud semantic segmentation result.

[0012] As an optional implementation, three-dimensional point cloud data is reconstructed based on the image data to obtain the image point cloud, specifically:

[0013] The SFM algorithm is used to calculate the sparse 3D point cloud of the image, and the dense reconstruction algorithm based on PatchMatch is used to densify the sparse 3D point cloud to obtain the image point cloud.

[0014] As an optional implementation, the laser point cloud and the image point cloud are registered in three-dimensional space, specifically:

[0015] First, the rigid body transformation between the laser point cloud and the image point cloud is preliminarily calculated through coarse registration, and then the rigid body transformation is corrected through fine registration to obtain the six-degree-of-freedom rigid body transformation matrix. Based on the six-degree-of-freedom rigid body transformation matrix, the laser point cloud and the image point cloud are registered in three-dimensional space.

[0016] As an optional implementation, the precise registration algorithm used is the ICP algorithm, and the Point-to-Plane distance is used as the geometric term in the ICP metric function. The ICP metric function is constructed as follows:

[0017] E=(1-λ c -λ s )E g +λ c E c +λ s E s

[0018] Among them, E g represents the geometric term, E c represents the photometric term, E s represents a semantic term, λ c and λ s represents the balance weight of the three error terms.

[0019] As an optional implementation, the semantic item is defined as follows:

[0020]

[0021] in, Indicates p i and q i Semantic histogram distance within a neighborhood.

[0022] As an optional implementation, a depth image estimation and a normal image estimation method based on a convolutional neural network are used to generate a depth image and a normal image respectively, so as to supplement the dense three-dimensional structure information lost in the laser point cloud projection process.

[0023] In a second aspect, the present invention provides a semantic segmentation system for laser point clouds collected by drones in power corridors, comprising:

[0024] The data acquisition module is configured to: collect laser point cloud data and image data of the power corridor;

[0025] The registration and projection module is configured to: reconstruct three-dimensional point cloud data based on the image data to obtain the image point cloud, use the semantic histogram distance to assist the laser point cloud and the image point cloud to perform three-dimensional spatial registration, project the registered laser point cloud onto the image, and obtain the projected laser point cloud;

[0026] The multi-mode feature aggregation module is configured to: perform depth image estimation and normal image estimation on the image data to obtain a depth image and a normal image respectively, and fuse the depth image and the normal image with the image and the projected laser point cloud to obtain multi-mode fusion data;

[0027] The semantic segmentation module is configured to: perform two-dimensional semantic segmentation on the multi-modal fusion data, perform back-projection operation on the two-dimensional semantic segmentation result, and obtain the final three-dimensional point cloud semantic segmentation result.

[0028] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0030] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present disclosure proposes a method and system for semantic segmentation of laser point clouds of power corridors collected by drones, which fuses the visible light image of the power corridor with the laser point cloud data, and uses the appearance information contained in the visible light image to achieve high-precision semantic segmentation of the laser point cloud, so as to improve the accuracy, efficiency and precision of the semantic segmentation of the laser point cloud of the power corridor. In order to establish the association between the visible light image and the laser point cloud, the visible light image of the power corridor collected by the drone is aligned with the laser point cloud data in six degrees of freedom space, and the error term is constructed based on the local histogram of the three-dimensional semantic segmentation of the image point cloud and the laser point cloud during the spatial alignment to improve the alignment accuracy. The laser point cloud is then projected to the image perspective for data fusion. Since the laser point cloud projection process loses three-dimensional structural information and the projected image is relatively sparse, in order to supplement the dense three-dimensional structural information, the present invention simultaneously uses the depth image estimation and normal image estimation methods based on convolutional neural networks to generate depth images and normal images, and uses a two-dimensional semantic segmentation convolutional neural network to process the fusion data of visible light images, depth images, normal images and laser point cloud projection images to obtain two-dimensional semantic segmentation results, and finally reversely projects the two-dimensional semantic segmentation results into three-dimensional space to obtain the power corridor laser point cloud semantic segmentation results. The present invention has good robustness, accuracy and scalability, and at the same time helps to improve the utilization rate of drone-collected data in safe distance inspections of transmission lines, and can be used in power corridor inspections, unmanned driving perception, geographic information surveying and mapping and other fields.

[0033] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0035] Figure 1 A flowchart of a method for semantic segmentation of laser point clouds collected by a drone in a power corridor provided in Example 1 of the present invention;

[0036] Figure 2 It is a schematic diagram of the structure of the convolutional neural network for two-dimensional semantic segmentation of laser point clouds of the present invention;

[0037] Figure 3 It is a schematic diagram of the structure of the multi-mode feature aggregation module of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0040] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0042] Example 1

[0043] like Figure 1 As shown, this embodiment provides a method for semantic segmentation of laser point clouds collected by a drone in a power corridor, comprising the following steps:

[0044] S1 collects laser point cloud data and image data of the power corridor;

[0045] S2 reconstructs three-dimensional point cloud data based on image data to obtain image point cloud, uses semantic histogram distance to assist laser point cloud and image point cloud in three-dimensional spatial registration, projects the registered laser point cloud onto the image, and obtains the projected laser point cloud;

[0046] S3 performs depth image estimation and normal image estimation on the image data to obtain a depth image and a normal image respectively, and fuses the depth image and the normal image with the image and the projected laser point cloud to obtain multi-mode fusion data;

[0047] S4 performs two-dimensional semantic segmentation on the multi-modal fusion data, and back-projects the two-dimensional semantic segmentation result to obtain the final three-dimensional point cloud semantic segmentation result.

[0048] Data collection:

[0049] Use commercial drones or handheld devices equipped with cameras and lidar to collect laser point cloud data and image data of the power corridor. The laser point cloud data format is in LAS format, and the image data format is in JPG or PNG format. For daytime scenes, the camera here refers to a visible light camera, and the image in this disclosure refers to a visible light image.

[0050] First, the acquired image must be aligned with the laser point cloud data:

[0051] The image data and point cloud data collected by the drone are not spatially aligned. In order to fuse the image and laser point cloud data, it is necessary to first establish an association relationship between the two different types of data. The present invention uses an image-based 3D point cloud reconstruction algorithm to reconstruct the 3D point cloud corresponding to the image, and then performs 3D spatial registration on the laser point cloud and the image point cloud, that is, calculating a six-degree-of-freedom rigid body transformation matrix, thereby transforming the two sets of point clouds into the same coordinate system, and finally projecting the laser point cloud onto the visible image for subsequent operations.

[0052] Image-based 3D point cloud reconstruction: Image-based 3D point cloud reconstruction includes two parts: sparse reconstruction and dense reconstruction. Considering the reconstruction accuracy requirements, the present disclosure uses the SFM algorithm to calculate sparse 3D point clouds. The sparse reconstruction algorithm here includes but is not limited to the SFM algorithm, and there are also SLAM algorithms. The SFM algorithm used in the present disclosure is an incremental SFM algorithm, and the overall process includes feature point detection and matching, scene graph construction, incremental camera pose calculation, and bundle adjustment. Among them, feature point detection can use SIFT features or SuperPoint features, feature matching can use the nearest neighbor or SuperGlue method, and the global descriptor can use the vocabulary tree or NetVlad method. The above process can restore the coordinates of the image in the local coordinate system, as well as the sparse 3D point cloud of the scene. The laser point cloud is a dense point cloud. In order to establish the association between the image point cloud and the laser point cloud, the sparse 3D point cloud calculated in the previous step needs to be densified. The commonly used 3D point cloud densification method is the MVS algorithm. The present disclosure adopts a dense reconstruction algorithm with PatchMatch as the main idea. The overall process includes four steps: neighbor image selection, depth map calculation, depth map filtering and fusion, and point cloud coloring. Through the above process, the dense point cloud of the scene corresponding to the drone image can be restored.

[0053] Image point cloud and laser point cloud registration: Since the image point cloud and laser point cloud belong to two different coordinate systems, the registration process of the two sets of point clouds is to first calculate the rigid body transformation between the two sets of point clouds through coarse registration, and then correct the rigid body transformation through fine registration. Commonly used registration methods run slowly when there are many three-dimensional points, so the two sets of point clouds are first downsampled. Since the overall structure of the power corridor is relatively uniform, including structures with insignificant geometric features such as transmission lines, ground, and roads, a curvature-based point cloud downsampling method can be used when downsampling, so as to ensure that more three-dimensional points are sampled in areas with large curvature (such as towers, green plants, etc.), and a small number of three-dimensional points are sampled in areas with small curvature (such as transmission lines, ground, etc.). The point cloud coarse registration adopts the classic coarse registration method based on FPFH features.

[0054] Since the above-mentioned coarse registration method is easily affected by noise and sampling points, the obtained rigid body transformation is inaccurate, and it is necessary to improve the accuracy of the rigid body transformation through fine registration. The commonly used fine registration algorithm is the ICP algorithm. Taking into account that the power corridor scene contains a large number of three-dimensional points, in order to speed up the convergence speed of ICP, the present disclosure adopts Point-to-Plane distance as the geometric term in the ICP metric function. In addition, since both the image point cloud and the laser point cloud contain color information, the grayscale error between the corresponding three-dimensional points is added as a photometric term. For visible light images, the photometric term is constructed by converting colors into grayscale values. At the same time, the three-dimensional semantic segmentation results are obtained for the image three-dimensional point cloud and the laser three-dimensional point cloud, respectively, and the error function is constructed based on the three-dimensional semantic segmentation local histogram to improve the accuracy of fine registration. Therefore, the ICP metric function is constructed as follows:

[0055] E=(1-λ c -λ s )E g +λ c E c +λ s E s

[0056] Among them, E g represents the geometric term, E c represents the photometric term, E s represents a semantic term, λ c and λ s represents the balance weight of the three error terms.

[0057] Geometry Term E g The definition is as follows:

[0058]

[0059] Among them, R and t represent the rotation vector and rotation matrix in the six-degree-of-freedom rigid body transformation, pi and qi represent the three-dimensional points in the image point cloud and laser point cloud, respectively, and n qi Indicates q i The normal vector at .

[0060] Photometric term E c The definition is as follows:

[0061]

[0062] in, and are the continuous luminosity functions on the tangent planes of the two point clouds respectively.

[0063] Semantic Item E s The definition is as follows:

[0064]

[0065] in, Indicates p i and q i Semantic histogram distance within the neighborhood. First, use the 3D point cloud semantic segmentation convolutional neural network (such as PointNet, PointNet++, etc.) to process the image point cloud and laser point cloud respectively to obtain the 3D semantic segmentation results. The semantic categories here are towers, wires, green plants, buildings, insulators, roads, upper cross spans, and lower cross spans. Due to the existence of mis-segmentation in 3D semantic segmentation, the same number of K 3D points are randomly sampled within the radius r of each 3D point to form a support set, and the semantic category label histogram corresponding to the support set is statistically calculated. Finally, the semantic histogram correlation between 3D points is calculated. The histogram distance is calculated as:

[0066]

[0067] in, and denote the label mean respectively. Since the semantic segmentation results are prone to errors, λ s Set to a smaller value.

[0068] Through the above process, the image point cloud and the laser point cloud are unified in the same coordinate system. Since the position and posture of the image in the image point cloud have been obtained through sparse reconstruction, the association between the image and the laser point cloud is established.

[0069] Semantic segmentation method of laser point cloud of power corridor by fusion of image and laser point cloud:

[0070] For semantic segmentation, image data contains more appearance information and is more continuous, but loses geometric information and is easily affected by lighting, weather, etc., while 3D point clouds contain geometric information and are more robust to lighting and weather changes. Therefore, image data and point cloud data are complementary. The present invention combines image and laser point cloud information to perform semantic segmentation of power corridor laser point cloud. The overall process is as follows: Figure 1 As shown. In order to use image data and point cloud data for semantic segmentation, it is necessary to represent the two types of data in the same space. Considering the calculation accuracy and efficiency, a two-dimensional image space is selected. First, the laser point cloud is projected onto the image, and the laser point cloud information and image information are fused on the image plane. Given a laser three-dimensional point p i , image X position rotation matrix R, translation vector t, projection point of the laser 3D point on the image It is expressed as:

[0071]

[0072] Where π(·) represents the perspective projection from a 3D point to the image plane. The corresponding image pixel coordinates are further calculated as During the projection process, multiple laser 3D points may be projected onto the same pixel coordinate of the image. According to the nearest occlusion principle, the 3D point closest to the image is taken as the corresponding point. The laser image Y feature after being projected onto the image X has eight dimensions, expressed as (d, x, y, z, r, g, b, γ), where (x, y, z) represents the coordinates of the laser 3D point. represents the distance from the 3D point to the image, (r, g, b) represents the color information of the 3D point, and γ represents the reflectivity of the 3D point.

[0073] Since the laser point cloud projection process loses three-dimensional structural information and the projected image is relatively sparse, in order to supplement the dense three-dimensional structural information, the present invention simultaneously uses a depth image estimation and a normal image estimation method based on a convolutional neural network to generate depth images and normal images, and uses a two-dimensional semantic segmentation convolutional neural network to process visible light images, depth images, normal images and laser projection fusion data, and finally reversely projects the two-dimensional semantic segmentation results into three-dimensional space to obtain the semantic segmentation results of the power corridor laser point cloud. The depth image estimation and normal image estimation of the present invention use the Omnidata deep learning model. Optionally, if there are many valid depth points in the laser point cloud projection, the depth image estimation method can also be replaced by a depth map completion algorithm based on sparse depth.

[0074] The subsequent laser point cloud semantic segmentation algorithm is implemented through a two-dimensional convolutional neural network. The overall structure of the network is as follows: Figure 2 As shown. The visible light image, depth image, normal image and three-dimensional point cloud projection image are respectively subjected to the feature extraction module to extract features of different modalities. The feature extraction module is composed of three convolutional layers and corresponding activation functions. Each basic convolutional layer is a 3×3 convolution with a step size of 2. Then, a residual-based multi-mode feature aggregation module is used to fuse multi-modal features such as visible light images, depth images, normal images and three-dimensional point cloud projection images. Since the feature extraction process is prone to lose high-frequency information, and the segmentation edges are usually in the high-frequency area, the present disclosure adopts a high-frequency feature attention mechanism (HFFM) in the multi-mode feature aggregation module to extract high-frequency information from images of different modalities. Specifically, the feature extraction module is used to extract high-frequency information from the visible image, Depth Image Normal Image Extract deep features containing fine structures from v ,f d ,f n}={E F (I v ),EF (I d ),E F (I n )}, where f v 、f d and f n Represent visible light features, depth image features and normal image features respectively. HFFM ensures the network’s ability to represent fine structures by extracting high-level features. Given the HFFM input f i , first extract high-frequency information:

[0075]

[0076] Among them, Conv(·) represents the convolutional layer, Conv n (·) represents the nth convolutional layer in the cascade. Symbol Represents the gradient operator, which can be regarded as a manually set convolution operation. The high-frequency convolution kernel can be used to extract high-frequency information. This patent uses the common Sobel gradient operator. In addition, It represents element-by-element addition. After that, it passes through a global pooling layer, a 1×1 convolution layer, a batch normalization layer, and a Sigmoid layer to generate an attention vector, which is multiplied with the original features to obtain high-frequency features. The high-frequency features are added element-by-element to the original features to obtain the processed image features of each modality.

[0077] Multi-mode feature aggregation module such as Figure 3As shown in the figure. First, the visible light image features, depth image features, normal image features and laser point cloud features are connected together and processed through a 3×3 convolution layer and an LReLU activation layer. Part of its output is sent to a sub-network consisting of two 3×3 convolution layers, an LReLU activation layer and a Sigmoid function, and is multiplied element-by-element with the original output to obtain preliminary aggregated features. However, since images are easily affected by factors such as lighting or climate, image features are not reliable in outdoor scenes. Therefore, the preliminary aggregated features are supplemented to the laser point cloud features, that is, the output of the previous step is finally added to the laser point cloud features pixel by pixel, and then a 1×1 convolution layer and a Tanh activation layer are passed to obtain the final aggregated features. After feature aggregation, the two-dimensional semantic segmentation results are obtained by designing a convolutional neural network structure. Since the aerial photography distance is usually far, in order to maintain detailed information in the three-dimensional point cloud semantic segmentation, the aggregated features are sent to a spatial branch and a context branch respectively, and the features of the two branches are fused through a multi-scale feature aggregation module. Finally, a decoder is used to generate a two-dimensional semantic segmentation result with the same image resolution. The structural design of the spatial branch, context branch, multi-scale feature aggregation and decoder can refer to the BiSeNet V2 algorithm published in IJCV2021. The spatial branch is mainly used to extract low-level spatial detail information. The network is generally relatively shallow, and avoids the use of residual connections to improve the inference speed and reduce the memory usage; the context branch is used to extract high-level semantic information. The number of channels in the branch is less than that of the spatial branch, and the receptive field is increased through multiple downsampling. The multi-scale feature aggregation module merges the spatial branch features with the context branch features.

[0078] The above two-dimensional semantic segmentation results are used to achieve the final three-dimensional point cloud semantic segmentation through back-projection operation. The back-projection operation is expressed as:

[0079]

[0080] Among them, (u, v) represents the image pixel, d is the laser point cloud depth value of the pixel, K -1 is the inverse of the camera intrinsic parameter matrix. The above operations transform the 2D semantic segmentation results into a 3D coordinate system. Since the same 3D point is usually visible in multiple images, and the 2D semantic segmentation results may be different, the semantic category with the most observations is assigned to the 3D point.

[0081] Taking into account the requirements of the power corridor scenario for high-precision semantic segmentation, the present disclosure adopts a supervised loss function design, that is, the first type of loss function of the present disclosure is constructed by the difference between the true value of the semantic label of each three-dimensional point and the predicted value of the algorithm. Since the three-dimensional point cloud is visible in multiple images, and there are differences in the semantic segmentation results of the fusion data of different images, the present disclosure introduces a second type of loss function in the loss function, which is called the multi-view semantic consistency loss function. The calculation method of this loss function is as follows: for each image, by calculating the corresponding viewing cone, find the image with the largest common viewing area with it, and obtain the corresponding pixel points on the two images by laser point cloud projection. Through the previous semantic segmentation step, the corresponding pixel points on each image can get a semantic label, and theoretically these semantic labels should be the same. Therefore, based on the difference between the semantic labels of the corresponding pixels of the two images, a multi-view semantic consistency loss function is constructed.

[0082] The technical solution disclosed in the present invention can realize high-precision semantic segmentation of laser point clouds based on visible light images or laser point clouds collected by drones. In addition, in order to establish the association between visible light images and laser point clouds, a method for aligning drone aerial images and laser point clouds based on a three-dimensional semantic segmentation local histogram is proposed. The above method can not only be used for power corridor safety distance analysis, but also for other tasks such as power corridor defect detection, unmanned driving scene perception, and real-life three-dimensional geographic mapping.

[0083] Example 2

[0084] This embodiment provides a power corridor laser point cloud semantic segmentation system collected by a drone, including:

[0085] The data acquisition module is configured to: collect laser point cloud data and image data of the power corridor;

[0086] The registration and projection module is configured to: reconstruct three-dimensional point cloud data based on the image data to obtain the image point cloud, use the semantic histogram distance to assist the laser point cloud and the image point cloud to perform three-dimensional spatial registration, project the registered laser point cloud onto the image, and obtain the projected laser point cloud;

[0087] The multi-mode feature aggregation module is configured to: perform depth image estimation and normal image estimation on the image data to obtain a depth image and a normal image respectively, and fuse the depth image and the normal image with the image and the projected laser point cloud to obtain multi-mode fusion data;

[0088] The semantic segmentation module is configured to: perform two-dimensional semantic segmentation on the multi-modal fusion data, perform back-projection operation on the two-dimensional semantic segmentation result, and obtain the final three-dimensional point cloud semantic segmentation result.

[0089] It should be noted that the above modules correspond to the steps described in Example 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0090] In further embodiments, there is also provided:

[0091] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, it will not be described in detail here.

[0092] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0093] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0094] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in Example 1 is completed.

[0095] The method in Example 1 can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it is not described in detail here.

[0096] A computer program product includes a computer program, and when the computer program is executed by a processor, the method described in embodiment 1 is implemented.

[0097] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer executable instructions, such as instructions included in a program module, which are executed in a device on a real or virtual processor of the target to perform the process / method as described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules can be combined or divided between program modules as needed. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0098] The computer program code for implementing the method of the present invention can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the computer or other programmable data processing device, causes the function / operation specified in the flow chart and / or block diagram to be implemented. The program code can be executed completely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer or completely on a remote computer or server.

[0099] In the context of the present invention, computer program codes or related data may be carried by any appropriate carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, and the like. Examples of signals may include electrical, optical, radio, acoustic or other forms of propagation signals, such as carrier waves, infrared signals, and the like.

[0100] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0101] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for laser point cloud of power corridor collected by drone, characterized in that: The following steps are involved: Collect laser point cloud data and image data of power corridors; Reconstruct three-dimensional point cloud data based on image data to obtain image point cloud, use semantic histogram distance to assist laser point cloud and image point cloud in three-dimensional spatial registration, project the registered laser point cloud onto the image, and obtain the projected laser point cloud; Performing depth image estimation and normal image estimation on the image data to obtain a depth image and a normal image respectively, and fusing the depth image and the normal image with the image and the projected laser point cloud to obtain multi-mode fusion data; Perform two-dimensional semantic segmentation on the multi-modal fusion data, and perform back-projection operation on the two-dimensional semantic segmentation result to obtain the final three-dimensional point cloud semantic segmentation result.

2. The method for semantic segmentation of laser point clouds collected by drones in power corridors according to claim 1, characterized in that: Reconstruct the three-dimensional point cloud data based on the image data to obtain the image point cloud, specifically: The SFM algorithm is used to calculate the sparse 3D point cloud of the image, and the dense reconstruction algorithm based on PatchMatch is used to densify the sparse 3D point cloud to obtain the image point cloud.

3. The method for semantic segmentation of laser point clouds collected by drones in power corridors according to claim 1, characterized in that: The laser point cloud and the image point cloud are registered in three-dimensional space, specifically: First, the rigid body transformation between the laser point cloud and the image point cloud is preliminarily calculated through coarse registration, and then the rigid body transformation is corrected through fine registration to obtain the six-degree-of-freedom rigid body transformation matrix. Based on the six-degree-of-freedom rigid body transformation matrix, the laser point cloud and the image point cloud are registered in three-dimensional space.

4. The method for semantic segmentation of laser point clouds collected by drones in power corridors as claimed in claim 3, characterized in that: The precise registration algorithm used is the ICP algorithm, and the Point-to-Plane distance is used as the geometric term in the ICP metric function. The ICP metric function is constructed as follows: E=(1-λ c -l s )E g +λ c E c +λ s E s Among them, E g represents the geometric term, E c represents the photometric term, E s represents a semantic term, λ c and λ s represents the balance weight of the three error terms.

5. The method for semantic segmentation of laser point clouds collected by drones in power corridors as claimed in claim 4, characterized in that: The semantic terms are defined as follows: in, Indicates p i and q i Semantic histogram distance within a neighborhood.

6. The method for semantic segmentation of laser point clouds collected by drones in power corridors according to claim 1, characterized in that: Depth image estimation and normal image estimation methods based on convolutional neural networks are used to generate depth images and normal images respectively, which are used to supplement the dense three-dimensional structure information lost in the laser point cloud projection process.

7. A semantic segmentation system for laser point clouds of power corridors collected by drones, characterized in that: include: The data acquisition module is configured to: collect laser point cloud data and image data of the power corridor; The registration and projection module is configured to: reconstruct three-dimensional point cloud data based on the image data to obtain the image point cloud, use the semantic histogram distance to assist the laser point cloud and the image point cloud to perform three-dimensional spatial registration, project the registered laser point cloud onto the image, and obtain the projected laser point cloud; The multi-mode feature aggregation module is configured to: perform depth image estimation and normal image estimation on the image data to obtain a depth image and a normal image respectively, and fuse the depth image and the normal image with the image and the projected laser point cloud to obtain multi-mode fusion data; The semantic segmentation module is configured to: perform two-dimensional semantic segmentation on the multi-modal fusion data, perform back-projection operation on the two-dimensional semantic segmentation result, and obtain the final three-dimensional point cloud semantic segmentation result.

8. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 6 is completed.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images