Point cloud data processing method and device for high voltage tower in power grid

Abstract

The present application relates to a method, device, computer equipment and storage medium for processing point cloud data of high-voltage towers in power grids. The method comprises: obtaining electromagnetic parameter information of high-voltage towers and high-voltage wires and real-time information data of the environment within a preset radius of the high-voltage towers; generating a three-dimensional model of a scene including high-voltage towers and high-voltage wires based on the real-time information data; performing semantic segmentation on the high-voltage towers and high-voltage wires in the three-dimensional model of the scene to obtain three-dimensional models of different components; and uniformly managing and controlling the point cloud data of components based on the point cloud coordinate information of the three-dimensional model of the component in the three-dimensional model of the scene. The use of this method can timely and accurately support defect detection, troubleshooting and other management of various components of power facilities, thereby improving the overall management efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of power facility management, and in particular to a method, device, computer equipment and storage medium for processing point cloud data of high-voltage towers in a power grid. Background Art

[0002] With the advancement of science and technology, people's demand for electric energy is increasing, and the scale of the power grid is expanding accordingly. Among them, power facilities are the carriers of electric energy production, transmission and supply, and are important social public facilities. The safety protection of power facilities is an important part of ensuring the safety of power supply and maintenance of social public safety. However, some power facilities, especially high-voltage towers and high-voltage transmission lines, are exposed to the outdoors for a long time, which makes the parts easily damaged, affecting the safety of power facilities. Therefore, it is crucial to timely and accurately detect defects in power facilities, troubleshoot and manage them, improve the quality of the entire circuit system, and effectively ensure the safety of power facilities.

[0003] At present, in the existing technology, the pole tower needs to be processed first to obtain the pole tower basic model, the key component basic model and the pole tower surrounding environment basic model respectively, and then the obtained models are combined to obtain the final pole tower virtual reality three-dimensional model. In addition, the pole tower virtual reality three-dimensional model and the digital pole tower archive need to be integrated and visualized to obtain a digital pole tower management and control system.

[0004] The processing process of the above-mentioned prior art is complicated and has the problem of low management and control efficiency. Summary of the invention

[0005] Based on this, it is necessary to provide a more efficient method, device, computer equipment and storage medium for processing point cloud data of high-voltage towers in power grids in response to the above-mentioned technical problems.

[0006] A method for processing point cloud data of a high-voltage tower in a power grid, comprising:

[0007] Obtain electromagnetic parameter information of high-voltage towers and high-voltage wires and real-time environmental information data within a preset radius of the high-voltage towers;

[0008] Generate a three-dimensional model of the scene including high-voltage towers and high-voltage wires based on real-time information data;

[0009] Perform semantic segmentation on the high-voltage tower and high-voltage wires in the 3D model of the scene to obtain 3D models of different components;

[0010] Based on the point cloud coordinate information of the component 3D model in the scene 3D model, the component point cloud data is uniformly managed and controlled.

[0011] In one embodiment, obtaining electromagnetic parameter information of a high-voltage iron tower and a high-voltage electric wire and real-time information data of the environment within a preset radius of the high-voltage iron tower also includes:

[0012] Push flight instructions to the drone;

[0013] Receive real-time information data on high-voltage towers and high-voltage wires sent back by drones, as well as information within a preset radius of the high-voltage towers.

[0014] In one embodiment, a three-dimensional model of a scene including a high-voltage iron tower and a high-voltage wire is generated based on real-time information data, and further includes:

[0015] According to the RTK data in the real-time information data, the neighbor relationship between the RGB images is determined, and the RGB images are sorted to obtain an image sequence;

[0016] The SIFT algorithm is used to extract the feature points of each image in the image sequence;

[0017] Match feature points between two images in the image sequence to obtain matching feature points;

[0018] The camera pose is obtained based on the position of the matching feature points in the image and the intrinsic parameter data of the camera.

[0019] According to the camera pose, the three-dimensional coordinates of the matching feature points are obtained to obtain a sparse three-dimensional point cloud model of the high-voltage tower and the high-voltage line;

[0020] According to the sparse 3D point cloud model, a dense 3D point cloud model is obtained;

[0021] Based on the dense 3D point cloud model, a 3D model of the scene including high-voltage towers and high-voltage wires is obtained.

[0022] In one embodiment, obtaining a dense 3D point cloud model according to the sparse 3D point cloud model further includes:

[0023] According to the camera pose, dense stereo matching is used to obtain the depth map corresponding to each image in the image sequence;

[0024] According to the sparse 3D point cloud model, the depth map is fused to obtain a dense 3D point cloud model.

[0025] In one embodiment, before semantic segmentation of the high-voltage iron tower and the high-voltage wire in the three-dimensional model of the scene is performed, the method further includes:

[0026] A deep learning fully convolutional network structure is used to perform semantic segmentation on the RGB image to obtain a two-dimensional image semantic segmentation result, where the two-dimensional image semantic segmentation result includes the semantic category corresponding to each pixel point of the high-voltage tower and high-voltage wire in the two-dimensional image.

[0027] In one embodiment, a deep learning fully convolutional network structure is adopted to perform semantic segmentation on the RGB image to obtain a two-dimensional image semantic segmentation result, and the two-dimensional semantic segmentation result is obtained, which also includes:

[0028] Get the deep learning fully convolutional network model;

[0029] Input the RGB image into the deep learning fully convolutional network structure model, perform convolution encoding on the RGB image, and obtain a feature map smaller than the original RGB image size;

[0030] Perform deconvolution decoding on the feature map of the RGB image to obtain a feature map of the size of the original RGB image;

[0031] According to the feature map of the original RGB image size, the two-dimensional image semantic segmentation result is obtained.

[0032] In one embodiment, semantic segmentation of high-voltage iron towers and high-voltage wires in a three-dimensional model of a scene is performed based on the semantic segmentation result of the two-dimensional image, and further includes:

[0033] Get the semantic segmentation result of the two-dimensional image;

[0034] Mapping the pixels of the two-dimensional image to the vertices of the high-voltage towers and high-voltage wires in the three-dimensional model of the scene;

[0035] According to the semantic category corresponding to each pixel point of the high-voltage iron tower and the high-voltage electric wire in the two-dimensional image, the semantic category of the three-dimensional high-voltage iron tower and the high-voltage electric wire vertex corresponding to the pixel point is obtained;

[0036] According to the semantic categories of the vertices of the three-dimensional high-voltage towers and high-voltage wires, the semantic segmentation results of the high-voltage towers and high-voltage wires in the three-dimensional model of the scene are obtained.

[0037] A point cloud data processing device for a high-voltage tower in a power grid, the device comprising:

[0038] A data acquisition module is used to acquire electromagnetic parameter information of high-voltage iron towers and high-voltage electric wires and real-time environmental information data within a preset radius of the high-voltage iron towers;

[0039] The data processing module generates a three-dimensional model of the scene including high-voltage towers and high-voltage wires according to real-time information data;

[0040] The semantic segmentation module is used to perform semantic segmentation on the high-voltage tower and high-voltage wires in the scene 3D model to obtain 3D models of different components;

[0041] The control module is used to uniformly control the point cloud data of parts according to the point cloud coordinate information of the three-dimensional models of the parts in the three-dimensional model of the scene.

[0042] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0043] Obtain electromagnetic parameter information of high-voltage towers and high-voltage wires and real-time environmental information data within a preset radius of the high-voltage towers;

[0044] Generate a three-dimensional model of the scene including high-voltage towers and high-voltage wires based on real-time information data;

[0045] Semantically segment the high-voltage towers and high-voltage wires in the 3D model of the scene to obtain 3D models of different components;

[0046] Based on the point cloud coordinate information of the component 3D model in the scene 3D model, the component point cloud data is uniformly managed and controlled.

[0047] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0048] Obtain electromagnetic parameter information of high-voltage towers and high-voltage wires and real-time environmental information data within a preset radius of the high-voltage towers;

[0049] Generate a three-dimensional model of the scene including high-voltage towers and high-voltage wires based on real-time information data;

[0050] Semantically segment the high-voltage towers and high-voltage wires in the 3D model of the scene to obtain 3D models of different components;

[0051] Based on the point cloud coordinate information of the component 3D model in the scene 3D model, the component point cloud data is uniformly managed and controlled.

[0052] The above-mentioned method, device, computer equipment and storage medium for processing point cloud data of high-voltage towers in the power grid obtain electromagnetic parameter information of high-voltage towers and high-voltage wires and real-time information data of the environment within a preset radius of the high-voltage towers; generate a three-dimensional model of the scene including high-voltage towers and high-voltage wires based on the real-time information data; perform semantic segmentation on the high-voltage towers and high-voltage wires in the three-dimensional model of the scene to obtain different three-dimensional models of components; and perform unified management and control of the point cloud data of components based on the point cloud coordinate information of the three-dimensional model of the component in the three-dimensional model of the scene. It can timely and accurately support the management of defects detection, troubleshooting and other management of various components of power facilities, and improve the overall management and control efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A diagram showing an application environment of a method for processing point cloud data of a high-voltage tower in a power grid in one embodiment;

[0054] Figure 2 A schematic diagram of a process for processing point cloud data of a high-voltage tower in a power grid in one embodiment;

[0055] Figure 3 A schematic diagram of a process of reconstructing a three-dimensional model of a scene in one embodiment;

[0056] Figure 4 A schematic diagram of a process of using a deep learning fully convolutional network structure to perform semantic segmentation on an RGB image to obtain a semantic segmentation result of a two-dimensional image in another embodiment;

[0057] Figure 5 A schematic diagram of a process of performing semantic segmentation on high-voltage iron towers and high-voltage wires in a three-dimensional model of a scene according to a semantic segmentation result of a two-dimensional image in another embodiment;

[0058] Figure 6 is a structural block diagram of a point cloud data processing device for a high-voltage tower in a power grid in one embodiment;

[0059] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The method for processing point cloud data of high-voltage towers in power grids provided in this application can be applied to Figure 1 In the application environment shown. Among them, the server 101 controls the drone 102 to take real-time photos of the high-voltage tower 103 to collect continuous images and transmit the data back to the server 101. The server 101 obtains the electromagnetic parameter information of the high-voltage tower and the high-voltage wire and the real-time information data of the environment within the preset radius of the high-voltage tower. According to the acquired real-time information data, a scene three-dimensional model including the high-voltage tower and the high-voltage wire is generated, and the high-voltage tower and the high-voltage wire in the scene three-dimensional model are semantically segmented to obtain different three-dimensional models of parts; according to the point cloud coordinate information of the three-dimensional model of the parts in the three-dimensional model of the scene, the point cloud data of the parts is uniformly managed and controlled. Among them, the server 101 can be implemented by an independent server or a server cluster composed of multiple servers.

[0062] In one embodiment, Figure 2 As shown, a method for processing point cloud data of high-voltage towers in a power grid is provided, and the method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0063] S100: Acquire electromagnetic parameter information of the high-voltage iron tower and the high-voltage electric wire and real-time information data of the environment within a preset radius of the high-voltage iron tower.

[0064] Among them, real-time information data can be electromagnetic parameter information about high-voltage towers and high-voltage wires and data on the environment within a preset radius of the high-voltage towers collected by continuous images taken by drones, including RGB (Red Green Blue) images and RTK (Real-time kinematic) data.

[0065] Specifically, the server 101 controls the drone 102 to photograph the high-voltage tower 103 , collects real-time environmental information data of the high-voltage tower and the high-voltage wires, as well as within a preset radius of the high-voltage tower, and transmits the collected real-time information data back to the server 101 .

[0066] S200: Generate a three-dimensional model of a scene including high-voltage towers and high-voltage wires based on real-time information data.

[0067] The scene three-dimensional model is a virtual reality scene three-dimensional model obtained by dense three-dimensional reconstruction of the collected scene image data.

[0068] Specifically, after acquiring the real-time information data of the environment within the preset radius of the high-voltage iron tower, the server 101 can construct a three-dimensional model of the scene based on this part of the data. The construction of the three-dimensional model of the scene can be achieved by traditional methods or existing three-dimensional scene construction software. Specifically, the real-time information data of the environment within the preset radius of the high-voltage iron tower can be imported into the existing three-dimensional scene construction software, and the three-dimensional model of the scene including the high-voltage iron tower and the high-voltage wire is generated by running the existing three-dimensional scene construction software.

[0069] S300: semantically segmenting high-voltage towers and high-voltage wires in the three-dimensional model of the scene to obtain three-dimensional models of different components.

[0070] Among them, semantic segmentation refers to the semantic classification of basic elements of two-dimensional images or three-dimensional models including high-voltage towers and high-voltage wires, so as to achieve a high-dimensional understanding of the content information of the power scene.

[0071] Specifically, in order to perform semantic segmentation on the obtained three-dimensional model of the scene and accurately segment it into components of high-voltage towers and high-voltage wires, for example: the high-voltage tower is segmented into components such as angle steel, connecting steel plates, and foot nails, and the high-voltage wire is segmented into components such as transformers and distribution equipment, lightning conductors, line hardware, insulators, and pole towers, the server 101 can first perform semantic segmentation on the collected two-dimensional image, perform pixel semantic level prediction and segmentation on the input image to obtain the two-dimensional image semantic segmentation result, and then migrate the two-dimensional image semantic segmentation result to the three-dimensional model. Under the guidance of the semantic segmentation result of the two-dimensional image, the corresponding three-dimensional model of the component in the scene three-dimensional model can be clearly segmented. Specifically, the pixel points after the semantic segmentation of the two-dimensional image can be mapped with the points in the three-dimensional model, so as to determine the semantic category to which each point of the three-dimensional model belongs.

[0072] S400: uniformly controlling the point cloud data of the components according to the point cloud coordinate information of the three-dimensional model of the components in the three-dimensional model of the scene.

[0073] The point cloud coordinate information refers to the set of three-dimensional point cloud coordinates of the segmented component three-dimensional model in the entire scene three-dimensional model. Specifically, the server 101 obtains the point cloud coordinate information of each component three-dimensional model in the scene three-dimensional model. Each component three-dimensional model has a corresponding point cloud coordinate data set in the scene three-dimensional model, and generates an independent ID for the point cloud coordinate data set, thereby uniformly managing and controlling the component point cloud data.

[0074] The above-mentioned method for processing point cloud data of high-voltage towers in power grids obtains electromagnetic parameter information of high-voltage towers and high-voltage wires and real-time information data of the environment within a preset radius of high-voltage towers; generates a three-dimensional model of the scene including high-voltage towers and high-voltage wires based on the real-time information data; semantically segments the high-voltage towers and high-voltage wires in the three-dimensional model of the scene to obtain different three-dimensional models of components; and uniformly manages the point cloud data of components based on the point cloud coordinate information of the three-dimensional model of the component in the three-dimensional model of the scene. It can timely and accurately support the management of defect detection, troubleshooting, etc. of various components of power facilities, and improve the overall management efficiency.

[0075] In one embodiment, S100 includes:

[0076] Step 1: Push flight instructions to the drone;

[0077] Step 2: Receive real-time information data of high-voltage towers and high-voltage wires, as well as data within a preset radius of the high-voltage towers, sent back by the drone.

[0078] Among them, the flight instruction refers to an instruction sent by the server 101 to the drone 102, which is used to control the drone 102 to take real-time photos and collect environmental data of high-voltage towers, high-voltage wires and within a preset radius of the high-voltage towers along a preset route; specifically, the server 101 sends a flight instruction to the drone 102, and the drone 102 receives the instruction, takes real-time photos and collects real-time information data of high-voltage towers, high-voltage wires and within a preset radius of the high-voltage towers along a preset bow-shaped route, and transmits the collected real-time information data back to the server 101, and the server 101 receives the returned real-time information data, wherein the real-time information data includes RGB images and RTK data.

[0079] In this embodiment, by sending flight instructions to the drone, controlling the drone 102 to take real-time photos and collect environmental information data of high-voltage towers, high-voltage wires, and within a preset radius of the high-voltage towers along a preset route and receiving real-time information data sent back by the drone, relevant RGB images and RTK data can be obtained, and the obtained image data is used as input data, and a plurality of algorithms are used for fusion processing to reconstruct the generated virtual reality three-dimensional model for subsequent semantic segmentation of the three-dimensional model.

[0080] In one embodiment, Figure 3 As shown, step S200 includes:

[0081] S220: Determine the neighbor relationship between the RGB images according to the RTK data in the real-time information data, sort the RGB images, and obtain an image sequence.

[0082] Among them, RTK data refers to the longitude and latitude coordinate data of the image collected by the drone; RGB image refers to the image displayed in RGB color mode. Specifically, according to the RTK data, the longitude and latitude coordinate information of each collected RGB image is determined, the neighbor relationship between the RGB images is determined, and the RGB images are sorted according to the obtained neighbor relationship to obtain an image sequence.

[0083] S240: extracting feature points of each image in the image sequence using the SIFI algorithm.

[0084] Among them, the SIFI (Scale Invariant Feature Transform) algorithm is an algorithm for extracting local features of an image, which can handle the matching problem between two images when translation, rotation or affine transformation occurs. Feature points refer to points with rich texture and easy identification in the images of high-voltage towers and high-voltage wires. Specifically, the points with rich texture and easy identification in the images of high-voltage towers and high-voltage wires are detected, and the SIFT algorithm is used to extract the feature point information of each image in the image sequence. The feature point information includes the two-dimensional coordinate value of the feature point in the image and the descriptor of the feature point, where the descriptor represents the direction information of the feature point.

[0085] S260: Match feature points between two images in the image sequence to obtain matching feature points.

[0086] Among them, matching feature points refer to feature point pairs obtained by matching feature points of two images in pairs, and the matching feature points can correspond to a specific point in the actual scene. Specifically, after the feature points are obtained by using the SIFT algorithm, the SIFT algorithm is also used to match the feature points in the two images one by one to obtain feature point pairs after successful matching, i.e., matching feature points.

[0087] S280: Obtain the camera posture according to the positions of the matching feature points in the image and the intrinsic parameter data of the image acquisition device.

[0088] The camera pose refers to the position and orientation of the image acquisition device when it acquires images. The intrinsic parameter data mainly refers to the focal length of the image acquisition device, the position of the principal point, etc. Specifically, two images with matching feature points are selected, the geometric relationship described by epipolar geometry is used to estimate the basic matrix, the basic matrix and the internal parameter data of the image acquisition device are used to estimate the essential matrix, and finally the camera pose is obtained according to the essential matrix.

[0089] S300: According to the camera posture, the three-dimensional coordinates of the matching feature points are obtained to obtain a sparse three-dimensional point cloud model including the high-voltage tower and the high-voltage wires.

[0090] Among them, the sparse 3D point cloud model refers to the set of 3D coordinates of matching feature points in space, with a small number of points and a large spacing between points. Specifically, after obtaining the camera pose, the 3D coordinates of the matching feature points between images in space are calculated through triangulation, thereby obtaining a sparse 3D point cloud model of the scene including high-voltage towers and high-voltage wires.

[0091] S320: Obtain a dense three-dimensional point cloud model based on the sparse three-dimensional point cloud model.

[0092] S340: Construct a three-dimensional model of the scene including high-voltage towers and high-voltage wires based on the dense three-dimensional point cloud model.

[0093] Specifically, based on the obtained camera pose, the depth map corresponding to each image is restored using dense stereo matching technology, and the depth map corresponding to each image is fused to obtain a dense 3D point cloud model. The obtained dense 3D point cloud model is meshed, that is, the dense 3D mesh of the high-voltage tower and high-voltage wire scene is extracted from the dense 3D point cloud model using surface mesh extraction technology, and the obtained dense 3D mesh is the final 3D model of the scene.

[0094] In this embodiment, the SIFT algorithm is first used to extract feature points in the matching image. It is invariant to the rotation and size change of the image and will not be affected by the size change or deformation of the image. It can extract enough feature points and can quickly and accurately match a large number of feature points, thereby improving the efficiency of feature point extraction and matching. Secondly, the camera posture is obtained according to the matching feature points, a sparse three-dimensional point cloud model is obtained, and then a dense three-dimensional point cloud model is obtained to obtain a three-dimensional model of the scene, and finally the three-dimensional reconstruction of the high-voltage iron tower and high-voltage wire scene is completed, and the actual high-voltage iron tower and high-voltage wire scene is converted into a virtual reality scene three-dimensional model and stored in the server, so as to facilitate the subsequent semantic segmentation of the reconstructed scene three-dimensional model, segment the corresponding three-dimensional model of the parts, and realize the unified management of the parts of the high-voltage iron tower and high-voltage wire.

[0095] In one embodiment, S320 includes:

[0096] S322: Based on the camera pose, dense stereo matching is performed to obtain a depth map corresponding to each image in the image sequence.

[0097] S324: According to the sparse 3D point cloud model, the depth map is fused to obtain a dense 3D point cloud model.

[0098] Among them, dense stereo matching refers to the technology of selecting multiple feature points, establishing the correspondence between feature points, matching the three-dimensional points in space with the feature points between different images, and recovering the image depth information from the two-dimensional image. The depth map is obtained by matching two or more images, and is used to represent the depth information of each point in the scene, that is, the distance between each point in the scene and the image acquisition device. Specifically, the dense stereo matching technology is used to recover the depth map corresponding to each image in the image sequence. On the basis of the sparse point cloud model, the depth map is fused based on the depth map fusion method to obtain a dense three-dimensional point cloud model.

[0099] In this embodiment, the depth map of the two-dimensional image is restored by dense stereo matching technology to obtain a dense three-dimensional point cloud model, which can obtain a more accurate depth map and thus a more accurate dense three-dimensional point cloud model, enabling better reconstruction of the scene three-dimensional model.

[0100] In one embodiment, Figure 4 As shown in the figure, a deep learning fully convolutional network structure is used to perform semantic segmentation on the RGB image to obtain the semantic segmentation results of the two-dimensional image, including:

[0101] S400: Obtain a deep learning fully convolutional network model.

[0102] S420: Input the RGB image into the fully convolutional deep learning network structure model, perform convolution encoding on the RGB image, and obtain a feature map smaller than the size of the original RGB image.

[0103] S440: Perform deconvolution decoding on the feature map of the RGB image to obtain a feature map of the size of the original RGB image.

[0104] Among them, the deep learning full convolution network model refers to a network structure model based on deep learning full convolution; convolutional coding refers to the process of extracting image features through a convolutional network. Specifically, after obtaining the deep learning full convolutional network model, the RGB image is input into the full convolutional deep learning network structure model. In the convolution layer of the model, the input image is convolved multiple times using the convolution kernel to extract the features of the input image. Different features are represented by different colors, and a feature map smaller than the original RGB image size is output. Then, the feature map smaller than the original RGB image size obtained by convolutional coding is deconvolutionally decoded, and the feature map is gradually enlarged to restore it to the original RGB image size, thereby obtaining a feature map of the original RGB image size, so that a prediction can be generated for each pixel while retaining the spatial information in the original RGB image.

[0105] S460: Obtain a two-dimensional image semantic segmentation result according to the feature map of the original RGB image size.

[0106] The 2D semantic segmentation result refers to the semantic category corresponding to each pixel in the 2D image obtained by semantic segmentation of the 2D image. Specifically, the original RGB image size feature map obtained by deconvolution decoding is classified pixel by pixel according to semantics to obtain the final 2D image semantic segmentation result.

[0107] In this embodiment, by adopting a deep learning full convolutional network model, the input RGB image is classified at the pixel level in a manner of encoding first and then decoding, so as to realize the semantic segmentation of high-voltage iron towers and high-voltage wire images of any size, improve the robustness of the full convolutional network, and make the image segmentation more efficient. In addition, the pixel points are classified according to semantics to obtain the semantic segmentation result of the two-dimensional image, which is convenient for providing a basis for the subsequent semantic segmentation of the three-dimensional model, and can make the semantic segmentation effect of the three-dimensional model better.

[0108] In one embodiment, Figure 5 As shown, based on the semantic segmentation results of the two-dimensional image, the high-voltage towers and high-voltage wires in the three-dimensional model of the scene are semantically segmented, including:

[0109] S500: Obtaining a two-dimensional image semantic segmentation result;

[0110] S520: Mapping the pixel points of the two-dimensional image with the vertices of the high-voltage tower and the high-voltage wire in the three-dimensional model of the scene;

[0111] S540: According to the semantic category corresponding to each pixel point of the high-voltage iron tower and the high-voltage electric wire in the two-dimensional image, the semantic category of the three-dimensional high-voltage iron tower and the high-voltage electric wire vertex corresponding to the pixel point is obtained;

[0112] S560: Obtain semantic segmentation results of the high-voltage iron tower and the high-voltage electric wire in the three-dimensional model of the scene according to the semantic categories of the vertices of the three-dimensional high-voltage iron tower and the high-voltage electric wire.

[0113] Specifically, since the two-dimensional image and the three-dimensional model of the scene belong to the same scene and the three-dimensional model is generated based on the reconstruction of the two-dimensional image, the pixels of the two-dimensional image can be matched with the vertices of the three-dimensional model of the scene; the two-dimensional coordinates of each pixel in the two-dimensional image are obtained, and the pixels in the two-dimensional image are mapped with the vertices in the three-dimensional model of the scene through the conversion relationship between the two-dimensional coordinates and the three-dimensional coordinates of each vertex in the three-dimensional model of the scene, wherein one vertex in the three-dimensional model only corresponds to one pixel in the two-dimensional image; multiple vertices in the three-dimensional model may correspond to one pixel in the two-dimensional image at the same time. According to the semantic category to which the pixel of the two-dimensional image belongs, the semantic category to which each vertex of the three-dimensional model of the scene belongs is determined, and then according to the semantic category of each vertex, the semantic segmentation result of the high-voltage iron tower and the high-voltage wire in the three-dimensional model of the scene is obtained.

[0114] In one embodiment, Figure 6 As shown, a point cloud data processing device for a high-voltage tower in a power grid is provided, comprising a data acquisition module 100, a data processing module 200, a semantic segmentation module 300 and a control module 400:

[0115] The data acquisition module 100 is used to acquire electromagnetic parameter information of the high-voltage iron tower and the high-voltage electric wire and real-time information data of the environment within a preset radius of the high-voltage iron tower;

[0116] The data processing module 200 generates a three-dimensional model of the scene including the high-voltage tower and the high-voltage wire according to the real-time information data;

[0117] The semantic segmentation module 300 is used to perform semantic segmentation on the high-voltage tower and high-voltage wires in the scene three-dimensional model to obtain three-dimensional models of different components;

[0118] The control module 400 is used to uniformly control the components of the high-voltage tower and the high-voltage wire according to the point cloud coordinate information of the three-dimensional models of different components in the three-dimensional model of the scene.

[0119] In one embodiment, the data acquisition module 100 is also used to push flight instructions to the drone; receive real-time information data of the high-voltage tower and high-voltage wires, as well as the high-voltage tower within a preset radius sent back by the drone.

[0120] In one embodiment, the data processing module 200 is also used to determine the proximity relationship between RGB images based on the RTK data in the real-time information data, sort the RGB images, and obtain an image sequence; use the SIFT algorithm to extract the feature points of each image in the image sequence; match the feature points of the images in the image sequence pair by pair to obtain matching feature points; obtain the camera pose based on the position of the matching feature points in the image in combination with the intrinsic parameter data of the image acquisition device, and the image acquisition device is a device for collecting real-time information data; obtain the three-dimensional coordinates of the matching feature points based on the camera pose to obtain a sparse three-dimensional point cloud model of the scene including high-voltage towers and high-voltage lines; obtain a dense three-dimensional point cloud model based on the sparse three-dimensional point cloud model; and construct a three-dimensional model of the scene including high-voltage towers and high-voltage wires based on the dense three-dimensional point cloud model.

[0121] In one embodiment, the data processing module 200 is further used to obtain a depth map corresponding to each image in the image sequence by dense stereo matching according to the camera posture; and to fuse the depth map according to the sparse three-dimensional point cloud model to obtain a dense three-dimensional point cloud model;

[0122] In one embodiment, the semantic segmentation module 300 is further used to perform semantic segmentation on the RGB image using a deep learning fully convolutional network structure to obtain a two-dimensional image semantic segmentation result; based on the two-dimensional image semantic segmentation result, perform semantic segmentation on the high-voltage iron tower and the high-voltage wire in the three-dimensional model of the scene;

[0123] In one embodiment, the semantic segmentation module 300 is also used to obtain a fully convolutional deep learning network model; input the RGB image into the fully convolutional deep learning network structure model, perform convolution encoding on the RGB image, and obtain a feature map that is smaller than the original RGB image size; perform deconvolution decoding on the feature map of the RGB image to obtain a feature map of the size of the original RGB image; and obtain a two-dimensional image semantic segmentation result based on the feature map of the size of the original RGB image.

[0124] In one embodiment, the semantic segmentation module 300 is also used to obtain the semantic segmentation results of the two-dimensional image; map the pixel points of the two-dimensional image with the vertices of the high-voltage towers and high-voltage wires in the three-dimensional model of the scene; obtain the semantic categories of the three-dimensional high-voltage towers and high-voltage wires vertices corresponding to the pixel points based on the semantic categories corresponding to each pixel point of the high-voltage towers and high-voltage wires in the two-dimensional image; obtain the semantic segmentation results of the high-voltage towers and high-voltage wires in the three-dimensional model of the scene based on the semantic categories of the vertices of the three-dimensional high-voltage towers and high-voltage wires.

[0125] For the specific definition of the point cloud data processing device for high-voltage towers in power grids, please refer to the definition of the point cloud data processing method for high-voltage towers in power grids mentioned above, which will not be repeated here. Each module in the above-mentioned point cloud data processing device for high-voltage towers in power grids can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0126] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store point cloud data of high-voltage towers in a power grid. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for processing point cloud data of high-voltage towers in a power grid is implemented.

[0127] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0129] Obtain electromagnetic parameter information of high-voltage towers and high-voltage wires and real-time information data of the environment within a preset radius of the high-voltage towers; generate a three-dimensional model of the scene including high-voltage towers and high-voltage wires based on the real-time information data; perform semantic segmentation on the high-voltage towers and high-voltage wires in the three-dimensional model of the scene to obtain different three-dimensional models of components; and uniformly manage and control the point cloud data of the components based on the point cloud coordinate information of the three-dimensional model of the components in the three-dimensional model of the scene.

[0130] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0131] Push flight instructions to the drone; receive real-time information data on high-voltage towers and high-voltage wires, as well as high-voltage towers within a preset radius, sent back by the drone.

[0132] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0133] According to the RTK data in the real-time information data, the neighboring relationship between the RGB images is determined, and the RGB images are sorted to obtain an image sequence; the SIFT algorithm is used to extract the feature points of each image in the image sequence; the feature points of the images in the image sequence are matched between each other to obtain matching feature points; according to the positions of the matching feature points in the image, combined with the intrinsic parameter data of the image acquisition device, the camera posture is obtained, and the image acquisition device is a device for collecting real-time information data; according to the camera posture, the three-dimensional coordinates of the matching feature points are obtained to obtain a sparse three-dimensional point cloud model including high-voltage towers and high-voltage lines; according to the sparse three-dimensional point cloud model, a dense three-dimensional point cloud model is obtained; according to the dense three-dimensional point cloud model, a three-dimensional model of the scene including high-voltage towers and high-voltage wires is constructed.

[0134] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0135] According to the camera pose, dense stereo matching is used to obtain the depth map corresponding to each image in the image sequence; according to the sparse 3D point cloud model, the depth map is fused to obtain a dense 3D point cloud model.

[0136] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0137] A deep learning fully convolutional network structure is used to perform semantic segmentation on RGB images and obtain the semantic segmentation results of two-dimensional images.

[0138] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0139] Obtain a deep learning fully convolutional network model; input the RGB image into the deep learning fully convolutional network structure model, perform convolution encoding on the RGB image, and obtain a feature map smaller than the original RGB image size; perform deconvolution decoding on the feature map of the RGB image to obtain a feature map of the original RGB image size; obtain a two-dimensional image semantic segmentation result based on the feature map of the original RGB image size.

[0140] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0141] Obtain the semantic segmentation result of the two-dimensional image; map the pixel points of the two-dimensional image with the vertices of the high-voltage towers and high-voltage wires in the three-dimensional model of the scene; according to the semantic category corresponding to each pixel point of the high-voltage towers and high-voltage wires in the two-dimensional image, obtain the semantic category of the three-dimensional high-voltage towers and high-voltage wire vertices corresponding to the pixel points; according to the semantic category of the three-dimensional high-voltage towers and high-voltage wire vertices, obtain the semantic segmentation result of the high-voltage towers and high-voltage wires in the three-dimensional model of the scene.

[0142] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0143] Obtain electromagnetic parameter information of high-voltage towers and high-voltage wires and real-time information data of the environment within a preset radius of the high-voltage towers; generate a three-dimensional model of the scene including high-voltage towers and high-voltage wires based on the real-time information data; perform semantic segmentation on the high-voltage towers and high-voltage wires in the three-dimensional model of the scene to obtain different three-dimensional models of components; and uniformly manage and control the point cloud data of the components based on the point cloud coordinate information of the three-dimensional model of the components in the three-dimensional model of the scene.

[0144] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0145] Push flight instructions to the drone; receive real-time information data on high-voltage towers and high-voltage wires, as well as high-voltage towers within a preset radius, sent back by the drone.

[0146] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0147] According to the RTK data in the real-time information data, the neighboring relationship between the RGB images is determined, and the RGB images are sorted to obtain an image sequence; the SIFT algorithm is used to extract the feature points of each image in the image sequence; the feature points of the images in the image sequence are matched between each other to obtain matching feature points; according to the positions of the matching feature points in the image, combined with the intrinsic parameter data of the image acquisition device, the camera posture is obtained, and the image acquisition device is a device for collecting real-time information data; according to the camera posture, the three-dimensional coordinates of the matching feature points are obtained to obtain a sparse three-dimensional point cloud model including high-voltage towers and high-voltage lines; according to the sparse three-dimensional point cloud model, a dense three-dimensional point cloud model is obtained; according to the dense three-dimensional point cloud model, a three-dimensional model of the scene including high-voltage towers and high-voltage wires is constructed.

[0148] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0149] According to the camera pose, dense stereo matching is used to obtain the depth map corresponding to each image in the image sequence; according to the sparse 3D point cloud model, the depth map is fused to obtain a dense 3D point cloud model.

[0150] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0151] A deep learning fully convolutional network structure is used to perform semantic segmentation on RGB images and obtain the semantic segmentation results of two-dimensional images.

[0152] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0153] Obtain a deep learning fully convolutional network model; input the RGB image into the deep learning fully convolutional network structure model, perform convolution encoding on the RGB image, and obtain a feature map smaller than the original RGB image size; perform deconvolution decoding on the feature map of the RGB image to obtain a feature map of the original RGB image size; obtain a two-dimensional image semantic segmentation result based on the feature map of the original RGB image size.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] Obtain the semantic segmentation result of the two-dimensional image; map the pixel points of the two-dimensional image with the vertices of the high-voltage towers and high-voltage wires in the three-dimensional model of the scene; according to the semantic category corresponding to each pixel point of the high-voltage towers and high-voltage wires in the two-dimensional image, obtain the semantic category of the three-dimensional high-voltage towers and high-voltage wire vertices corresponding to the pixel points; according to the semantic category of the three-dimensional high-voltage towers and high-voltage wire vertices, obtain the semantic segmentation result of the high-voltage towers and high-voltage wires in the three-dimensional model of the scene.

[0156] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0157] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for processing point cloud data of high-voltage towers in a power grid, characterized in that: The method comprises: Acquire electromagnetic parameter information of high-voltage iron towers and high-voltage electric wires and real-time environmental information data within a preset radius of the high-voltage iron towers, wherein the real-time environmental information data includes RGB images; Generating a three-dimensional model of a scene including a high-voltage iron tower and high-voltage wires according to the real-time environmental information data; Using a deep learning fully convolutional network structure, the RGB image is semantically segmented to obtain a two-dimensional image semantic segmentation result, wherein the two-dimensional image semantic segmentation result includes a semantic category corresponding to each pixel point of a high-voltage iron tower and a high-voltage wire in the two-dimensional image; According to the semantic segmentation result of the two-dimensional image, semantic segmentation is performed on the high-voltage iron tower and the high-voltage wire in the three-dimensional model of the scene to obtain three-dimensional models of different components; Based on the point cloud coordinate information of the component three-dimensional model in the scene three-dimensional model, the component point cloud data is uniformly controlled.

2. The method according to claim 1, characterized in that: The acquisition of electromagnetic parameter information of the high-voltage iron tower and the high-voltage electric wire and real-time information data of the environment within a preset radius of the high-voltage iron tower includes: Push flight instructions to the drone; Receive electromagnetic parameter information of the high-voltage iron tower and the high-voltage electric wire and real-time information data of the environment within a preset radius of the high-voltage iron tower sent back by the UAV.

3. The method according to claim 1, characterized in that The environmental real-time information data also includes RTK data; Generating a three-dimensional model of a scene including a high-voltage iron tower and high-voltage wires according to the real-time environmental information data includes: Determine the neighbor relationship between RGB images according to the RTK data in the environmental real-time information data, sort the RGB images, and obtain an image sequence; Using SIFT algorithm to extract feature points of each image in the image sequence; Matching feature points between two images in the image sequence to obtain matching feature points; According to the position of the matching feature point in the image, the camera posture is obtained in combination with the intrinsic parameter data of the image acquisition device, and the image acquisition device is a device for collecting the real-time information data of the environment; According to the camera position, the three-dimensional coordinates of the matching feature points are obtained to obtain a sparse three-dimensional point cloud model including a high-voltage tower and a high-voltage wire; Obtaining a dense three-dimensional point cloud model according to the sparse three-dimensional point cloud model; A three-dimensional model of a scene including high-voltage towers and high-voltage wires is constructed based on the dense three-dimensional point cloud model.

4. The method according to claim 3, characterized in that The step of obtaining a dense three-dimensional point cloud model according to the sparse three-dimensional point cloud model comprises: According to the camera pose, dense stereo matching is performed to obtain a depth map corresponding to each image in the image sequence; According to the sparse three-dimensional point cloud model, the depth map is fused to obtain a dense three-dimensional point cloud model.

5. The method according to any one of claims 1 to 4, characterized in that: The deep learning fully convolutional network structure is used to perform semantic segmentation on the RGB image to obtain a two-dimensional image semantic segmentation result including: Get the deep learning fully convolutional network structure; Inputting the RGB image into the deep learning fully convolutional network structure, performing convolution encoding processing on the RGB image, and obtaining a feature map smaller than the size of the original RGB image; Performing deconvolution decoding on the feature map of the RGB image to obtain a feature map of the size of the original RGB image; A two-dimensional image semantic segmentation result is obtained according to the feature map of the original RGB image size.

6. The method according to any one of claims 1 to 4, characterized in that: The semantic segmentation of the high-voltage iron tower and the high-voltage wire in the three-dimensional model of the scene according to the semantic segmentation result of the two-dimensional image comprises: Obtaining the semantic segmentation result of the two-dimensional image; Mapping the pixel points of the two-dimensional image to the vertices of the high-voltage iron tower and the high-voltage wire in the three-dimensional model of the scene; According to the semantic category corresponding to each pixel point of the high-voltage iron tower and the high-voltage electric wire in the two-dimensional image, the semantic category of the three-dimensional high-voltage iron tower and the high-voltage electric wire vertex corresponding to the pixel point is obtained; According to the semantic categories of the vertices of the three-dimensional high-voltage iron tower and the high-voltage electric wire, the semantic segmentation results of the high-voltage iron tower and the high-voltage electric wire in the three-dimensional model of the scene are obtained.

7. A point cloud data processing device for a high-voltage tower in a power grid, characterized in that: The device comprises: A data acquisition module is used to acquire electromagnetic parameter information of a high-voltage iron tower and a high-voltage electric wire and real-time environmental information data within a preset radius of the high-voltage iron tower, wherein the real-time environmental information data includes an RGB image; A data processing module, used to generate a three-dimensional model of a scene including a high-voltage iron tower and high-voltage wires according to the real-time environmental information data; A semantic segmentation module, which is used to perform semantic segmentation on the RGB image using a deep learning fully convolutional network structure to obtain a two-dimensional image semantic segmentation result, wherein the two-dimensional image semantic segmentation result includes a semantic category corresponding to each pixel point of a high-voltage iron tower and a high-voltage wire in the two-dimensional image, and obtain different three-dimensional models of components according to the two-dimensional image semantic segmentation result; The control module is used to uniformly control the point cloud data of the components according to the point cloud coordinate information of the three-dimensional model of the components in the three-dimensional model of the scene.

8. The device according to claim 7, characterized in that The data acquisition module is also used to push flight instructions to the drone, and receive electromagnetic parameter information of the high-voltage tower and high-voltage wires and real-time environmental information data within a preset radius of the high-voltage tower sent back by the drone.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images