Pole tower part identification method and equipment based on combination of point cloud and image data, and medium

Abstract

The invention discloses a tower part identification method and equipment based on combination of point cloud and image data, and a medium, and relates to the technical field of tower inspection and identification. The identification method comprises the following steps: acquiring point cloud data and image data of a tower; segmenting the point cloud data by using a region growing algorithm to obtain a point cloud segmentation result of the component of the tower; performing semantic segmentation on the image data by using a deep learning model to obtain an image segmentation result of the tower component; constructing an association graph of the point cloud segmentation result and the image segmentation result of each component; and based on a matching result of the association graph and a pre-stored typical association graph, obtaining an identification type of each component. According to the method, multi-source data are fused through an effective mechanism, and a matching identification method is combined, so that detail features of the tower component can be better captured and identified, and the tower component identification capability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pole tower inspection and identification, and in particular to a pole tower component identification method, equipment and medium based on the combination of point cloud and image data. Background Art

[0002] With the rapid development of power systems and the promotion of smart grids, the scale of distribution networks continues to expand, which puts higher requirements on the management and maintenance of distribution network towers. Traditional distribution network tower detection and identification methods mainly rely on manual inspections, which have problems such as low efficiency, high cost, and high safety risks.

[0003] In recent years, with the development of computer vision and artificial intelligence technology, automated detection methods based on image processing have gradually emerged. However, most methods rely on only a single data source, such as using only images or only point clouds, and cannot fully utilize the complementary advantages of multi-source data, resulting in insufficient accuracy and robustness of recognition results. In particular, single data source detection methods cannot verify accuracy and accurately identify precise details of tower components. For small components or components with similar structures on towers, existing recognition methods often have difficulty in accurately distinguishing them and are insufficient in their ability to identify component details. Summary of the invention

[0004] The present invention provides a pole tower component recognition method, device and medium based on the combination of point cloud and image data to solve the problem that the existing recognition method based on a single data source has insufficient recognition ability for pole tower component details, resulting in insufficient recognition accuracy.

[0005] The present invention is achieved through the following technical solutions:

[0006] A first aspect of the present invention provides a method for identifying tower components based on a combination of point cloud and image data, comprising:

[0007] Obtain point cloud data and image data of the tower;

[0008] Segmenting the point cloud data using a region growing algorithm to obtain point cloud segmentation results of components of the tower;

[0009] Performing semantic segmentation on the image data using a deep learning model to obtain image segmentation results of components of the tower;

[0010] Constructing a correlation graph between the point cloud segmentation results and the image segmentation results of each component;

[0011] Based on the matching result between the association graph and the pre-stored typical association graph, the identification type of each component is obtained.

[0012] The present invention fuses three-dimensional point cloud data and two-dimensional image data to identify pole tower components. By combining two-dimensional data and three-dimensional data, more detailed information of pole tower components can be obtained, so that the recognition accuracy of small components or components with similar structures on the pole tower is higher. When using multi-source data, the three-dimensional point cloud data is first segmented by a regional growing algorithm to obtain the point cloud segmentation results of each component, and at the same time, the two-dimensional image segmentation results of each component are obtained by image semantic segmentation. The point cloud data of each component is associated with the image data to form an effective multi-source data fusion mechanism. The obtained association diagram is matched and identified with the typical association diagram of the pre-stored component, so as to accurately identify the type of the component.

[0013] Furthermore, the point cloud data is segmented using a region growing algorithm to obtain a point cloud segmentation result of the components of the tower, including:

[0014] S2-1, calculating the normal and curvature of each point in the point cloud data, selecting the point with the smallest curvature as the initial seed point, and creating a current seed point sequence and a current area according to the initial seed point;

[0015] S2-2, taking the initial seed point as the center, searching all neighboring points of the initial seed point;

[0016] S2-3, obtaining the normal vector angle between the neighborhood point and the initial seed point according to the normal line of the neighborhood point, and if the normal vector angle is less than a preset angle value, adding the neighborhood point to the current area;

[0017] S2-4, checking the curvature of the neighborhood points in the current area, and adding the neighborhood points whose curvature is less than a preset curvature value to the current seed point sequence;

[0018] S2-5, select the next point from the current seed point sequence as a new initial seed point, repeat the growth process of S2-2 to S2-4 until no new point is added to the current seed point sequence, and add the current area that has been grown to the cluster array;

[0019] S2-6, executing S2-1 to S2-5 again for the point cloud data that has not entered the current area until all points are traversed, outputting a clustering array, and obtaining the point cloud segmentation results of each component.

[0020] Furthermore, before segmenting the point cloud data and the image data, preprocessing the point cloud data and the image data is performed respectively;

[0021] Preprocessing of the point cloud data includes: denoising, downsampling and smoothing;

[0022] The preprocessing of the image data includes: histogram equalization processing.

[0023] Furthermore, the constructing of a correlation graph between the point cloud segmentation results and the image segmentation results of each component includes:

[0024] Projecting the image segmentation result into the same three-dimensional coordinate system as the point cloud data according to the intrinsic parameters and extrinsic parameters of the camera, and obtaining projection points of pixel points in the image segmentation result in the three-dimensional coordinate system;

[0025] Extracting a two-dimensional image contour of the image segmentation result, extracting a plurality of target corner points from the two-dimensional image contour, and acquiring target projection points corresponding to the target corner points from the projection points;

[0026] The point in the point cloud segmentation result that is closest to the target projection point is obtained, and the closest point is matched and connected with the target projection point to obtain a correlation graph between the point cloud segmentation result and the image segmentation result.

[0027] Furthermore, the identification type of each component is obtained based on the matching result of the association graph and the pre-stored typical association graph, including:

[0028] Identify the component type according to the point cloud segmentation result to obtain a first identification type;

[0029] Obtaining a second recognition type according to the two-dimensional image contour recognition component type;

[0030] If the first identification type is the same as the second identification type for the same component, a typical association diagram of the first identification type is retrieved from a pre-stored database;

[0031] The matching degree between the association diagram and the typical association diagram is calculated. If the matching degree is greater than a standard value, the type of the typical association diagram is used as the identification type of the component.

[0032] Furthermore, the calculating the matching degree between the association graph and the typical association graph includes:

[0033] Calculate the connection deviation of a plurality of target corner points in the association graph;

[0034] Extracting the outer contour to be detected formed by connecting a plurality of target corner points through an edge detection algorithm;

[0035] Calculating a similarity value between the outer contour to be detected and the typical outer contour in the typical association graph;

[0036] The matching degree D is calculated according to the connection deviation and the similarity value:

[0037]

[0038] Among them, θi represents the connection deviation of the target corner point i, i∈[1,N], N is the total number of target corner points; S represents the similarity value between the outer contour to be detected and the typical outer contour, S∈[0,1], 1 represents complete similarity, and 0 represents complete difference; α is the weight parameter for balancing the connection distance, and β is the weight parameter for balancing the outer contour similarity.

[0039] Furthermore, the connection deviation is calculated as follows:

[0040] Calculate the connection distance of the target corner point:

[0041] d i =distance(X,Y);

[0042] Among them, d i represents the connection distance of the target corner point i, X represents the coordinates of the target projection point of the target corner point i in the three-dimensional coordinate system, Y represents the coordinates of the point closest to the target projection point in the point cloud segmentation result, and distance is the distance calculation formula;

[0043] Calculate the average connection distance of the target corner points:

[0044]

[0045] in, represents the average connection distance, d i Represents the connection distance of target corner point i, i∈[1,N], N is the total number of target corner points;

[0046] Calculate the connection deviation of the target corner point:

[0047]

[0048] Among them, θ i Indicates the connection deviation of target corner point i, d i represents the connection distance of the target corner point i, Represents the average connection distance.

[0049] Furthermore, the method also includes: extracting a preliminary outline of the point cloud segmentation result of each component, fusing the preliminary outline with the two-dimensional image outline to obtain a three-dimensional outline of each component, extracting geometric features and point cloud attribute features from the three-dimensional outline, and outputting the extracted features and the identification type of the component.

[0050] A second aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for identifying tower components based on the combination of point cloud and image data as described in any one of the first aspects of the present invention is implemented.

[0051] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for identifying tower components based on the combination of point cloud and image data as described in any one of the first aspects of the present invention is implemented.

[0052] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0053] The association graph between the point cloud segmentation results and the two-dimensional image segmentation results of each tower component is constructed, and the final type of each component is determined based on the association graph, which effectively fuses multi-source data to obtain more details of the components, especially improving the ability to distinguish small components and components with similar structures. The point cloud segmentation results with preliminary contours are fused with the image segmentation results with two-dimensional image contours to obtain the fused three-dimensional contours of each tower component. Feature extraction is performed based on the fused three-dimensional contours to extract the target features of each tower component. The extracted target features of the tower component and the identification information are output together as the recognition results of the tower component. By using the association graph and the fused three-dimensional contours, the detailed features of the tower components can be better captured, the recognition ability of small or complex components can be improved, and the applicability in practical applications can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0055] Figure 1 It is a flow chart of a tower component identification method based on the combination of point cloud and image data according to an embodiment of the present invention;

[0056] Figure 2 is a calculation flow chart of an improved region growing algorithm according to an embodiment of the present invention;

[0057] Figure 3 is a flow chart of a method for establishing an association graph according to an embodiment of the present invention;

[0058] Figure 4 It is a matching calculation flow chart of a component association diagram and a typical association diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0060] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to or inherent to other steps or units of the device.

[0061] The terms used in the various embodiments of the application are only used to describe the purpose of specific embodiments and are not intended to limit the various embodiments of the application. As used herein, the singular form is intended to also include the plural form, unless the context clearly indicates otherwise. Unless otherwise limited, all terms used here (including technical terms and scientific terms) have the same meaning as the meanings commonly understood by ordinary technicians in the field of the various embodiments of the application. The terms (such as the terms defined in the dictionary generally used) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning, unless clearly defined in the various embodiments of the application.

[0062] The embodiment of the present invention provides a pole tower component recognition method based on the combination of point cloud and image data, which is suitable for inspection and recognition of pole towers, and is conducive to improving the ability to recognize details of pole tower components and improving the accuracy of recognition results.

[0063] like Figure 1 The flowchart of the tower component identification method based on the combination of point cloud and image data shown includes the following steps.

[0064] S1, obtaining point cloud data and image data of the tower;

[0065] S2, using the region growing algorithm to segment the point cloud data to obtain the point cloud segmentation results of the tower components;

[0066] S3, using the deep learning model to perform semantic segmentation on the image data to obtain image segmentation results of the tower components;

[0067] S4, constructing a correlation graph between the point cloud segmentation results and the image segmentation results of each component;

[0068] S5, obtaining the identification type of each component based on the matching result between the association graph and the pre-stored typical association graph.

[0069] The present invention constructs an association graph between point cloud segmentation results and two-dimensional image segmentation results, fuses multi-source data, and combines matching and recognition methods to better capture and recognize detailed features of tower components, thereby improving detail recognition capabilities.

[0070] The region growing algorithm is a method for image segmentation based on pixel similarity. Its basic idea is to start from one or more seed points and merge adjacent pixels into the same region according to the similarity criterion until there are no adjacent pixels that meet the criterion. In this embodiment, an improved region growing algorithm is used to segment point cloud data, such as Figure 2 The calculation flow chart of the improved region growing algorithm shown includes the following steps.

[0071] S2-1, calculate the normal and curvature of each point in the point cloud data, select the point with the smallest curvature in the point cloud data as the initial seed point, and create the current seed point sequence and the current area based on the initial seed point.

[0072] S2-2, with the initial seed point as the center, search all neighboring points of the initial seed point.

[0073] S2-3, according to the normal of the neighborhood point, obtain the normal vector angle between the neighborhood point and the initial seed point (that is, the angle between the two normals between the neighborhood point and the initial seed point). If the normal vector angle is less than the preset angle value, the neighborhood point is added to the current area.

[0074] S2-4, checking the curvature of all neighborhood points in the current area, if it is less than a preset curvature value, adding the corresponding neighborhood point to the current seed point sequence.

[0075] S2-5, select the next point from the current seed point sequence as the new initial seed point, repeat the growth process of S2-2 to S2-4 until no new points are added to the seed point sequence, and add the current area that has been grown to the cluster array.

[0076] S2-6, for the point cloud data that has not entered the current area, execute S2-1 to S2-5 again until all points are traversed and the cluster array is output, that is, the point cloud segmentation results of each component are obtained.

[0077] It should be noted that in the region growing algorithm, initially, a seed point is selected, which is usually the point with the smallest curvature. The region is grown from this seed point, and the neighboring points that meet the conditions are added to the current region. At the same time, if the curvature of some neighboring points is less than the threshold, they are also added to the seed point sequence. After processing the current seed point, the scheme of the embodiment will take out the next point from the seed point sequence as a new seed point and continue the growth process. This process will continue until no new points can be added to the seed point sequence, that is, the seed point sequence becomes empty, at which point a complete region is grown. For the remaining point cloud data, a new initial seed point is reselected and the regional growth process is repeated until the collected tower point cloud data is divided into multiple regions. The point cloud segmentation result obtained may include various components of the tower, and the corresponding component contours and component types can be obtained based on point cloud contour extraction and contour recognition.

[0078] In step S3, the semantic segmentation of the image is to let the computer segment according to the semantics of the image. In the image field, semantics refers to the content of the image and the understanding of the meaning of the image. This step can use some common semantic segmentation deep models, such as fully convolutional neural network (FCN), U-Net, etc.

[0079] In step S4, a correlation diagram between the point cloud segmentation results and the image segmentation results of each component is constructed, such as Figure 3 The method flow chart of establishing the association graph shown in the figure specifically includes:

[0080] S4-1, projecting the image segmentation result into the same three-dimensional coordinate system as the point cloud data according to the intrinsic and extrinsic parameters of the camera, and obtaining the projection points of the pixel points in the image segmentation result in the three-dimensional coordinate system.

[0081] S4-2, extracting multiple target corner points from the two-dimensional image contour, and obtaining target projection points corresponding to the target corner points from the projection points.

[0082] S4-3, obtain the point closest to the target projection point in the point cloud segmentation result (ie, the nearest point cloud point), establish a matching connection between the nearest point and the target projection point, and obtain a correlation diagram between the point cloud segmentation result and the image segmentation result.

[0083] The camera's intrinsic parameters are used to describe the geometric characteristics of the image, such as focal length (f), principal point coordinates (cx, cy), etc., and the extrinsic parameters describe the position and posture of the camera in the world coordinate system, such as rotation matrix R and translation vector T. The multi-angle 2D image segmentation results are projected into the 3D space in the same coordinate system as the 3D point cloud data, and then the pixel coordinates in the 2D image are converted into the 3D space coordinates of the current pixel to obtain the projection point corresponding to the current pixel.

[0084] Specifically, it includes projection and coordinate transformation. The pixel points (u, v) in the two-dimensional image are converted into coordinates (X, Y, Z) in the three-dimensional space through internal and external parameters. The mapping can be expressed as:

[0085]

[0086] Among them, Z is the depth information, R -1 It is the inverse of the rotation matrix R. The depth information usually refers to the distance along the optical axis of the camera. In the camera coordinate system, the depth is usually represented by the Z-axis coordinate.

[0087] The acquisition of projection points is to convert each pixel point in the two-dimensional segmentation result into a projection point in three-dimensional space through the above formula, that is, to convert it into coordinates (X, Y, Z) in three-dimensional space. This step allows the two-dimensional image segmentation result to be compared and fused with the three-dimensional point cloud data in the same coordinate system.

[0088] The two-dimensional image contour is obtained by performing contour extraction on the image segmentation result of the tower component. The method of extracting multiple target corner points from the two-dimensional image contour includes: performing corner point detection on the two-dimensional image contour using the Harris corner point detection algorithm to obtain the target corner points. The basic steps of the Harris corner point detection algorithm are as follows.

[0089] (1) Calculate gradient: Calculate the gradient of the image in the x and y directions. You can use gradient operators such as the Sobel operator.

[0090] (2) Constructing the matrix M: For each pixel in the image, construct the matrix M. The required parameters are: the window function w(x, y), and the gradient information Ix and Iy of all pixels in the window.

[0091] (3) Calculate the corner response function R: Use the Harris corner evaluation function to determine whether there is a corner point in the rectangular box. The formula is R = λ1λ2-k(λ1+λ2)2, where λ1 and λ2 are the eigenvalues ​​of the matrix M, and k is an empirical constant.

[0092] (4) Threshold screening: Set a threshold, then slide the window across the entire image to find the position (x, y) where R(λ1,λ2)>threshold. This (x, y) is the corner point we need to find.

[0093] (5) Non-maximum suppression: Perform non-maximum suppression in a 3×3 neighborhood to find the local maximum point, which is the corner point in the image.

[0094] The present invention can also use the Shi-Tomasi corner point detection algorithm to extract the target corner points. Shi-Tomasi is an improvement of Harris. It directly uses the smaller eigenvalue as the score, so there is no need to adjust the k value. If the smaller of the two eigenvalues ​​of the matrix M is greater than the set threshold, then this point is a corner point. If both eigenvalues ​​are less than the threshold, then the smaller eigenvalue must also be less than the threshold, then this point is a point in the flat area. If one of the eigenvalues ​​is greater than the threshold and the other eigenvalue is less than the threshold, then this point is an edge point.

[0095] Furthermore, the matching degree between the current association graph and the typical association graph of the corresponding tower component is calculated by the connection distance between the i-th target corner point and the nearest point cloud point of the association graph and the typical association graph, the average connection distance deviation of the target corner points of the overall association graph, the total number N of all target corner points, and the similarity of the outer contour. Figure 4 The matching calculation flow chart of the association diagram of the components shown is compared with the typical association diagram.

[0096] S5-1, calculating the connection deviation of the target corner points of the two-dimensional image contour in the association graph.

[0097] (1) Calculate the connection distance of the target corner points:

[0098] d i =distance(X,Y);

[0099] Among them, d i represents the connection distance of target corner point i, X represents the coordinates of the target projection point in the three-dimensional coordinate system space of target corner point i, Y represents the coordinates of the point closest to the target projection point in the point cloud segmentation result, and distance is the distance calculation formula;

[0100] (2) Calculate the average connection distance of the target corner points:

[0101]

[0102] in, represents the average connection distance, d i Represents the connection distance of target corner point i, i∈[1,N], N is the total number of target corner points;

[0103] (3) Calculate the connection deviation of the target corner point:

[0104]

[0105] Among them, θ i Indicates the connection deviation of target corner point i, d i represents the connection distance of the target corner point i, Represents the average connection distance.

[0106] S5-2, extract multiple target corner points of the two-dimensional image contour through an edge detection algorithm (Canny algorithm), and connect them to form an outer contour to be detected.

[0107] S5-3, calculating the similarity value between the outer contour to be detected and the typical outer contour in the typical association graph.

[0108] The similarity value S between the outer contour to be detected and the typical outer contour of the tower component type corresponding to the association graph is calculated, and the similarity value S is in the range of [0,1]. The larger the value, the more similar it is. 1 means completely similar, and 0 means completely different.

[0109] S5-4, calculate the matching degree D according to the connection deviation and similarity value:

[0110]

[0111] Among them, θ i It represents the connection deviation of target corner point i, i∈[1,N], N is the total number of target corner points; S represents the similarity value of the outer contour; α is the weight parameter for balancing the connection distance, and β is the weight parameter for balancing the outer contour similarity, that is, α and β respectively affect the matching result value.

[0112] The output matching result value is still between 0 and 1. The closer the matching result value is to 1, the better the matching effect is, which combines the connection distance and the shape similarity of the outer contour.

[0113] As an optional implementation, the method further includes determining the similarity between the type of the pending tower component and the type of the typical association diagram of the corresponding tower component according to the matching result value. The closer the current matching result value is to 1, the higher the similarity between the type of the pending tower component and the type of the typical association diagram of the corresponding tower component is.

[0114] As an optional implementation manner, identifying the tower type according to the association diagram further includes:

[0115] Identify the component type according to the point cloud segmentation result to obtain a first identification type;

[0116] Identify the component type according to the two-dimensional image contour to obtain a second identification type;

[0117] If the first identification type and the second identification type are the same for the same component, a typical association diagram of the first identification type (or the second identification type) is retrieved from a pre-stored database;

[0118] The matching degree between the association diagram of the component and the typical association diagram is calculated. If the matching degree is greater than a standard value, the type of the typical association diagram is used as the identification type of the component.

[0119] The matching degree calculation method between the association diagram and the typical association diagram of the corresponding tower type is shown in the above embodiment. The types of tower components include cross arms, insulators, and hardware (including wire clamps, connection hardware, anti-vibration hammers, etc.).

[0120] In this embodiment, it is determined whether the image recognition result of the current tower component identified by the current two-dimensional image contour and the point cloud recognition result of the current tower component identified by the point cloud segmentation result are consistent. If they are consistent, the type of the tower component to be determined is preliminarily determined, and the typical association diagram of the tower component corresponding to the type of the tower component to be determined is called from the pre-stored database, and a matching calculation is performed, and finally the matching result value is output. If the current matching result value is greater than the standard value, it is determined that the type of the tower component to be determined is consistent with the type of the typical association diagram of the corresponding tower component, and the type of the typical association diagram of the tower component is output as the type of the tower component to be determined. The component type is identified using point cloud data and image data respectively. For components with consistent identification results, the association diagram of the point cloud segmentation result and the two-dimensional image segmentation result is identified again to ensure that the identification results are consistent. Multiple identification results can also be used as mutual verification to ensure the accuracy of identification.

[0121] In the above recognition process, if the point cloud recognition result is inconsistent with the two-dimensional image recognition result or the matching degree does not reach the preset standard value, it means that the currently judged area may not be a tower component or a tower component that has not been included yet. It can be marked after manual judgment or not processed.

[0122] As an optional implementation, before segmenting the point cloud data and the image data, the point cloud data and the image data are preprocessed respectively. The preprocessing of the point cloud data includes: denoising, downsampling and smoothing; the preprocessing of the image data includes: using a histogram equalization method to perform image enhancement on the current digital image data.

[0123] As an optional implementation, it also includes: extracting the preliminary outline of each component according to the point cloud segmentation result, extracting the two-dimensional image outline according to the image segmentation result, fusing the preliminary outline of each component and the two-dimensional image outline to obtain the three-dimensional outline of each component, performing feature extraction based on the fused three-dimensional outline, extracting the target features of each tower component, and outputting the extracted target features of the tower component and the recognition result of the tower together as the recognition result of the current tower component. The recognition result includes the name of the current tower component and the type of the tower.

[0124] Among them, the preliminary outline and the two-dimensional image outline are fused to extract the key feature points of the point cloud and the image through the SIFT algorithm or SURF, and the corresponding relationship between the key points is established through feature matching to achieve the matching of the above two.

[0125] The target features of each tower component include geometric features and point cloud attribute features. The geometric features include volume features and surface area features, and the point cloud attribute features include point cloud density features, shape distribution features, and spatial distribution entropy features.

[0126] The calculation formula of volume characteristic V is:

[0127]

[0128] Where V represents the volume feature, v i is the volume of the i-th voxel in the three-dimensional contour, i∈[1,N], and N is the total number of voxels.

[0129] The calculation formula of surface area characteristic S is:

[0130]

[0131] Where m is the number of triangles on the surface of the component, s is i is the area of ​​the i-th triangle, i∈[1,M], and M is the total number of triangles.

[0132] The above technical solution can effectively obtain the geometric features of each tower component, including volume features and surface area features, by extracting features from the fused three-dimensional contour. Specifically, by extracting volume and surface area features, the geometric shape and size of the tower components can be fully understood, providing basic data for subsequent analysis and achieving more comprehensive feature acquisition. At the same time, this technical solution uses the number of voxels and triangular facets for precise calculation to ensure the accuracy of feature extraction, and is suitable for the analysis of towers with complex shapes. This method can adapt to tower components of different shapes and sizes, and at the same time retains more detailed features of the tower components through precise calculation of volume and surface area, which is helpful for subsequent analysis.

[0133] The calculation formula of point cloud density feature ρ is:

[0134] ρ=N / V

[0135] Among them, N is the number of points contained in the component, that is, point is the number of point cloud points, and V is the volume of the component.

[0136] The calculation formula of spatial distribution entropy feature E is:

[0137]

[0138] Where M is the number of grids used to divide the space, n i is the number of points in the i-th grid, and N is the total number of points, that is, the number of point cloud points.

[0139] The above technical solution can deeply analyze the geometric and spatial characteristics of tower components based on actual data by calculating point cloud density and spatial distribution entropy. The point cloud density feature provides the point cloud distribution of the component, which can effectively reflect the structural integrity and uniformity of the component. The spatial distribution entropy feature can quantify the distribution law of the point cloud in space and help identify the spatial information of the point cloud distribution.

[0140] In specific applications, the three-dimensional point cloud data and digital image data of the distribution network tower are obtained; the obtained three-dimensional point cloud data is preprocessed, including denoising, downsampling and smoothing. The preprocessed point cloud data is segmented using an improved region growing algorithm to obtain the preliminary outlines of the various components of the tower. The digital image data is preprocessed, including image enhancement. The preprocessed digital image data is semantically segmented using a pre-trained deep learning model to obtain the two-dimensional image outlines of the various components of the tower. The point cloud segmentation results with the preliminary outlines are fused with the image segmentation results with the two-dimensional image outlines to obtain the fused three-dimensional outlines of the various components of the tower. At the same time, a correlation diagram between the point cloud segmentation results and the two-dimensional image segmentation results of the various components of the tower is constructed, and the final type of the three-dimensional outline is determined based on the correlation diagram.

[0141] The above steps can achieve high accuracy and efficient feature extraction. By fusing 3D point cloud data and 2D image data, combined with an improved region growing algorithm and deep learning model, this method can significantly improve the accuracy of tower component recognition. The combination of multi-source data can complement each other's advantages and overcome the limitations of a single data source. The fused 3D contour provides a rich source of information for feature extraction, including geometric features and point cloud attribute features, which helps to more comprehensively describe the characteristics of tower components.

[0142] Based on the fused three-dimensional contour, feature extraction is performed to extract the target features of each tower component, including geometric features and point cloud attribute features. The target features extracted from the current tower component and the identification information are output together as the identification result of the current tower component. Through the above classification, not only the name of the tower component can be identified, but also the type of the tower can be determined, providing more comprehensive identification information.

[0143] In summary, the distribution network tower component detection and recognition method provided by the present invention, which combines three-dimensional point cloud data with digital image data, can better capture and recognize the detailed features of the tower components by constructing a correlation diagram between the point cloud segmentation result and the two-dimensional image segmentation result, thereby improving the detail recognition capability and finally providing comprehensive recognition information. Combining three-dimensional point cloud data with two-dimensional image data can accurately perceive the spatial position and shape of the tower components, which is helpful for precise positioning and measurement and improves the spatial perception capability.

[0144] The output identification information includes not only the component name, but also the type of the tower, providing more comprehensive information support for subsequent management analysis and maintenance work. The target features extracted from the current tower component and the identification information are output together as the identification result of the current tower component. The output data can also be used to build the output result deep model. The target features extracted from the current tower component are stored and used as samples to enrich the training output result deep model, and finally the converged output result deep model is obtained. In this way, the final type of the three-dimensional contour can also be verified and identified according to the output result deep model.

[0145] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory, and the number of processors may be one or more. The memory, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and modules. The processor executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory, so as to implement the tower component identification method based on the combination of point cloud and image data of any of the above embodiments of the present invention.

[0146] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for identifying tower components based on the combination of point cloud and image data according to any embodiment of the present invention is implemented. The storage medium may be a ROM / RAM, a magnetic disk, an optical disk, etc.

[0148] An embodiment of the present invention further provides a computer program product. When the computer program product is run on a computer, the computer executes the tower component identification method based on the combination of point cloud and image data according to any of the above embodiments of the present invention.

[0149] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A tower component recognition method based on the combination of point cloud and image data, characterized in that: include: Obtain point cloud data and image data of the tower; Segmenting the point cloud data using a region growing algorithm to obtain point cloud segmentation results of components of the tower; Performing semantic segmentation on the image data using a deep learning model to obtain image segmentation results of components of the tower; Constructing a correlation graph between the point cloud segmentation results and the image segmentation results of each component; Based on the matching result between the association graph and the pre-stored typical association graph, the identification type of each component is obtained.

2. The tower component identification method based on the combination of point cloud and image data according to claim 1 is characterized in that: The step of segmenting the point cloud data using a region growing algorithm to obtain a point cloud segmentation result of components of the tower includes: S2-1, calculating the normal and curvature of each point in the point cloud data, selecting the point with the smallest curvature as the initial seed point, and creating a current seed point sequence and a current area according to the initial seed point; S2-2, taking the initial seed point as the center, searching all neighboring points of the initial seed point; S2-3, obtaining the normal vector angle between the neighborhood point and the initial seed point according to the normal line of the neighborhood point, and if the normal vector angle is less than a preset angle value, adding the neighborhood point to the current area; S2-4, checking the curvature of the neighborhood points in the current area, and adding the neighborhood points whose curvature is less than a preset curvature value to the current seed point sequence; S2-5, select the next point from the current seed point sequence as a new initial seed point, repeat the growth process of S2-2 to S2-4 until no new point is added to the current seed point sequence, and add the current area that has been grown to the cluster array; S2-6, executing S2-1 to S2-5 again for the point cloud data that has not entered the current area until all points are traversed, outputting a clustering array, and obtaining the point cloud segmentation results of each component.

3. The tower component identification method based on the combination of point cloud and image data according to claim 1 is characterized in that: Before segmenting the point cloud data and the image data, preprocessing the point cloud data and the image data respectively; Preprocessing of the point cloud data includes: denoising, downsampling and smoothing; The preprocessing of the image data includes: histogram equalization processing.

4. The tower component identification method based on the combination of point cloud and image data according to claim 1 is characterized in that: The constructing of a correlation diagram between the point cloud segmentation results and the image segmentation results of each component includes: Projecting the image segmentation result into the same three-dimensional coordinate system as the point cloud data according to the intrinsic parameters and extrinsic parameters of the camera, and obtaining projection points of pixel points in the image segmentation result in the three-dimensional coordinate system; Extracting a two-dimensional image contour of the image segmentation result, extracting a plurality of target corner points from the two-dimensional image contour, and acquiring target projection points corresponding to the target corner points from the projection points; The point in the point cloud segmentation result that is closest to the target projection point is obtained, and the closest point is matched and connected with the target projection point to obtain a correlation graph between the point cloud segmentation result and the image segmentation result.

5. The tower component identification method based on the combination of point cloud and image data according to claim 4 is characterized in that: The identification type of each component is obtained based on the matching result of the association graph and the pre-stored typical association graph, including: Identify the component type according to the point cloud segmentation result to obtain a first identification type; Obtaining a second recognition type according to the two-dimensional image contour recognition component type; If the first identification type is the same as the second identification type for the same component, a typical association diagram of the first identification type is retrieved from a pre-stored database; The matching degree between the association diagram and the typical association diagram is calculated. If the matching degree is greater than a standard value, the type of the typical association diagram is used as the identification type of the component.

6. The tower component identification method based on the combination of point cloud and image data according to claim 5 is characterized in that: The calculating the matching degree between the association graph and the typical association graph includes: Calculate the connection deviation of a plurality of target corner points in the association graph; Extracting multiple target corner points through edge detection algorithm and connecting them to form an outer contour to be detected; Calculating a similarity value between the outer contour to be detected and the typical outer contour in the typical association graph; The matching degree D is calculated according to the connection deviation and the similarity value: Among them, θ i represents the connection deviation of the target corner point i, i∈[1,N], N is the total number of target corner points; S represents the similarity value between the outer contour to be detected and the typical outer contour, S∈[0,1], 1 represents complete similarity, and 0 represents complete difference; α is the weight parameter for balancing the connection distance, and β is the weight parameter for balancing the outer contour similarity.

7. The tower component identification method based on the combination of point cloud and image data according to claim 6 is characterized in that: The calculation method of the connection deviation is: Calculate the connection distance of the target corner point: d i =distance(X,Y); Among them, d i represents the connection distance of the target corner point i, X represents the coordinates of the target projection point of the target corner point i in the three-dimensional coordinate system, Y represents the coordinates of the point closest to the target projection point in the point cloud segmentation result, and distance is the distance calculation formula; Calculate the average connection distance of the target corner points: Among them, d represents the average connection distance, d i Represents the connection distance of target corner point i, i∈[1,N], N is the total number of target corner points; Calculate the connection deviation of the target corner point: Among them, θ i Indicates the connection deviation of target corner point i, d i represents the connection distance of the target corner point i, Represents the average connection distance.

8. The tower component identification method based on the combination of point cloud and image data according to claim 4 is characterized in that: The method also includes: extracting a preliminary outline of the point cloud segmentation result of each component, fusing the preliminary outline with the two-dimensional image outline to obtain a three-dimensional outline of each component, extracting geometric features and point cloud attribute features from the three-dimensional outline, and outputting the extracted features and the identification type of the component.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for identifying tower components based on the combination of point cloud and image data according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying tower components based on the combination of point cloud and image data according to any one of claims 1 to 8 is implemented.

Images