Semantic segmentation method of point cloud data and related device

Abstract

The invention discloses a semantic segmentation method for point cloud data and a related device, and the method comprises the steps: carrying out the sampling processing of to-be-segmented point cloud data, and obtaining central point sampling data and neighborhood point sampling data; obtaining relative distance data according to the first coordinate data and the second coordinate data; performing feature extraction on the central point sampling data to obtain central point features; performing feature extraction on the relative distance data and the second multi-dimensional feature data to obtain neighborhood point features; performing feature aggregation processing on the central point sampling data, the central point features and the neighborhood point features to obtain third multi-dimensional feature data of each central point; determining fourth multi-dimensional feature data of each point in the to-be-segmented point cloud data according to the third multi-dimensional feature data of each center point; and inputting the fourth multi-dimensional feature data into a preset model to obtain a segmentation result of each point in the to-be-segmented point cloud data. According to the invention, the segmentation precision of the point cloud data can be improved.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction of high-voltage transmission lines, and particularly to a semantic segmentation method for point cloud data and related devices. Background Art

[0002] As a carrier for long-distance power transmission, transmission lines are an important part of the power grid infrastructure, and their daily monitoring and maintenance are the focus of attention of management departments. For the daily maintenance of transmission lines, the traditional inspection method is to rely on manual inspection, which is time-consuming and laborious, has a long inspection cycle, and may not be able to detect potential safety hazards in a timely manner, failing to meet the requirements of power inspection.

[0003] With the rapid development of airborne platforms and light and small lidar systems, it is now mainly through controlling drones equipped with lidar for inspection to obtain point cloud data of transmission lines, and then through semantic segmentation operations on the point cloud data of transmission lines for fault detection and potential hazard investigation. However, due to factors such as the complex scene of transmission lines and the large difference in the sizes of various objects, the segmentation accuracy of the point cloud data of transmission lines is relatively low. Summary of the Invention

[0004] Embodiments of this application provide a semantic segmentation method for point cloud data and related devices to improve the segmentation accuracy of point cloud data.

[0005] In a first aspect, embodiments of this application provide a semantic segmentation method for point cloud data, including:

[0006] Performing sampling processing on the point cloud data to be segmented to obtain center point sampling data and neighborhood point sampling data, where each center point in the center point sampling data corresponds to multiple neighborhood points in the neighborhood point sampling data. The center point sampling data includes the first coordinate data and the first multi-dimensional feature data of each center point in the center point sampling data, and the neighborhood point sampling data includes the second coordinate data and the second multi-dimensional feature data of each neighborhood point in the neighborhood point sampling data;

[0007] Determining the distance difference between each center point and each neighborhood point according to the first coordinate data and the second coordinate data to obtain relative distance data;

[0008] Performing feature extraction on the center point sampling data to obtain the center point features of each center point; and performing feature extraction on the relative distance data and the second multi-dimensional feature data to obtain the neighborhood point features of each neighborhood point;

[0009] Performing feature aggregation processing on the center point sampling data, the center point features of each center point, and the neighborhood point features of each neighborhood point to obtain the third multi-dimensional feature data of each center point;

[0010] Determine the fourth multi-dimensional feature data of each point in the point cloud data to be segmented according to the third multi-dimensional feature data of each center point;

[0011] Input the fourth multi-dimensional feature data into a preset model to obtain the segmentation result of each point in the point cloud data to be segmented.

[0012] Among them, the feature extraction of the relative distance data and the second multi-dimensional feature data to obtain the neighborhood point features of each neighborhood point includes:

[0013] Extract features from the relative distance data through a first feature extraction network to obtain the first neighborhood point features of each neighborhood point;

[0014] Extract features from the relative distance data through a second feature extraction network to obtain the second neighborhood point features of each neighborhood point;

[0015] Extract features from the second multi-dimensional feature data through a third feature extraction network to obtain the third neighborhood point features of each neighborhood point;

[0016] Determine the first association degree between the first neighborhood point features of each neighborhood point and the second neighborhood point features of each neighborhood point;

[0017] Determine the neighborhood point features of each neighborhood point according to the third neighborhood point features of each neighborhood point and the first association degree.

[0018] Among them, the feature aggregation process of the center point sampling data, the center point features of each center point, and the neighborhood point features of each neighborhood point to obtain the third multi-dimensional feature data of each center point includes:

[0019] Determine multiple neighborhood point features corresponding to each center point according to the neighborhood point features of each neighborhood point;

[0020] Perform a first aggregation process on the multiple neighborhood point features corresponding to each center point to obtain the fifth multi-dimensional feature data of each center point;

[0021] Perform a second aggregation process on the fifth multi-dimensional feature data, the center point sampling data, and the center point features to obtain the third multi-dimensional feature data.

[0022] Among them, the performing a first aggregation process on the multiple neighborhood point features corresponding to each center point to obtain the fifth multi-dimensional feature data of each center point includes:

[0023] Sort the multiple neighborhood point features corresponding to each center point to obtain multiple neighborhood point feature sequences, where the center point corresponds to the neighborhood point feature sequence one by one;

[0024] Extract the first number of neighborhood point features from each neighborhood point feature sequence among the multiple neighborhood point feature sequences according to a preset ratio to obtain multiple neighborhood point feature groups. The number of neighborhood point features corresponding to each center point is the second number, and the first number is less than the second number;

[0025] For the multiple neighborhood point feature groups, perform feature fusion on the first number of neighborhood point features in each neighborhood point feature group to obtain the fusion feature of each neighborhood point feature group;

[0026] Determine the fusion feature of each neighborhood point feature group as the fifth multi-dimensional feature data of each center point.

[0027] Among them, the second aggregation process on the fifth multi-dimensional feature data, the center point sampling data, and the center point feature to obtain the third multi-dimensional feature data includes:

[0028] Obtain multiple first dimensions of the fifth multi-dimensional feature data; and, obtain multiple second dimensions of the center point sampling data; and, obtain multiple third dimensions of the center point feature;

[0029] According to the multiple first dimensions, the multiple second dimensions, and the multiple third dimensions, splice the fifth multi-dimensional feature data, the center point sampling data, and the center point feature to obtain the third multi-dimensional feature data.

[0030] Among them, the determination of the fourth multi-dimensional feature data of each point in the point cloud data to be segmented according to the third multi-dimensional feature data of each center point includes:

[0031] According to the first coordinate data, determine a group of center points having a distance constraint relationship with each non-center point to obtain multiple groups of center points. A single group of center points includes P center points, where P is greater than or equal to 4, and the non-center point is other points in the point cloud data to be segmented except the center point;

[0032] According to the distance constraint relationship and the third multi-dimensional feature data of each center point, determine the seventh multi-dimensional feature data of each non-center point;

[0033] According to the seventh multi-dimensional feature data of each non-center point and the third multi-dimensional feature data of each center point, obtain the fourth multi-dimensional feature data of each point in the point cloud data to be segmented.

[0034] Among them, determining the seventh multi-dimensional feature data of each non-central point according to the distance constraint relationship and the third multi-dimensional feature data of each central point includes:

[0035] Determining the second degree of association between each non-central point and each central point in the corresponding central point group;

[0036] Determining the third degree of association between every two central points in each central point group;

[0037] Determining the weight of each central point in each central point group according to the second degree of association and the third degree of association;

[0038] Determining multiple third multi-dimensional feature data of each central point group according to the third multi-dimensional feature data of each central point;

[0039] Determining the sixth multi-dimensional feature data of each non-central point according to the weight of each central point in each central point group and the multiple third multi-dimensional feature data of each central point group;

[0040] Obtaining the fourth multi-dimensional feature data of each point in the to-be-segmented point cloud data according to the sixth multi-dimensional feature data of each non-central point and the third multi-dimensional feature data of each central point.

[0041] In a second aspect, an embodiment of the present application provides a semantic segmentation device for point cloud data, including:

[0042] A sampling unit, configured to perform sampling processing on the to-be-segmented point cloud data to obtain central point sampling data and neighborhood point sampling data, where each central point in the central point sampling data corresponds to multiple neighborhood points in the neighborhood point sampling data, the central point sampling data includes the first coordinate data and the first multi-dimensional feature data of each central point in the central point sampling data, and the neighborhood point sampling data includes the second coordinate data and the second multi-dimensional feature data of each neighborhood point in the neighborhood point sampling data;

[0043] A first determination unit, configured to determine the distance difference between each central point and each neighborhood point according to the first coordinate data and the second coordinate data to obtain relative distance data;

[0044] An extraction unit, configured to perform feature extraction on the central point sampling data to obtain the central point features of each central point; and perform feature extraction on the relative distance data and the second multi-dimensional feature data to obtain the neighborhood point features of each neighborhood point.

[0045] An aggregation unit, configured to perform feature aggregation processing on the central point sampling data, the central point features of each central point, and the neighborhood point features of each neighborhood point, to obtain third multi-dimensional feature data of each central point;

[0046] A second determination unit, configured to determine fourth multi-dimensional feature data of each point in the point cloud data to be segmented according to the third multi-dimensional feature data of each central point;

[0047] An input unit, configured to input the fourth multi-dimensional feature data into a preset model to obtain a segmentation result of each point in the point cloud data to be segmented.

[0048] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and executable program code stored on the memory and executable on the processor. When the processor executes the executable program code, it performs the steps of the method described in the first aspect.

[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which executable program code is stored. The executable program code includes execution instructions for performing the steps of the method described in the first aspect.

[0050] In a fifth aspect, an embodiment of the present application provides a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to cause a computer to perform some or all of the steps described in the first aspect of the embodiments of the present application. The computer program product may be a software installation package.

[0051] It can be seen that in the embodiment of the present application, first, the point cloud data to be segmented is sampled to obtain central point sampling data and neighborhood point sampling data. Each central point in the central point sampling data corresponds to multiple neighborhood points in the neighborhood point sampling data. The central point sampling data includes the first coordinate data and the first multi-dimensional feature data of each central point in the central point sampling data, and the neighborhood point sampling data includes the second coordinate data and the second multi-dimensional feature data of each neighborhood point in the neighborhood point sampling data. Then, according to the first coordinate data and the second coordinate data, the distance difference between each central point and each neighborhood point is determined to obtain relative distance data. After that, feature extraction is performed on the central point sampling data to obtain the central point features of each central point. Also, feature extraction is performed on the relative distance data and the second multi-dimensional feature data to obtain the neighborhood point features of each neighborhood point. Then, feature aggregation processing is performed on the central point sampling data, the central point features of each central point, and the neighborhood point features of each neighborhood point to obtain the third multi-dimensional feature data of each central point. After that, the fourth multi-dimensional feature data of each point in the point cloud data to be segmented is determined according to the third multi-dimensional feature data of each central point. Finally, the fourth multi-dimensional feature data is input into a preset model to obtain the segmentation result of each point in the point cloud data to be segmented.

[0052] The present application reconstructs the neighborhood point sampling data and performs neighborhood point feature extraction on the reconstructed neighborhood point sampling data, which is beneficial to deeply excavating the local feature information of the neighborhood points. After feature extraction, the central point features and the neighborhood point features are aggregated to obtain high-dimensional features containing more dimensional information and spatial relationships. Then, upsampling processing is performed on the high-dimensional features of each central point obtained by aggregation to obtain the high-dimensional features of each point in the point cloud data, restoring the data volume of the point cloud, ensuring the integrity of the data, improving the quality of the data, and enabling accurate per-point classification based on the comprehensive and accurate point cloud data features to achieve precise segmentation of the point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 It is a schematic diagram of the architecture of a semantic segmentation system for point cloud data provided by an embodiment of the present application;

[0055] Figure 2 It is a schematic flowchart of a semantic segmentation method for point cloud data provided by an embodiment of the present application;

[0056] Figure 3 It is a schematic flowchart of another semantic segmentation method for point cloud data provided by an embodiment of the present application;

[0057] Figure 4 It is a schematic flowchart of a sampling data processing provided by an embodiment of the present application;

[0058] Figure 5 It is a schematic flowchart of a feature extraction provided by an embodiment of the present application;

[0059] Figure 6 It is a schematic diagram of a segmentation result provided by an embodiment of the present application;

[0060] Figure 7 It is a block diagram of the functional units of a semantic segmentation device for point cloud data provided by an embodiment of the present application;

[0061] Figure 8 It is a block diagram of the functional units of another semantic segmentation device for point cloud data provided by an embodiment of the present application;

[0062] Figure 9 It is a schematic structural diagram of an electronic device proposed by an embodiment of the present application. Detailed implementation manners

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present application.

[0064] The terms "first", "second", etc. in the specification and claims of the present application and the above accompanying drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0065] References to "embodiments" in this specification mean that the particular features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0066] Currently, for the daily maintenance of transmission lines, the traditional inspection method relies on manual detection. This method is time-consuming and laborious, has a long inspection cycle, and may not be able to detect potential safety hazards in a timely manner, failing to meet the requirements of power inspection.

[0067] With the rapid development of airborne platforms and lightweight lidar systems, inspections are now mainly carried out by controlling drones equipped with lidar to obtain point cloud data of transmission lines. Then, through semantic segmentation operations on the point cloud data of transmission lines, fault detection and hidden danger investigation are carried out. However, due to factors such as the complex scene of transmission lines and the large size difference of various objects, the segmentation accuracy of the point cloud data of transmission lines is relatively low.

[0068] In view of the above problems, the embodiments of the present application provide a semantic segmentation method for point cloud data and related devices. The embodiments of the present application will be introduced in detail below with reference to the accompanying drawings.

[0069] Please refer to Figure 1 , Figure 1 which is the system architecture diagram of a semantic segmentation system provided by the embodiments of the present application. As Figure 1 shown, the semantic segmentation system 100 includes a sampling module 101, a first extraction module 102, a second extraction module 103, a fusion module 104, an upsampling module 105, and a segmentation module 106. Among them, the sampling module 101, the first extraction module 102, the second extraction module 103, the fusion module 104, the upsampling module 105, and the segmentation module 106 are interconnected.

[0070] Among them, the sampling module 101 is used to obtain the point cloud data of the transmission line and collect the center point data and neighborhood point data from the point cloud data to be segmented; the sampling module 101 is also used to reconstruct the neighborhood point data, that is, decompose the neighborhood point data to obtain coordinate feature data and high-dimensional feature data, calculate the relative coordinate feature data with the center point based on the coordinate feature data, and form new neighborhood point data according to the relative coordinate feature data and the high-dimensional feature data, and then output the center point data and the reconstructed neighborhood point data.

[0071] Among them, the first extraction module 102 is used to receive the center point data collected by the sampling module 101, extract features from the center point data, and output high-dimensional center point features. Specifically, the first extraction module 102 can be a multi-layer perceptron deep network, including a 1×1 convolutional layer, a normalization layer, an activation function layer, a 1×1 convolutional layer, a normalization layer, and an activation function layer, and outputs high-dimensional center point features through multi-layer perceptron operations.

[0072] Among them, the second extraction module 103 is used to receive the neighborhood point data reconstructed by the sampling module 101, extract the features of the neighborhood point data based on the attention mechanism, and output high-dimensional neighborhood point features. Specifically, the second extraction module 103 can include three multi-layer perceptron deep networks, whose network structures can be the same as the network structure for extracting the features of the center point data, but the weight parameters in the networks are different. Two of the multi-layer perceptron deep networks are used to extract the features of the relative coordinate feature data, and one of the multi-layer perceptron deep networks is used to extract high-dimensional feature data, and then comprehensively extract and connect the three obtained neighborhood point features to output high-dimensional neighborhood point features.

[0073] Among them, the fusion module 104 is used to receive the center point data collected by the sampling module 101, the high-dimensional center point features output by the first extraction module 102, and the high-dimensional neighborhood point features output by the second extraction module 103. The fusion module 104 first compresses the number of high-dimensional neighborhood point features, and then performs feature aggregation processing on the compressed high-dimensional neighborhood point features, center point data, and high-dimensional center point features, and outputs the high-dimensional features of each center point.

[0074] Among them, the upsampling module 105 is used to receive the high-dimensional features of each center point output by the fusion module 104, and restore the data volume of the transmission line point cloud data by means of feature interpolation to obtain the high-dimensional features of each point in the transmission line point cloud data.

[0075] Among them, the segmentation module 106 is used to receive the high-dimensional features of each point in the transmission line point cloud data output by the upsampling module 105, and perform segmentation according to the high-dimensional features of each point to obtain the segmentation result of each point.

[0076] Based on this, the present application provides a semantic segmentation method and related device for point cloud data. The present application will be described in detail below with reference to the accompanying drawings.

[0077] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of a semantic segmentation method for point cloud data provided by an embodiment of the present application. As Figure 2 shown, the method includes the following steps:

[0078] S210, sample the point cloud data to be segmented to obtain central point sampling data and neighborhood point sampling data.

[0079] Among them, each central point in the central point sampling data corresponds to multiple neighborhood points in the neighborhood point sampling data. The central point sampling data includes the first coordinate data and the first multi-dimensional feature data of each central point in the central point sampling data. The neighborhood point sampling data includes the second coordinate data and the second multi-dimensional feature data of each neighborhood point in the neighborhood point sampling data.

[0080] Among them, the point cloud data of the transmission line can be collected by an airborne lidar. The collected point cloud data of the transmission line is the point cloud data to be segmented. The point cloud data to be segmented includes multiple points, with the number being K. Each point contains three-dimensional spatial coordinates and other high-dimensional characteristic information other than the three-dimensional spatial coordinates. Exemplarily, the high-dimensional characteristic information includes reflection intensity, echo times, etc.

[0081] Among them, the dimension of the three-dimensional spatial coordinates is 3, the dimension of the high-dimensional characteristic information is C0, and the data size of the point cloud data of the transmission line is K×(C0 + 3).

[0082] In a possible embodiment, after obtaining the point cloud data to be segmented, sample the point cloud data to be segmented. By setting sampling parameters, collect central point data and neighborhood point data from the input point cloud data of the transmission line.

[0083] Among them, the number M of sampling central points, the number N of neighborhood points, and the maximum radius R of neighborhood points can be set; then input the point cloud data of the transmission line, and then perform central point sampling according to the point cloud data of the transmission line.

[0084] Exemplarily, the input point cloud data of the transmission line can be sampled in two batches. The first batch collects M / 2 central points by uniform sampling; the second batch collects M / 2 central points by random sampling. The finally sampled central point data has a data size of M×(C0 + 3). Among them, M is the number of central points, and (C0 + 3) is the feature dimension of the central points.

[0085] Among them, after the central point sampling is completed, take the central point as the origin and perform neighborhood point sampling with a radius of R. For each point in the point cloud data, calculate its Euclidean distance from the central point, traverse all points in the point cloud data, and filter out the points whose distance from the central point is less than or equal to the radius R. These points constitute a neighborhood point set with the central point as the origin and a radius of R.

[0086] Exemplarily, if N is 8, the area where the center point is located can be divided into 8 intervals along the XYZ coordinate axes of the point cloud data. Then, the point closest to the center point is sequentially extracted from the 8 intervals as the neighborhood points. If there is no data in a certain interval, that interval is skipped. If the number of neighborhood points within the maximum radius range R is less than 8, in order to make the number of neighborhood points reach 8, it will be complemented with the center point, that is, the relevant information of the center point is used to generate new points, and these newly generated points are added to the neighborhood point set until the total number of neighborhood points reaches 8. Finally, the neighborhood point data is obtained, and its data size is M×N×(C0 + 3). Among them, M×N is the number of neighborhood points, and (C0 + 3) is the feature dimension of the neighborhood points.

[0087] Among them, the first coordinate data is the three-dimensional space coordinates of the center point, and the first multi-dimensional feature data is the high-dimensional characteristic information of the center point except for the three-dimensional space coordinates; the second coordinate data is the three-dimensional space coordinates of the neighborhood points, and the second multi-dimensional feature data is the high-dimensional characteristic information of the neighborhood points except for the three-dimensional space coordinates.

[0088] In a possible embodiment, the number of dimensions of the first multi-dimensional feature data is the same as the number of dimensions of the second multi-dimensional feature data, and is also the same as the number of dimensions of the point cloud data to be segmented.

[0089] In a possible embodiment, transmission line point cloud data is obtained, with a data volume of 10,000 and a dimension of (6 + 3). Then, sampling parameters are set, with the number of sampling center points set to 1024, the number of neighborhood points set to 32, and the maximum radius of the neighborhood points set to 0.2. The transmission line point cloud data is input, and center point sampling and neighborhood point sampling are performed on the transmission line point cloud data. After sampling, the data size of the center point data is 1024×(6 + 3), and the data size of the neighborhood point data is 1024×32×(6 + 3).

[0090] S220, according to the first coordinate data and the second coordinate data, determine the distance difference between each center point and each neighborhood point to obtain relative distance data.

[0091] Among them, for the reconstruction of the neighborhood point data, first, the sampled neighborhood point data is subjected to feature separation, decomposed into second coordinate data that only contains the XYZ coordinate information of the neighborhood points, and second multi-dimensional feature data that only contains the high-dimensional features of the neighborhood points. Among them, the data size of the second coordinate data is M×N×3, and the data size of the second multi-dimensional feature data is M×N×C0.

[0092] Among them, the center point data obtained by synchronous sampling is subjected to feature separation, decomposed into first coordinate data containing only the XYZ coordinate information of the center point, and first multi-dimensional feature data containing only the high-dimensional features of the center point. Among them, the data size of the first coordinate data is M×N×3, and the data size of the first multi-dimensional feature data is M×N×C0.

[0093] Furthermore, according to the coordinates of the neighborhood points and the coordinates of the center point obtained by decomposition, the three-dimensional coordinate distance difference between the neighborhood points and the center point is calculated to obtain the relative distance data between the neighborhood points and the center point. Specifically, the relative distance data (ΔX i , ΔY i , ΔZ i ) is expressed as shown in formula (1):

[0094] ΔX i =X i -X c

[0095] ΔY i =Y i -Y c

[0096] ΔZ i =Z i -Z c ,

[0097] where, (X c , Y c , Z c ) represents the three-dimensional coordinates of the center point, and (X i , Y i , ΔZ i ) represents the three-dimensional coordinates of the i-th neighborhood point (i = 1, 2,..., N).

[0098] In a possible embodiment, after feature separation of the neighborhood point data with a data size of 1024×32×(6 + 3), coordinate data with a data size of 1024×32×3 is obtained, and the data stored therein are all the XYZ coordinate data of the neighborhood points; and high-dimensional feature data with a data size of 1024×32×6 is obtained, and the data stored therein is high-dimensional feature information other than the three-dimensional coordinates. Similarly, the center point data is subjected to feature decomposition to obtain the center point coordinate data, and then the coordinates of the neighborhood points are subtracted from the corresponding coordinates of the center point to obtain relative distance data with a data size of 1024×32×3.

[0099] Specifically, new neighborhood point data is composed of the relative distance data and the second multi-dimensional feature data. For facilitating subsequent feature extraction. Among them, the data size of the relative distance data is M×N×3.

[0100] S230, performing feature extraction on the center point sampling data to obtain the center point feature of each center point; and performing feature extraction on the relative distance data and the second multidimensional feature data to obtain the neighborhood point feature of each neighborhood point.

[0101] A multi-layer perceptron deep network can be constructed to extract features from the center point data. The multi-layer perceptron deep network can include an input layer, a multi-layer perceptron, and an output layer, wherein the multi-layer perceptron is a stack of fully connected layers, exemplarily including a fully connected layer, a normalization layer, an activation function layer, a fully connected layer, a normalization layer, and an activation function layer. Furthermore, the fully connected layer can be replaced with a convolutional layer, such as a 1×1 convolutional layer.

[0102] Among them, neurons between adjacent layers in the multilayer perceptron are connected by weights to realize feature conversion and combination, that is, weighted summation of inputs is performed through neurons, and then processed by activation functions. This process is repeated in each layer of the multilayer perceptron to realize feature extraction and transformation of input data.

[0103] Specifically, based on the multi-layer perceptron deep network, after extracting the features of the center point data through the multi-layer perceptron operation, the feature dimension of the center point is expanded from C0+3 dimension to C1+3 dimension. The specific calculation expression is shown in formula (2): c =MPL c (P c ), where MPL c Represents the multi-layer perceptron operation 1, O c is the center point feature, P c The center point data.

[0104] Among them, the value of C1 is a preset value and can be adjusted according to the actual scenario.

[0105] In a possible embodiment, the extracting features from the relative distance data and the second multidimensional feature data to obtain the neighborhood point features of each neighborhood point includes: extracting features from the relative distance data through a first feature extraction network to obtain the first neighborhood point features of each neighborhood point; extracting features from the relative distance data through a second feature extraction network to obtain the second neighborhood point features of each neighborhood point; extracting features from the second multidimensional feature data through a third feature extraction network to obtain the third neighborhood point features of each neighborhood point; determining a first degree of association between the first neighborhood point features of each neighborhood point and the second neighborhood point features of each neighborhood point; and determining the neighborhood point features of each neighborhood point based on the third neighborhood point features of each neighborhood point and the first degree of association.

[0106] Among them, a spatial attention depth network can be constructed to extract features from the reconstructed neighborhood point data. Among them, the spatial attention depth network can include: an input layer, a convolutional layer, a spatial attention layer, and an output layer. Among them, the spatial attention layer includes multiple feature extraction networks and feature weighting operations. Each feature extraction network has the same network structure as a multi-layer perceptron, but the weight values of the connections between neurons in different multi-layer perceptrons are different.

[0107] Among them, based on the first feature extraction network, after extracting features from the relative distance data through multi-layer perceptron operations, the feature dimension of the relative distance data is expanded from 3 dimensions to C2 dimensions. The specific calculation expression is shown in formula (3): O Q = MPL Q (P cn ), where MPL Q represents multi-layer perceptron operation 2, O Q is the first neighborhood point feature, corresponding to the query vector, and P cn is the relative distance data.

[0108] Among them, based on the second feature extraction network, after extracting features from the relative distance data through multi-layer perceptron operations, the feature dimension of the relative distance data is expanded from 3 dimensions to C2 dimensions. The specific calculation expression is shown in formula (4): O K = MPL K (P cn ), where MPL K represents multi-layer perceptron operation 3, O K is the second neighborhood point feature, corresponding to the key vector, and P cn is the relative distance data.

[0109] Among them, based on the third feature extraction network, after extracting features from the relative distance data through multi-layer perceptron operations, the feature dimension of the second multi-dimensional feature data is expanded from C0 dimensions to C2 dimensions. The specific calculation expression is shown in formula (5): O V = MPL V (P nn ), where MPL V represents multi-layer perceptron operation 4, O V is the third neighborhood point feature, corresponding to the value vector, and P nn is the second multi-dimensional feature data.

[0110] After that, the first neighborhood point feature is transposed and then matrix-multiplied with the second neighborhood point feature, and a normalization operation is performed to obtain the attention weight, which is used to represent the matching degree or correlation degree between the first neighborhood point feature and the second neighborhood point feature. The specific calculation expression is shown in formula (6): Among them, O His the attention weight, and softmax represents the normalized exponential function. represents the transpose of the first neighborhood point feature.

[0111] Finally, according to the attention weight, the information most relevant to the first neighborhood point feature is extracted from the third neighborhood point feature, so as to realize the focusing on important information, and the information screened by attention is fused with the original information, which not only highlights the important features but also retains some basic information in the third neighborhood point feature, playing a role in enhancing the key information.

[0112] Specifically, the product of the third neighborhood point feature and the attention weight is added to the third neighborhood point feature to obtain the feature extraction result of the neighborhood point. The specific calculation expression is shown in formula (7): O n = O V + O V × O H , where O n is the feature extraction result of the neighborhood point.

[0113] Among them, the dimension of the feature extraction result of the neighborhood point is C2, and the data size is M×N×C2.

[0114] Among them, the value of C2 is a preset value and can be adjusted according to the actual scenario.

[0115] It can be seen that in the embodiment of the present application, based on the spatial attention mechanism, the neighborhood point feature extraction is performed on the reconstructed neighborhood point sampling data, which is beneficial to deeply excavate the local feature information of the neighborhood point and realize the expansion of the feature dimension.

[0116] S240. Perform feature aggregation processing on the center point sampling data, the center point features of each center point, and the neighborhood point features of each neighborhood point to obtain the third multi-dimensional feature data of each center point.

[0117] In a possible embodiment, the performing feature aggregation processing on the center point sampling data, the center point features of each center point, and the neighborhood point features of each neighborhood point to obtain the third multi-dimensional feature data of each center point includes: determining multiple neighborhood point features corresponding to each center point according to the neighborhood point features of each neighborhood point; performing first aggregation processing on the multiple neighborhood point features corresponding to each center point to obtain the fifth multi-dimensional feature data of each center point; and performing second aggregation processing on the fifth multi-dimensional feature data, the center point sampling data, and the center point features to obtain the third multi-dimensional feature data.

[0118] In a possible embodiment, the first aggregation process is used to aggregate the multiple neighborhood point features corresponding to each center point to obtain the fifth multi-dimensional feature data corresponding to each center point. Through the above aggregation process, the feature information of multiple neighborhood points can be integrated to form a more representative feature vector, which can more comprehensively describe each center point and its surrounding local features.

[0119] In a possible embodiment, based on the mean method, the multiple neighborhood point features corresponding to each center point can be aggregated, and the average value of the multiple neighborhood point features corresponding to each center point can be calculated and used as the aggregation feature of the center point, that is, the fifth multi-dimensional feature data. This aggregation process is beneficial to smoothing noise and reflecting the overall level of neighborhood point features.

[0120] In a possible embodiment, based on the weighted average method, the multiple neighborhood point features corresponding to each center point can be aggregated, weights are assigned to each neighborhood point feature according to the correlation or importance between the neighborhood point and the center point, and then the weighted average value is calculated as the aggregation feature of the center point.

[0121] In a possible embodiment, based on the max pooling process, the maximum value among the multiple neighborhood point features corresponding to each center point can be selected as the aggregation feature.

[0122] In a possible embodiment, the first aggregation process for the multiple neighborhood point features corresponding to each center point to obtain the fifth multi-dimensional feature data of each center point includes: sorting the multiple neighborhood point features corresponding to each center point to obtain multiple neighborhood point feature sequences, with the center point corresponding to each neighborhood point feature sequence one by one; extracting the first number of neighborhood point features from each neighborhood point feature sequence in the multiple neighborhood point feature sequences according to a preset ratio to obtain multiple neighborhood point feature groups, where the number of neighborhood point features corresponding to each center point is the second number, and the first number is less than the second number; for the multiple neighborhood point feature groups, performing feature fusion on the first number of neighborhood point features in each neighborhood point feature group to obtain the fusion feature of each neighborhood point feature group; and determining the fusion feature of each neighborhood point feature group as the fifth multi-dimensional feature data of each center point.

[0123] Among them, the multiple neighborhood point features corresponding to the center point can be sorted in descending order. Then, according to a preset ratio, multiple larger-value neighborhood point features are selected from the head of the neighborhood point feature sequence to construct a neighborhood point feature group. Exemplarily, if each center point corresponds to 8 neighborhood point features and the preset ratio is 50%, then the neighborhood point features are sorted in descending order to obtain a neighborhood point feature sequence. At the head of the neighborhood point feature sequence, 4 neighborhood point features are selected to construct a neighborhood point feature group.

[0124] Among them, each center point corresponds to a neighborhood point feature group, and the neighborhood point features included in each neighborhood point feature group are fused. Specifically, fusion can be performed by weighted averaging according to weights.

[0125] Among them, the weight of each neighborhood point feature can be determined according to the spatial distance, where the spatial distance refers to the distance between a single neighborhood point corresponding to a single neighborhood point feature and the center point corresponding to the single neighborhood point. Among them, the closer the spatial distance, the greater the weight.

[0126] Among them, through the first aggregation process, the data size of the neighborhood point features is compressed from M×N×C2 to M×C2.

[0127] It can be seen that in the embodiments of the present application, by selecting multiple neighborhood point features with larger values, it helps to extract key features in the data. At the same time, the weight is determined according to the distance between the neighborhood point and the center point, fully considering the spatial distribution information of the neighborhood points, so that the fused features can better reflect the local structure and spatial features of the data.

[0128] In a possible embodiment, the second aggregation process on the fifth multi-dimensional feature data, the center point sampling data, and the center point features to obtain the third multi-dimensional feature data includes: obtaining multiple first dimensions of the fifth multi-dimensional feature data; and obtaining multiple second dimensions of the center point sampling data; and obtaining multiple third dimensions of the center point features; according to the multiple first dimensions, the multiple second dimensions, and the multiple third dimensions, the fifth multi-dimensional feature data, the center point sampling data, and the center point features are concatenated to obtain the third multi-dimensional feature data.

[0129] Among them, each of the fifth multi-dimensional feature data, the center point sampling data, and the center point features includes multiple dimensions, and there are identical dimensions or mergeable dimensions among the multiple dimensions. For example, both the center point sampling data and the center point features include three-dimensional spatial coordinates. Determine whether there are duplicate dimensions or mergeable dimensions between the fifth multi-dimensional feature data, the center point sampling data, and the center point features. For the feature data with duplicate dimensions, the feature data with duplicate dimensions are merged, and only one copy of the feature data needs to be retained; for the feature data with mergeable dimensions, the feature data can be fused based on weighted averaging or other methods; for the feature data with different dimensions, they are concatenated in a preset order. Combining the feature data obtained in each of the above cases, the high-dimensional features of each center point are obtained.

[0130] Specifically, for the 3 dimensions of the three-dimensional space coordinates included in both the central point sampling data and the central point features, they are merged, and one three-dimensional space coordinate is retained; the data of different dimensions are concatenated in dimension, so that the feature dimension of each central point is C0 + C1 + C2 + 3.

[0131] In a possible embodiment, when the value of C1 is set to 128, after feature extraction is performed on the central point data with a data size of 1024×(6 + 3), its data size is expanded to 1024×128. When the value of C2 is set to 256, after feature extraction and max pooling are performed on the neighborhood point data with a data size of 1024×32×(6 + 3), its data size is expanded to 1024×256. Finally, the central point data, the extracted central point features, and the neighborhood point features are aggregated through feature aggregation to obtain feature data with a data size of 1024×(6 + 3 + 128 + 256).

[0132] It can be seen that in the embodiment of the present application, through the idea of multi-source data features, feature extraction is performed on the point cloud data, and higher-dimensional features of the point cloud data are obtained.

[0133] S250. Determine the fourth multi-dimensional feature data of each point in the to-be-segmented point cloud data according to the third multi-dimensional feature data of each central point.

[0134] In a possible embodiment, the determining the fourth multi-dimensional feature data of each point in the to-be-segmented point cloud data according to the third multi-dimensional feature data of each central point includes: determining, according to the first coordinate data, a group of central points having a distance constraint relationship with each non-central point, obtaining a plurality of groups of central points, and a single group of central points includes P central points, where P is greater than or equal to 4, and the non-central point is other points in the to-be-segmented point cloud data except the central points; determining the seventh multi-dimensional feature data of each non-central point according to the distance constraint relationship and the third multi-dimensional feature data of each central point; and obtaining the fourth multi-dimensional feature data of each point in the to-be-segmented point cloud data according to the seventh multi-dimensional feature data of each non-central point and the third multi-dimensional feature data of each central point.

[0135] Among them, a single group of central points includes multiple central points closest to the corresponding non-central point, and the distance constraint relationship is determined by the spatial distance between points. Exemplarily, P can be 8.

[0136] In a possible embodiment, the weight of each central point in each group of central points relative to the corresponding non-central point can be determined according to the spatial distance between each non-central point and the multiple central points in the corresponding group of central points. The specific calculation expression is as shown in formula (8): where, w i(x) is the weight of the i-th center point in the center point group relative to the corresponding non-center point, and d i is the spatial distance between the i-th center point in the center point group and the corresponding non-center point.

[0137] Among them, through the way of feature interpolation, according to the weight of each center point in each center point group relative to the corresponding non-center point and the multiple high-dimensional features of each center point group, the high-dimensional features of each non-center point can be calculated. The specific calculation expression is shown in formula (9): Among them, f(x) represents the high-dimensional feature of the non-center point x, and f i represents the high-dimensional feature of the i-th center point in the center point group.

[0138] It can be seen that in the embodiment of the present application, according to the existing point cloud data features, the missing part is reasonably filled, so that the point cloud data is more complete and can more comprehensively describe the geometric information of the object or scene. At the same time, in the interpolation process, the position and attributes of each point can be adjusted more finely according to the feature information of the surrounding points. By using the spatial relationship and feature change trend of adjacent points, the value of the missing point can be estimated more accurately, thereby improving the accuracy of the entire point cloud data. In addition, the density of the point cloud is increased, making the contour and details of the object clearer, reducing the jagged or blocky phenomenon in the visualization result, improving the visual effect, and helping users to more intuitively understand the object or scene represented by the point cloud.

[0139] In a possible embodiment, the determining the seventh multi-dimensional feature data of each non-center point according to the distance constraint relationship and the third multi-dimensional feature data of each center point includes: determining the second association degree between each non-center point and each center point in the corresponding center point group; determining the third association degree between every two center points in each center point group; determining the weight of each center point in each center point group according to the second association degree and the third association degree; determining multiple third multi-dimensional feature data of each center point group according to the third multi-dimensional feature data of each center point; determining the sixth multi-dimensional feature data of each non-center point according to the weight of each center point in each center point group and the multiple third multi-dimensional feature data of each center point group; obtaining the fourth multi-dimensional feature data of each point in the to-be-segmented point cloud data according to the sixth multi-dimensional feature data of each non-center point and the third multi-dimensional feature data of each center point.

[0140] Among them, the second association degree can be represented by the distance between a single center point and the corresponding non-center point, and is the same as the calculation method of w i (x).

[0141] Among them, the third correlation degree can be represented by the density of the central points of the central point group. The density is determined according to the distance between every two central points in the central point group. The closer the distance is, the higher the density is, thereby reflecting that the point cloud distribution in this area is relatively dense.

[0142] In a possible embodiment, according to the second correlation degree, the weight of each central point in each central point group relative to the corresponding non-central point is obtained; the weight adjustment coefficient is determined through the third correlation degree, and then the weight is adjusted.

[0143] Among them, when the third correlation degree is in different ranges, it corresponds to different adjustment coefficients.

[0144] Among them, through the way of feature interpolation, according to the adjusted weight and the multiple high-dimensional features of each central point group, the high-dimensional features of each non-central point can be calculated.

[0145] It can be seen that in the embodiments of the present application, combining density to adjust the weight can more precisely reflect the spatial distribution characteristics of the point cloud data and improve the interpolation accuracy. At the same time, it can adaptively process non-uniform data, making the interpolation result smoother and more reasonable, so as to better reflect the overall trend and local characteristics of the data.

[0146] S260, input the fourth multi-dimensional feature data into a preset model to obtain the segmentation result of each point in the point cloud data to be segmented.

[0147] Among them, the preset model is a fully connected neural network model, including an input layer, a normalization layer, an activation function layer, a linear layer, a dropout layer, a normalization layer, and an output layer; the output layer can have 7 nodes, and each node corresponds to a segmentation result, including tower, ground, vegetation, insulator, conductor, ground wire, and jumper wire.

[0148] Among them, the high-dimensional feature data of each point is input into the fully connected neural network model, and the segmentation result of each point is output.

[0149] It can be seen that in the embodiments of the present application, the present application reconstructs the neighborhood point sampling data, and extracts the neighborhood point features from the reconstructed neighborhood point sampling data, which is beneficial to deeply excavate the local feature information of the neighborhood points; after the feature extraction is completed, the central point features and the neighborhood point features are aggregated to obtain high-dimensional features containing more dimensional information and spatial relationships; then, the upsampling process is performed on the high-dimensional features of each central point obtained by aggregation to obtain the high-dimensional features of each point in the point cloud data, restoring the data volume of the point cloud, ensuring the integrity of the data, improving the quality of the data, and finally performing point-by-point classification according to the comprehensive and accurate point cloud data features to achieve precise segmentation of the point cloud data.

[0150] In a possible embodiment, please refer toFigure 3 , Figure 3 is a schematic flowchart of another method for semantic segmentation of point cloud data provided by an embodiment of the present application. As Figure 3 shown, the method for semantic segmentation of point cloud data includes the following steps:

[0151] S310, point cloud sampling.

[0152] Among them, sampling parameters are set, and M central points and M×N neighborhood points are collected from the transmission line point cloud data with a data size of K×(C0 + 3) obtained, to obtain M pieces of central point data with (C0 + 3) dimensions and M×N pieces of neighborhood point data with (C0 + 3) dimensions.

[0153] S320, sampling data processing.

[0154] Among them, the sampling data includes central point data and neighborhood point data.

[0155] Among them, please refer to Figure 4 , Figure 4 which is a schematic flowchart of a sampling data processing provided by an embodiment of the present application. As Figure 4 shown, the sampling data processing can first perform feature separation on the neighborhood point data to obtain 3D coordinate feature data and C0 - dimensional high - dimensional feature data; similarly, perform feature separation on the central point data to obtain 3D coordinate feature data and C0 - dimensional high - dimensional feature data. Then, according to the coordinate feature data separated from the central point data and the coordinate feature data separated from the neighborhood point data, calculate the relative coordinate feature data of the neighborhood point and the central point to obtain 3 - dimensional relative distance data. Furthermore, form new neighborhood point data according to the relative distance data and the high - dimensional feature data separated from the neighborhood point data to obtain reconstructed neighborhood point data.

[0156] S330, feature extraction.

[0157] Among them, feature extraction includes central point feature extraction and neighborhood point feature extraction.

[0158] Among them, please refer to Figure 5 , Figure 5 which is a schematic flowchart of a feature extraction provided by an embodiment of the present application. As Figure 5 shown, construct a multi - layer perceptron deep network MLP c , and perform feature extraction on the central point data through MLP c . Expand the feature dimension of the central point from C0 + 3 dimensions to C1 + 3 dimensions.

[0159] Among them, constructing multiple multi - layer perceptron deep networks may include MLP Q , MLP K and MLP V, feature extraction is performed on the high-dimensional feature data separated from the relative distance data and the neighborhood point sampling data respectively.

[0160] Specifically, the weight parameters corresponding to each multi-layer perceptron deep network are different. Through the MLP Q Extract the features of the relative coordinate feature data to obtain the first neighborhood point features, whose dimension is extended from 3 to the C2 dimension; through the MLP K Extract the features of the relative coordinate feature data to obtain the second neighborhood point features, whose dimension is extended from 3 to the C2 dimension; and through the MLP V Extract the features of the high-dimensional feature data separated from the neighborhood point sampling data to obtain the third neighborhood point features, whose dimension is extended from C0 to the C2 dimension; finally, based on the spatial attention mechanism, the above three neighborhood point features are synthesized to obtain M×N neighborhood point features O of the C2 dimension n .

[0161] Then perform max pooling on the neighborhood point features O n to compress the data size from M×N×C2 to M×C2, and then combine the center point data and the center point features output by the MLP c to perform feature aggregation processing to obtain the final feature extraction result, that is, the high-dimensional features of each center point.

[0162] S340, upsampling.

[0163] Among them, through the method of feature interpolation, according to the high-dimensional features of each center point, the data volume of the point cloud data is restored to determine the high-dimensional features of each point in the point cloud data.

[0164] S350, point cloud segmentation.

[0165] Input the high-dimensional features of each point into the fully connected neural network model, and output the segmentation results of each point, including tower poles, ground, vegetation, insulators, conductors, ground wires, and jumper wires, etc.

[0166] Exemplarily, please refer to Figure 6 , Figure 6 is a schematic diagram of a segmentation result provided by an embodiment of the present application. As Figure 6 shown, in the figure, the segmentation results are distinguished by different colors. The green part represents vegetation, showing the distribution form of vegetation on the ground; the red part represents the ground, outlining the contour of the ground; the blue part presents the tower poles, clearly showing the structure and shape of the tower poles; in addition, conductors, ground wires, and jumper wires, etc. are shown in other line forms, reflecting the layout of the transmission line. The partial enlarged view in the upper right corner presents the details of the tower poles more carefully, facilitating the observation of the segmentation of the relevant structures of the tower poles, and the overall intuitively presents the segmentation results of the transmission line point cloud data.

[0167] Consistent with the above embodiments, please refer to Figure 7 , Figure 7 which is a functional unit composition block diagram of a semantic segmentation device for point cloud data provided by an embodiment of the present application. As Figure 7 shown, the semantic segmentation device 70 for point cloud data includes: a sampling unit 71 for sampling the point cloud data to be segmented to obtain central point sampling data and neighborhood point sampling data, where each central point in the central point sampling data corresponds to multiple neighborhood points in the neighborhood point sampling data. The central point sampling data includes the first coordinate data and the first multi-dimensional feature data of each central point in the central point sampling data, and the neighborhood point sampling data includes the second coordinate data and the second multi-dimensional feature data of each neighborhood point in the neighborhood point sampling data; a first determination unit 72 for determining the distance difference between each central point and each neighborhood point according to the first coordinate data and the second coordinate data to obtain relative distance data; an extraction unit 73 for extracting features from the central point sampling data to obtain the central point features of each central point; and extracting features from the relative distance data and the second multi-dimensional feature data to obtain the neighborhood point features of each neighborhood point; an aggregation unit 74 for performing feature aggregation processing on the central point sampling data, the central point features of each central point, and the neighborhood point features of each neighborhood point to obtain the third multi-dimensional feature data of each central point; a second determination unit 75 for determining the fourth multi-dimensional feature data of each point in the point cloud data to be segmented according to the third multi-dimensional feature data of each central point; and an input unit 76 for inputting the fourth multi-dimensional feature data into a preset model to obtain the segmentation result of each point in the point cloud data to be segmented.

[0168] In a possible embodiment, in terms of extracting features from the relative distance data and the second multi-dimensional feature data to obtain the neighborhood point features of each neighborhood point, the extraction unit 73 is specifically further configured to: extract features from the relative distance data through a first feature extraction network to obtain the first neighborhood point features of each neighborhood point; extract features from the relative distance data through a second feature extraction network to obtain the second neighborhood point features of each neighborhood point; extract features from the second multi-dimensional feature data through a third feature extraction network to obtain the third neighborhood point features of each neighborhood point; determine the first correlation degree between the first neighborhood point features of each neighborhood point and the second neighborhood point features of each neighborhood point; and determine the neighborhood point features of each neighborhood point according to the third neighborhood point features of each neighborhood point and the first correlation degree.

[0169] In a possible embodiment, in terms of performing feature aggregation processing on the sampled data of the center point, the center point features of each center point, and the neighborhood point features of each neighborhood point to obtain the third multi-dimensional feature data of each center point, the aggregation unit 74 is specifically further configured to: determine multiple neighborhood point features corresponding to each center point according to the neighborhood point features of each neighborhood point; perform first aggregation processing on the multiple neighborhood point features corresponding to each center point to obtain the fifth multi-dimensional feature data of each center point; perform second aggregation processing on the fifth multi-dimensional feature data, the center point sampled data, and the center point features to obtain the third multi-dimensional feature data.

[0170] In a possible embodiment, in terms of performing first aggregation processing on the multiple neighborhood point features corresponding to each center point to obtain the fifth multi-dimensional feature data of each center point, the aggregation unit 74 is specifically further configured to: sort the multiple neighborhood point features corresponding to each center point to obtain multiple neighborhood point feature sequences, where the center point corresponds one-to-one with the neighborhood point feature sequence; extract a first number of neighborhood point features in each neighborhood point feature sequence of the multiple neighborhood point feature sequences according to a preset ratio to obtain multiple neighborhood point feature groups, the number of neighborhood point features corresponding to each center point is a second number, and the first number is less than the second number; for the multiple neighborhood point feature groups, perform feature fusion on the first number of neighborhood point features in each neighborhood point feature group to obtain the fusion feature of each neighborhood point feature group; determine the fusion feature of each neighborhood point feature group as the fifth multi-dimensional feature data of each center point.

[0171] In a possible embodiment, in terms of performing second aggregation processing on the fifth multi-dimensional feature data, the center point sampled data, and the center point features to obtain the third multi-dimensional feature data, the aggregation unit 74 is specifically further configured to: obtain multiple first dimensions of the fifth multi-dimensional feature data; and obtain multiple second dimensions of the center point sampled data; and obtain multiple third dimensions of the center point features; splice the fifth multi-dimensional feature data, the center point sampled data, and the center point features according to the multiple first dimensions, the multiple second dimensions, and the multiple third dimensions to obtain the third multi-dimensional feature data.

[0172] In a possible embodiment, in terms of determining the fourth multi-dimensional feature data of each point in the point cloud data to be segmented according to the third multi-dimensional feature data of each of the center points, the second determination unit 75 is specifically further configured to: determine, according to the first coordinate data, a group of center points having a distance constraint relationship with each non-center point, to obtain a plurality of groups of center points, where a single group of center points includes P center points, P is greater than or equal to 4, and the non-center points are other points in the point cloud data to be segmented except the center points; determine the seventh multi-dimensional feature data of each non-center point according to the distance constraint relationship and the third multi-dimensional feature data of each center point; and obtain the fourth multi-dimensional feature data of each point in the point cloud data to be segmented according to the seventh multi-dimensional feature data of each non-center point and the third multi-dimensional feature data of each center point.

[0173] In a possible embodiment, in terms of determining the seventh multi-dimensional feature data of each non-center point according to the distance constraint relationship and the third multi-dimensional feature data of each center point, the second determination unit 75 is specifically further configured to: determine the second association degree between each non-center point and each center point in the corresponding group of center points; determine the third association degree between every two center points in each group of center points; determine the weight of each center point in each group of center points according to the second association degree and the third association degree; determine a plurality of third multi-dimensional feature data of each group of center points according to the third multi-dimensional feature data of each center point; determine the sixth multi-dimensional feature data of each non-center point according to the weight of each center point in each group of center points and the plurality of third multi-dimensional feature data of each group of center points; and obtain the fourth multi-dimensional feature data of each point in the point cloud data to be segmented according to the sixth multi-dimensional feature data of each non-center point and the third multi-dimensional feature data of each center point.

[0174] It can be understood that since the method embodiment and the device embodiment are different presentation forms of the same technical concept, the content of the method embodiment part in this application should be synchronously adapted to the device embodiment part, and will not be elaborated here.

[0175] In the case of adopting an integrated unit, please refer to Figure 8 , Figure 8 which is a functional unit composition block diagram of another semantic segmentation device for point cloud data provided by an embodiment of this application. As shown in Figure 8As shown, the semantic segmentation device 70 for point cloud data includes: a processing module 702 and a communication module 701. The processing module 702 is used to control and manage the operations of the semantic segmentation device 70 for point cloud data. For example, it executes the steps of the sampling unit 71, the first determination unit 72, the extraction unit 73, the aggregation unit 74, the second determination unit 75, and the input unit 76, and / or is used to execute other processes of the technologies described herein. The communication module 701 is used for the interaction between the semantic segmentation device 70 for point cloud data and other devices. As Figure 8 shown, the semantic segmentation device 70 for point cloud data may further include a storage module 703, and the storage module 703 is used to store the program code and data of the semantic segmentation device 70 for point cloud data.

[0176] Among them, the processing module 702 may be a processor or a controller. For example, it may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on. The communication module 701 may be a transceiver, an RF circuit, or a communication interface, etc. The storage module 703 may be a memory.

[0177] Among them, all relevant contents of each scenario involved in the above method embodiments can be cited in the function descriptions of the corresponding functional modules, and will not be elaborated here. The above semantic segmentation device 70 for point cloud data can execute the above Figure 2 shown semantic segmentation method for point cloud data.

[0178] Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of an electronic device proposed in an embodiment of this application. As Figure 9As shown, the electronic device 900 includes a processor 910, a memory 920, a communication interface 930, and one or more programs 921. The one or more programs 921 are stored in the memory and are configured to be executed by the processor. When the program is executed, it includes some or all of the steps of any semantic segmentation method of point cloud data described in the method embodiments. The processor, the memory, and the communication interface are interconnected and complete communication with each other.

[0179] Among them, the memory can be a volatile memory such as a dynamic random access memory (DRAM), or a non-volatile memory such as a mechanical hard disk. The memory is used to store a set of executable program codes, and the processor is used to call the executable program codes stored in the memory and can execute some or all of the steps of any semantic segmentation method of point cloud data described in the semantic segmentation method embodiments of the point cloud data as above.

[0180] It can be seen that for the electronic device 900 described in the embodiments of the present application, first, the point cloud data to be segmented is sampled to obtain center point sampling data and neighborhood point sampling data. Each center point in the center point sampling data corresponds to multiple neighborhood points in the neighborhood point sampling data. The center point sampling data includes first coordinate data and first multi-dimensional feature data of each center point in the center point sampling data, and the neighborhood point sampling data includes second coordinate data and second multi-dimensional feature data of each neighborhood point in the neighborhood point sampling data; then, according to the first coordinate data and the second coordinate data, the distance difference between each center point and each neighborhood point is determined to obtain relative distance data; then, feature extraction is performed on the center point sampling data to obtain the center point features of each center point; and, feature extraction is performed on the relative distance data and the second multi-dimensional feature data to obtain the neighborhood point features of each neighborhood point; then, feature aggregation processing is performed on the center point sampling data, the center point features of each center point, and the neighborhood point features of each neighborhood point to obtain third multi-dimensional feature data of each center point; then, according to the third multi-dimensional feature data of each center point, fourth multi-dimensional feature data of each point in the point cloud data to be segmented is determined; finally, the fourth multi-dimensional feature data is input into a preset model to obtain the segmentation result of each point in the point cloud data to be segmented.

[0181] This application reconstructs the neighborhood point sampling data and extracts the neighborhood point features from the reconstructed neighborhood point sampling data, which is beneficial to deeply excavate the local feature information of the neighborhood points. After the feature extraction is completed, the central point features and the neighborhood point features are aggregated to obtain high-dimensional features containing more dimensional information and spatial relationships. Then, for each central point's high-dimensional feature obtained by aggregation, upsampling processing is performed to obtain the high-dimensional features of each point in the point cloud data, restoring the data volume of the point cloud, ensuring the integrity of the data, improving the quality of the data, and enabling accurate per-point classification based on the comprehensive and accurate point cloud data features to achieve precise segmentation of the point cloud data.

[0182] An embodiment of this application also provides a computer storage medium. The computer storage medium stores a computer program for electronic data exchange, and the computer program enables the computer to execute some or all of the steps of any method described in the above method embodiments. The above computer includes an electronic device.

[0183] An embodiment of this application also provides a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to enable the computer to execute some or all of the steps of any method described in the above method embodiments. The computer program product can be a software installation package, and the above computer includes an electronic device.

[0184] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0185] In the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0186] In several embodiments provided by this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0187] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software program module.

[0189] If the above-mentioned integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned memory includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0190] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable memory, and the memory can include: flash drives, read-only memories, random access memories, magnetic disks, or optical discs, etc.

[0191] The above has introduced the embodiments of the present application in detail. Specific examples are used in this article to elaborate on the principle and embodiments of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific embodiments and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A semantic segmentation method for point cloud data, characterized in that Including: Sampling the point cloud data to be segmented to obtain central point sampling data and neighborhood point sampling data. Each central point in the central point sampling data corresponds to multiple neighborhood points in the neighborhood point sampling data. The central point sampling data includes the first coordinate data and the first multi-dimensional feature data of each central point in the central point sampling data, and the neighborhood point sampling data includes the second coordinate data and the second multi-dimensional feature data of each neighborhood point in the neighborhood point sampling data; Determining the distance difference between each central point and each neighborhood point according to the first coordinate data and the second coordinate data to obtain relative distance data; Performing feature extraction on the central point sampling data to obtain the central point features of each central point; and performing feature extraction on the relative distance data and the second multi-dimensional feature data to obtain the neighborhood point features of each neighborhood point; Performing feature aggregation processing on the central point sampling data, the central point features of each central point, and the neighborhood point features of each neighborhood point to obtain the third multi-dimensional feature data of each central point; Determining the fourth multi-dimensional feature data of each point in the point cloud data to be segmented according to the third multi-dimensional feature data of each central point; Inputting the fourth multi-dimensional feature data into a preset model to obtain the segmentation result of each point in the point cloud data to be segmented.

2. The method according to claim 1, wherein The performing feature extraction on the relative distance data and the second multi-dimensional feature data to obtain the neighborhood point features of each neighborhood point includes: Performing feature extraction on the relative distance data through a first feature extraction network to obtain the first neighborhood point features of each neighborhood point; Performing feature extraction on the relative distance data through a second feature extraction network to obtain the second neighborhood point features of each neighborhood point; Performing feature extraction on the second multi-dimensional feature data through a third feature extraction network to obtain the third neighborhood point features of each neighborhood point; Determining the first association degree between the first neighborhood point features of each neighborhood point and the second neighborhood point features of each neighborhood point; Determining the neighborhood point features of each neighborhood point according to the third neighborhood point features of each neighborhood point and the first association degree.

3. The method according to claim 1, characterized in that, The performing feature aggregation processing on the central point sampling data, the central point features of each central point, and the neighborhood point features of each neighborhood point to obtain the third multi-dimensional feature data of each central point includes: Determining the multiple neighborhood point features corresponding to each central point according to the neighborhood point features of each neighborhood point; Performing a first aggregation process on the multiple neighborhood point features corresponding to each central point to obtain the fifth multi-dimensional feature data of each central point; Performing a second aggregation process on the fifth multi-dimensional feature data, the central point sampling data, and the central point features to obtain the third multi-dimensional feature data.

4. The method according to claim 3, wherein The performing a first aggregation process on the multiple neighborhood point features corresponding to each central point to obtain the fifth multi-dimensional feature data of each central point includes: Sort the multiple neighborhood point features corresponding to each of the center points to obtain multiple neighborhood point feature sequences, with one-to-one correspondence between the center points and the neighborhood point feature sequences; Extract the first number of neighborhood point features from each of the multiple neighborhood point feature sequences according to a preset ratio to obtain multiple neighborhood point feature groups, where the number of neighborhood point features corresponding to each center point is the second number, and the first number is less than the second number; For the multiple neighborhood point feature groups, perform feature fusion on the first number of neighborhood point features in each neighborhood point feature group to obtain the fusion feature of each neighborhood point feature group; Determine the fusion feature of each neighborhood point feature group as the fifth multi-dimensional feature data of each center point.

5. The method according to claim 3, characterized in that, The second aggregation process on the fifth multi-dimensional feature data, the center point sampling data, and the center point feature to obtain the third multi-dimensional feature data includes: Obtain multiple first dimensions of the fifth multi-dimensional feature data; and, obtain multiple second dimensions of the center point sampling data; and, obtain multiple third dimensions of the center point feature; According to the multiple first dimensions, the multiple second dimensions, and the multiple third dimensions, splice the fifth multi-dimensional feature data, the center point sampling data, and the center point feature to obtain the third multi-dimensional feature data.

6. The method according to any one of claims 1-5, characterized in that, The determination of the fourth multi-dimensional feature data of each point in the point cloud data to be segmented according to the third multi-dimensional feature data of each center point includes: According to the first coordinate data, determine a group of center points having a distance constraint relationship with each non-center point to obtain multiple groups of center points, where a single group of center points includes P center points, P is greater than or equal to 4, and the non-center point is other points in the point cloud data to be segmented except the center points; According to the distance constraint relationship and the third multi-dimensional feature data of each center point, determine the seventh multi-dimensional feature data of each non-center point; According to the seventh multi-dimensional feature data of each non-center point and the third multi-dimensional feature data of each center point, obtain the fourth multi-dimensional feature data of each point in the point cloud data to be segmented.

7. The method according to claim 6, wherein The determination of the seventh multi-dimensional feature data of each non-center point according to the distance constraint relationship and the third multi-dimensional feature data of each center point includes: Determine the second association degree between each non-center point and each center point in the corresponding group of center points; Determine the third association degree between every two center points in each group of center points; According to the second association degree and the third association degree, determine the weight of each center point in each group of center points; Determine multiple third multi-dimensional feature data of each group of center points according to the third multi-dimensional feature data of each center point; According to the weight of each center point in each group of center points and the multiple third multi-dimensional feature data of each group of center points, determine the sixth multi-dimensional feature data of each non-center point; Based on the sixth multi-dimensional feature data of each non-central point and the third multi-dimensional feature data of each central point, the fourth multi-dimensional feature data of each point in the point cloud data to be segmented is obtained.

8. A semantic segmentation device for point cloud data, characterized in that Including: A sampling unit for performing sampling processing on the point cloud data to be segmented to obtain central point sampling data and neighborhood point sampling data. Each central point in the central point sampling data corresponds to multiple neighborhood points in the neighborhood point sampling data. The central point sampling data includes the first coordinate data and the first multi-dimensional feature data of each central point in the central point sampling data, and the neighborhood point sampling data includes the second coordinate data and the second multi-dimensional feature data of each neighborhood point in the neighborhood point sampling data; A first determination unit for determining the distance difference between each central point and each neighborhood point according to the first coordinate data and the second coordinate data to obtain relative distance data; An extraction unit for performing feature extraction on the central point sampling data to obtain the central point features of each central point; and performing feature extraction on the relative distance data and the second multi-dimensional feature data to obtain the neighborhood point features of each neighborhood point; An aggregation unit for performing feature aggregation processing on the central point sampling data, the central point features of each central point, and the neighborhood point features of each neighborhood point to obtain the third multi-dimensional feature data of each central point; A second determination unit for determining the fourth multi-dimensional feature data of each point in the point cloud data to be segmented according to the third multi-dimensional feature data of each central point; An input unit for inputting the fourth multi-dimensional feature data into a preset model to obtain the segmentation result of each point in the point cloud data to be segmented.

9. An electronic device, characterized in that, The device includes: A memory, a processor, and executable program code stored on the memory and executable on the processor. When the processor executes the executable program code, it executes the steps of the semantic segmentation method of the point cloud data according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Executable program code is stored on the computer-readable storage medium. The executable program code includes execution instructions for executing the steps of the semantic segmentation method of the point cloud data according to any one of claims 1-7.

Images