A method and system for three-dimensional point cloud tree structure segmentation based on deep learning

Abstract

The present invention belongs to the technical field of forest point cloud segmentation, and discloses a method and system for three-dimensional point cloud tree structure segmentation based on deep learning, obtains a data set of forest point cloud data, and divides it into a training set, a validation set, and a test set according to a certain ratio; uses a multi-scale sampling method to sample the data set and establish a projection, and the sampled data constitutes a new data set; configures a network operating environment, and uses the training set and validation set in the new data set to train the network; the test set uses the trained network for prediction, and uses the projection information to project the label of the data set into the original point cloud, visualizes the results, and realizes the tree structure segmentation of the entire point cloud. The present invention can realize the seamless integration of data collection and point cloud segmentation; has an excellent recognition effect on tree branch points, and the recognition coverage is far ahead; can not only recognize the primary branches, but also the secondary branches.

Description

Technical Field

[0001] The present invention belongs to the technical field of forest point cloud segmentation, and in particular relates to a three-dimensional point cloud tree structure segmentation method and system based on deep learning. Background Art

[0002] In order to effectively manage and protect forest resources and understand the health of the ecosystem, forestry surveys urgently need to accurately segment and identify tree structures in forests. Traditional forestry survey methods are often labor-intensive, with problems such as high cost, long time consumption, and small number of samples. In recent years, laser point cloud data collected by 3D laser scanning technology can quickly and accurately obtain the three-dimensional spatial structure of large areas of forests, and has emerged in forestry surveys. However, the point cloud obtained by 3D laser scanning technology is discrete, disordered, and irregular, and cannot directly provide effective forest parameters. Accurate segmentation of forest point cloud data is of great significance and value, and is a key issue in improving the efficiency and accuracy of forestry surveys.

[0003] The research on 3D forest point cloud segmentation methods originated from the separation of wood and leaves, that is, separating the wood part from the leaf part from the entire point cloud scene. In traditional wood and leaf separation research, the processing of point cloud data mostly depends on geometric features, and the data type is usually single-wood point cloud data or multi-wood data with a simple environment. Among them, the shortest path algorithm is one of the effective and commonly used algorithms for wood and leaf separation. However, the forest environment is complex, and point cloud data is easily lost due to shade during the collection process, which makes the method of extracting and separating branches and leaves using single tree features difficult to apply to regional forest scenes. Therefore, algorithms based on machine learning and deep learning have been studied. Among them, theories such as random forests, support vector machines, and graph theory in machine learning algorithms are widely used. However, machine learning algorithms still cannot do without artificial feature selection. Benefiting from the improvement of computer performance, deep learning methods have gradually shown amazing performance. Deep learning constructs a machine learning architecture model with multiple hidden layers, trains large-scale data, and obtains a large amount of more representative feature information, thereby classifying and predicting samples and improving the accuracy of classification and prediction.

[0004] At present, the existing forest point cloud segmentation methods based on deep learning have the following problems: (1) It is difficult to obtain datasets. The complex tree structure and forest environment make it challenging to accurately identify and label the various components of trees, thus hindering the acquisition of high-quality labeled datasets for forest point clouds. Existing methods mostly use tree-level datasets (i.e., each point cloud is a tree). Compared with scene-level datasets (i.e., each point cloud is a forest scene), this dataset is simpler and has less data. However, in practical applications, it is necessary to extract a single tree from the entire point cloud scene before segmenting it, which is not conducive to the seamless integration of field data collection and point cloud segmentation. (2) Existing research on deep learning semantic segmentation algorithms mainly relies on urban point cloud datasets. Compared with forest point cloud data, urban point cloud data exhibits less complexity and irregularity, and this difference is particularly evident in tree branches.

[0005] Through the above analysis, the problems and defects of the existing technology are as follows: Limited by the forest point cloud dataset, the potential of the existing semantic segmentation algorithm based on deep learning for tree structure segmentation in complex forest environments has not been fully explored. Considering the actual application needs, this method should be applicable to scene-level point cloud data to achieve seamless integration of field collection and structure segmentation, and be optimized for the more important branch categories in the forest. Summary of the invention

[0006] In response to the problems existing in the prior art, the present invention provides a three-dimensional point cloud tree structure segmentation method and system based on deep learning.

[0007] The present invention is implemented as follows: a three-dimensional point cloud tree structure segmentation method based on deep learning is called a Multi-Scale Spatially Enhanced Network (MSSE-Net). MSSE-Net is an end-to-end network with an encoder-decoder architecture. The network of the invention focuses on complex features and optimizes spatial awareness, thereby improving the accuracy of segmentation, and can achieve a more refined segmentation effect for three-dimensional point cloud tree structures. The network of the invention introduces multi-scale fusion in the sampling process to reduce the loss of edge points and increase the number of small sample points, thereby improving accuracy without increasing computational overhead. The network of the invention proposes a feature extraction module unit for extracting features from point cloud data. In this module, the invention introduces position encoding in distance weighting, performs cross-fusion of position and feature encoding, and applies multi-head attention pooling to extract and aggregate features.

[0008] Furthermore, the method for implementing tree structure segmentation includes the following steps:

[0009] Step 1: Obtain a dataset of forest point cloud data and divide it into a training set, a validation set, and a test set according to a certain ratio;

[0010] Step 2: Use a multi-scale sampling method to sample the data set and create a projection. The sampled data constitute a new data set.

[0011] Step 3: Configure the network operating environment and use the training set and validation set in the new data set to train the network;

[0012] Step 4: The test set uses the trained network to make predictions, and uses the projection information to project the labels of the dataset into the original point cloud, visualize the results, and achieve tree structure segmentation of the entire point cloud.

[0013] Furthermore, in step 2, a multi-scale sampling method is used to sample the data set and establish a projection. The specific method of forming a new data set from the sampled data includes:

[0014] S21, use the grid G ​​to sample the point cloud data; divide the entire point cloud data into different grids, and retain a point in each grid. The coordinates of this point are the centroid of the coordinates of all points in the grid, and the label is the label of the point with the most categories in the grid; assume that the sampled data obtained is S G , the data is the central sampling data;

[0015] S22, a series of grids of different sizes are used to sample the point cloud; the point cloud data is sampled using grids of G / 2, 2G, and 4G respectively, and the final sampled data are S G / 2 , S 2G , S 4G .

[0016] This data is reference sampling data;

[0017] S23, with the center sampling data S G As the core, other reference sampling data S G / 2 , S 2G , S 4G Supplement to the central sampling data; the data set obtained after all the reference sampling data are inserted into the central sampling data is called a multi-scale fusion point cloud data set;

[0018] S24, using the original point cloud data to perform nearest neighbor indexing on the multi-scale fused point cloud data set, and recording the nearest coordinates of each point in the original point cloud data in the fused data set as its projection to assist in data restoration.

[0019] Furthermore, the rule for adding other reference sampling data to the central sampling data in S23 is that for each point in the reference sampling data, the point closest to the point is searched in the central sampling data and compared with it; if the distance between the two points is greater than a certain threshold, the point in the reference sampling data is added to the central sampling data; if the labels of the two points are different, they are also added; otherwise, they are all discarded.

[0020] Furthermore, the feature extraction module of the network in step 3 is used to capture and aggregate the position and feature information of neighboring points, and the feature aggregation unit and multi-head attention pooling unit are used to explicitly embed the geometric and distribution characteristics of the surrounding points into each point through effective encoding and representation.

[0021] Furthermore, the feature aggregation module of the network in step three is used to aggregate the spatial and feature information of each point, and the information is mainly obtained from the neighboring points of the point; the aggregation method includes: based on the point-to-point L2 distance (Euclidean distance), using the k-nearest neighbor algorithm to obtain k neighboring points and their point features; using the point and its k neighboring points for spatial encoding and feature encoding, and then cross-merging the spatial encoding and feature encoding to aggregate into a feature vector for the point.

[0022] Furthermore, in the spatial coding, the xyz coordinates of the point (hereinafter referred to as the origin) and its neighboring points are used to encode the position-related information; the spatial coding includes: L1 distance (Manhattan distance), L2 distance (Euclidean distance), xyz coordinates of the origin and neighboring points, and the relative coordinates between the origin and the neighboring points; the formula is as follows:

[0023]

[0024] Among them, p i is the xyz coordinate of the origin i; are the xyz coordinates of the kth nearest neighbor point j; ⊕ is the concatenation operation; ||·||1 calculates the L1 distance; ||·||2 calculates the L2 distance.

[0025] Use a function g() to weight the spatial encoding vector; weight function as follows:

[0026]

[0027] Among them, L k is the L2 distance between the origin and the kth nearest neighbor; L i_max is the maximum L2 distance between the origin and all neighboring points. The weight value is limited to the range of (0, 0.5); the encoding result of the adjacent points is represented by an 11-dimensional feature vector; a shared multi-layer perceptron is used to extract features and coordinate dimensions; position encoding It is expressed as follows:

[0028]

[0029] Furthermore, in feature encoding, feature-related information is encoded using the feature vectors of the origin and its neighboring points; the feature encoding is the feature of each neighboring point of the origin minus the feature of the origin; feature encoding The formula is:

[0030]

[0031] Among them, f i is the eigenvector of the origin i, is the eigenvector of the kth nearest neighbor point j.

[0032] Furthermore, in the cross-fusion process, the position encoding vector and the neighboring point feature vector Connect them in series to encode the feature vector and the unweighted position vector Then, a shared multi-layer perceptron is used to unify the dimensions of the two sets of vectors and connect them in series. The mathematical formula of this process is as follows:

[0033]

[0034]

[0035]

[0036] in, The spatial encoding vector and the neighboring point feature vector are combined; The feature encoding vector and the unweighted spatial encoding vector are fused; is the aggregate feature vector.

[0037] Furthermore, the multi-head attention pooling module of the network in step 3 is located after the feature aggregation module to learn and retain the main features; the vector features are extracted by using a shared multi-layer perceptron and the attention score is obtained by the SoftMax function; the mathematical representation of the attention score is as follows:

[0038]

[0039] in, is the attention score; is the aggregate feature vector.

[0040] Through the above formula, N groups of attention scores are obtained. Then, these N groups of attention scores are multiplied by the aggregation vector respectively, and the results are summed and maximized respectively, and then connected in series; finally, a shared multi-layer perceptron is used to obtain the one-dimensional feature vector of the origin. The formula is expressed as follows:

[0041]

[0042] Among them, F i_feature is the eigenvector of the origin i.

[0043] Another object of the present invention is to provide a three-dimensional point cloud tree structure segmentation method based on deep learning and a three-dimensional point cloud tree structure segmentation system based on deep learning, the system comprising:

[0044] The data set acquisition and division module is used to obtain the data set of forest point cloud data and divide it into training set, verification set and test set according to a certain ratio;

[0045] The data set sampling and projection establishment module is used to sample the data set and establish projection using a multi-scale sampling method. The sampled data constitute a new data set.

[0046] The network training module is used to configure the network operating environment and use the training set and validation set in the new data set to train the network;

[0047] The prediction and label projection module is used to use the trained network to make predictions using the test set, and use the projection information to assign the labels of the dataset to the original point cloud, visualize the results, and achieve tree structure segmentation of the entire point cloud.

[0048] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0049] First, the present invention will provide a semantic segmentation network for complex forest environments. The network is designed as an end-to-end network framework that directly uses the original 3D coordinates as input and outputs semantic labels for the point clouds of each component of the tree. In order to solve the problems caused by occlusion and irregular distribution patterns in forest laser scanning data, the network of the present invention focuses on enhancing spatial features, with the focus on multi-scale sampling and feature extraction modules. Multi-scale sampling allows a more comprehensive representation of the global point cloud distribution without adding additional computational costs. At the same time, the network can be trained with a smaller amount of data, which improves the learning efficiency of the network. In the feature extraction module, each feature vector is position encoded and feature encoded respectively, and distance-weighted position encoding and neighbor point features, as well as cross-fusion between position encoding and feature encoding, are performed, and multi-head attention pooling is used to aggregate local information, thereby promoting effective feature extraction.

[0050] The present invention innovatively implements forest point cloud segmentation to achieve point cloud segmentation of the ground, tree trunks, branches, and other categories. The end-to-end network architecture ensures that the network can directly train and test the scene-level forest point cloud dataset. The dataset used by the network does not process the noise generated during the data acquisition process, and directly annotates the forest point cloud. Therefore, the trained network can directly process the data obtained on site, thereby achieving seamless integration of data acquisition and point cloud segmentation. At the same time, the network is also greatly superior to other networks in terms of accuracy, and its segmentation results are more reliable.

[0051] Second, the present invention designs a semantic segmentation network around a complex forest environment to achieve efficient and accurate segmentation of forest trees. The network has been improved around spatial enhancement, focusing on tree branch points that are distributed in long strips and have fewer categories in the point cloud. The improved network has an excellent recognition effect on tree branch points, and the recognition coverage is far ahead. At the same time, the network can not only identify primary branches, but also secondary branches, which is an advantage that other deep learning semantic segmentation networks do not have. The invention can accurately identify various parts of trees, providing a powerful tool for forest ecological monitoring and research.

[0052] In modern forest resource management and monitoring, rapid acquisition of tree information is crucial for decision-making. Compared with traditional methods, the present invention has excellent data processing capabilities and can directly process data collected in the forest field. At the same time, the present invention has the ability to quickly process large-scale point cloud data, and can provide semantic labels for forest point cloud data at the sample plot level, providing reliable and valuable information for forest parameter acquisition. At the same time, these parameters will be beneficial to forest assessments, such as timber assessment, carbon reserve calculation, biomass assessment, etc., thereby further promoting the sustainable development, protection and utilization of the forest environment.

[0053] Third, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0054] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are as follows: The semantic segmentation network of the present invention has broad application prospects in the field of forest tree segmentation. By providing efficient, accurate and real-time tree segmentation services, the present technical solution will play a key role in the fields of forest resource management, ecological monitoring, environmental protection, etc. It is expected that in commercial applications such as decision support systems, forest resource surveys, and environmental monitoring, the present invention will bring significant economic benefits to enterprises and research institutions, provide high-quality information services to their users, and thus bring considerable market benefits.

[0055] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad: Due to the lack of high-precision and reliable data sets at home and abroad, there is still a large gap in forest tree segmentation technology. Traditional methods often face problems such as insufficient accuracy, poor real-time performance, and weak adaptability. The present invention successfully fills this technical gap by introducing an advanced semantic segmentation network and optimizing it for complex forest environments, improving the accuracy and efficiency of tree segmentation, providing a new solution for forest resource management, and having a good market competitive advantage.

[0056] (3) The technical solution of the present invention solves a technical problem that people have been eager to solve but have never succeeded in solving: traditional technical solutions often require manual intervention and it is difficult to process large-scale point cloud data. The present invention can achieve accurate segmentation of trees from forest point cloud data at the sample plot level through a deep learning network. At the same time, the present invention can directly process large-scale collected data and reduce manual intervention. Therefore, the present invention will provide more reliable technical support for forest ecological research and resource management, solving the long-standing bottleneck problem in the industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 It is a flow chart of a method for segmenting tree structure of three-dimensional point cloud based on deep learning provided by an embodiment of the present invention;

[0059] Figure 2 It is a structural diagram of a three-dimensional point cloud tree structure segmentation system based on deep learning provided by an embodiment of the present invention;

[0060] Figure 3 Schematic diagram of a data set for verification provided by an embodiment of the present invention; (A) Four types of visualization results: ground, tree trunk, branches and others; (B) Three types of visualization results: ground, tree trunk, branches;

[0061] Figure 4 is a schematic diagram of multi-scale sampling provided by an embodiment of the present invention;

[0062] Figure 5 is a detailed architecture diagram of a network provided by an embodiment of the present invention;

[0063] Figure 6 is a schematic diagram of a feature extraction module in a network provided by an embodiment of the present invention;

[0064] Figure 7 is a schematic diagram of position coding provided by an embodiment of the present invention;

[0065] Figure 8 This is a panoramic visualization comparison result diagram provided by an embodiment of the present invention and other methods;

[0066] Fig. 9 is a canopy area visualization comparison result diagram provided by an embodiment of the present invention and other methods;

[0067] Fig.10 This is a visualization comparison result diagram of the middle area provided by an embodiment of the present invention and other methods;

[0068] Fig.11 This is a visualization comparison result diagram of the ground area provided by an embodiment of the present invention and other methods. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0070] Compared with traditional 3D point cloud processing technology, the 3D point cloud tree structure segmentation method based on deep learning provided by the present invention has shown the following significant technical progress:

[0071] 1. Improved segmentation accuracy: By leveraging deep learning models, this method is able to more accurately identify and segment individual tree structures, especially when processing complex 3D point cloud data, and maintain high segmentation accuracy even in environments with dense trees or high occlusion.

[0072] 2. Enhanced detail capture capability: Multi-scale sampling technology enables this method to capture the characteristics of tree structures at different scales, allowing the model to retain the characteristics of the point cloud data itself to the greatest extent without increasing the amount of computation, thereby improving the accuracy of the model and its ability to capture fine-grained features.

[0073] 3. Enhanced data processing capabilities: The network of this method is designed to process large-scale, large-scale point cloud data sets, and its computing speed is not affected by the data size, making the trained model highly applicable and directly usable for field data processing.

[0074] 4. Automated processing is achieved: Compared with the traditional point cloud processing method that relies on expert manual analysis or simple algorithms, the present invention achieves automated processing and analysis of point cloud data, significantly improves work efficiency and reduces labor costs.

[0075] 5. Enhanced interpretability of results: By projecting the prediction results of the deep learning model back to the original point cloud data and visualizing it, the segmentation results are more intuitive and easy to understand, making it easier for professionals to conduct further analysis and application.

[0076] 6. Promoted multi-field applications: Accurate tree structure segmentation is of great significance to many fields such as forest management, urban planning, and environmental monitoring. The technical progress of this method provides strong technical support for these fields and helps promote the development of related research and applications.

[0077] By combining deep learning and multi-scale sampling technology, the present invention significantly improves the accuracy, efficiency and application scope of three-dimensional point cloud data processing, and provides an efficient and accurate tree structure segmentation solution for fields such as forest resource management and environmental monitoring.

[0078] like Figure 1 As shown, a three-dimensional point cloud tree structure segmentation method based on deep learning provided by an embodiment of the present invention includes the following steps:

[0079] Step 1: Obtain a dataset of forest point cloud data and divide it into training set, validation set, and test set;

[0080] Step 2: Use a multi-scale sampling method to sample the data set and create a projection. The sampled data constitute a new data set.

[0081] Step 3: Configure the network operating environment and use the training set and validation set in the new data set to train the network;

[0082] Step 4: The test set uses the trained network to make predictions, and uses the projection information to project the labels of the dataset into the original point cloud, visualize the results, and achieve tree structure segmentation of the entire point cloud.

[0083] The three-dimensional point cloud tree structure segmentation method based on deep learning described in the embodiment of the present invention is mainly implemented through the following working principles:

[0084] Step 1: Acquisition and division of data sets

[0085] Dataset acquisition: First, a dataset containing forest point cloud data needs to be obtained. These point cloud data are usually obtained through laser scanning (such as LIDAR) or photogrammetry technology, which can reflect the three-dimensional structure of trees and their surroundings in detail.

[0086] Dataset division: The acquired dataset is divided into training set, validation set and test set. The training set is used to train the deep learning model, the validation set is used to evaluate the performance of the model and adjust the hyperparameters during the training process, and the test set is used to finally evaluate the generalization ability of the model.

[0087] Step 2: Multi-scale sampling and projection establishment

[0088] Multi-scale sampling: In order to improve the generalization ability and processing efficiency of the model, a multi-scale sampling method is used to sample the original point cloud data. This means sampling the point cloud at different scales to obtain features of different granularities, so that the model can learn tree structure information from local to global.

[0089] Projection creation: The sampled data is converted into a new data representation through some form of projection (such as orthogonal projection or perspective projection) to facilitate subsequent deep learning model processing. This step is to reduce the complexity of the data while retaining important structural features.

[0090] Step 3: Network training

[0091] Configure the network environment: Configure the corresponding software and hardware operating environment according to the selected deep learning model architecture (such as convolutional neural network CNN, graph neural network GNN, etc.).

[0092] Network training: The model is trained using training set data that has been processed with multi-scale sampling and projection. During the training process, the validation set is used to monitor the performance of the model so that appropriate adjustments and optimizations can be made.

[0093] Step 4: Prediction and Visualization

[0094] Prediction: The trained model is used to make predictions using the test set to evaluate the performance of the model on unseen data.

[0095] Visualization: Projecting the labels in the prediction results back to the original point cloud data allows you to visually observe the tree structure segmented by the model. Through visualization, researchers and professionals can verify the accuracy and practicality of the model.

[0096] The core of this method is to use deep learning technology to process and analyze 3D point cloud data, so as to achieve efficient and accurate segmentation of tree structures. Through multi-scale sampling and data projection, the model can effectively learn and identify tree structure features in point clouds. In addition, projecting the prediction results back to the original point cloud and visualizing them can intuitively demonstrate the segmentation effect, providing strong technical support for fields such as forest resource management, urban planning and environmental monitoring.

[0097] Further, the detailed description and sub-steps are as follows:

[0098] (1) Obtain a forest point cloud dataset and divide it into three independent subsets according to a certain ratio, which are used as training set, validation set, and test set. The training set is used to train the network, and the validation set is used to test the state and convergence of the model during training and adjust the model hyperparameters. The test set is used to evaluate the generalization ability of the model.

[0099] (2) Since the number of point clouds is large and they are highly clustered, data redundancy can be reduced by sampling. However, if the sampling scale is too large, information will be lost, and if the sampling scale is too small, the amount of data cannot be effectively reduced. The multi-scale sampling method adopted in the present invention can greatly reduce the redundancy of the data while retaining the detailed information. At the same time, in order not to destroy the integrity of the data, a projection is established to ensure that the labels of the sampled data set can be projected into the original data set. In theory, this step can be ignored, but too many point clouds will bring higher computer performance requirements and longer training time. In actual processes, this step is inevitable due to limited computer performance.

[0100] First, use the grid G ​​to sample the point cloud data. Divide the entire point cloud data into different grids, and retain a point in each grid. The coordinates of this point are the centroid of the coordinates of all points in the grid, and the label is the label of the point with the most categories in the grid. Assume that the sampled data obtained is S G , which is called the center sampled data.

[0101] Secondly, a series of grids of different sizes are used to sample the point cloud. For example, the point cloud data is sampled using grids of G / 2, 2G, and 4G respectively. The final sampled data are S G / 2 , S 2G , S 4G . This data is called reference sampling data.

[0102] Sampling data at the center S G As the core, other reference sampling data S G / 2 , S 2G , S 4G Supplement to the central sampling data. The rule is that for each point in the reference sampling data, search for the point in the central sampling data closest to the point and compare it with it. If the distance between the two points is greater than a certain threshold, the point in the reference sampling data is supplemented to the central sampling data; if the labels of the two points are different, they are also supplemented; otherwise, they are discarded.

[0103] After all the reference sampling data are inserted into the central sampling data, a multi-scale fused point cloud dataset will be obtained. The original point cloud data is used to perform nearest neighbor indexing on the dataset, and the nearest coordinates of each point in the original point cloud data in the new dataset are recorded as its projection to assist data restoration.

[0104] (3) The network of the present invention is a framework of an encoder and a decoder. In general, the point cloud data is directly input into the network, and the point cloud features are extracted through a shared multi-layer perceptron. Then it passes through four encoders, each of which includes a feature extraction module and downsampling. After that, it passes through four decoders, each of which includes a shared multi-layer perceptron and upsampling. Among them, jump connections are used to promote information transmission between the encoding and decoding stages. Subsequently, three fully connected layers and a random inactivation layer (dropout layer) are passed to achieve the prediction of the semantic label of each point.

[0105] The key module of the network is the feature extraction module in the encoder. Other parts, such as the shared multi-layer perceptron, up and down sampling, fully connected layers, and dropout layers follow the general framework of deep learning and are not described in detail. This module is designed to capture and aggregate the position and feature information of neighboring points. Through effective encoding and representation, it can explicitly embed the geometric and distribution characteristics of the surrounding points into each point, thereby facilitating the network's ability to learn the complex internal structure of the forest point cloud. The feature extraction module uses feature aggregation units and multi-head attention pooling units to achieve this function.

[0106] The feature aggregation module aggregates the spatial and feature information of each point, and the information is mainly obtained from the neighboring points of the point. First, based on the point-wise L2 distance (Euclidean distance), the k nearest neighbor algorithm is used to obtain k neighboring points and their point features. The point and its k neighboring points are used for spatial encoding and feature encoding, and then the spatial encoding and feature encoding are cross-fused and aggregated into a feature vector for the point.

[0107] In spatial coding, the xyz coordinates of the point (hereinafter referred to as the origin) and its neighboring points are used to encode the location-related information; the spatial coding includes: L1 distance (Manhattan distance), L2 distance (Euclidean distance), xyz coordinates of the origin and neighboring points, and the relative coordinates between the origin and neighboring points; the formula is as follows:

[0108]

[0109] Among them, p i is the xyz coordinate of the origin i; are the xyz coordinates of the kth nearest neighbor point j; ⊕ is the concatenation operation; ||·||1 calculates the L1 distance; ||·||2 calculates the L2 distance.

[0110] Use a function g() to weight the spatial encoding vector; weight function as follows:

[0111]

[0112] Among them, L k is the L2 distance between the origin and the kth nearest neighbor; L i_max is the maximum L2 distance between the origin and all neighboring points. The weight value is roughly limited to the range of (0, 0.5). The range of the weight function is not normalized by linear transformation because these parameters will be learned and adjusted through subsequent training. The encoding result of the neighboring points is represented by an 11-dimensional feature vector. Therefore, in order to facilitate the extensive exploration of position features and subsequent cascade operations, a shared multi-layer perceptron is used to extract features and coordinate dimensions. Finally, the position encoding It is expressed as follows:

[0113]

[0114] In feature encoding, the feature vectors of the origin and its neighboring points are used to encode feature-related information. The feature encoding is the feature of each neighboring point of the origin minus the feature of the origin. Feature encoding The formula is:

[0115]

[0116] Among them, f i is the eigenvector of the origin i, is the eigenvector of the kth nearest neighbor point j.

[0117] During the cross-fusion process, the position encoding vector and the neighboring point feature vector Connect them in series to encode the feature vector and the unweighted position vector Then, a shared multi-layer perceptron is used to unify the dimensions of the two sets of vectors and connect them in series. The mathematical formula of this process is as follows:

[0118]

[0119]

[0120]

[0121] in, The spatial encoding vector and the neighboring point feature vector are combined; The feature encoding vector and the unweighted spatial encoding vector are fused; is the aggregate feature vector.

[0122] The multi-head attention pooling module is located after the feature aggregation module and is used to learn and retain the main features. This model is inspired by the multi-head attention mechanism, but with some changes in the computation. The traditional multi-head attention mechanism requires mapping the input to the query space, key space, and value space through a linear transformation, and then performing matrix multiplication on the query space and key space to produce an attention matrix. However, matrix multiplication brings a considerable computational memory burden. Therefore, a shared multi-layer perceptron is used to extract vector features and obtain the attention score through the SoftMax function. Attention score The mathematical representation of is as follows:

[0123]

[0124] in, is the attention score; is the aggregate feature vector.

[0125] In order to fully understand the aggregate feature vector, N groups of attention scores are obtained through the above formula. Then, these N groups of attention scores are multiplied by the aggregate vector respectively, and the results are summed and maximized respectively, and then connected in series; finally, a shared multi-layer perceptron is used to obtain the one-dimensional feature vector of the origin, which is expressed as follows:

[0126]

[0127] Among them, F i_feature is the eigenvector of the origin i.

[0128] (4) The test set will use the trained network to make predictions. In the prediction results, each point will have a label that represents the category of the point. The labels of the dataset are assigned to the original point cloud using projection information, and the results are visualized to finally achieve tree structure segmentation of the entire point cloud.

[0129] like Figure 2 As shown, a 3D point cloud tree structure segmentation method based on deep learning and a 3D point cloud tree structure segmentation system based on deep learning are provided in an embodiment of the present invention, and the system comprises:

[0130] The data set acquisition and division module is used to obtain the data set of forest point cloud data and divide it into training set, verification set and test set according to a certain ratio;

[0131] The data set sampling and projection establishment module is used to sample the data set and establish projection using a multi-scale sampling method. The sampled data constitute a new data set.

[0132] The network training module is used to configure the network operating environment and use the training set and validation set in the new data set to train the network;

[0133] The prediction and label projection module is used to use the trained network to make predictions using the test set, and use the projection information to assign the labels of the dataset to the original point cloud, visualize the results, and achieve tree structure segmentation of the entire point cloud.

[0134] The three-dimensional point cloud tree structure segmentation technology based on deep learning provided by the specific implementation example of the present invention includes the following steps:

[0135] Step 1: Obtain a dataset of forest point cloud data and divide it into training set, validation set, and test set in a ratio of 2:1:1. Figure 3 As shown, Forest 0 is the validation set. Forest 1 and Forest 2 are the training sets. Forest 3 is the test set. The forest point cloud data needs to include the xyz coordinates of each point and the label of each point. The labels include four categories, namely ground (0), trunk (1), branch (2), and others (3). Figure 3 All categories are shown in (A). Figure 3 Other categories are removed in (B) to provide better display effect.

[0136] Step 2: Use a multi-scale sampling method to sample the data set. The sampling method is as follows: Figure 4 As shown in the figure, the central sampling size is 1 cm, and the auxiliary sampling sizes are 0.25 cm, 0.5 cm, and 2 cm. Taking the central sampling point cloud as the core, the auxiliary sampling point cloud is divided into Figure 4 The rule is inserted into the center sampling to form a fused point cloud, which is fed into the network as a new data set for training. Subsequently, the projection relationship is established between the original point cloud data and the fused point cloud data.

[0137] Step three, put the data into the network for training and verification. Among them, the network proposed in the present invention is implemented by coding based on PyTorch. The computer configuration of the experiment is: Intel (R) Xeon (R) Gold5218CPU, 2.30GHz, 16 cores. The computer is equipped with two NVIDIA GeForce RTX 3080Ti graphics cards, each of which provides 11GB of memory capacity. The model was trained using 240 epochs and 16 batch sizes, each batch consisting of 16,384 sampling points. Some techniques were used to optimize model training and enhance its performance. The AdamW optimizer was used, with a learning rate of 1e-3 and a decay weight of 1e-4. Class-balanced loss was used to deal with unbalanced distribution, and Leaky Relu was selected as the activation function. These training strategies can effectively optimize network performance and maximize the capabilities of the model of the present invention. The overall framework of the network is as follows Figure 5As shown. The point cloud data passes through a multi-layer perceptron to extract point cloud features, and the dimension becomes (N, 8), where N is the number of point clouds. Then it passes through four encoders, each of which includes a feature extraction module and downsampling, and the dimensions become (N / 4, 32), (N / 16, 128), (N / 64, 256), (N / 256, 512) respectively. After that, it passes through four decoders, each of which includes a multi-layer perceptron and upsampling, and the dimensions are restored to (N / 64, 256), (N / 16, 128), (N / 4, 32), (N, 8). Skip connections are used in the encoding and decoding process to promote information transmission. Subsequently, three fully connected layers and one dropout layer are used to finally achieve semantic segmentation. The structure of the key module of the network, the Feature Extraction Module (FEM), is shown as follows Figure 6 As shown in , the module outputs different feature vectors according to the different input feature vectors. Each point indexes the information of the 16 neighboring points around it, and performs spatial encoding and feature encoding. The schematic diagram of spatial encoding is shown in Figure 7 Afterwards, a multi-head attention pooling unit is used to learn and emphasize key information, and finally output a one-dimensional feature for each point.

[0138] Step 4: Use the trained network to predict the validation set and use projection to project the labels into the original point cloud.

[0139] Accurate and fast segmentation of forest point cloud data is an important prerequisite for forest assessment and research. Through detailed research on segmented forest point cloud data, including feature extraction of crown shape, trunk thickness, branching, etc., it can be used to gain a deeper understanding of forest ecosystems. This analysis can reveal the growth pattern, structural characteristics and spatial distribution of trees, which will help provide basic data for research fields such as forest health status monitoring, biodiversity assessment, carbon storage estimation, and better promote the sustainable development and utilization of forests.

[0140] 1) Growth pattern and health status assessment. By monitoring the structure of trees, including crown shape, trunk thickness, and branching, the growth pattern of trees can be inferred. At the same time, using the reflection intensity and color information in the point cloud data, the health status of trees can be identified and assessed, and potential diseases, pests, or other physiological abnormalities can be detected. This comprehensive point cloud analysis helps to detect ecological problems in forests in a timely manner and provide accurate and comprehensive information for sustainable forest management and ecosystem monitoring.

[0141] 2) Biodiversity assessment. By identifying different tree species and analyzing their distribution density, the biodiversity in the forest can be quantitatively assessed. This information is essential for monitoring ecosystem health, inferring vegetation structure and species composition, and assessing the adaptability and stability of ecosystems. The practical application value is reflected in the provision of non-invasive biodiversity monitoring methods, providing a scientific basis for ecological research, environmental protection and sustainable resource management, helping to formulate effective protection strategies and promote the sustainable development of ecosystems.

[0142] 3) Carbon stock estimation research. Through detailed analysis of parameters such as tree volume, density and structure, the woody biomass in the forest can be accurately calculated, thereby estimating carbon storage. Combined with the segmented tree structure information, such as trunks, branches and leaves, the volume and biomass of each tree can be quantified more accurately, providing more refined data for the carbon storage of the entire forest. This method not only helps to gain a deeper understanding of the relationship between carbon cycle and climate change, but also provides a basis for scientific and reasonable carbon emission management and forest protection decision-making, and promotes sustainable natural resource management.

[0143] The segmented forest point cloud data can provide key ecological information to assist in monitoring forest conditions. By deeply understanding the dynamic changes of forest ecosystems, environmental protection research can formulate more precise protection strategies, promote sustainable forestry management, ensure the health and ecological balance of forest ecosystems, and respond to climate change, providing important support for protecting the earth's environment.

[0144] 3. Evidence of the effects of the embodiments. The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the prior art. The following content is described in conjunction with the data, charts, etc. of the test process.

[0145] In order to evaluate the advantages of the method, the recall rate, precision, intersection over union, and F1 score of the network were calculated, and comparative experiments were conducted. The comparative experiments used a series of excellent semantic segmentation networks, namely: PointNet (2017), PointNet++ (2017), KPConv (2019), RandLA-Net (2020), BAAF-Net (2021), PointTransformer (2021). All networks were tested on computer systems with the same configuration. The panoramic visualization comparison results are shown in Figure 2. Figure 8 As shown; the canopy area visualization comparison results are shown in Fig. 9 As shown; the visualization comparison results of the middle area are as follows Fig.10 As shown; the ground area visualization comparison results are shown Fig.11 The quantitative results are shown in Table 1 and Table 2:

[0146] Table 1 Comparison results of recall rate and precision

[0147]

[0148] Table 2 Comparison results of F1 score and IoU

[0149]

[0150] According to the quantitative evaluation, it can be seen that the accuracy change gap between the ground and trunk categories is relatively small (except PointNet), but it is more obvious in the branch category. Therefore, the recognition of branches is focused on. Overall, the model of the present invention and the model of KPConv have the highest accuracy, but interestingly, the two show different characteristics. KPConv has a low recall rate (0.5251) and a high precision (0.7857), which indicates that it has a better ability to recognize branches while ignoring certain instances. In contrast, the model of the present invention has a high recall rate (0.8383) and a low precision (0.6436), indicating that the recognition coverage of branches is wider, but the recognition accuracy is reduced. The visualization results also verify this finding. The prediction results of KPConv are very consistent with the true value of the branches, but there are obvious omissions, especially in the secondary branches. On the other hand, the prediction of branches by the present invention is more comprehensive; but it appears to be thicker than the true value. This means that the network of the present invention is affected by noise and mistakenly identifies the surrounding noise as branches. It is worth mentioning that the network of the present invention is the only network that can identify secondary branches, which is a unique advantage that other networks do not have.

[0151] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A 3D point cloud tree structure segmentation method based on deep learning, characterized in that: The following steps are involved: Step 1: Obtain a dataset of forest point cloud data and divide it into a training set, a validation set, and a test set according to a certain ratio; Step 2: Use a multi-scale sampling method to sample the data set and create a projection. The sampled data constitute a new data set. Step 3: Configure the network operating environment and use the training set and validation set in the new data set to train the network; Step 4: The test set uses the trained network to make predictions, and uses the projection information to project the labels of the dataset into the original point cloud, visualize the results, and achieve tree structure segmentation of the entire point cloud; In step 3, the network includes a feature extraction module, which is used to capture and aggregate the position and feature information of neighboring points. The feature aggregation module and the multi-head attention pooling module are used to explicitly embed the geometric and distribution characteristics of the surrounding points into each point through effective encoding and representation. In step 3, the network includes a feature aggregation module, which is used to aggregate the spatial and feature information of each point, and the information is mainly obtained from the neighboring points of the point; The clustering method includes: based on L2 distance, the L2 distance is Euclidean distance, using k nearest neighbor algorithm to obtain k neighbor points and their point features; using the point and its k nearest neighbor points to perform spatial coding and feature coding, and then cross-merging the spatial coding and feature coding to aggregate into a feature vector of the point; In step 3, the network includes a multi-head attention pooling module, which is located after the feature aggregation module and is used to learn and retain the main features; the vector features are extracted by using a shared multi-layer perceptron and the attention score is obtained by the SoftMax function; the attention score The mathematical representation of is as follows: in, is the attention score; is the aggregate feature vector; Through the above formula, N groups of attention scores are obtained. Then, these N groups of attention scores are multiplied by the aggregation vector respectively, and the results are summed and maximized respectively, and then connected in series; finally, a shared multi-layer perceptron is used to obtain the one-dimensional feature vector of the origin. The formula is expressed as follows: Among them, F i_feature is the eigenvector of the origin i.

2. The method for segmenting tree structures in three-dimensional point clouds based on deep learning according to claim 1, characterized in that: In step 2, a multi-scale sampling method is used to sample the data set and establish a projection. The specific method of forming a new data set from the sampled data includes: S21, use the grid G ​​to sample the point cloud data; divide the entire point cloud data into different grids, and retain a point in each grid. The coordinates of this point are the centroid of the coordinates of all points in the grid, and the label is the label of the point with the most categories in the grid; assume that the sampled data obtained is S G , the data is the central sampling data; S22, a series of grids of different sizes are used to sample the point cloud; the point cloud data is sampled using grids of G / 2, 2G, and 4G respectively, and the final sampled data are S G / 2 , S 2G , S 4G ; This data is reference sampling data; S23, with the center sampling data S G As the core, the reference sampling data S G / 2 , S 2G , S 4G Supplement to the central sampling data; the data set obtained after all the reference sampling data are inserted into the central sampling data is called a multi-scale fusion point cloud data set; S24, using the original point cloud data to perform nearest neighbor indexing on the multi-scale fused point cloud data set, and recording the nearest coordinates of each point in the original point cloud data in the fused data set as its projection to assist in data restoration.

3. The method for segmenting tree structure in three-dimensional point cloud based on deep learning as claimed in claim 2, characterized in that: The rule for adding the reference sampling data to the central sampling data in S23 is that for each point in the reference sampling data, the point closest to the point in the central sampling data is searched and compared with it; if the distance between the two points is greater than a certain threshold, the point in the reference sampling data is added to the central sampling data; if the labels of the two points are different, they are also added; otherwise, they are discarded.

4. The method for segmenting tree structures in three-dimensional point clouds based on deep learning as claimed in claim 1, characterized in that: In spatial coding, the xyz coordinates of the point and its neighboring points are used to encode the position-related information. This point is subsequently referred to as the origin. The spatial coding includes: L1 distance, L2 distance, xyz coordinates of the origin and neighboring points, and relative coordinates between the origin and the neighboring points. The L1 distance is the Manhattan distance, and the formula is as follows: Among them, p i is the xyz coordinate of the origin i; is the xyz coordinate of the kth nearest neighbor point j; It is a concatenation operation; ||·||1 calculates the L1 distance; ||·||2 calculates the L2 distance; Use a function g( ) to weight the spatial encoding vector; weight function as follows: Among them, L k is the L2 distance between the origin and the kth nearest neighbor; L i_max is the maximum L2 distance between the origin and all its neighboring points; The weight value is limited to the range of (0, 0.5); the encoding results of adjacent points are represented by 11-dimensional feature vectors; a shared multi-layer perceptron is used to extract features and coordinate dimensions; position encoding It is expressed as follows:

5. The method for segmenting tree structures in three-dimensional point clouds based on deep learning as claimed in claim 1, characterized in that: In feature encoding, feature-related information is encoded using the feature vectors of the origin and its neighboring points; the feature encoding is the feature of each neighboring point of the origin minus the feature of the origin; Feature Encoding The formula is: Among them, f i is the eigenvector of the origin i, is the eigenvector of the kth nearest neighbor point j.

6. The method for segmenting tree structures in three-dimensional point clouds based on deep learning according to claim 1, characterized in that: During the cross-fusion process, the position encoding vector and the neighboring point feature vector Connect them in series to encode the feature vector and the unweighted position vector Then, a shared multi-layer perceptron is used to unify the dimensions of the two sets of vectors and connect them in series. The mathematical formula of this process is as follows: in, The spatial encoding vector and the neighboring point feature vector are combined; The feature encoding vector and the unweighted spatial encoding vector are fused; is the aggregate feature vector.

7. A 3D point cloud tree structure segmentation system based on deep learning according to any one of claims 1 to 6, characterized in that: The system includes: The data set acquisition and division module is used to obtain the data set of forest point cloud data and divide it into training set, verification set and test set according to a certain ratio; The data set sampling and projection establishment module is used to sample the data set and establish projection using a multi-scale sampling method. The sampled data constitute a new data set. The network training module is used to configure the network operating environment and use the training set and validation set in the new data set to train the network; The prediction and label projection module is used to use the trained network to make predictions using the test set, and use the projection information to assign the labels of the dataset to the original point cloud, visualize the results, and achieve tree structure segmentation of the entire point cloud.

Images