Tree crown breadth and trunk classification method, device and equipment and storage medium

Abstract

The invention discloses a tree crown breadth and trunk classification method and device, equipment and a storage medium, and the method comprises the steps: collecting a point cloud data sample of a tree, and carrying out the downsampling to obtain the cloud data of each sub-point; performing spatial sparse labeling on each piece of sub-point cloud data according to the tree part type to obtain each piece of weak label data and each piece of unlabeled data; training a preliminary classification model by using the weak label data, and processing the unlabeled data to obtain pseudo label data corresponding to the unlabeled data; performing data conversion processing on each piece of pseudo tag data; optimizing the preliminary classification model by using the weak label data, the pseudo label data and the transformation data to obtain a trained target classification model; and collecting point cloud data of a to-be-classified tree, and processing the point cloud data by using the target classification model to obtain a crown breadth and trunk classification result of the to-be-classified tree. According to the method, the crown breadth and the trunk of the tree can be accurately classified, and reliable data support is provided for safe operation of electric power facilities.

Description

Technical Field

[0001] The present application relates to the technical field of tree classification, and in particular to a method, device, equipment and storage medium for classifying tree crowns and trunks. Background Art

[0002] In today's society, it is extremely important to ensure the safety of equipment or areas such as power lines and transmission corridors. For example, in the transmission corridor scenario, overgrown or weirdly growing trees are often an important cause of power safety accidents. In terms of transmission corridor protection, the precise distinction between tree crowns and trunks provides refined support for identifying dangerous trees in transmission corridors, greatly improving the scientificity and efficiency of transmission line safety management. Through the precise classification of tree parts, the specific structural information of each tree along the transmission corridor can be obtained, and the specific structural information provides a reliable basis for accurately assessing the potential tree barrier risks in the transmission corridor.

[0003] However, the current tree crown and trunk classification methods do not take into account the complex and changeable conditions of trees, have low adaptability, and cannot meet the classification needs of the crown and trunk of diverse trees. Summary of the invention

[0004] In view of this, the present application provides a tree crown and trunk classification method, device, equipment and storage medium, which are used to solve the problem that the current tree crown and trunk classification method does not take into account the complex and changeable conditions of trees, has low adaptability, and cannot meet the classification needs of the crown and trunk of diverse trees.

[0005] To achieve the above objectives, the proposed solution is as follows:

[0006] In a first aspect, a method for classifying tree crowns and trunks comprises:

[0007] Collecting point cloud data samples of trees, and downsampling the point cloud data samples to obtain sub-point cloud data;

[0008] Performing spatial sparse annotation on each of the sub-point cloud data according to each pre-set tree part type to obtain each weak label data;

[0009] Using each of the weakly labeled data to train a preliminary classification model;

[0010] Using the preliminary classification model to process each sub-point cloud data except the weakly labeled data, to obtain pseudo-label data corresponding to each sub-point cloud data;

[0011] Performing data transformation processing on each of the pseudo-label data to obtain each transformed data;

[0012] Optimizing the preliminary classification model using the weakly labeled data, pseudo-labeled data, and transformed data to obtain a trained target classification model;

[0013] The point cloud data of the trees to be classified are collected, and the point cloud data are processed using the target classification model to obtain the crown width and trunk classification results of the trees to be classified.

[0014] Preferably, downsampling the point cloud data samples to obtain each sub-point cloud data includes:

[0015] Obtaining the density of the point cloud data sample;

[0016] Setting a three-dimensional voxel grid according to the density, and dividing the point cloud data samples according to the three-dimensional voxel grid to obtain individual voxels;

[0017] Selecting a representative point in each of the voxels;

[0018] The data corresponding to each representative point is used as the sub-point cloud data.

[0019] Preferably, the taking the data corresponding to each representative point as the sub-point cloud data includes:

[0020] For each representative point, extract the three-dimensional spatial coordinates and reflection intensity of the representative point as basic features;

[0021] Extract the normal vector, curvature and point density of the representative point as high-level geometric features;

[0022] The basic features and advanced geometric features of the representative point are used as sub-point cloud data of the representative point.

[0023] Preferably, the spatial sparse labeling of each of the sub-point cloud data according to each preset tree part type to obtain each weak label data and each unlabeled data includes:

[0024] Determining the number of sub-point cloud data;

[0025] Calculating the number of sub-point cloud data in proportion to a preset annotation ratio to obtain the number of annotations;

[0026] Determine the number of tree part types;

[0027] Dividing the number of annotations according to the number of tree part types, and determining the number of annotated sub-point clouds for each tree part type;

[0028] Based on the number of labeled sub-point clouds, corresponding sub-point cloud data to be labeled are selected from each sub-point cloud data for labeling to obtain each weak label data; and each unlabeled sub-point cloud data is used as each unlabeled data.

[0029] Preferably, the optimizing the preliminary classification model by using the weak label data, pseudo label data and transformed data to obtain a trained target classification model comprises:

[0030] Combining the pseudo-label data and the transformed data to form a plurality of consistent data pairs;

[0031] Establish a consistency loss function;

[0032] Inputting a plurality of the consistency data pairs into the preliminary classification model, optimizing the preliminary classification model with the goal of minimizing the consistency loss function, and obtaining a first classification model;

[0033] The first classification model is optimized using the weak label data and the pseudo label data to obtain a trained target classification model.

[0034] Preferably, the optimizing the first classification model by using the weak label data and the pseudo label data to obtain a trained target classification model includes:

[0035] Using the weak label data to establish a supervision loss function, using the pseudo label data to establish a pseudo label loss function, and simultaneously establishing an entropy regularization loss function;

[0036] Combining the supervision loss function, the pseudo label loss function, the entropy regularization loss function and the consistency loss function to obtain a joint loss function;

[0037] The weak label data and pseudo label data are input into the first classification model, and the joint loss function is optimized with the goal of minimizing the joint loss function to obtain a trained target classification model.

[0038] Preferably, the collecting of point cloud data samples of trees includes:

[0039] Using a drone to collect point cloud data of trees, obtaining first drone point cloud data;

[0040] Using a mobile backpack laser scanning system to collect point cloud data of trees, obtaining first backpack point cloud data;

[0041] Performing a coordinate system conversion on the first UAV point cloud data and the first backpack point cloud data to obtain second UAV point cloud data and second backpack point cloud data in the same coordinate system;

[0042] Performing scale normalization processing on the second UAV point cloud data and the second backpack point cloud data to obtain third UAV point cloud data and third backpack point cloud data;

[0043] Simultaneously, the third UAV point cloud data and the third backpack point cloud data are subjected to denoising to obtain fourth UAV point cloud data and fourth backpack point cloud data;

[0044] In the fourth drone point cloud data, point cloud data having a resolution less than a first preset threshold is determined as low-resolution data, and in the fourth backpack point cloud data, point cloud data having a resolution greater than a second preset threshold is determined as high-resolution data;

[0045] Matching the low-resolution data with the high-resolution data to form a matching pair;

[0046] According to the matching pair, the fourth UAV point cloud data and the fourth backpack point cloud data are fused to obtain a point cloud data sample.

[0047] In a second aspect, a tree crown and trunk classification device includes:

[0048] A downsampling module, used for collecting point cloud data samples of trees, and downsampling the point cloud data samples to obtain each sub-point cloud data;

[0049] A labeling module, used for performing spatial sparse labeling on each of the sub-point cloud data according to each preset tree part type to obtain each weak label data;

[0050] A preliminary training module, used for training a preliminary classification model using each of the weak label data;

[0051] A pseudo label data obtaining module, used for processing each sub-point cloud data except the weak label data by using the preliminary classification model to obtain pseudo label data corresponding to each sub-point cloud data;

[0052] A data transformation module, used for performing data transformation processing on each of the pseudo-label data to obtain each transformed data;

[0053] An optimization module, used to optimize the preliminary classification model using the weak label data, pseudo label data and transformed data to obtain a trained target classification model;

[0054] The classification result determination module is used to collect point cloud data of the trees to be classified, process the point cloud data using the target classification model, and obtain the crown width and trunk classification results of the trees to be classified.

[0055] In a third aspect, a tree crown width and trunk classification device includes a memory and a processor;

[0056] The memory is used to store programs;

[0057] The processor is used to execute the program to implement each step of the tree crown width and trunk classification method as described in any one of the first aspects.

[0058] In a fourth aspect, a storage medium stores a computer program, which, when executed by a processor, implements the steps of the tree crown and trunk classification method as described in any one of the first aspects.

[0059] It can be seen from the above technical scheme that the present application obtains each sub-point cloud data by collecting point cloud data samples of trees and downsampling the point cloud data samples; spatially sparsely annotates each sub-point cloud data according to each pre-set tree part type to obtain each weak label data and each unlabeled data; uses each weak label data to train a preliminary classification model; uses the preliminary classification model to process each unlabeled data to obtain pseudo label data corresponding to each unlabeled data; performs data transformation processing on each pseudo label data to obtain each transformed data; uses the weak label data, pseudo label data and transformed data to optimize the preliminary classification model to obtain a trained target classification model; collects point cloud data of trees to be classified, uses the target classification model to process the point cloud data, and obtains the crown width and trunk classification results of the trees to be classified. This application first downsamples the collected point cloud data samples to remove redundant points and improve efficiency, and then performs spatial sparse annotation to obtain weak label data to achieve the purpose of streamlining the data volume, and uses weak label data to train a preliminary classification model. The preliminary classification model can process the sub-point cloud data that is not labeled in the spatial sparse annotation process to obtain various pseudo-label data. Next, the pseudo-label data is transformed to expand the data volume, and the preliminary classification model is optimized using weak label data, pseudo-label data and transformed data, so that the training samples can cover the complex and changeable conditions of trees, improve the adaptability of the model, and achieve better classification results. Then, using the trained target classification model to classify the crown width and trunk of the classified trees can provide significant support for the safety management of equipment or scenes such as power lines and transmission corridors. This fine-grained analysis of tree parts and structures not only helps to timely identify potential tree obstacle risks, but also provides reliable data support for the safe operation of power facilities, so that operation and maintenance personnel can accurately obtain key information such as the growth status, spatial distribution and distance from the lines of trees along power lines and transmission corridors, making inspection and maintenance more intelligent and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0061] Figure 1 An optional flow chart of a tree crown width and trunk classification method provided in an embodiment of the present application;

[0062] Figure 2 A schematic diagram of a pseudo-label data generation process provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of a model optimization process provided in an embodiment of the present application;

[0064] Figure 4 An optional flow chart of another method for classifying tree crown width and trunk provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the structure of a tree crown and trunk classification device provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of the structure of a tree crown and trunk classification device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] In today's society, it is extremely important to ensure the safety of equipment or areas such as power lines and transmission corridors. For example, in the transmission corridor scenario, overgrown or weirdly growing trees are often important causes of power safety accidents. In terms of transmission corridor protection, the precise distinction between tree crowns (leaf and branch areas) and trunks provides refined support for identifying dangerous trees in transmission corridors, greatly improving the scientificity and efficiency of transmission line safety management. Through the precise classification of tree parts, dangerous trees that may threaten the safety of the line can be accurately identified, and the specific structural information of each tree along the transmission corridor can be obtained, such as the height, thickness of the trunk, and the size and shape of the crown. The specific structural information can ensure accurate assessment of whether the tree will pose a threat to the transmission line, and facilitate rapid defensive measures, especially during extreme weather or natural disasters. These data can help quickly find trees that may fall or be dangerous, ensure the smooth and safe transmission lines, and provide a reliable basis for accurately assessing the potential tree barrier risks in the transmission corridor.

[0069] However, the current tree crown and trunk classification methods do not take into account the complex and changeable conditions of trees, have low adaptability, and cannot meet the classification needs of the crown and trunk of diverse trees.

[0070] In order to solve the defects of the above-mentioned prior art, an embodiment of the present invention provides a tree crown width and trunk classification method, which can be applied to various computer terminals or intelligent terminals, and its execution subject can be a processor or server of the computer terminal or intelligent terminal. The method flow chart of the method is as follows Figure 1 As shown, specifically including:

[0071] S1: Collect point cloud data samples of trees, and downsample the point cloud data samples to obtain sub-point cloud data.

[0072] In order to train the preliminary classification model more fully, point cloud data samples of countless trees can be collected in different ways, thereby expanding the amount of sample data and obtaining comprehensive point cloud data of trees from different perspectives and resolutions.

[0073] After collection, the point cloud data is downsampled. Considering that a large sample size will affect the training efficiency of the model, after collecting data from multiple aspects, multiple angles, multiple resolutions, and multiple perspectives in the first step, it is necessary to appropriately reduce the amount of data. Downsampling can ensure the accuracy and stability of the overall features of the point cloud data while reducing the amount of data, thereby obtaining each sub-point cloud data. This can also reduce noise and highlight obvious and important data features.

[0074] S2: performing spatial sparse labeling on each of the sub-point cloud data according to each pre-set tree part type to obtain each weak label data and each unlabeled data.

[0075] The crown width of a tree mainly includes the branches and leaves of the canopy layer and the range of its growth space, such as branches, leaves, flowers, fruits, etc. The classification of tree parts includes: roots, trunks, branches, leaves, flowers, fruits, etc.

[0076] In the process of spatial sparse labeling, the present application regards the labeled data as weakly labeled data and the unlabeled data as unlabeled data.

[0077] Existing technologies often rely heavily on large-scale labeled data, but this is difficult for point cloud data, especially point cloud data of trees. Trees have complex structures and a large amount of point cloud data. If all of them are labeled, it will take a lot of time and professional knowledge, waste manpower and material resources, and the labeling cost is very high. Therefore, if this problem cannot be solved, it will affect the accuracy and efficiency of tree crown and trunk classification, and affect the safety and normal operation of equipment or areas such as power transmission lines. Therefore, this application adopts a spatial sparse labeling method. Spatial sparse labeling is a commonly used technical means in the field of spatial data processing and analysis. It is a method for labeling a part of representative or critical spatial locations or objects in point cloud data. In this way, the purpose of summarizing and expressing the main features of point cloud data with less labeled information can be achieved, thereby reducing the labeling cost and solving the problem of difficulty in labeling point cloud data and difficulty in fully acquiring data. This not only greatly reduces the dependence on labeled data, but also expands the application of subsequent trained target classification models in data-scarce environments or scenarios with high labeling costs, providing a practical solution for the accurate classification of tree point cloud data.

[0078] S3: Using each of the weakly labeled data to train a preliminary classification model.

[0079] You can choose KPConv (Kernel Point Convolution) network to build a preliminary classification model. KPConv network has strong adaptability and local feature extraction capabilities for point cloud data. It can identify preliminary features of different categories under the premise of sparse spatial annotation, and can directly perform convolution operations on irregular point cloud data to capture the three-dimensional structural features of trees, helping the preliminary classification model to identify subtle structural differences in trees. Through weak supervision training, useful geometric and spatial features can be extracted from weakly labeled data of non-large data volumes, laying the foundation for the subsequent output of pseudo-label data.

[0080] S4: Using the preliminary classification model to process each of the unlabeled data, to obtain pseudo-label data corresponding to each of the unlabeled data.

[0081] The preliminary classification model that has completed preliminary training can process unlabeled data to obtain pseudo-label data. The output of pseudo-label data depends on the features learned by the preliminary classification model from weakly labeled data during the training process. At the same time, pseudo-label data can also expand the training samples and further improve the classification effect of the preliminary classification model.

[0082] The entropy regularization method can be used to optimize the quality of pseudo-labeled data, thereby improving the classification standard and stability of the model under weak supervision. In the optimization process, the entropy value of unlabeled data can be minimized to encourage the model to be more confident in the generation of pseudo-labeled data, thereby reducing the generation of low-confidence and noise labels and ensuring that the preliminary classification model obtains clearer classification boundaries on unlabeled data. Specifically, entropy is a measure used to describe the uncertainty of the model's prediction results. Assuming that the predicted category probability distribution of a data point is ,in represents the probability that the point cloud data belongs to the i-th category, and C is the total number of categories. Then the entropy of this data point can be expressed as:

[0083]

[0084] When the category prediction probability is close to uniform distribution (that is, the prediction probability of each category is close to 1 / C), the entropy value is high, which indicates that the preliminary classification model has a large uncertainty in the classification of the data point; and when the prediction probability of the preliminary classification model for a certain category is close to 1 (that is, close to a certain classification), the entropy value is low, which indicates that the model prediction is more certain. The purpose of the entropy regularization method used in this application is to reduce the predicted entropy value of unlabeled data, so that the preliminary classification model is more confident when outputting pseudo-label data and reducing the appearance of fuzzy labels. Specifically, an entropy regularization term can be introduced into the loss function of the preliminary classification model. By setting a standard entropy value to constrain the confidence of the preliminary classification model in prediction, the pseudo-label data will also be iterated synchronously during the iterative optimization process to adjust the standard entropy value, thereby further optimizing the quality of the pseudo-label data and finally obtaining high-quality pseudo-label data.

[0085] In addition to the entropy regularization method, the overlapping area loss method can also be used to achieve this. The core idea of ​​the overlapping area loss method is to use the overlapping areas between adjacent sub-point clouds to increase the classification network's perception of subtle semantic differences, thereby generating higher quality pseudo labels.

[0086] S5: Perform data transformation processing on each of the pseudo-label data to obtain each transformed data.

[0087] The purpose of data transformation on pseudo-label data is to expand the scope and angle of data volume on the one hand, and to make the model training more comprehensive and maintain consistency on the other hand.

[0088] The data transformation process may include rotation, scaling, translation, etc.

[0089] S6: Optimize the preliminary classification model using the weak label data, pseudo label data and transformed data to obtain a trained target classification model.

[0090] The optimization process can include multiple methods, such as optimizing using weakly labeled data / pseudo-labeled data / transformed data separately, optimizing by combining two of them, or inputting the three types of data into the preliminary classification model for optimization at the same time.

[0091] Data with large amounts and multiple data angles can cover point cloud data of trees of different shapes and types, making the final trained target classification model more adaptable and applicable to a wider range of applications.

[0092] S7: collecting point cloud data of the trees to be classified, and processing the point cloud data using the target classification model to obtain crown width and trunk classification results of the trees to be classified.

[0093] Finally, the point cloud data of the trees to be classified can be input into the trained target classification model, which will process them to obtain accurate crown and trunk classification results. Since crown is a general term that includes various parts, the target classification model contains a module for counting and merging the various parts belonging to the crown to distinguish between the crown and the trunk.

[0094] This application first downsamples the collected point cloud data samples to remove redundant points and improve efficiency, and then performs spatial sparse annotation to obtain weak label data to achieve the purpose of streamlining the data volume, and uses weak label data to train a preliminary classification model. The preliminary classification model can process the sub-point cloud data that is not labeled in the spatial sparse annotation process to obtain various pseudo-label data. Next, the pseudo-label data is transformed to expand the data volume, and the preliminary classification model is optimized using weak label data, pseudo-label data and transformed data, so that the training samples can cover the complex and changeable conditions of trees, improve the adaptability of the model, and achieve better classification results. Then, using the trained target classification model to classify the crown width and trunk of the classified trees can provide significant support for the safety management of equipment or scenes such as power lines and transmission corridors. This fine-grained analysis of tree parts and structures not only helps to timely identify potential tree obstacle risks, but also provides reliable data support for the safe operation of power facilities, so that operation and maintenance personnel can accurately obtain key information such as the growth status, spatial distribution and distance from the lines of trees along power lines and transmission corridors, making inspection and maintenance more intelligent and efficient.

[0095] The process of collecting point cloud data samples of trees can be done as follows:

[0096] Using a drone to collect point cloud data of trees, obtaining first drone point cloud data;

[0097] Using a mobile backpack laser scanning system to collect point cloud data of trees, obtaining first backpack point cloud data;

[0098] Performing a coordinate system conversion on the first UAV point cloud data and the first backpack point cloud data to obtain second UAV point cloud data and second backpack point cloud data in the same coordinate system;

[0099] Performing scale normalization processing on the second UAV point cloud data and the second backpack point cloud data to obtain third UAV point cloud data and third backpack point cloud data;

[0100] Simultaneously, the third UAV point cloud data and the third backpack point cloud data are subjected to denoising to obtain fourth UAV point cloud data and fourth backpack point cloud data;

[0101] In the fourth drone point cloud data, point cloud data having a resolution less than a first preset threshold is determined as low-resolution data, and in the fourth backpack point cloud data, point cloud data having a resolution greater than a second preset threshold is determined as high-resolution data;

[0102] Matching the low-resolution data with the high-resolution data to form a matching pair;

[0103] According to the matching pair, the fourth UAV point cloud data and the fourth backpack point cloud data are fused to obtain a point cloud data sample.

[0104] Specifically, the present application adopts two different platforms, namely, UAV and backpack, which can obtain comprehensive tree point cloud data from different perspectives and resolutions. Selecting these two platforms to collect point cloud data can ensure the adequacy and completeness of the point cloud data. For example, the point cloud data collected by drones has a relatively wide coverage range. Although it can be adapted to large-scale tree monitoring, its resolution is relatively low; while the backpack point cloud data has a relatively high resolution and can provide more detailed tree parts and structural information, but its coverage is relatively small. Therefore, considering the advantages and disadvantages of these two methods, point cloud data of the two methods are collected at the same time, and the point cloud data collected by the two methods, namely the first UAV point cloud data and the first backpack point cloud data, are integrated. At the same time, in order to ensure the precision and unification of the point cloud data, data processing is performed during the integration process. First of all, considering the differences in coordinate systems between different platforms, the first UAV point cloud data and the first backpack point cloud data are uniformly converted to the same coordinate system, such as the geographic coordinate system or the local plane coordinate system, to ensure the first The UAV point cloud data and the first backpack point cloud data can be compared and fused under the same spatial benchmark. Secondly, the scale is normalized to make the two consistent in spatial range and density, so as to ensure that there will be no problems due to differences in resolution in the subsequent matching and fusion process; in addition, considering that the point cloud data is often disturbed by the external environment during the collection process, there will be noise and abnormal points, especially in complex areas such as leaves and branches, so denoising operations can be performed to remove these noise and abnormal points to make the point cloud data clearer and more accurate; finally, the resolution is adjusted to match the fourth UAV point cloud data with a lower resolution with the fourth backpack point cloud data with a higher resolution, so that the resolution of the fourth UAV point cloud data and the fourth backpack point cloud data are similar when they are fused, ensuring the accuracy after splicing.

[0105] In the method provided in the embodiment of the present invention, the process of downsampling the point cloud data samples to obtain each sub-point cloud data is specifically described as follows:

[0106] Obtaining the density of the point cloud data sample;

[0107] Setting a three-dimensional voxel grid according to the density, and dividing the point cloud data samples according to the three-dimensional voxel grid to obtain individual voxels;

[0108] Selecting a representative point in each of the voxels;

[0109] The data corresponding to each representative point is used as the sub-point cloud data.

[0110] Specifically, the process can use a voxelization method to significantly reduce the data size while ensuring the overall structure and main structural features of the tree crown and trunk, as well as the consistency of resolution, and will not generate too many voxels. A voxel is a cubic unit, and the voxel size can be set to 0.2 meters.

[0111] Among them, the spatial range of the point cloud data sample is divided into three-dimensional voxel grids of equal size according to the density. In each voxel, there are usually multiple points. In order to simplify the data, a point is selected as the representative of the voxel, that is, the representative point, which means replacing all the points in the voxel. When selecting, points that can highlight the features of the point cloud are given priority, such as points located on the central axis of the trunk and the tip of the crown. In this way, the purpose of segmenting the point cloud data sample can be achieved, and the entire point cloud data sample can be downsampled into sub-point cloud data with similar resolution, and the spatial structure of the point cloud data sample is retained.

[0112] Furthermore, the present application uses basic point features and advanced geometric features as data of representative points, as sub-point cloud data, specifically:

[0113] For each representative point, extract the three-dimensional spatial coordinates and reflection intensity of the representative point as basic features;

[0114] Extract the normal vector, curvature and point density of the representative point as high-level geometric features;

[0115] The basic features and advanced geometric features of the representative point are used as sub-point cloud data of the representative point.

[0116] Among them, the basic point features can be read directly from the representative points. The basic point features can provide preliminary spatial position and tree surface information for the preliminary classification model, thereby assisting in identifying the basic attributes of different parts. The three-dimensional spatial coordinates (X, Y, Z) of each representative point provide the position of the representative point in space. The three-dimensional spatial coordinates are the core information and can be used to construct the overall shape and spatial structure of the tree. The reflection intensity represents the intensity of the reflected echo after the laser beam hits the surface of the object, which is usually related to the material and surface roughness of the object. In the point cloud data of the tree, the trunk and leaves may have different reflection intensities, so the reflection intensity, a basic feature, can be used to distinguish different tree surfaces, which is a crucial feature.

[0117] Advanced geometric features such as normal vectors, curvature, and point density can be used to describe more complex spatial structures and morphological information in point cloud data. These features can be used to analyze and calculate the local neighborhood of point cloud data to show the geometric morphology of the tree surface, which can help the preliminary classification model to more accurately distinguish different types of tree parts and structures.

[0118] Specifically, the normal vector is used to describe the orientation of the point cloud surface at the point position, and is obtained by calculating the local neighborhood plane of the representative point. For each representative point, define a neighborhood with a smaller radius (for example, take other representative points within a certain range around the representative point), use the principal component analysis (PCA) method to analyze the distribution of representative points in the neighborhood, and find the main direction of the neighborhood. The normal vector refers to the normal direction of the local plane. The eigenvector corresponding to the minimum eigenvalue calculated by PCA is the normal vector; the curvature is used to describe the degree of curvature of the point in its local neighborhood, that is, the complexity of the surface morphology. A neighborhood with a smaller radius can also be defined, and the ratio of the minimum eigenvalue to the sum of the total eigenvalues ​​is used as the curvature of the representative point. The larger the curvature value, the higher the degree of curvature of the surface. In the point cloud data of trees, the curvature of the crown area is generally higher, while the trunk part is relatively smooth. Therefore, determining the curvature feature can assist in distinguishing different parts of the tree; point density is used to indicate the density of points in a certain area. Usually, the number of points in a fixed radius neighborhood is calculated. Point density can reflect the distribution characteristics when the data is collected. For each representative point, a radius neighborhood is set and the number of neighborhood points within the radius is counted. The more representative points there are in the neighborhood, the higher the point density at the representative point. Point density features can help distinguish different structural areas. The point density in the trunk area is usually higher, while the point density in the crown or sparse leaf area is relatively low.

[0119] The following is an explanation of the process of performing spatial sparse labeling on each of the sub-point cloud data according to each pre-set tree part type in the present application to obtain each weak label data and each unlabeled data.

[0120] Determining the number of sub-point cloud data;

[0121] Calculating the number of sub-point cloud data in proportion to a preset annotation ratio to obtain the number of annotations;

[0122] Determine the number of tree part types;

[0123] Dividing the number of annotations according to the number of tree part types, and determining the number of annotated sub-point clouds for each tree part type;

[0124] Based on the number of labeled sub-point clouds, corresponding sub-point cloud data to be labeled are selected from each sub-point cloud data for labeling to obtain each weak label data; and each unlabeled sub-point cloud data is used as each unlabeled data.

[0125] Specific as Figure 2As shown in the figure, a labeling ratio of 1‰ can be adopted, that is, one thousandth of the sub-point cloud data is selected for labeling in all the sub-point cloud data to generate sparsely distributed weak label data. Since the amount of weak label data is less than that of all sub-point cloud data, it is necessary to carefully design the labeling strategy so that the distribution of weak label data in space is more reasonable, and it can cover multiple parts such as trunks and crowns, and cover various structural characteristics of trees, so as to be more sufficient for the subsequent training of the preliminary classification model. Finally, various weak label data are formed. These weak label data correspond to the preset types of tree parts and have rich spatial characteristics, which are helpful for the model to preliminarily distinguish various structures under weak supervision conditions and treat the unlabeled data as unlabeled data.

[0126] Furthermore, in order to make the preliminary classification model predict the consistency under different forms of point cloud data, the pseudo-label data initially output by the model is converted to obtain transformed data, and then the preliminary classification model is optimized using weak label data, pseudo-label data and transformed data to obtain a trained target classification model, which can enhance the stability and adaptability of the preliminary classification model. The specific process includes:

[0127] Combining the pseudo-label data and the transformed data to form a plurality of consistent data pairs;

[0128] Establish a consistency loss function;

[0129] Inputting a plurality of the consistency data pairs into the preliminary classification model, optimizing the preliminary classification model with the goal of minimizing the consistency loss function, and obtaining a first classification model;

[0130] The first classification model is optimized using the weak label data and the pseudo label data to obtain a trained target classification model.

[0131] Specific as Figure 3As shown, firstly, the pseudo-label data and the transformed data are optimized, and the pseudo-label data and the transformed data are combined to form multiple consistent data pairs. The consistent constraint method is used to act on the consistent data pairs, and a consistent input function is established, so that the preliminary classification model outputs the same or similar classification results when predicting the pseudo-label data and the transformed data, so that the model can learn more stable and universal feature expressions under various input conditions, and maintain consistent classification performance even under different acquisition conditions or changes in viewing angles. This method can significantly enhance the robustness and generalization ability of the preliminary classification model, and can more accurately and reliably process point cloud data in different scenes. Among them, the category probability distribution and category label of the pseudo-label data will be obtained. The consistency loss function can measure the difference in the prediction results between the pseudo-label data and the transformed data. The mean square error can be used as the constraint principle of the consistency loss function. Specifically, the preliminary classification model keeps the prediction consistent by minimizing the square error between the prediction results of the same data in different forms. The consistency loss function is shown as follows:

[0132]

[0133] Where N represents the number of data points; is the prediction result of the i-th data point in the pseudo-label data; is the prediction result of the i-th data point in the transformed data.

[0134] In order to further optimize the accuracy of the preliminary classification model, after the optimization is completed using the pseudo-label data and the transformed data, the first classification model can also be optimized using the weak label data and the pseudo-label data to obtain a trained target classification model. The specific steps include:

[0135] Using the weak label data to establish a supervision loss function, using the pseudo label data to establish a pseudo label loss function, and simultaneously establishing an entropy regularization loss function;

[0136] Combining the supervision loss function, the pseudo label loss function, the entropy regularization loss function and the consistency loss function to obtain a joint loss function;

[0137] The weak label data and pseudo label data are input into the first classification model, and the joint loss function is optimized with the goal of minimizing the joint loss function to obtain a trained target classification model.

[0138] Specifically, in this step, weak label data and pseudo label data are input into the classification model for joint training, and the classification performance of the model is gradually optimized through multiple rounds of iterations. The supervised loss function is established using the weak label data. This is to use the real labels to guide the preliminary classification names to learn basic classification features, and ensure that the model can accurately distinguish between tree trunks and crowns. Secondly, the pseudo label data is used to establish a pseudo label loss function. The high-confidence pseudo labels can indirectly supervise the unlabeled data, expand the effective range of the training samples, and enable the preliminary classification model to learn sufficient classification features on a variety of training samples, further improving the model performance. At the same time, the above-mentioned embodiments are used to improve the classification performance of the preliminary classification model. The entropy regularization method is used to optimize the quality of pseudo-label data and construct an entropy regularized loss function, which can reduce the uncertainty of the preliminary classification model on unlabeled data, reduce the noise in the pseudo-label data, and ensure that the prediction results of the preliminary classification model are more stable. With the iterative training, the pseudo-label data is constantly updated and optimized. After each iteration, high-confidence pseudo-labels are screened out, thereby expanding the coverage of pseudo-label data. At the same time, low-confidence labels are removed or adjusted to gradually improve the accuracy of pseudo-labels. This dynamically updated pseudo-label mechanism provides the model with increasingly rich and accurate supervision signals, thereby improving its learning effect on unlabeled data. Then introduce the consistency loss function, construct a joint loss function, and optimize the joint loss function with the goal of minimizing the joint loss function. After each round of training, the classification performance of the preliminary classification model will be evaluated and adjusted. A separate test set can be introduced for verification to ensure the accuracy, stability and generalization ability of the preliminary classification model in the classification of tree crowns and trunks. Through such multiple rounds of optimization and adjustment, a trained target classification model is finally obtained, which achieves efficient and accurate classification of tree crowns and trunks under sparse annotation conditions. The overall flow chart of a tree crown and trunk classification method provided in this application can be shown as follows: Figure 4 shown.

[0139] and Figure 1 Corresponding to the method described above, the embodiment of the present invention also provides a tree crown width and trunk classification device for classifying Figure 1 In the specific implementation of the method, the tree crown width and trunk classification device provided by the embodiment of the present invention can be used in a computer terminal or various mobile devices, combined with Figure 5 , introduce the tree crown width and trunk classification device, such as Figure 5 As shown, the device may include:

[0140] A downsampling module 10 is used to collect point cloud data samples of trees and downsample the point cloud data samples to obtain each sub-point cloud data;

[0141] A labeling module 20 is used to perform spatial sparse labeling on each of the sub-point cloud data according to each preset tree part type to obtain each weak label data and each unlabeled data;

[0142] A preliminary training module 30, used for training a preliminary classification model using each of the weak label data;

[0143] A pseudo label data obtaining module 40 is used to process each of the unlabeled data using the preliminary classification model to obtain pseudo label data corresponding to each of the unlabeled data;

[0144] A data transformation module 50 is used to perform data transformation processing on each of the pseudo-label data to obtain each transformed data;

[0145] An optimization module 60, configured to optimize the preliminary classification model using the weak label data, pseudo label data and transformed data to obtain a trained target classification model;

[0146] The classification result determination module 70 is used to collect point cloud data of the trees to be classified, process the point cloud data using the target classification model, and obtain the crown width and trunk classification results of the trees to be classified.

[0147] This application first downsamples the collected point cloud data samples to remove redundant points and improve efficiency, and then performs spatial sparse annotation to obtain weak label data to achieve the purpose of streamlining the data volume, and uses weak label data to train a preliminary classification model. The preliminary classification model can process the sub-point cloud data that is not labeled in the spatial sparse annotation process to obtain various pseudo-label data. Next, the pseudo-label data is transformed to expand the data volume, and the preliminary classification model is optimized using weak label data, pseudo-label data and transformed data, so that the training samples can cover the complex and changeable conditions of trees, improve the adaptability of the model, and achieve better classification results. Then, using the trained target classification model to classify the crown width and trunk of the classified trees can provide significant support for the safety management of equipment or scenes such as power lines and transmission corridors. This fine-grained analysis of tree parts and structures not only helps to timely identify potential tree obstacle risks, but also provides reliable data support for the safe operation of power facilities, so that operation and maintenance personnel can accurately obtain key information such as the growth status, spatial distribution and distance from the lines of trees along power lines and transmission corridors, making inspection and maintenance more intelligent and efficient.

[0148] Furthermore, the embodiment of the present application provides a tree crown width and trunk classification device. Optionally, Figure 6 The hardware structure diagram of the tree crown and trunk classification equipment is shown in FIG. Figure 6The hardware structure of the tree crown and trunk classification device may include: at least one processor 01, at least one communication interface 02, at least one memory 03 and at least one communication bus 04.

[0149] In the embodiment of the present application, the number of the processor 01 , the communication interface 02 , the memory 03 , and the communication bus 04 is at least one, and the processor 01 , the communication interface 02 , and the memory 03 communicate with each other through the communication bus 04 .

[0150] The processor 01 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0151] The memory 03 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory.

[0152] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to execute the following tree crown width and trunk classification method, including:

[0153] Collecting point cloud data samples of trees, and downsampling the point cloud data samples to obtain sub-point cloud data;

[0154] Performing spatial sparse labeling on each of the sub-point cloud data according to each preset tree part type to obtain each weak label data and each unlabeled data;

[0155] Using each of the weakly labeled data to train a preliminary classification model;

[0156] Using the preliminary classification model to process each of the unlabeled data, to obtain pseudo-label data corresponding to each of the unlabeled data;

[0157] Performing data transformation processing on each of the pseudo-label data to obtain each transformed data;

[0158] Optimizing the preliminary classification model using the weakly labeled data, pseudo-labeled data, and transformed data to obtain a trained target classification model;

[0159] The point cloud data of the trees to be classified are collected, and the point cloud data are processed using the target classification model to obtain the crown width and trunk classification results of the trees to be classified.

[0160] Optionally, the detailed functions and extended functions of the program may refer to the description of the tree crown width and trunk classification method in the method embodiment.

[0161] The embodiment of the present application further provides a storage medium, which can store a program suitable for execution by a processor, and when the program is executed, controls the device where the storage medium is located to execute the following tree crown width and trunk classification method, including:

[0162] Collecting point cloud data samples of trees, and downsampling the point cloud data samples to obtain sub-point cloud data;

[0163] Performing spatial sparse labeling on each of the sub-point cloud data according to each preset tree part type to obtain each weak label data and each unlabeled data;

[0164] Using each of the weakly labeled data to train a preliminary classification model;

[0165] Using the preliminary classification model to process each of the unlabeled data, to obtain pseudo-label data corresponding to each of the unlabeled data;

[0166] Performing data transformation processing on each of the pseudo-label data to obtain each transformed data;

[0167] Optimizing the preliminary classification model using the weakly labeled data, pseudo-labeled data, and transformed data to obtain a trained target classification model;

[0168] The point cloud data of the trees to be classified are collected, and the point cloud data are processed using the target classification model to obtain the crown width and trunk classification results of the trees to be classified.

[0169] Specifically, the storage medium may be a computer-readable storage medium, and the computer-readable storage medium may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk or a ROM.

[0170] Optionally, the detailed functions and extended functions of the program may refer to the description of the tree crown width and trunk classification method in the method embodiment.

[0171] In addition, the functional modules in the various embodiments of the present disclosure can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a live broadcast device, or a network device, etc.) to perform all or part of the steps of the methods of the various embodiments of the present disclosure.

[0172] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0173] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0174] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for classifying tree crowns and trunks, characterized in that: include: Collecting point cloud data samples of trees, and downsampling the point cloud data samples to obtain sub-point cloud data; Performing spatial sparse labeling on each of the sub-point cloud data according to each preset tree part type to obtain each weak label data and each unlabeled data; Using each of the weakly labeled data to train a preliminary classification model; Using the preliminary classification model to process each of the unlabeled data, to obtain pseudo-label data corresponding to each of the unlabeled data; Performing data transformation processing on each of the pseudo-label data to obtain each transformed data; Optimizing the preliminary classification model using the weakly labeled data, pseudo-labeled data, and transformed data to obtain a trained target classification model; The point cloud data of the trees to be classified are collected, and the point cloud data are processed using the target classification model to obtain the crown width and trunk classification results of the trees to be classified.

2. The method according to claim 1, characterized in that The step of downsampling the point cloud data samples to obtain each sub-point cloud data comprises: Obtaining the density of the point cloud data sample; Setting a three-dimensional voxel grid according to the density, and dividing the point cloud data samples according to the three-dimensional voxel grid to obtain individual voxels; Selecting a representative point in each of the voxels; The data corresponding to each representative point is used as the sub-point cloud data.

3. The method according to claim 2, characterized in that The step of taking the data corresponding to each representative point as the sub-point cloud data includes: For each representative point, extract the three-dimensional spatial coordinates and reflection intensity of the representative point as basic features; Extract the normal vector, curvature and point density of the representative point as high-level geometric features; The basic features and advanced geometric features of the representative point are used as sub-point cloud data of the representative point.

4. The method according to claim 1, characterized in that: The spatial sparse labeling of each sub-point cloud data according to each preset tree part type is performed to obtain each weak label data and each unlabeled data, including: Determining the number of sub-point cloud data; Calculating the number of sub-point cloud data in proportion to a preset annotation ratio to obtain the number of annotations; Determine the number of tree part types; Dividing the number of annotations according to the number of tree part types, and determining the number of annotated sub-point clouds for each tree part type; Based on the number of labeled sub-point clouds, corresponding sub-point cloud data to be labeled are selected from each sub-point cloud data for labeling to obtain each weak label data; and each unlabeled sub-point cloud data is used as each unlabeled data.

5. The method according to claim 1, characterized in that The step of optimizing the preliminary classification model by using the weak label data, pseudo label data and transformed data to obtain a trained target classification model includes: Combining the pseudo-label data and the transformed data to form a plurality of consistent data pairs; Establish a consistency loss function; Inputting a plurality of the consistency data pairs into the preliminary classification model, optimizing the preliminary classification model with the goal of minimizing the consistency loss function, and obtaining a first classification model; The first classification model is optimized using the weak label data and the pseudo label data to obtain a trained target classification model.

6. The method according to claim 5, characterized in that The step of optimizing the first classification model by using the weakly labeled data and the pseudo-labeled data to obtain a trained target classification model includes: Using the weak label data to establish a supervision loss function, using the pseudo label data to establish a pseudo label loss function, and simultaneously establishing an entropy regularization loss function; Combining the supervision loss function, the pseudo label loss function, the entropy regularization loss function and the consistency loss function to obtain a joint loss function; The weak label data and pseudo label data are input into the first classification model, and the joint loss function is optimized with the goal of minimizing the joint loss function to obtain a trained target classification model.

7. The method according to any one of claims 1 to 6, characterized in that: The collecting of tree point cloud data samples includes: Using a drone to collect point cloud data of trees, obtaining first drone point cloud data; Using a mobile backpack laser scanning system to collect point cloud data of trees, obtaining first backpack point cloud data; Performing a coordinate system conversion on the first UAV point cloud data and the first backpack point cloud data to obtain second UAV point cloud data and second backpack point cloud data in the same coordinate system; Performing scale normalization processing on the second UAV point cloud data and the second backpack point cloud data to obtain third UAV point cloud data and third backpack point cloud data; Simultaneously, the third UAV point cloud data and the third backpack point cloud data are subjected to denoising to obtain fourth UAV point cloud data and fourth backpack point cloud data; In the fourth drone point cloud data, point cloud data having a resolution less than a first preset threshold is determined as low-resolution data, and in the fourth backpack point cloud data, point cloud data having a resolution greater than a second preset threshold is determined as high-resolution data; Matching the low-resolution data with the high-resolution data to form a matching pair; According to the matching pair, the fourth UAV point cloud data and the fourth backpack point cloud data are fused to obtain a point cloud data sample.

8. A tree crown and trunk classification device, characterized in that: include: A downsampling module, used for collecting point cloud data samples of trees, and downsampling the point cloud data samples to obtain each sub-point cloud data; A labeling module, used for performing spatial sparse labeling on each of the sub-point cloud data according to each preset tree part type, to obtain each weak label data and each unlabeled data; A preliminary training module, used for training a preliminary classification model using each of the weak label data; A pseudo-label data obtaining module, used to process each of the unlabeled data using the preliminary classification model to obtain pseudo-label data corresponding to each of the unlabeled data; A data transformation module, used for performing data transformation processing on each of the pseudo-label data to obtain each transformed data; An optimization module, used to optimize the preliminary classification model using the weak label data, pseudo label data and transformed data to obtain a trained target classification model; The classification result determination module is used to collect point cloud data of the trees to be classified, process the point cloud data using the target classification model, and obtain the crown width and trunk classification results of the trees to be classified.

9. A tree crown and trunk classification device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the tree crown and trunk classification method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the tree crown and trunk classification method as described in any one of claims 1 to 7 is implemented.

Images