Two-stage bridge point cloud segmentation method fusing deep learning and heuristic method

Through the two-stage bridge point cloud segmentation method, combined with deep learning and heuristic methods, the problem of large-scale classification errors in bridge point cloud segmentation is solved, achieving higher segmentation accuracy and controllability, and is suitable for 3D reconstruction work .

CN118967700APending Publication Date: 2024-11-15DALIAN MARITIME UNIVERSITY
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410983444.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing bridge point cloud segmentation methods can easily lead to large-scale point cloud classification errors, and deep learning methods are difficult to be directly used in 3D reconstruction work when data is insufficient and accuracy is limited.

Method used

Using a two-stage bridge point cloud segmentation method that integrates deep learning and heuristic methods, firstly, a bridge point cloud virtual data set is established through the ShapeNet data structure, deep learning network model is trained, and the real point cloud data set is roughly segmented, and then the improvement is used. The heuristic method performs a second segmentation to obtain the final bridge point cloud model.

Benefits of technology

It improves the controllability of segmentation accuracy, reduces the requirements for deep learning results, simplifies the steps of heuristic methods, avoids large-scale classification errors of point clouds, and ensures accurate segmentation of bridge point clouds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967700A_ABST
    Figure CN118967700A_ABST
Patent Text Reader

Abstract

The invention discloses a two-stage bridge point cloud segmentation method fusing deep learning and a heuristic method. The method comprises the following steps: S1, establishing a bridge point cloud virtual data set based on a ShapeNet data structure; s2, constructing a deep learning network model, and training the deep learning network model based on the bridge point cloud virtual data set to obtain a trained deep learning network model; s3, acquiring a real point cloud data set, and performing first rough segmentation on the real point cloud data set based on the trained deep learning network model to obtain a first segmented point cloud model; and S4, performing second segmentation on the point cloud model after the first segmentation by using an improved heuristic method to obtain a final bridge point cloud model. According to the method, the bridge point cloud virtual data set, the deep learning method segmentation and the heuristic method segmentation are complementary in optimization and deficiency, mutual simplification can be realized on the premise of improving the segmentation precision, accurate segmentation of the bridge point cloud is ensured, and the condition of large-scale classification errors of the point cloud is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of bridge point cloud segmentation, and in particular to a two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods. Background Art

[0002] Existing point cloud segmentation methods can generally be divided into three methods: top-down, bottom-up, and deep learning. Among them, the bottom-up method has great limitations in the process of bridge point cloud segmentation, while the top-down method is usually a heuristic method, that is, the components are segmented by summarizing and step-by-step utilizing information such as the bridge's geometric structure and topological constraints. This need for prior knowledge means that different bridges require different methods, resulting in a large amount of iteration costs in practice.

[0003] Different types of bridges can be segmented using the same deep learning method, but in the field of bridges, the lack of point cloud data sets is more serious. Unlike image data sets, point cloud data are more difficult to collect and the environment is more severe. In practice, point cloud data sets are usually produced by first building a BIM model and then generating a point cloud data set. There are two problems in these works: 1) Batch modeling means sacrificing the geometric structure of the bridge, and fine modeling means that it is difficult to produce data sets in batches. The lack of data sets and insufficient data sets will affect the segmentation results, so the segmentation accuracy of deep learning is limited; 2) The segmentation effect obtained by using pure synthetic data sets as training data is ultimately limited and lacks robustness, which can easily cause large-scale classification errors in point clouds and is not easy to be directly used for 3D reconstruction. Summary of the invention

[0004] The present invention provides a two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods, so as to overcome the technical problem that the existing point cloud segmentation method easily causes large-scale classification errors of point clouds and is not easy to be directly used for 3D reconstruction work.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] A two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods, the specific steps include:

[0007] S1: Establish a bridge point cloud virtual dataset based on ShapeNet data structure;

[0008] S2: constructing a deep learning network model, and training the deep learning network model based on the bridge point cloud virtual data set to obtain a trained deep learning network model;

[0009] S3: Acquire a real point cloud data set, and perform a first rough segmentation on the real point cloud data set based on the trained deep learning network model to obtain a point cloud model after the first segmentation;

[0010] S4: The improved heuristic method is used to perform a second segmentation on the point cloud model after the first segmentation to obtain the final bridge point cloud model.

[0011] Furthermore, in S1, the specific steps of establishing a bridge point cloud virtual dataset based on the ShapeNet structure include:

[0012] S11: Obtain bridge component data, and establish a bridge entity model based on the bridge component data:

[0013] S12: performing point cloud processing on the bridge entity model by using point cloud processing software to obtain a point cloud model, and assigning labels to the point cloud model according to the type of bridge components to obtain a labeled point cloud model;

[0014] S13: Adjust the labeled point cloud model to finally obtain a bridge point cloud virtual dataset.

[0015] Furthermore, in S11, the process of acquiring bridge component data and establishing a bridge entity model based on the bridge component data is as follows:

[0016] S111: Acquire N different types of bridge component data, each bridge component data includes size data and relative position data of the bridge component;

[0017] S112: Convert the bridge component data into data recognizable by Revit through Dynamo, create an adaptive family through Revit, then input the data recognizable by Revit after conversion by Dynamo into the adaptive family, and obtain a bridge solid model through Revit batch production, and export the bridge solid model in .fbx format to obtain a .fbx file.

[0018] Further, in S12, point cloud processing is performed on the bridge entity model by point cloud processing software to obtain a point cloud model, and labels are assigned to the point cloud model according to the type of bridge components. The process of obtaining the labeled point cloud model is as follows:

[0019] S121: reading the .fbx file, adjusting the size of the bridge entity model in proportion, and then generating a point cloud model and a normal line based on the bridge entity model according to a set density ρ;

[0020] S122: classifying the point cloud model generated in S121 according to the type of bridge components, and merging point clouds of the same type of components to obtain N groups of point clouds;

[0021] S123: assigning corresponding labels to the N groups of point clouds according to the types of bridge components to obtain N groups of point clouds with labels;

[0022] S124: Merge the N groups of labeled point clouds to obtain a labeled point cloud model, and output it as a txt file, which stores the X-axis coordinates, Y-axis coordinates and Z-axis coordinates of the points in the labeled point cloud model, as well as the normal information of the labeled point cloud model.

[0023] Further, in S13, the format of the labeled point cloud model is adjusted by Python to obtain a bridge point cloud virtual dataset, which specifically includes performing the following steps in Python software:

[0024] S131: Delete the point clouds at the overlapped parts of the bridge components in the labeled point cloud model, and remove the point clouds at both ends of the beam;

[0025] S132: rotating the point cloud model processed by S131 around the Y axis and the Z axis in the spatial coordinate system respectively, so that the processed point cloud model can generate a longitudinal slope with a random direction;

[0026] S133: deleting data that does not conform to the current specification and is randomly generated in the point cloud model processed in the above step S132;

[0027] S134: Automatically distribute the data processed by S133 based on the ShapeNet structure to obtain a bridge point cloud virtual data set.

[0028] Further, in S2, the deep learning network model is a PointNet++ network model;

[0029] The PointNet++ network model is trained based on the bridge point cloud virtual data set, and the normal is used as a training parameter; the number of training rounds is set, and when mAcc and mIoU stop increasing, the training is stopped to obtain the trained PointNet++ network model;

[0030] Furthermore, in S4, the improved heuristic method is used to perform a second segmentation on the point cloud model after the first segmentation, and the specific steps of obtaining the final bridge point cloud model include:

[0031] S41: performing principal component analysis PCA processing on the point cloud model after the first segmentation, so that the point cloud model after the first segmentation is parallel to the X-axis in the spatial coordinate system, and obtaining the displacement x1 and the rotation angle β1 of the point cloud model after the first segmentation;

[0032] S42: if the point cloud model processed by S41 includes two special components, the diaphragm and the tie beam, which are located in special positions, then the points of the special components are separated from the point cloud model processed by S41;

[0033] The point cloud model of the points with special components separated is cut into P slices along the X axis, and the slice thickness is the same, and the slice set S = {S i :i=1,2,…,P}, using each slice S i The height of the slice is divided into slices containing pier groups α p and slice α without pier group D Two categories, namely:

[0034]

[0035] H=|max{z i |S}-min{z i |S}|

[0036] In the formula, f(S i ) is a piecewise function, H is the height difference between the highest and lowest points of the full bridge point cloud, and ρ is a constant, which is used as a discriminant parameter to distinguish the two types of slices; i is the Z-axis coordinate of the i-th point;

[0037] S43: Slice the bridge pier group α p The optimal rotation angle is obtained by iterating the rotation angle. The process is:

[0038] Slice the pier group α p According to the set rotation angles, the points are rotated around the X-axis and Y-axis in the spatial coordinate system, and then projected to the YZ plane. The point cloud density histogram of the Z axis is made according to the rotation angle. p The Z coordinate of the point is used as the horizontal coordinate, and the number of point clouds in the unit area along the Z axis is used as the vertical coordinate to make a point cloud density histogram;

[0039] The maximum standard deviation is obtained by iteratively rotating the angle, that is, the slice α containing the bridge pier group is obtained. p The optimal rotation angle β2 around the X axis and the optimal rotation angle β3 around the Y axis, and the slice containing the pier group α p and slice α without pier group D Both rotate around the X-axis and Y-axis according to the optimal rotation angle;

[0040] The rotation angle of the separated special component points is iterated to obtain the optimal rotation angle. The process is as follows: the separated special component points are rotated around the X-axis and Y-axis in the space coordinate system according to the set new rotation angle, and then projected to the YZ plane. According to the new rotation angle, the point cloud density histograms of the Z-axis and Y-axis are made respectively. The maximum standard deviation is obtained by iterating the new rotation angle, and the optimal rotation angle corresponding to the standard deviation is obtained, and the special component points are rotated according to the optimal rotation angle;

[0041] S44: Assume that the point cloud model after the first segmentation includes M components, and M≤N, and the component set is D={D i :i=1,2,…,M},different components D i The point clouds are projected onto the YZ plane respectively, and a point cloud density histogram about the Z axis is made. The boundary information, i.e., the position information of the contact surface between components, is obtained based on the extreme values ​​in the point cloud density histogram.

[0042] S45: Use the obtained boundary information to separate each component D i The point cloud is divided into non-component point cloud FD i ={FD i |D i} and the point cloud TD of this component i ={TD i |D i}, and then FD is i Assign them to adjacent components, and finally combine all components to obtain the initial bridge point cloud model;

[0043] S46: Based on the displacement x1 and the rotation angle β1 obtained in S41, and the optimal rotation angles β2 and β3 obtained in S43, the initial bridge point cloud model is inversely transformed to restore the position of the bridge and obtain the final bridge point cloud model.

[0044] Further, in S131, the process of deleting the point cloud at the overlapping position of the bridge components in the labeled point cloud model is as follows:

[0045] 1) Classify the labeled point cloud model according to the labels to obtain N components;

[0046] 2) The components are grouped based on the size of the contact surface between two components. That is, from the N components, the number of points in each component whose point cloud height difference is less than 2ρ at the contact surface of the two components is determined in turn, and the number of points is compared. The two components with more points are regarded as large components, and the components with fewer points are regarded as small components.

[0047] 3) Allocating the N components into N-1 groups according to the size of the contact surface, each group including a large component b and a small component s;

[0048] 4) Obtain the point clouds whose distance between point clouds in all point clouds of large components and small components is less than the set distance D1, D1 = ρ, and then determine the position of the point clouds in the large components and small components, and delete the point clouds that meet the following conditions, including:

[0049] Determine whether the distance between each point cloud in the large component and each point cloud in the small component is less than the set distance D2, and after the determination, delete the point clouds whose distance between the point clouds in all point clouds of the large component and the small component is less than the set distance D2, where D2 = 0.2ρ;

[0050] Determine whether the Z-axis coordinates of each point cloud in the large component and each point cloud in the small component are less than the set distance D3. After determination, delete the point clouds in all point clouds of the large component and the small component whose Z-axis coordinates are less than the set distance D3, where D3 = 0.01ρ.

[0051] Beneficial effect: The present invention establishes a virtual data set of bridge point cloud, trains the deep learning network model, and uses the trained deep learning network model to roughly segment the real point cloud data set to obtain a point cloud model after the first segmentation, and uses an improved heuristic method to perform a second segmentation on the point cloud model after the first segmentation, thereby obtaining the final bridge point cloud model; the present invention uses a heuristic method as the second segmentation method, which can not only give play to the high segmentation accuracy of the heuristic method, but also make the segmentation result more controllable. This step-by-step segmentation method can not only simplify the current heuristic method, but also reduce the requirements for deep learning results. Even when the accuracy of deep learning segmentation is limited, it can be compensated by the second segmentation to ensure the final segmentation accuracy; at the same time, the requirements for the training data set used by the deep learning network model are reduced, which means that the data set can adopt a strategy of reducing accuracy and batch production. Therefore, the advantages and disadvantages of the bridge point cloud virtual data set, deep learning method segmentation and heuristic method segmentation complement each other, and can simplify each other on the premise of improving segmentation accuracy, thereby ensuring accurate segmentation of the bridge point cloud and avoiding large-scale classification errors of the point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 A flowchart of a two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods in the present invention;

[0054] Figure 2 A flowchart of deleting point clouds at overlapping locations of bridge components in a labeled point cloud model in an embodiment of the present invention;

[0055] Figure 3The projection diagram and density histogram of the bridge point cloud on the Z axis and the Y axis without the first segmentation in the embodiment of the present invention;

[0056] Figure 4 This is a flowchart of the second segmentation in an embodiment of the present invention;

[0057] Figure 5 A schematic diagram of a training set and a test content in an embodiment of the present invention;

[0058] Figure 6 A diagram showing a comparison of the training effects of various deep learning network models under different conditions in an embodiment of the present invention;

[0059] Figure 7 This is a comparison diagram of the optimal segmentation results of Bridge1 by various deep learning network models in an embodiment of the present invention;

[0060] Figure 8 A point cloud slice map and a grouping map based on the first segmentation result in an embodiment of the present invention;

[0061] Fig. 9 In the embodiment of the present invention, P1 The relationship between the rotation angle and the standard deviation under uniform sampling;

[0062] Fig.10 In the embodiment of the present invention, P2 Projection diagram on the YZ plane;

[0063] Fig.11 It is a point cloud density histogram made by label based on the pre-segmentation result in an embodiment of the present invention;

[0064] Fig.12 Schematic diagram of the segmentation result of Bridge1 and the slight error at the boundary in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] This embodiment provides a two-stage bridge point cloud segmentation method that integrates deep learning and heuristic methods, such as Figure 1 As shown, the specific steps include:

[0067] S1: Establish a bridge point cloud virtual dataset based on ShapeNet data structure;

[0068] Specifically, in the field of bridges, the shortage of point cloud data sets is more serious. Unlike image data sets, the collection of bridge point cloud data is more difficult and the environment is more severe. Therefore, this embodiment trains the deep learning network model through synthetic point cloud data to avoid the lack of data sets and the lack of precision of data sets affecting the segmentation results of the deep learning network model. The construction of the bridge point cloud virtual data set required for point cloud component segmentation must meet two aspects: 1) It can be simple and fast to batch build a data set that meets the requirements; 2) It is closer to the real bridge point cloud, such as piers may be of different lengths due to occlusion, there is no point cloud between components, the left and right sections of the main beam cannot obtain point clouds, the orientation is random, there is a slope, etc., so that the output result of the deep learning network model meets the input requirements of the second segmentation.

[0069] In a specific embodiment, in S1, the specific steps of establishing a bridge point cloud virtual dataset based on the ShapeNet structure include:

[0070] S11: Obtain bridge component data, and establish a bridge entity model based on the bridge component data:

[0071] In a specific embodiment, in S11, the process of acquiring bridge component data and establishing a bridge entity model based on the bridge component data is as follows:

[0072] S111: Acquire N different types of bridge component data, each bridge component data includes size data and relative position data of the bridge component;

[0073] S112: Convert the bridge component data into data recognizable by Revit through Dynamo, create an adaptive family through Revit, then input the data recognizable by Revit after conversion by Dynamo into the adaptive family, and obtain a bridge solid model through Revit batch production, and export the bridge solid model in .fbx format to obtain a .fbx file.

[0074] Specifically, the purpose of batch modeling is to realize batch construction of bridge entity models for conversion into point cloud models, and to realize random size including topological relationships. Bridge component data is constructed through Excel, not only because Excel can interact directly with Dynamo, but also because the process of inputting data is also the process of realizing the first data randomness and mutual constraints between data. Bridge data containing the same components are saved as the same set of data, and at the same time, the size range of each component containing topological relationships is formulated. For example, the height of the pier can be randomly sampled at more than 3.5m according to the clear height of the car channel, but it will also be affected by the height of the pier cap, and the position of the pier will be affected by the length, width, and number of piers of the bridge. In Dynamo, topological relationships and randomness that are difficult to construct in Excel can be further realized, and the more cumbersome functions in Dynamo can also be achieved by adjusting the structure of Excel.

[0075] S12: performing point cloud processing on the bridge entity model by using point cloud processing software to obtain a point cloud model, and assigning labels to the point cloud model according to the type of bridge components to obtain a labeled point cloud model;

[0076] In a specific embodiment, in S12, point cloud processing is performed on the bridge entity model by using point cloud processing software CloudCompare to obtain a point cloud model, and labels are assigned to the point cloud model according to the type of bridge components. The process of obtaining the labeled point cloud model is as follows:

[0077] S121: reading the .fbx file through CloudCompare, adjusting the size of the bridge entity model in proportion, and then generating a point cloud model and a normal line based on the bridge entity model according to the set density ρ;

[0078] Specifically, CloudCompare groups .fbx files by components when reading them, which makes it easier to assign tags later.

[0079] S122: classifying the point cloud model generated in S121 according to the type of bridge components, and merging point clouds of the same type of components to obtain N groups of point clouds;

[0080] S123: assigning corresponding labels to the N groups of point clouds according to the types of bridge components to obtain N groups of point clouds with labels;

[0081] S124: Merge the N groups of labeled point clouds to obtain a labeled point cloud model, and output it as a txt file, which stores the X-axis coordinates, Y-axis coordinates and Z-axis coordinates of the points in the labeled point cloud model, as well as the normal information of the labeled point cloud model.

[0082] S13: The labeled point cloud model is adjusted to make it close to the real point cloud, and finally a virtual data set of bridge point cloud is obtained.

[0083] In a specific embodiment, in S13, the format of the labeled point cloud model is adjusted by Python to obtain a bridge point cloud virtual dataset, which specifically includes performing the following steps in Python software:

[0084] S131: Delete the point clouds at the overlapping parts of the bridge components in the labeled point cloud model, and remove the point clouds at both ends of the beam to obtain the processed point cloud model; specifically, since the present embodiment generates point clouds based on components, there will be point clouds on the contact surfaces of the components, which will cause adverse effects in the deep learning process. For example, many Encoder modules of the current point cloud classification network rely on the K-Nearest Neighbor (KNN) algorithm. For some components, these overlapping points greatly affect the proximity values ​​of the sample points in the KNN, resulting in the inability to generalize the features of the synthetic point cloud to the real point cloud. Therefore, it is necessary to delete the point clouds at the overlapping parts of the bridge components in the labeled point cloud model. For the case where there is no point cloud on the left and right sections of the main beam, it is only necessary to delete them by obtaining the maximum values ​​at both ends.

[0085] In a specific embodiment, in S131, if Figure 2 As shown in the figure, the process of deleting the point cloud at the overlapping parts of the bridge components in the labeled point cloud model is as follows:

[0086] 1) Classify the labeled point cloud model according to the labels to obtain N components;

[0087] 2) The components are grouped based on the size of the contact surface between two components. That is, from the N components, the number of points in each component whose point cloud height difference is less than 2ρ at the contact surface of the two components is determined in turn, and the number of points is compared. The two components with more points are regarded as large components, and the components with fewer points are regarded as small components.

[0088] 3) Allocating the N components into N-1 groups according to the size of the contact surface, each group including a large component b and a small component s;

[0089] 4) Obtain the point clouds whose distance between point clouds in all point clouds of large components and small components is less than the set distance D1, D1 = ρ, and then determine the position of the point clouds in the large components and small components, and delete the point clouds that meet the following conditions, including:

[0090] Determine whether the distance between each point cloud in the large component and each point cloud in the small component is less than the set distance D2, and after the determination, delete the point clouds whose distance between the point clouds in all point clouds of the large component and the small component is less than the set distance D2, where D2 = 0.2ρ;

[0091] Determine whether the Z-axis coordinates of each point cloud in the large component and each point cloud in the small component are less than the set distance D3. After determination, delete the point clouds in all point clouds of the large component and the small component whose Z-axis coordinates are less than the set distance D3, where D3 = 0.01ρ.

[0092] Specifically, in this embodiment, D1=1; D2=0.2; D3=0.01.

[0093] S132: rotating the point cloud model processed by S131 around the Y axis and the Z axis in the spatial coordinate system respectively, so that the processed point cloud model can generate a longitudinal slope with a random direction;

[0094] S133: deleting data that does not conform to current specifications due to random data generation in the point cloud model processed in the above step S132. The current specifications include "General Specifications for Highway Bridge and Culvert Design", "Technical Standards for Highway Engineering", and "Design Specifications for Urban Road Engineering".

[0095] Specifically, since the point cloud text information input into the deep learning network is not a real spatial structure, a small rotation in space will also result in a large change in the text information. That is, the segmentation effect of the point cloud of the same bridge will change significantly after being rotated at different angles. Therefore, each synthetic point cloud is normally distributed with a slope between -4% and 4% and randomly rotated around the Z axis.

[0096] S134: Automatically distribute the data processed by S133 based on the ShapeNet structure to obtain a bridge point cloud virtual data set.

[0097] S2: constructing a deep learning network model, and training the deep learning network model based on the bridge point cloud virtual data set to obtain a trained deep learning network model;

[0098] In a specific embodiment, in S2, the deep learning network model is a PointNet++ network model;

[0099] The PointNet++ network model is trained based on the bridge point cloud virtual data set, and the normal is used as a training parameter; the number of training rounds is set, and when mAcc and mIoU stop increasing, the training is stopped to obtain the trained PointNet++ network model;

[0100] Specifically, when the number of training rounds is greater than 500, if mAc and mIoU stop increasing, the training is stopped to obtain the trained PointNet++ network model; if mAc and mIoU continue to increase, they are increased in units of 100 rounds until mAc and mIoU stop increasing.

[0101] S3: Acquire a real point cloud data set, and perform a first rough segmentation on the real point cloud data set based on the trained deep learning network model to obtain a point cloud model after the first segmentation, wherein the point cloud model after the first segmentation includes M components, and M≤N;

[0102] Specifically, in this embodiment, since the effect of segmentation through deep learning methods is limited, the segmentation results based on deep learning and combined with geometric methods can fully realize the component segmentation. However, the geometric characteristics of each bridge are different, so they need to be assigned segmentation methods respectively. The ShapeNet structure can achieve a one-to-one correspondence with the secondary segmentation method. For example, in the ShapeNet bridge data set structure, Box_girder_Pier_caps_Circular_piers can correspond to the secondary segmentation method of the three components of box girder, pier cap, and circular pier. Therefore, the ShapeNet structure is more suitable for establishing a bridge point cloud data set, and its modularity of "data set + secondary segmentation method" as a unit is also easier to expand to other types of bridges. Specifically, for bridges of the same type but with different components, different parameters can be set for the same segmentation method.

[0103] S4: The improved heuristic method is used to perform a second segmentation on the point cloud model after the first segmentation to ensure that the boundary of each component in the point cloud model after the first segmentation is clear, thereby obtaining the final bridge point cloud model.

[0104] Specifically, based on the classification ability of current point cloud classification technology, under the condition of pure synthetic data sets as training sets, the effect of deep learning networks on point cloud segmentation is ultimately limited. It is difficult to achieve application-level component segmentation by deep learning alone. Therefore, it is necessary to perform secondary segmentation on the already segmented point clouds. The first segmentation is to correctly classify most of the point clouds. The classification results often have two problems: (1) mixed erroneous points in each component; (2) unclear boundaries of each component. These two problems will seriously affect the subsequent work based on the classified point clouds. The second segmentation is to solve these two problems. Unlike traditional heuristic methods, pre-segmented point clouds do not require complex segmentation methods and segmentation steps. They only need to solve the above two problems in a targeted manner and make the final classification results controllable. For example, for reinforced concrete beam bridges, except for special components such as diaphragms, the various components of the bridge generally have obvious hierarchical relationships in the vertical or horizontal direction. With this feature and combined with incompletely classified point clouds, secondary classification can be achieved.

[0105] In a specific embodiment, in S4, if Figure 4 As shown in ①-⑥, the improved heuristic method is used to perform a second segmentation on the point cloud model after the first segmentation, and the specific steps of obtaining the final bridge point cloud model include:

[0106] S41: performing principal component analysis PCA processing on the point cloud model after the first segmentation, so that the point cloud model after the first segmentation is parallel to the X-axis in the spatial coordinate system, and obtaining the displacement x1 and the rotation angle β1 of the point cloud model after the first segmentation;

[0107] S42: if the point cloud model processed by S41 includes two special components, the diaphragm and the tie beam, which are located in special positions, then the points of the special components are separated from the point cloud model processed by S41;

[0108] The point cloud model of the points with special components separated is cut into P slices along the X axis, and the slice thickness is the same, and the slice set S = {S i :i=1,2,…,P}, using each slice S i The height of the slice is divided into slices containing pier groups α p and slice α without pier group D Two categories, namely:

[0109]

[0110] H=|max{z i |S}-min{z i |S}|

[0111] In the formula, f(S i ) is a piecewise function, H is the height difference between the highest and lowest points of the full bridge point cloud, and ρ is a constant, which is used as a discriminant parameter to distinguish the two types of slices; i is the Z-axis coordinate of the i-th point;

[0112] S43: Slice the bridge pier group α p The optimal rotation angle is obtained by iterating the rotation angle. The process is:

[0113] Slice the pier group α p Rotate around the X-axis and Y-axis according to the set rotation angles, and then project to the YZ plane. Draw the point cloud density histogram of the Z axis according to the rotation angles, that is, take the Z coordinate of the point as the horizontal coordinate and the number of point clouds in the unit area along the Z axis as the vertical coordinate to draw the point cloud density histogram;

[0114] Since each rotation angle corresponds to a histogram and a standard deviation, and the distribution of the standard deviation follows a normal distribution for the rotation angle, the maximum standard deviation is obtained by iterating the rotation angle, that is, the slice α containing the bridge pier group is obtained. p The optimal rotation angle β2 around the X axis and the optimal rotation angle β3 around the Y axis, and the slice containing the pier group α p and slice α without pier group DBoth rotate around the X-axis and Y-axis according to the optimal rotation angle;

[0115] The rotation angle of the separated special component points is iterated to obtain the optimal rotation angle. The process is as follows: the separated special component points are rotated around the X-axis and Y-axis in the space coordinate system according to the set new rotation angle, and then projected to the YZ plane. According to the new rotation angle, the point cloud density histograms of the Z-axis and Y-axis are made respectively. The maximum standard deviation is obtained by iterating the new rotation angle, and the optimal rotation angle corresponding to the standard deviation is obtained, and the special component points are rotated according to the optimal rotation angle;

[0116] S44: The point cloud model after the first segmentation includes M components, and the component set is D=

[0117] {D i :i=1,2,…,M|,different components D i The point clouds are projected onto the YZ plane, and a point cloud density histogram about the Z axis is made. The boundary information, that is, the position information of the contact surface between components, is obtained based on the extreme values ​​in the point cloud density histogram. Figure 3 Different from the direct boundary division without pre-segmentation in , the boundary of the point cloud density histogram is clearer after separating the components, such as Figure 4 The method of drawing separately by component will associate the boundary information with the component itself, which can greatly simplify the heuristic segmentation method and eliminate the need to combine normal information and other information for tedious subsequent segmentation.

[0118] S45: Use the obtained boundary information to separate each component D i The point cloud is divided into non-component point cloud FD i ={FD i |D i} and the point cloud TD of this component i ={TD i |D i}, and then FD is i Assign them to adjacent components, and finally combine all components to obtain the initial bridge point cloud model;

[0119] S46: Based on the displacement x1 and the rotation angle β1 obtained in S41, and the optimal rotation angles β2 and β3 obtained in S43, the initial bridge point cloud model is inversely transformed to restore the position of the bridge and obtain the final bridge point cloud model.

[0120] In this embodiment, the feasibility of this embodiment is verified by conducting the following experiments. The experiments involve four types of components: beam slab, pier cap, cylindrical pier, and trapezoidal pier. The experiments are divided into two types of bridges: one with beam, pier cap, and cylindrical pier, and the other with beam and trapezoidal pier. Each type of bridge is regarded as an experimental group. Figure 5As shown in Figure 1, each experimental group contains three sections: short cantilever box girder section 1 based on the real point cloud (test set), conventional box girder section 2 and rectangular section 3, where the rectangular section does not include the pier cap. Each section is divided into three forms: 3x3, 3x2 and 2x2 according to the arrangement of the piers. A set of data is produced for each arrangement, as shown in Figure 1. Figure 8 As shown. Take 25 data (25 bridges) as a group, remove 5 unreasonable data from each group, that is, 20 data in each group, then each Train1 and Train2 have 3×3×20=180 bridge data as training sets; allocate bridge point clouds according to Table 1, Group1 and Group2 are divided into three training sets, namely {Group1: A1, B1, C1}, {Group2: A2, B2, C2}, A1 can also be expressed as Train1-A, used to test the influence of training set content on segmentation effect, the test sets are all from Lu et al. Public bridge point cloud dataset, where Bridge1-1 and Bridge1-2 are split and modified from Bridge1, Bridge3 and Bridge4 are both in curved sections, the beam cross sections in the test sets are similar to Bridge1 and Bridge2, Bridge5 and Bridge6 are single trapezoidal piers and double trapezoidal piers under straight sections respectively.

[0121] Table 1:

[0122]

[0123] Based on the above dataset, three point cloud deep learning networks, PointNet++, PCT, and DGCNN, are used for segmentation tasks. The stochastic gradient descent (SGD) optimizer with a learning rate of 0.1 and a momentum of 0.9 is used. The learning rate is also reduced to 0.001 using cosine annealing. The batch size is 4, and the number of training rounds is set to 100. The experiments will be downsampled to 4096 points and 8192 points respectively for comparative experiments. PointNet++ and PCT will also input normal information as experimental groups, namely PointNet++ (normal) and PCT (normal). Normal is one of the important geometric information of bridge point clouds and plays a vital role in classification tasks. Unlike RGB information, which is easily disturbed during the point cloud acquisition process, normal information is not only not easily disturbed, but can also be generated later. When used as training information input, it generally improves the segmentation effect. In the segmentation task, the average accuracy (mAcc) and the average intersection over union (mIoU) will be selected as evaluation indicators. Different from the conventional deep learning-based segmentation task, the segmentation result of the point cloud in the present invention serves the secondary segmentation rather than being directly used as the result. Therefore, there is no need for too many evaluation indicators. It is only necessary to ensure that the key indicators meet the requirements. Therefore, for the secondary segmentation method used in this embodiment, mIoU will directly affect the secondary segmentation effect, and mAc is used as an auxiliary indicator, which are respectively expressed as:

[0124]

[0125] Where n represents the number of bridge components, p ij Indicates that the point with label i is incorrectly predicted as label j; Table 2 shows the original data of the experiment. In the segmentation results of Group 2, mAcc and mIoU are both greater than 97%, which means that the pure synthetic dataset can meet the input requirements of the second segmentation as a training set for the segmentation task. For the multi-classification task represented by Group 1, it is more complicated.

[0126] Table 2:

[0127]

[0128] First, from the perspective of network performance, Figure 6 The data comes from the original experimental data (combined with Figure 7 ),pass Figure 6It can be clearly seen from the two figures (a) and (b) that PointNet++ has higher mAc and mIoU under various conditions, and can reach more than 80% mIoU with the participation of normal information using only a simple pure synthetic dataset. Although PCT has better segmentation capabilities than the other two models in the segmentation of large datasets, it is not ideal when based on pure synthetic datasets. Figure 6 As shown in (b), mIoU is at the lowest level without the participation of normals.

[0129] From the analysis of the response degree of normal information, it is very unstable to perform segmentation tasks based only on the three-dimensional coordinates of the point cloud. Figure 6 In (b), when PointNet does not involve normals, the mIoU of the A1 training group is much lower than the average, and the DGCNN in the B2-8192 experimental group is also much lower than the average. In addition to the stability problem, Figure 6 (d)(f) Comparison of the two figures shows that both PointNet and PCT improve the segmentation effect with the participation of normals, and PointNet has a greater degree of response. In fact, unlike PointNet++, PCT does not involve normals in routine work, but due to its simpler network framework and more scalable mechanism, it can easily use normals as training parameters and produce positive responses in experimental results. However, graph convolutional neural networks such as DGCNN are not easy to respond to normal information from a mechanism perspective, and may even produce negative responses. Therefore, the degree of response of PointNet++ to normals of synthetic datasets is also one of the important indicators of the experiment.

[0130] From the perspective of data set, Figure 6 It can be seen that different data sets are adapted to different network models. DGCNN obtains geometric information from different dimensions. The geometric information of the data set has a greater impact on the segmentation effect. Therefore, the C1 group with monotonous data content has the highest mAc and mIoU, while PointNet++_n with 8192 points is closer to the ideal situation. With the increase of the richness of the training set content, the segmentation effect gradually improves.

[0131] In summary, network models like PointNet++, which are highly adaptable to synthetic datasets, responsive to normals and have higher responsiveness, and whose performance improves with the enrichment of training data, are more suitable for training based on synthetic datasets for the first segmentation work.

[0132] Specifically, the second segmentation of the real bridge takes Bridge1 as an example. For the convenience of calculation, PointNet++_normal is used to output 100,000 points for the experiment. First, PCA is performed to eliminate the geographic information carried by the point cloud. During the process, the displacement x1 and the rotation angle β1 are recorded; the slice thickness δ = 0.5m, and ρ = 0.30; the following is obtained: Figure 8 (c) The pier group α P1 and without pier group α D1 ;

[0133] Next is the iteration of the rotation angle. During the experiment, it was found that the real bridge point cloud often falls into the local optimal solution during the iteration process due to factors such as noise and irregular deformation of the bridge. Considering that the rotation angle value range [J, K] is small, the slope rotation angle (around the Y axis) only needs to be controlled at J = -K = 6° based on the specification. The exhaustive method can meet the accuracy requirements. The number of samples is:

[0134]

[0135] Among them, P R It is the accuracy, the unit is degree; for the projection of point cloud, the accuracy of angle is of no use value, only the recognition accuracy of point cloud spacing is valuable:

[0136] P L =L·tanP R

[0137] Among them, P L is the minimum recognition accuracy, in meters, and L is the length of the bridge; to ensure that R =0.1°, only 120 samples need to be taken evenly, the length of Bridge 1 is 6m, then P L ≈0.01m. Since the distribution of the standard deviation is close to the normal distribution under uniform sampling, such as Fig. 9 As shown, the normal sampling can further improve the accuracy; similarly, the X-axis is controlled at J = -K = 1°. Through cross-enumeration, the maximum standard deviation of 1309.14 is obtained by rotating β2 = 0.039° and β3 = -0.157° around the x-axis and y-axis respectively.

[0138] After iteration, from Fig.10 It can be seen that the boundaries of each component are basically overlapped. The point cloud after the optimal rotation transformation is the point cloud containing the pier group α P2 ; For the splicing model, the same transformation does not include the pier group α D1 Group α does not contain piers D2 .

[0139] The boundary is not obtained directly through the histogram, but the maximum value of n histograms is obtained by the number of components.i :i=1,2,…,n}, thus obtaining the bounding box B, the axis coordinates of the upper and lower bounds of the bounding box are B U ,B L In the real point cloud, the boundary information is not as ideal as that of the synthetic point cloud. As the width of the histogram column decreases, the concentration of the point cloud will decrease relatively. Therefore, the boundary points of the real point cloud will not be a line after projection, but a small area. The points in the boundary box are mainly the boundary points of the bridge components. For Bridge1, Fig.11 As shown,

[0140] B={B1:Bound box 1,B2:Bound box 2},B U = {B 1U :4.91,B 2U :3.93}, B L =

[0141] {B 1L :4.73,B 2L :3.85}. There are two problems involved in the process of finding B:

[0142] (1) The value of the bin width of the histogram. Although ideally the smaller the bin width, the more accurate the bounding box, the accuracy of the point cloud density and rotation angle iteration steps is limited. The point cloud density can basically reach the accuracy P with the current point cloud acquisition equipment. P = 0.001m, and the accuracy of angle iteration is P L , then the overall accuracy P T =max{P P ,P L}. In order to ensure the accuracy of the boundary points, then:

[0143] bin=2P T

[0144] For Bridge 1 in this experiment, bin = 0.02m.

[0145] (2) The lower limit ρmin of the column height of the upper and lower bounds of the bounding box. The column height value should be based on the maximum value of the histogram and is related to the bin height, that is, ρmin∝ρmax, ρmin∝bin. Combined with the experiment, we can get:

[0146] ρmin=10ρmax·bin

[0147] For this experiment, B1:ρmin1=10×5039×0.02=1007.8, B2:ρmin2=10×1113×0.02=222.6.

[0148] After determining the boundary information, first use each component D i The boundary information B obtained by itself i The point cloud of the component is divided into non-component point cloud FD i And the point cloud TD of this component i Two parts, and then use the boundary information obtained from other components to FD i Correctly classified. Finally, use x1, β1, β2, β3 to do inverse transformation to restore the geographical location information of the bridge.

[0149] Since the entire secondary segmentation process is simpler and the error is easier to control than the general heuristic method, it is more robust than the pure deep learning method or heuristic method. R Control, S U ,P L ,bin,ρmin all achieve the same R The association of , realizes secondary segmentation while controlling the global error, such as Fig.12 As shown, in P R =0.1°, which illustrates the feasibility of the method provided in this embodiment.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods, characterized in that: The specific steps include: S1: Establish a bridge point cloud virtual dataset based on ShapeNet data structure; S2: constructing a deep learning network model, and training the deep learning network model based on the bridge point cloud virtual data set to obtain a trained deep learning network model; S3: Acquire a real point cloud data set, and perform a first rough segmentation on the real point cloud data set based on the trained deep learning network model to obtain a point cloud model after the first segmentation; S4: The improved heuristic method is used to perform a second segmentation on the point cloud model after the first segmentation to obtain the final bridge point cloud model.

2. The two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods according to claim 1, characterized in that: In S1, the specific steps of establishing a bridge point cloud virtual dataset based on the ShapeNet structure include: S11: Obtain bridge component data, and establish a bridge entity model based on the bridge component data: S12: performing point cloud processing on the bridge entity model by using point cloud processing software to obtain a point cloud model, and assigning labels to the point cloud model according to the type of bridge components to obtain a labeled point cloud model; S13: Adjust the labeled point cloud model to finally obtain a bridge point cloud virtual dataset.

3. The two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods according to claim 2, characterized in that: In S11, the process of acquiring bridge component data and establishing a bridge entity model based on the bridge component data is as follows: S111: Acquire N different types of bridge component data, each bridge component data includes size data and relative position data of the bridge component; S112: Convert the bridge component data into data recognizable by Revit through Dynamo, create an adaptive family through Revit, then input the data recognizable by Revit after conversion by Dynamo into the adaptive family, and obtain a bridge solid model through Revit batch production, and export the bridge solid model in .fbx format to obtain a .fbx file.

4. The two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods according to claim 3, characterized in that: In S12, point cloud processing is performed on the bridge entity model by point cloud processing software to obtain a point cloud model, and labels are assigned to the point cloud model according to the type of bridge components. The process of obtaining the labeled point cloud model is as follows: S121: reading the .fbx file, adjusting the size of the bridge entity model in proportion, and then generating a point cloud model and a normal line based on the bridge entity model according to a set density ρ; S122: classifying the point cloud model generated in S121 according to the type of bridge components, and merging point clouds of the same type of components to obtain N groups of point clouds; S123: assigning corresponding labels to the N groups of point clouds according to the types of bridge components to obtain N groups of point clouds with labels; S124: Merge the N groups of labeled point clouds to obtain a labeled point cloud model, and output it as a txt file, which stores the X-axis coordinates, Y-axis coordinates and Z-axis coordinates of the points in the labeled point cloud model, as well as the normal information of the labeled point cloud model.

5. The two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods according to claim 4, characterized in that: In S13, the format of the labeled point cloud model is adjusted by Python to obtain a bridge point cloud virtual dataset, which specifically includes performing the following steps in Python software: S131: Delete the point clouds at the overlapped parts of the bridge components in the labeled point cloud model, and remove the point clouds at both ends of the beam; S132: rotating the point cloud model processed by S131 around the Y axis and the Z axis in the spatial coordinate system respectively, so that the processed point cloud model can generate a longitudinal slope with a random direction; S133: deleting data that does not conform to the current specification and is randomly generated in the point cloud model processed in the above step S132; S134: Automatically distribute the data processed by S133 based on the ShapeNet structure to obtain a bridge point cloud virtual data set.

6. The two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods according to claim 5, characterized in that: In S2, the deep learning network model is a PointNet++ network model; The PointNet++ network model is trained based on a virtual data set of bridge point clouds, and the normal is used as a training parameter; Set the number of training rounds. When mAcc and mIoU stop increasing, stop training and get the trained PointNet++ network model.

7. The two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods according to claim 6, characterized in that: In S4, the improved heuristic method is used to perform a second segmentation on the point cloud model after the first segmentation, and the specific steps of obtaining the final bridge point cloud model include: S41: performing principal component analysis PCA processing on the point cloud model after the first segmentation, so that the point cloud model after the first segmentation is parallel to the X-axis in the spatial coordinate system, and obtaining the displacement x1 and the rotation angle β1 of the point cloud model after the first segmentation; S42: if the point cloud model processed by S41 includes two special components, the diaphragm and the tie beam, which are located in special positions, then the points of the special components are separated from the point cloud model processed by S41; The point cloud model of the points with special components separated is cut into P slices along the X axis, and the slice thickness is the same, and the slice set S = {S i :i=1,2,…,P}, using each slice S i The height of the slice is divided into slices containing pier groups α p and slice α without pier group D Two categories, namely: H=|max{z i |S}-min{z i |S}| In the formula, f(S i ) is a piecewise function, H is the height difference between the highest and lowest points of the full bridge point cloud, and ρ is a constant, which is used as a discriminant parameter to distinguish the two types of slices; i is the Z-axis coordinate of the i-th point; S43: Slice the bridge pier group α p The optimal rotation angle is obtained by iterating the rotation angle. The process is: Slice the pier group α p According to the set rotation angles, the points are rotated around the X-axis and Y-axis in the spatial coordinate system, and then projected to the YZ plane. The point cloud density histogram of the Z axis is made according to the rotation angle. p The Z coordinate of the point is used as the horizontal coordinate, and the number of point clouds in the unit area along the Z axis is used as the vertical coordinate to make a point cloud density histogram; The maximum standard deviation is obtained by iteratively rotating the angle, that is, the slice α containing the bridge pier group is obtained. p The optimal rotation angle β2 around the X axis and the optimal rotation angle β3 around the Y axis, and the slice containing the pier group α p and slice α without pier group D Both rotate around the X-axis and Y-axis according to the optimal rotation angle; The rotation angle of the separated special component points is iterated to obtain the optimal rotation angle. The process is as follows: the separated special component points are rotated around the X-axis and Y-axis in the space coordinate system according to the set new rotation angle, and then projected to the YZ plane. According to the new rotation angle, the point cloud density histograms of the Z-axis and Y-axis are made respectively. The maximum standard deviation is obtained by iterating the new rotation angle, and the optimal rotation angle corresponding to the standard deviation is obtained, and the special component points are rotated according to the optimal rotation angle; S44: Assume that the point cloud model after the first segmentation includes M components, and M≤N, and the component set is D={D t :i=1,2,…,M},different components D t The point clouds are projected onto the YZ plane respectively, and a point cloud density histogram about the Z axis is made. The boundary information, i.e., the position information of the contact surface between components, is obtained based on the extreme values ​​in the point cloud density histogram. S45: Use the obtained boundary information to separate each component D i The point cloud is divided into non-component point cloud FD i ={FD i |D i } and the point cloud TD of this component i ={TD i |D i }, and then FD is i Assign them to adjacent components, and finally combine all components to obtain the initial bridge point cloud model; S46: Based on the displacement x1 and the rotation angle β1 obtained in S41, and the optimal rotation angles β2 and β3 obtained in S43, the initial bridge point cloud model is inversely transformed to restore the position of the bridge and obtain the final bridge point cloud model.

8. The two-stage bridge point cloud segmentation method integrating deep learning and heuristic methods according to claim 7, characterized in that: In S131, the process of deleting the point cloud at the overlapping position of the bridge components in the labeled point cloud model is as follows: 1) Classify the labeled point cloud model according to the labels to obtain N components; 2) The components are grouped based on the size of the contact surface between two components. That is, from the N components, the number of points in each component whose point cloud height difference is less than 2ρ at the contact surface of the two components is determined in turn, and the number of points is compared. The two components with more points are regarded as large components, and the components with fewer points are regarded as small components. 3) Allocating the N components into N-1 groups according to the size of the contact surface, each group including a large component b and a small component s; 4) Obtain the point clouds whose distance between point clouds in all point clouds of large components and small components is less than the set distance D1, D1 = ρ, and then determine the position of the point clouds in the large components and small components, and delete the point clouds that meet the following conditions, including: Determine whether the distance between each point cloud in the large component and each point cloud in the small component is less than the set distance D2, and after the determination, delete the point clouds whose distance between the point clouds in all point clouds of the large component and the small component is less than the set distance D2, where D2 = 0.2ρ; Determine whether the Z-axis coordinates of each point cloud in the large component and each point cloud in the small component are less than the set distance D3. After determination, delete the point clouds in all point clouds of the large component and the small component whose Z-axis coordinates are less than the set distance D3, where D3 = 0.01ρ.

Citation Information

Cited By

  • Hybrid model-based bridge member geometric feature determination method

    CN119941827A

  • Method for Determining Geometric Features of Bridge Components Based on Hybrid Model

    CN119941827B

  • Mountain large-span bridge assembly point cloud segmentation method based on large model

    CN122265300A