A component automatic segmentation and identification method suitable for bridge point cloud

Abstract

The application discloses a kind of component automated segmentation identification method suitable for bridge point cloud, the method includes: extracting multi-view visible three-dimensional grid to bridge point cloud and rendering as two-dimensional video frame, realize the dimensionality reduction of three-dimensional point cloud;Using visual large model SAM and SAM2 obtains each component segmentation mask under each view angle;The segmentation result under diagonal view angle is projected, based on the nearest pairing algorithm of segmentation component cluster center, the segmentation result of same component on two sides is matched, according to the score of index point calculated from each view angle segmentation mask, realize the component level segmentation of bridge point cloud;Based on error point iterative repair algorithm, the segmentation result is optimized;Using multimodal large model CLIP realizes the classification identification of bridge component.The application only needs to collect the point cloud data of bridge point cloud for analysis and processing, can quickly and automatically realize the component level segmentation of bridge point cloud, based on the classification of component class based on segmentation result, realize the automated instance segmentation of bridge point cloud.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bridge point cloud instance segmentation, and particularly relates to a component automatic segmentation and identification method suitable for bridge point cloud. BACKGROUND

[0002] Bridges are an important part of transportation infrastructure, and instance segmentation of bridge point cloud is of great significance to the reconstruction of the existing bridge status model. However, the common method for collecting bridge point cloud needs the help of a three-dimensional laser scanner, and the obtained three-dimensional point cloud has the problems of large data volume, few semantics, and difficulty in segmentation. When three-dimensional reconstruction of the bridge is needed, the obtained point cloud data needs to be further segmented, and the method based on manual annotation segmentation is difficult, time-consuming and prone to errors. Existing point cloud segmentation is mostly based on neural network automatic segmentation, which has the problems of long training time, lack of training data, and gap between precision and modeling accuracy. Therefore, it is necessary to provide a component automatic segmentation and identification method suitable for bridge point cloud, which can improve the segmentation efficiency of bridge point cloud and has important application value. SUMMARY

[0003] The present application aims at the deficiencies of the prior art, and provides a component automatic segmentation and identification method suitable for bridge point cloud. The present application only needs to collect bridge point cloud data for analysis and processing, so as to quickly and accurately realize component-level instance segmentation.

[0004] The purpose of the present application is achieved by the following technical scheme: a component automatic segmentation and identification method suitable for bridge point cloud, comprising the following steps:

[0005] (1) The principal component analysis method is used to obtain the principal direction of the bridge point cloud, and coordinate transformation is performed to obtain the calibrated bridge point cloud; the calibrated bridge point cloud is divided into three-dimensional grids based on spatial position information, and the index information of the points in each grid is saved, and the multi-view visible three-dimensional grid is extracted based on the visibility principle when the camera is shooting, and is rendered into a two-dimensional video frame, realizing the dimension reduction processing of the three-dimensional point cloud;

[0006] (2) The two-dimensional video frame main view is taken as the input of the visual large model SAM to obtain the initial segmentation mask of each component, and the prompt box generated from each segmentation mask is taken as the input of the SAM2 model together with the two-dimensional video frame to obtain the segmentation mask of each component under each view;

[0007] (3) The segmentation result under the diagonal view obtained in step (2) is projected, the same component segmentation results on both sides are matched based on the nearest pairing algorithm of the clustering center of the segmented components, and the score of the index point is calculated according to the segmentation mask of each view, realizing the component-level segmentation of the bridge point cloud;

[0008] (4) Based on the error point iterative repair algorithm, the component-level segmentation result obtained in step (3) is optimized;

[0009] (5) Based on the optimized component-level segmentation result obtained in step (4), pixel points are extracted using the perspective segmentation mask coordinates, blank pixels around are added to generate component images, and the cosine similarity between each component image and prior knowledge text information is calculated based on a multi-modal large language model CLIP to realize classification and identification of bridge components.

[0010] Further, in the step (1), the principal direction of the bridge point cloud is obtained by using the principal component analysis method, and coordinate transformation is performed to obtain the calibrated bridge point cloud, specifically comprising:

[0011] The bridge point cloud is formally represented as an n-row 3-column three-dimensional point cloud coordinate matrix P. For the coordinate matrix P, the point cloud coordinate data therein is first centralized, and then the covariance matrix of the coordinate matrix is calculated. Singular value decomposition is performed on the covariance matrix to obtain three eigenvalues and their corresponding eigenvectors. The eigenvector corresponding to the maximum eigenvalue is taken as the principal direction of the three-dimensional point cloud, the eigenvector corresponding to the minimum eigenvalue is taken as the normal vector of the three-dimensional point cloud, and the eigenvector corresponding to the other eigenvalue is taken as the secondary principal direction of the three-dimensional point cloud. The principal direction is set as the x-axis, the secondary principal direction is set as the y-axis, and the normal vector is set as the z-axis. The three-dimensional point cloud of the bridge is rotated based on the set x-axis, y-axis and z-axis, and coordinate transformation is performed to obtain the calibrated bridge point cloud.

[0012] Further, in the step (1), the two-dimensional video frame is rendered, specifically comprising:

[0013] The point cloud of the visible three-dimensional grid is rendered into a two-dimensional video frame using a point cloud to video frame rendering algorithm to generate each perspective video frame image. The calculation formula is:

[0014]

[0015] In the formula, represents the i-th point in the point cloud the angle between the reference perspective and the perspective the corresponding three-dimensional grid index under the perspective of , , and represent the coordinates of the i-th point in the point cloud on the x-axis, y-axis and z-axis, , and represent the indices of the three-dimensional grid on the x-axis, y-axis and z-axis; is the size of the three-dimensional grid; is the rotation matrix, ; for point cloud coordinates, ; denotes an offset vector, , for point cloud center coordinates, for the minimum value of the point cloud in three coordinate directions; correspondingly, each group of , the minimum value of The grid closest to the virtual camera point is the visible point grid, and the remaining grids are invisible point grids. The RGB average of all points in the visible point grid is calculated as the pixel value of the grid.

[0016] Further, the step (2) specifically comprises:

[0017] The global segmentation of the main view of the reference view, i.e., the two-dimensional video frame, is performed using the visual large model SAM to obtain the initial segmentation mask of each component of the bridge, and the background mask is removed according to the pixel RGB value in the mask to obtain the initial segmentation mask of each component of the bridge. Select the upper left corner and the lower right corner pixel coordinates of each component initial segmentation mask to form a coordinate pair, and generate a prompt box using the coordinate pair; the two-dimensional video frame images under each view rendered are divided into two groups according to the rotation angle clockwise, and are input into the SAM2 model together with the prompt box to perform segmentation to obtain the segmentation mask of each component under the view, wherein the view selection is 0°, 15°, 165°, 180°, 195°, and 345°.

[0018] Further, the step (3) specifically comprises:

[0019] First, all visible points under 0° and 180° views are extracted and projected to the xOz plane, and the centers of the visible points corresponding to different masks after projection under 0° and 180° views are calculated, wherein the visible points are determined according to the index information of the points contained in the pixel coordinates; then for each center under the 0° view, the nearest center coordinates under the 180° view are found and paired, and the center coordinate index under the 180° view is modified based on the class index of the mask under the 0° view; the segmentation results under different views are integrated by counting the class scores, and for each point in the bridge point cloud corresponding to the mask, the score matrix of each point for each class is calculated, and the class with the highest score is selected as the class of the point, so as to realize the component-level segmentation of the bridge point cloud.

[0020] Further, the step (4) specifically comprises:

[0021] Based on the component-level segmentation result obtained in step (3), for the points with erroneous segmentation, under the constraint of the normal vector threshold, the radius interval Iteratively calculate the class to which the maximum number of points within the radius r of each point belongs from small to large, and take the class as the class of the error segmentation point; wherein and respectively represent the minimum radius at the beginning of iteration and the maximum radius at the end of iteration.

[0022] Further, the step (5) specifically comprises:

[0023] Based on the optimized component-level segmentation result obtained in step (4), the blank pixel points with a specified width are added to the periphery of each component segmentation mask under the perspective to generate an image group as the image input of the multi-modal large language model CLIP, m text statements containing the semantic information and geometric feature information of the bridge structure are taken as the text input of the CLIP, the CLIP obtains the corresponding probability by calculating the cosine similarity of each component image and the prior knowledge text information, and outputs the probability that each mask image belongs to each statement, and finally the prediction probabilities of each segmentation mask under all perspectives are summed to obtain the final score probability of all masks; the text information corresponding to the highest score probability of each segmentation mask contains the component category of the mask, and the final classification result is , and the calculation formula is:

[0024]

[0025] In the formula, is the category corresponding to the component n, is the xth text vector, is the mask image vector of the component n in the i th view, and k is the perspective corresponding to the total number of views; through this process, the text information corresponding to each component segmentation mask is determined, and the classification and identification of the bridge components are realized.

[0026] The present application has the advantages that: the present application only needs to collect bridge point cloud data for analysis and processing, can quickly realize component-level segmentation of bridge point cloud, and realizes rapid classification and identification of components through the segmentation result, so as to quickly and accurately realize component-level instance segmentation, and further classifies the component categories based on the segmentation result, and finally realizes automatic instance segmentation of bridge point cloud; the present application analyzes and processes the bridge point cloud obtained by the laser scanner through a series of methods such as video frame rendering, image segmentation and cosine similarity calculation, can realize accurate and stable instance segmentation of the bridge, and can solve the problems of difficult and inefficient manual segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is the flowchart of the component automatic segmentation and identification method suitable for bridge point cloud of the present application;

[0028] Figure 2 is the effect picture of the present application for realizing point cloud dimension reduction to generate two-dimensional video frames based on multi-view virtual grid visible points;

[0029] Figure 3 is the effect schematic diagram of the present application for segmenting each view image based on visual large model SAM and SAM2;

[0030] Figure 4 is the effect schematic diagram of the present application for bilateral view matching based on the nearest pairing algorithm of the clustering center of the segmentation component;

[0031] Figure 5 is the effect picture of the present application for segmenting each component of the bridge point cloud; wherein, Figure 5 (a) in is the segmentation effect picture before optimization; Figure 5 (b) in is the segmentation effect picture after optimization;

[0032] Figure 6 is the effect picture of the present application for semantic classification of each component of the bridge. DETAILED DESCRIPTION

[0033] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar elements, unless otherwise represented. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims. It is understood that the general description above and the detailed description below are exemplary and explanatory only and are not restrictive of the present application.

[0034] The terms used in the present application are merely for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "an" and "the" as used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein means and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that, although the terms first, second, third, etc. can be employed in this application to describe various information, the information should not be limited to these terms. These terms are only used to differentiate one piece of information from another. For example, without departing from the scope of the application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining" as in this manner. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0036] The application will be described in detail below with reference to the drawings. The features in the following examples and embodiments can be combined with each other without conflict.

[0037] Referring to Figure 1 The component automated segmentation and identification method for bridge point cloud of the application specifically includes the following steps:

[0038] (1) The principal component analysis (PCA) method is used to obtain the main direction of the bridge point cloud, and coordinate transformation is performed to obtain the calibrated bridge point cloud; the calibrated bridge point cloud is divided into three-dimensional grids based on spatial position information, and the index information of the points in each grid is saved; based on the visibility principle when the camera is shooting, the multi-view visible three-dimensional grids are extracted, and they are rendered into two-dimensional video frames, realizing the dimension reduction processing of the three-dimensional point cloud.

[0039] Further, in step (1), the principal component analysis method is used to obtain the principal direction of the bridge point cloud, and coordinate transformation is performed to obtain the calibrated bridge point cloud, which specifically includes: the bridge point cloud is formally represented as a three-dimensional point cloud coordinate matrix P of n rows and 3 columns, for the coordinate matrix P, the point cloud coordinate data therein is first decentralized, then the covariance matrix of the coordinate matrix is calculated, the singular value decomposition of the covariance matrix is performed to obtain three eigenvalues and their corresponding eigenvectors, the eigenvector corresponding to the maximum eigenvalue is taken as the principal direction (i.e. the traffic direction) of the three-dimensional point cloud, the eigenvector corresponding to the minimum eigenvalue is taken as the normal vector of the three-dimensional point cloud, and the eigenvector corresponding to the other eigenvalue is taken as the secondary principal direction of the three-dimensional point cloud, the principal direction is set as the x-axis, the secondary principal direction is set as the y-axis, and the normal vector is set as the z-axis; and the three-dimensional point cloud of the bridge is rotated based on the set x-axis, y-axis and z-axis, and coordinate transformation is performed thereon to obtain the calibrated bridge point cloud.

[0040] Further, in step (1), it is rendered into a two-dimensional video frame, specifically including: using a point cloud to video frame rendering algorithm of a visible three-dimensional grid to render the multi-view visible three-dimensional grid into a two-dimensional video frame to generate each view video frame image, and the calculation formula is:

[0041]

[0042] In the formula, represents the i-th point in the point cloud the included angle between the reference view and the view the corresponding three-dimensional grid index under the view , , and respectively represent the coordinates of the i-th point in the point cloud on the x-axis, y-axis and z-axis, , and respectively represent the indices of the three-dimensional grid on the x-axis, y-axis and z-axis; is the size of the three-dimensional grid; is a rotation matrix, ; is the point cloud coordinate, ; represents an offset vector, , is the point cloud center coordinate, is the minimum value of the point cloud in three coordinate directions. Correspondingly, the minimum value of each group , in the two-dimensional video frame The grid closest to the virtual camera point is the visible point grid, and the remaining grids are invisible point grids. The average RGB value of all points within the visible point grid is calculated as the pixel value of that grid. This ultimately achieves dimensionality reduction of the 3D point cloud, as shown in the image. Figure 2 As shown, Figure 2 This is a rendering of the effect of generating two-dimensional video frames by reducing the dimensionality of point clouds based on visible points of a multi-view virtual mesh.

[0043] It should be understood that the visible 3D mesh point cloud to video frame rendering algorithm is an existing technology that converts dynamic 3D point cloud data into continuous video frames. Its core lies in efficiently utilizing mesh visibility information to optimize the rendering process.

[0044] (2) The main viewpoint of the two-dimensional video frame is used as the input of the visual large model SAM (Segment Anything Model, abbreviated as SAM) to obtain the initial segmentation mask of each component. The prompt box generated by each segmentation mask is used together with the two-dimensional video frame as the input of the SAM2 (Segment Anything in Images and Videos, abbreviated as SAM2) model to obtain the segmentation mask of each component under each viewpoint.

[0045] It should be understood that SAM is a general-purpose image segmentation model designed to achieve fast and flexible zero-shot image segmentation tasks. It is a fundamental model in the field of computer vision, capable of segmenting any object in an image through simple user interaction without requiring training for a specific task. SAM2 is an extension of SAM in the video domain, aiming to transfer SAM's powerful zero-shot image segmentation capabilities to video, enabling frame-by-frame or cross-frame segmentation of objects in videos. This approach combines SAM's general segmentation capabilities with video temporal information.

[0046] Specifically, the visual large model (SAM) is used to analyze the baseline view. That is, global segmentation is performed on the main view of the 2D video frame (i.e., the 2D video frame with a 0° view) to obtain the initial segmentation mask for each component of the bridge. This includes a background mask that does not belong to any component of the bridge. Because the image is rendered based on point clouds, the background is a single black. Therefore, the background mask can be removed based on the RGB values ​​of the pixels in the mask, and finally the initial segmentation mask of each component of the bridge after the background mask is removed is obtained. This represents the initial segmentation mask for the first component. This represents the initial segmentation mask for the second component. This represents the initial segmentation mask for the nth component. For each of the bridge components' initial segmentation masks, the pixel coordinates of the top-left and bottom-right corners are selected to form a coordinate pair. ,in a coordinate pair representing the first component, a coordinate pair representing the second component, a coordinate pair representing the nth component, and respectively represent the upper left corner and the lower right corner pixel coordinates. The coordinate pairs obtained above are used to generate a bounding box A bounding box is generated; in addition, in order to balance the segmentation performance and the calculation speed of SAM2 on the video, the least number of rendered images is selected to achieve segmentation, and the two-dimensional video frame images obtained by rendering at each viewing angle are divided into two groups according to the rotation angle The two groups are respectively synthesized into video frames, and the video frames and the bounding box are input into the SAM2 model for segmentation, so that the component segmentation mask at each viewing angle The viewing angles are 0°, 15°, 165°, 180°, 195°, and 345°. The final segmentation effect is shown in Figure 3

[0047] In some other embodiments, new prompt points can also be generated according to the segmentation mask and input into the SAM model for further segmentation. The index information of the points contained in the pixel coordinate pair can determine the visible points and the invisible points under each mask.

[0048] (3) Project the segmentation results at the diagonal viewing angles obtained in step (2), match the segmentation results of the same components on both sides based on the nearest pairing algorithm of the clustering centers of the segmented components, calculate the scores of the index points according to the segmentation masks at each viewing angle, and realize the component-level segmentation of the bridge point cloud.

[0049] In this embodiment, the segmentation results at the diagonal viewing angles are projected, and the segmentation results of the same components on both sides are matched based on the nearest pairing algorithm of the clustering centers of the segmented components. Specifically, first, all the visible points at 0° and 180° viewing angles are extracted and projected onto the xOz plane, and the centers of the visible points corresponding to different masks after projection at 0° and 180° viewing angles are calculated and , wherein is the center of the visible points corresponding to each component mask at 0° viewing angle, is the center of the visible points corresponding to each component mask at 180° viewing angle. Then, for each center at 0° viewing angle, the nearest center coordinates at 180° viewing angle are found and paired, and the center coordinate index at 180° viewing angle is modified based on the category index of the mask at 0° viewing angle, so that the results obtained by the segmentation of the two groups of images can be associated, and the matching effect is shown in Figure 4 . The category scores are then counted to integrate the segmentation results at different viewing angles. For the mask​​ For each point p in the corresponding bridge point cloud, the score formula is:

[0050]

[0051]

[0052]

[0053] In the formula, is the final score of point p belonging to category n, k is the number of view angles, is the score of the point under the view angle is the point under the view angle represents that the point is a visible point, represents that the point is an invisible point. Through the above formula, the score matrix H of each point p in the bridge point cloud for each category can be calculated, and the category with the highest score is selected as the category of the point, so that the component-level segmentation of the bridge point cloud can be realized.

[0054] (4) The component-level segmentation result obtained in step (3) is optimized based on an error point iterative repair algorithm, as shown in Figure 5 .

[0055] It should be noted that the component-level segmentation result of the bridge point cloud obtained in step (3) has high segmentation accuracy, but there are still some points that are incorrectly segmented. These points can be divided into three categories: points that are not classified because they are not contained in any mask , points with the same score of multiple categories , and a small number of points with segmentation errors at the edges of bridge components , as shown in (a) of Figure 5 , wherein the first and second types of points with segmentation errors are represented in black. and can be easily obtained according to the score matrix H. By observation, it can be found that most will be in a state of separation from the main body of the category. The point cloud of each category can be clustered by DBSCAN, and all small clusters other than the largest cluster are removed from the category to obtain . Therefore, in this embodiment, the optimization of the segmentation result is realized based on an error point iterative repair algorithm, and the segmentation effect diagram after optimization is shown in (b) of Figure 5 .

[0056] Specifically, based on the component-level segmentation result obtained in step (3), for the three types of points with incorrect segmentation , ,​​​ Under the normal vector threshold constraint, the class to which the most points in the radius range of each point is calculated by iteration from small to large, and the class is taken as the class of the error segmentation point; wherein and respectively represent the minimum radius at the beginning of iteration and the maximum radius at the final iteration.

[0057] It should be understood that the normal vector of the plane is obtained by plane fitting using the points within a certain neighborhood radius of each error point. The normal vector indicates that the two points are more likely to be on the same plane, i.e., more likely to belong to the same bridge component. The normal vector threshold constraint means that only when the normal vectors of the error point and the surrounding points are similar, the subsequent radius range iteration optimization process is performed.

[0058] (5) Based on the optimized component-level segmentation result obtained in step (4), pixel points are extracted using the component segmentation mask coordinates of each view, blank pixels are added around the component segmentation mask, component images are generated, the cosine similarity between each component image and prior knowledge text information is calculated based on the multi-modal large language model CLIP (Contrastive Language-Image Pre-training, CLIP), and the classification and recognition of bridge components are realized.

[0059] Specifically, based on the optimized component-level segmentation result obtained in step (4), the component segmentation mask of each component in each view is added with a specified width of blank pixels to generate an image group. As the image input of the multi-modal large language model CLIP, m text sentences containing the semantic information and geometric feature information of the bridge structure are taken as the text input of the CLIP. The CLIP obtains the corresponding probability by calculating the cosine similarity between each component image and the prior knowledge text information, and outputs the probability of each mask image belonging to each sentence. Finally, the prediction probabilities of each segmentation mask M under all views are summed to obtain the final score probability of all masks. The highest score probability of each segmentation mask corresponds to the component category of the mask, and the final classification result is

[0060]

[0061] In the formula, is the category corresponding to component n, is the xth text vector, is the mask image vector of component n in the ith view, and k is the view ​​​​The corresponding total number of views. Through this process, the corresponding text information of each component segmentation mask can be determined. According to the association between the mask and the points in the point cloud, the semantic recognition of the bridge component in the point cloud is realized.

[0062] To sum up, the present application only needs to collect bridge point cloud data for analysis and processing, which can quickly realize component-level segmentation of bridge point cloud, and realize rapid classification and identification of components through the segmentation result, so as to quickly and accurately realize instance segmentation; The present application analyzes and processes the bridge point cloud obtained by the laser scanner through a series of methods such as video frame rendering, image segmentation, cosine similarity calculation, etc., which can realize accurate and stable instance segmentation of the bridge, and can solve the problems of difficult and inefficient manual segmentation.

[0063] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for automated segmentation identification of components suitable for bridge point clouds, characterized in that, The method comprises the following steps: (1) obtaining the main direction of the bridge point cloud by using the principal component analysis method, and performing coordinate transformation to obtain the calibrated bridge point cloud; The calibrated bridge point cloud is divided into three-dimensional grids based on spatial position information, and the index information of the points in each grid is saved. Based on the visibility principle when the camera is shooting, the multi-view visible three-dimensional grid is extracted, and it is rendered into a two-dimensional video frame to realize the dimension reduction processing of the three-dimensional point cloud; (2) taking the main view angle of the two-dimensional video frame as the input of the visual large model SAM to obtain the initial segmentation mask of each component, and taking the prompt box generated from each segmentation mask together with the two-dimensional video frame as the input of the SAM2 model to obtain the segmentation mask of each component under each view angle; (3) projecting the segmentation result under the diagonal view angle obtained in step (2), matching the segmentation results of the same components on both sides based on the nearest pairing algorithm of the clustering centers of the segmented components, and calculating the scores of the index points according to the segmentation masks under each view angle to realize the component-level segmentation of the bridge point cloud; (4) optimizing the component-level segmentation result obtained in step (3) based on the error point iterative repair algorithm; (5) based on the optimized component-level segmentation result obtained in step (4), extracting pixel points using the segmentation mask coordinates of each view angle, adding blank pixels around to generate component images, and calculating the cosine similarity between each component image and the prior knowledge text information based on the multi-modal large language model CLIP to realize the classification and identification of the bridge components; the step (5) specifically comprises: Based on the optimized component-level segmentation result obtained in step (4), the following steps are performed: An image group is generated by adding a specified width of blank pixel points to the periphery of each component segmentation mask under the perspective, which is used as the image input of the multi-modal large language model CLIP. m text statements containing semantic information and geometric feature information of the bridge structure are used as the text input of the CLIP. The CLIP calculates the cosine similarity between each component image and the prior knowledge text information to obtain the corresponding probability, and outputs the probability that each mask image belongs to each statement. Finally, the prediction probabilities of each segmentation mask under all perspectives are summed to obtain the final score probability of all masks. The text information corresponding to the highest score probability of each segmentation mask contains the component category of the mask, and the final classification result is The calculation formula is: In the formula, is the category corresponding to component n, is the xth text vector, is the mask image vector of component n in the ith view, and k is the view angle corresponding to the total number of views; Through this process, the text information corresponding to each component segmentation mask is determined, and the classification and identification of the bridge components are realized.

2. The method for automated segmentation identification of components suitable for bridge point clouds of claim 1, wherein, In the step (1), the principal component analysis method is used to obtain the main direction of the bridge point cloud, and coordinate transformation is performed to obtain the calibrated bridge point cloud, which specifically comprises: The bridge point cloud is formally represented as a three-dimensional point cloud coordinate matrix P of n rows and 3 columns. For the coordinate matrix P, the point cloud coordinate data is first decentralized, then the covariance matrix of the coordinate matrix is calculated, the singular value decomposition of the covariance matrix is performed, three eigenvalues and their corresponding eigenvectors are obtained, the eigenvector corresponding to the maximum eigenvalue is taken as the main direction of the three-dimensional point cloud, the eigenvector corresponding to the minimum eigenvalue is taken as the normal vector of the three-dimensional point cloud, and the eigenvector corresponding to the other eigenvalue is taken as the secondary main direction of the three-dimensional point cloud. The main direction is set as the x-axis, the secondary main direction is set as the y-axis, and the normal vector is set as the z-axis. The three-dimensional point cloud of the bridge is rotated based on the set x-axis, y-axis and z-axis, and coordinate transformation is performed to obtain the calibrated bridge point cloud.

3. The method for automated segmentation identification of components suitable for bridge point clouds of claim 1, wherein, In the step (1), the two-dimensional video frame is rendered, specifically comprising: The multi-view visible three-dimensional grid is rendered into a two-dimensional video frame using a three-dimensional grid point cloud to video frame rendering algorithm to generate video frame images under each view angle, and the calculation formula is: wherein, represents the i-th point in the point cloud the angle between the reference view angle and the view angle of the i-th point in the point cloud the corresponding three-dimensional grid index under the view angle of , , and respectively represent the coordinates of the i-th point in the point cloud on the x-axis, y-axis and z-axis, , and respectively represent the indices of the three-dimensional grid on the x-axis, y-axis and z-axis; is the size of the three-dimensional grid; is a rotation matrix, ; is the point cloud coordinate, ; represents the offset vector, , is the point cloud center coordinate, is the minimum value of the point cloud in the three coordinate directions; correspondingly, the minimum value of each group , in the two-dimensional video frame the grid corresponding to the grid distance closest to the virtual camera point is the visible point grid, and the remaining grids are the invisible point grids, and the RGB average value of all points in the visible point grid is calculated as the pixel value of the grid.

4. The method for automated segmentation identification of components suitable for bridge point clouds of claim 1, wherein, The step (2) specifically comprises: The Visual Acuity Model (SAM) is used to globally segment the reference view (i.e., the main viewpoint of the 2D video frame) to obtain initial segmentation masks for each bridge component. The background mask is then removed based on the RGB values ​​of the pixels in the mask, resulting in initial segmentation masks for each bridge component. Coordinate pairs are formed by selecting the top-left and bottom-right pixel coordinates of each component's initial segmentation mask, and these coordinate pairs are used to generate tooltips. The rendered 2D video frame images from each viewpoint are then rotated according to the specified angles. The video frames are divided into two groups clockwise and synthesized separately. Together with the prompt box, they are input into the SAM2 model for segmentation to obtain the final result. The component segmentation mask from the perspective of each component, among which The viewing angles are 0°, 15°, 165°, 180°, 195°, and 345°.

5. The method for automated segmentation identification of components suitable for bridge point clouds of claim 1, wherein, The step (3) specifically comprises: Firstly, all visible points under 0° and 180° view angles are extracted and projected to the xOz plane, and the centers of the visible points corresponding to different masks after projection under 0° and 180° view angles are calculated, wherein the visible points are determined according to the index information of the points contained in the pixel coordinates; then for each center under 0° view angle, the nearest center coordinate under 180° view angle is found and paired, and the center coordinate index under 180° view angle is modified based on the category index of the mask under 0° view angle; then the category score is counted to integrate the segmentation results under different view angles, for each point in the bridge point cloud corresponding to the mask, the score matrix of each point to each category is calculated, and the category with the highest score is selected as the category of the point, so as to realize the component-level segmentation of the bridge point cloud.

6. The method for automated segmentation identification of components suitable for bridge point clouds of claim 1, wherein, The step (4) specifically comprises: Based on the component-level segmentation results obtained in step (3), for the incorrectly segmented points, under the normal vector threshold constraint, according to the radius interval... Iteratively calculate the category of the most frequent point within the radius r of each point, from smallest to largest, and use that category as the category of the incorrectly segmented point; where and These represent the minimum radius at the start of the iteration and the maximum radius at the end of the iteration, respectively.

Images