An Interactive Bridge Point Cloud Semantic Segmentation Method Based on a Large Visual Model

Abstract

This invention discloses an interactive bridge point cloud semantic segmentation method based on a large visual model, comprising the following steps: 1) Projecting the bridge point cloud data into a sequence of N RGB images, recording the projection relationship between the bridge point cloud data and the image pixels; 2) Segmenting the RGB image sequence to obtain the corresponding segmentation mask sequence; 3) Tracing the RGB image sequence back to three-dimensional space according to the projection relationship, and extracting the three-dimensional point cloud index corresponding to each image to obtain N three-dimensional point cloud segmentation results; 4) Processing the N point cloud segmentation results to obtain preliminary segmented point clouds; 5) Based on the geometric features and spatial location of the bridge, using multi-plane spatial relationships to filter the spatial location of the preliminary segmented point clouds to obtain the final bridge component segmentation results. This invention effectively solves the problem of automatic segmentation and extraction of key components from various types of bridge point cloud data, providing reliable semantic information and high-precision segmented point cloud data for bridge three-dimensional reverse modeling.

Description

Technical Field

[0001] This invention relates to the fields of information technology and bridge safety management, specifically an interactive bridge point cloud semantic segmentation method based on a large visual model. Background Technology

[0002] Bridges are a crucial component of infrastructure, playing a vital role in transportation and socio-economic development. However, with the passage of time and the impact of the natural environment, bridges may experience wear and tear and defects, leading to a decline in safety and service performance. Therefore, continuous and effective health monitoring and maintenance of bridges are essential. Traditional bridge health assessment and maintenance primarily rely on manual inspections and routine measurement techniques. However, these methods are time-consuming, labor-intensive, unable to cover comprehensive areas, and cannot obtain high-precision structural information. In recent years, 3D laser scanning technology has developed rapidly, enabling efficient acquisition of high-resolution 3D, color, and texture information of scanned objects. However, point cloud data does not contain semantic information about the scanned objects. Therefore, reliable semantic segmentation of bridge point cloud data to obtain high-precision information on various structural components is a crucial prerequisite for bridge health monitoring, operation, and maintenance applications. Existing bridge point cloud segmentation methods are either manual or require manually creating large datasets to train neural networks for segmentation, which is not only time-consuming and labor-intensive but also has poor generalization ability. Therefore, proposing a general zero-shot semantic segmentation method is highly necessary. Summary of the Invention

[0003] The purpose of this invention is to provide an interactive bridge point cloud semantic segmentation method based on a large visual model, comprising the following steps:

[0004] 1) Acquire bridge point cloud data and project the bridge point cloud data into a sequence of N RGB images, and record the projection relationship between the bridge point cloud data and the image pixels;

[0005] 2) Use image semantic annotation software to obtain the mask of the component to be segmented from the first image and use it as the initial cue;

[0006] 3) Employ cross-perspective feedback enhancement strategies to generate feedback information;

[0007] 4) The RGB image sequence is segmented sequentially using prompts and a large visual model to obtain the corresponding segmentation mask sequence;

[0008] 5) Based on the segmentation mask, the RGB image sequence is backtracked to the three-dimensional space according to the projection relationship, and the three-dimensional point cloud index corresponding to each image is extracted to obtain N three-dimensional point cloud segmentation results;

[0009] 6) Process the N point cloud segmentation results to obtain a preliminary segmented point cloud;

[0010] 7) Based on the geometric features and spatial location of the bridge, the spatial location of the preliminary segmented point cloud is filtered using multi-plane spatial relationships to obtain the final segmentation results of the bridge components.

[0011] Furthermore, in the RGB image sequence, adjacent numbered images partially overlap.

[0012] Furthermore, the step of projecting the bridge point cloud data into a sequence of N RGB images includes:

[0013] 1.1) Using the point cloud data center as the center of rotation, rotate 360° / N around the Z-axis each time; after each rotation, project the point cloud data onto the XZ plane, with a projection range of (x... max -x min )×(z max -z min );x max x min These represent the upper and lower limits of the projection range on the X-axis; z max z min These are the upper and lower limits of the projection range on the Z-axis;

[0014] 1.2) Expand the projection range by 0.1*(x) to the left and right respectively. max -x min Expand by 0.1*(z) both above and below. max -z min This yields the final RGB image with a size of 1.2*(x). max -x min )×1.2*(z max -z min );

[0015] 1.3) Set the RGB image resolution to the set grid size v.

[0016] 1.4) Map the point cloud data to the corresponding image pixels and record the correspondence between the point cloud and the pixels, where the 3D point (x,y,z) corresponds to the pixel.

[0017] Furthermore, when each pixel corresponds to multiple 3D points, the average RGB value of all 3D points in the same pixel is taken as the RGB value of that pixel.

[0018] Furthermore, the steps for generating prompt information using a cross-perspective prompt enhancement strategy include:

[0019] 3.1) Calculate the mask pixel p in the initial prompt. init ;

[0020] 3.2) Calculate the segmentation mask pixel p of the previous image.i-1 ;

[0021] 3.3) If The initial prompt and the previous segmented image are used as the current prompt simultaneously. i Otherwise, use the initial prompt p. init As the current prompt message promt i .

[0022] Furthermore, the steps for segmenting the RGB image sequence include:

[0023] 4.1) Based on the segmentation of bridge component types, image semantic annotation software is used to annotate the segmentation targets in the first image of the RGB image sequence and obtain the corresponding segmentation mask to achieve the segmentation of the first image, which is then used as the initial prompt.

[0024] 4.2) A cross-view cue enhancement strategy is adopted to generate cue information. The images in the image sequence are segmented using a large visual model to obtain the projected image I. i The segmentation mask; i = 2, 3, ..., N;

[0025] 4.3) Repeat step 4.2) to complete the segmentation of the N RGB image sequence and obtain the corresponding segmentation mask sequence.

[0026] Furthermore, all point clouds corresponding to the segmentation mask sequence are extracted, and the point clouds that appear more than k times are used as the initial segmentation point clouds.

[0027] Furthermore, the steps for spatial location filtering of the initially segmented point cloud using multi-planar spatial relationships include:

[0028] 7.1) Use a plane fitting algorithm to fit the preliminary segmented point cloud to obtain multiple fitting planes; the plane fitting algorithm includes, but is not limited to, the RANSAC algorithm and the least squares algorithm;

[0029] 7.2) Based on the geometric features and spatial location of the bridge components, the planar location is filtered using the fitted plane, and noise points are removed to obtain the final bridge component segmentation result.

[0030] Furthermore, when there are multiple types of segmented bridge components, these types of segmented bridge components are sorted according to the hierarchical relationship between the components. First, the segmentation result of the bridge component corresponding to the j-th type of segmented bridge component is obtained through the interactive bridge point cloud semantic segmentation method. Then, the point cloud data corresponding to this type of segmented bridge component is deleted, and the interactive bridge point cloud semantic segmentation method is repeated until the segmentation result of each type of segmented bridge component is obtained.

[0031] Furthermore, the large visual model includes, but is not limited to, SegGPT, SAM, and SEEM models.

[0032] The technical effects of this invention are undeniable. This invention proposes a general semantic segmentation method for bridge point cloud data based on the SegGPT model. This method is based on a cross-view cue enhancement strategy and uses the visual large model SegGPT to achieve semantic segmentation of point cloud images from multiple views without sample annotation learning. It completes the automatic semantic segmentation and extraction of key components in the bridge system, effectively solving the problem of automatic segmentation and extraction of key components in point cloud data of various types of bridges. It provides reliable semantic information and high-precision segmented point cloud data for 3D reverse modeling of bridges, and thus provides reliable as-built model information for applications such as bridge engineering health monitoring, operation and maintenance. Attached Figure Description

[0033] Figure 1 A flowchart illustrating an embodiment of the present invention.

[0034] Figure 2 A schematic diagram illustrating the bridge system and key component classification of an embodiment of the present invention.

[0035] Figure 3 Example diagram of input bridge point cloud data for an embodiment of the present invention.

[0036] Figure 4 (a)-(j) are example projection images of beam bridges in embodiments of the present invention.

[0037] Figure 5 This is a schematic diagram illustrating the generation of a segmentation mask based on Labelme from the first projected image of the beam bridge in an embodiment of the present invention.

[0038] Figure 6 (a)-(j) are example diagrams of the segmentation mask corresponding to the projection image of the beam bridge in the embodiments of the present invention.

[0039] Figure 7 (a)-(j) are example images of point cloud segmentation results obtained by backtracking the projected image of the beam bridge to three-dimensional space in an embodiment of the present invention.

[0040] Figure 8 This is a schematic diagram of the hard voting results for the bridge piers of a beam bridge in an embodiment of the present invention.

[0041] Figure 9 This is an example diagram showing the hard voting results and final segmentation results of different components of beam bridges and suspension bridges in an embodiment of the present invention.

[0042] Figure 10 For the SegGPT training framework;

[0043] Figure 11 For context integration strategy;

[0044] Figure 12This is the In-Context Tuning method. Detailed Implementation

[0045] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.

[0046] Example 1:

[0047] See Figures 1 to 12 An interactive bridge point cloud semantic segmentation method based on a large visual model includes the following steps:

[0048] 1) Acquire interactive bridge point cloud data and project the bridge point cloud data into a sequence of N RGB images, and record the projection relationship between the bridge point cloud data and the image pixels;

[0049] 2) Use image semantic annotation software (such as Labelme, LabelIMG, etc.) to obtain the mask of the component to be segmented from the first image and use it as the initial prompt;

[0050] 3) Employ cross-perspective feedback enhancement strategies to generate feedback information;

[0051] 4) The RGB image sequence is segmented sequentially using prompts and a large visual model to obtain the corresponding segmentation mask sequence;

[0052] 5) Based on the segmentation mask, the RGB image sequence is backtracked to the three-dimensional space according to the projection relationship, and the three-dimensional point cloud index corresponding to each image is extracted to obtain N three-dimensional point cloud segmentation results;

[0053] 6) Process the N point cloud segmentation results to obtain a preliminary segmented point cloud;

[0054] 7) Based on the geometric features and spatial location of the bridge, the spatial location of the preliminary segmented point cloud is filtered using multi-plane spatial relationships to obtain the final segmentation results of the bridge components.

[0055] In an RGB image sequence, adjacent numbered images partially overlap.

[0056] The steps to project bridge point cloud data into a sequence of N RGB images include:

[0057] 1.1) Using the point cloud data center as the center of rotation, rotate 360° / N around the Z-axis each time; after each rotation, project the point cloud data onto the XZ plane, with a projection range of (x... max -x min )×(z max -zmin );x max x min These represent the upper and lower limits of the projection range on the X-axis; z max z min These are the upper and lower limits of the projection range on the Z-axis;

[0058] 1.2) Expand the projection range by 0.1*(x) to the left and right respectively. max -x min Expand by 0.1*(z) both above and below. max -z min This yields the final RGB image with a size of 1.2*(x). max -x min )×1.2*(z max -z min );

[0059] 1.3) Set the RGB image resolution to the set grid size v.

[0060] 1.4) Map the point cloud data to the corresponding image pixels and record the correspondence between the point cloud and the pixels, where the 3D point (x,y,z) corresponds to the pixel.

[0061] When each pixel corresponds to multiple 3D points, the average RGB value of all 3D points in the same pixel is taken as the RGB value of that pixel.

[0062] The steps for generating prompt information using a cross-perspective prompt enhancement strategy include:

[0063] 3.1) Calculate the mask pixel p in the initial prompt. init ;

[0064] 3.2) Calculate the segmentation mask pixel p of the previous image. i-1 ;

[0065] 3.3) If The initial prompt and the previous segmented image are used as the current prompt simultaneously. i Otherwise, use the initial prompt p. init As the current prompt message promt i .

[0066] The steps for segmenting an RGB image sequence include:

[0067] 4.1) Based on the segmentation of bridge component types, image semantic annotation software is used to annotate the segmentation targets in the first image of the RGB image sequence and obtain the corresponding segmentation mask to achieve the segmentation of the first image, which is then used as the initial prompt.

[0068] 4.2) A cross-view cue enhancement strategy is adopted to generate cue information. The images in the image sequence are segmented using a large visual model to obtain the projected image I. i The segmentation mask; i = 2, 3, ..., N;

[0069] 4.3) Repeat step 4.2) to complete the segmentation of the N RGB image sequence and obtain the corresponding segmentation mask sequence.

[0070] v Extract all point clouds corresponding to the segmentation mask sequence, and count the point clouds that appear more than k times as the initial segmentation point clouds.

[0071] The steps for spatial location filtering of the preliminary segmented point cloud using multi-planar spatial relationships include:

[0072] 7.1) Use a plane fitting algorithm to fit the preliminary segmented point cloud to obtain multiple fitting planes; the plane fitting algorithm includes, but is not limited to, the RANSAC algorithm and the least squares algorithm;

[0073] 7.2) Based on the geometric features and spatial location of the bridge components, the planar location is filtered using the fitted plane, and noise points are removed to obtain the final bridge component segmentation result.

[0074] When there are multiple types of segmented bridge components, these types of segmented bridge components are sorted according to the hierarchical relationship between the components. First, the segmentation result of the bridge component corresponding to the j-th type of segmented bridge component is obtained through the interactive bridge point cloud semantic segmentation method. Then, the point cloud data corresponding to the segmented bridge component of that type is deleted, and the interactive bridge point cloud semantic segmentation method is repeated until the segmentation result of each type of segmented bridge component is obtained.

[0075] The large visual model includes, but is not limited to, SegGPT, SAM, and SEEM models.

[0076] Example 2:

[0077] An interactive bridge point cloud semantic segmentation method based on a large visual model includes the following steps:

[0078] 1) Acquire bridge point cloud data and project the bridge point cloud data into a sequence of N RGB images, and record the projection relationship between the bridge point cloud data and the image pixels;

[0079] 2) Use image semantic annotation software to obtain the mask of the component to be segmented from the first image and use it as the initial cue;

[0080] 3) Employ cross-perspective feedback enhancement strategies to generate feedback information;

[0081] 4) The RGB image sequence is segmented sequentially using prompts and a large visual model to obtain the corresponding segmentation mask sequence;

[0082] 5) Based on the segmentation mask, the RGB image sequence is backtracked to the three-dimensional space according to the projection relationship, and the three-dimensional point cloud index corresponding to each image is extracted to obtain N three-dimensional point cloud segmentation results;

[0083] 6) Process the N point cloud segmentation results to obtain a preliminary segmented point cloud;

[0084] 7) Based on the geometric features and spatial location of the bridge, the spatial location of the preliminary segmented point cloud is filtered using multi-plane spatial relationships to obtain the final segmentation results of the bridge components.

[0085] Example 3:

[0086] An interactive bridge point cloud semantic segmentation method based on a large visual model, with the same technical content as in Embodiment 2, further wherein adjacent sequence images partially overlap in the RGB image sequence.

[0087] Example 4:

[0088] An interactive bridge point cloud semantic segmentation method based on a large visual model, with the same technical content as any one of embodiments 2-3, further comprising the step of projecting the bridge point cloud data into a sequence of N RGB images, including:

[0089] 1.1) Using the point cloud data center as the center of rotation, rotate 360° / N around the Z-axis each time; after each rotation, project the point cloud data onto the XZ plane, with a projection range of (x... max -x min )×(z max -z min );x max x min These represent the upper and lower limits of the projection range on the X-axis; z max z min These are the upper and lower limits of the projection range on the Z-axis;

[0090] 1.2) Expand the projection range by 0.1*(x) to the left and right respectively. max -x min Expand by 0.1*(z) both above and below. max -z min This yields the final RGB image with a size of 1.2*(x). max -x min )×1.2*(z max -z min );

[0091] 1.3) Set the RGB image resolution to the set grid size v.

[0092] 1.4) Map the point cloud data to the corresponding image pixels and record the correspondence between the point cloud and the pixels, where the 3D point (x,y,z) corresponds to the pixel.

[0093] Example 5:

[0094] An interactive bridge point cloud semantic segmentation method based on a large visual model, with the same technical content as any one of embodiments 2-4, further wherein when each pixel corresponds to multiple three-dimensional points, the average RGB value of all three-dimensional points in the same pixel is taken as the RGB value of that pixel.

[0095] Example 6:

[0096] An interactive bridge point cloud semantic segmentation method based on a large visual model, with technical content identical to any one of embodiments 2-5, further comprising the step of generating prompt information using a cross-view prompt enhancement strategy, including:

[0097] 3.1) Calculate the mask pixel p in the initial prompt. init ;

[0098] 3.2) Calculate the segmentation mask pixel p of the previous image. i-1 ;

[0099] 3.3) If The initial prompt and the previous segmented image are used as the current prompt simultaneously. i Otherwise, use the initial prompt p. init As the current prompt message promt i .

[0100] Example 7:

[0101] An interactive bridge point cloud semantic segmentation method based on a large visual model, with the same technical content as any one of embodiments 2-6, further comprising the following steps for segmenting the RGB image sequence:

[0102] 4.1) Based on the segmentation of bridge component types, image semantic annotation software is used to annotate the segmentation targets in the first image of the RGB image sequence and obtain the corresponding segmentation mask to achieve the segmentation of the first image, which is then used as the initial prompt.

[0103] 4.2) A cross-view cue enhancement strategy is adopted to generate cue information. The images in the image sequence are segmented using a large visual model to obtain the projected image I. i The segmentation mask; i = 2, 3, ..., N;

[0104] 4.3) Repeat step 4.2) to complete the segmentation of the N RGB image sequence and obtain the corresponding segmentation mask sequence.

[0105] Example 8:

[0106] An interactive bridge point cloud semantic segmentation method based on a large visual model, with the same technical content as any one of embodiments 2-7, further comprising: extracting all point clouds corresponding to the segmentation mask sequence, and counting the point clouds that appear more than k times as the initial segmented point clouds.

[0107] Example 9:

[0108] An interactive bridge point cloud semantic segmentation method based on a large visual model, with the same technical content as any one of embodiments 2-8, further comprising the step of using multi-plane spatial relationships to filter the spatial location of the initially segmented point cloud, including:

[0109] 7.1) Use a plane fitting algorithm to fit the preliminary segmented point cloud to obtain multiple fitting planes; the plane fitting algorithm includes, but is not limited to, the RANSAC algorithm and the least squares algorithm;

[0110] 7.2) Based on the geometric features and spatial location of the bridge components, the planar location is filtered using the fitted plane, and noise points are removed to obtain the final bridge component segmentation result.

[0111] Example 10:

[0112] An interactive bridge point cloud semantic segmentation method based on a large visual model is provided. The technical content is the same as any one of embodiments 2-9. Further, when there are multiple types of segmented bridge components, these types of segmented bridge components are sorted according to the hierarchical relationship between the components. First, the segmentation result of the bridge component corresponding to the j-th type of segmented bridge component is obtained through the interactive bridge point cloud semantic segmentation method. Then, the point cloud data corresponding to this type of segmented bridge component is deleted, and the interactive bridge point cloud semantic segmentation method is repeated until the segmentation result of each type of segmented bridge component is obtained.

[0113] Example 11:

[0114] An interactive bridge point cloud semantic segmentation method based on a large visual model, with the same technical content as any one of embodiments 2-10. Furthermore, the large visual model includes, but is not limited to, SegGPT, SAM, and SEEM models.

[0115] Example 12:

[0116] An interactive bridge point cloud semantic segmentation method based on a large visual model, with the same technical content as any one of embodiments 2-11, further wherein the large visual model is a SegGPT model.

[0117] like Figure 10 As shown, SegGPT is a general-purpose image segmentation model that can automatically adapt to different types of data and can be trained without a large amount of labeled data.

[0118] SegGPT applies the generative pre-trained model GPT-3 from the field of natural language processing to image segmentation tasks. The model employs a context learning framework to unify various segmentation tasks and uses In-Context Coloring for training. In-Context Coloring uses random color mappings to accomplish different tasks and forces the model to refer to contextual information to complete a specified task, rather than relying on specific colors.

[0119] Meanwhile, SegGPT employs a context ensemble strategy. Figure 11 The Context Ensemble aims to enhance a model's representational capabilities by leveraging multiple layers of contextual information. Specifically, it comprises two parts: spatial ensemble and feature ensemble. Spatial ensemble captures contextual information at different scales by fusing feature maps from different scales. Feature ensemble, on the other hand, captures contextual information at different levels by fusing features from different layers.

[0120] Then, SegGPT uses the In-Context Tuning method ( Figure 12 This method fine-tunes a pre-trained model to adapt it to a specific task. It uses learnable cues to indicate different tasks and can be conveniently used as an expert model to adapt to specific use cases by adjusting specific cues without updating model parameters. Specifically, the method defines the task cues as learnable tensors, freezes the entire model, and then uses the same training loss to optimize the task cues.

[0121] Example 13:

[0122] An interactive bridge point cloud semantic segmentation method based on a large visual model, comprising the following steps:

[0123] S101. Project the input bridge point cloud data into a sequence of N RGB images, with a certain degree of overlap between adjacent numbered images, and record the projection relationship between the point cloud and the image pixels.

[0124] S102. Based on the user-selected bridge component segmentation type, label the segmentation targets in the first image of the projected image sequence, and read the JSON file to obtain the corresponding segmentation mask; combine the segmentation mask obtained from the previous image with the segmentation mask of the first image as prompt information, and use the visual large model to segment the images in the image sequence to obtain the projected image I. iThe segmentation mask is obtained; the operation is repeated until the image sequence is segmented and the corresponding segmentation mask sequence is obtained.

[0125] S103. For each image in the projection sequence, according to its corresponding segmentation mask, backtrack to the three-dimensional space according to the projection relationship recorded in step S101, extract the three-dimensional point cloud index corresponding to each image, and obtain the three-dimensional point cloud segmentation result for each angle.

[0126] S104. Perform "hard voting" on the N point cloud segmentation results obtained in step S103 to obtain preliminary point cloud segmentation results;

[0127] S105. Based on the geometric features and spatial location of the segmentation target specified in step S101, the preliminary segmentation point cloud obtained in step S104 is used to filter the spatial location using multi-plane spatial relationships to obtain the final bridge component segmentation result.

[0128] S106. If the user has multiple segmentation targets, then according to the hierarchical relationship between components, delete the segmentation results obtained in step S105 to obtain the remaining point cloud data, delete redundant noise points, and obtain the final target segmentation result; repeat steps S102 to S105 based on the remaining point cloud until all segmentation targets are obtained.

[0129] Step S101 includes:

[0130] First, using the point cloud data center as the center of rotation, rotate 360° / N around the Z-axis each time; after each rotation, project the point cloud data onto the XZ plane, with the projection range being (x... max -x min )×(z max -z min Expand the projection range by 0.1*(x) to the left and right respectively. max -x min Expand by 0.1*(z) both above and below. max -z min The final image size is 1.2*(x). max -x min )×1.2*(z max -z min Then, based on the set grid size v, set the image resolution as follows:

[0131]

[0132] Point cloud data is mapped to corresponding image pixels, and the correspondence between point cloud and pixel is recorded, where a 3D point (x, y, z) corresponds to a pixel. Each pixel may correspond to multiple three-dimensional points;

[0133] The RGB value of a pixel is the average of the RGB values ​​of all three-dimensional points within that pixel.

[0134] Step S102 includes:

[0135] The images and prompts in the image sequence are sequentially input into the visual large model to obtain the segmentation mask corresponding to each image. The prompts for each image are the segmentation mask of the previous image. If the number of pixels in the segmentation mask of the previous image is less than 1 / 2 of the segmentation mask of the first image, the prompts for the previous image are considered invalid. In this case, the segmentation mask of the first image is input as the prompt.

[0136] Step S104 includes:

[0137] Based on the 3D segmentation index, point clouds that appear more than k times are selected as the initial segmentation results of the 3D point cloud. Generally, the default number of selections k is N / 2.

[0138] Step S105 includes:

[0139] The RANSAC algorithm is used to fit multiple planes in the hard voting results of step S104; based on the geometric features and spatial relationships of the components, the fitted planes are used to filter the plane positions and remove noise points to obtain the final segmentation target.

[0140] Example 14:

[0141] An interactive bridge point cloud semantic segmentation method based on a large visual model, comprising the following steps:

[0142] S101. Project the input point cloud data into a sequence of N RGB images, with adjacent images having a certain degree of overlap, and record the projection relationship between the point cloud and the image pixels.

[0143] In practice, the point cloud data center is used as the rotation center, and the system rotates 360° / N around the Z-axis each time. After each rotation, the point cloud data is projected onto the XZ plane, with a projection range of (x... max -x min )×(z max -z min Expand the projection range by 0.1*(x) to the left and right respectively. max -x min Expand by 0.1*(z) both above and below. max -z min The final image size is 1.2*(x). max -x min )×1.2*(z max -z min Then, based on the set grid size v, set the image resolution as follows:

[0144]

[0145] Point cloud data is mapped to corresponding image pixels, and the correspondence between point cloud and pixel is recorded, where a 3D point (x, y, z) corresponds to a pixel. Each pixel may correspond to multiple three-dimensional points;

[0146] The RGB value of a pixel is the average of the RGB values ​​of all three-dimensional points within that pixel.

[0147] Examples of this invention are beam bridges and suspension bridges (see...) Figure 3 For beam bridges, the segmentation targets are piers, pier caps, span structures, and retaining structures; for suspension bridges, the segmentation targets are piers, span structures, main cables, suspension cables, and towers. In the implementation example calculation, the projection viewpoint N was set to 10, meaning the point cloud data was rotated 36° for projection each time, ultimately obtaining 10 projected images. The projection results are shown below. Figure 4 .

[0148] S102. Based on the bridge component segmentation type selected by the user, the segmentation targets in the first image of the projected image sequence are labeled with Labelme, and the corresponding segmentation mask is obtained. Using the segmentation mask obtained from the previous image and the segmentation mask of the first image as prompts, the images in the image sequence are segmented using a large visual model to obtain the projected image I. i The segmentation mask is obtained; the operation is repeated until the image sequence is segmented and the corresponding segmentation mask sequence is obtained.

[0149] In practice, the segmentation type of bridge components is selected for each different bridge type. Labelme software is used to manually annotate the segmentation targets in the first image of the sequence and save them as JSON files. Then, the JSON file containing the segmentation target regions is read and converted into a mask for the first image. The second image of the acquired image sequence and the mask of the first image are input into the SegGPT model to obtain the semantic segmentation mask of the second image. Images and prompts in the image sequence are sequentially input into the SegGPT model to obtain the segmentation mask corresponding to each image. The prompt for each image is the segmentation mask of the previous image. If the pixel count of the segmentation mask of the previous image is less than half that of the segmentation mask of the first image, the prompt for the previous image is considered invalid, and the segmentation mask of the first image is used as the prompt.

[0150] In this embodiment of the invention, the first projected image of the beam bridge is generated based on Label annotation and corresponding segmentation mask, as shown below. Figure 5 As shown; in this embodiment of the invention, the 10 segmentation masks corresponding to the 10 projected images of the beam bridge are as follows: Figure 6 As shown.

[0151] S103. For each image in the projection sequence, according to its corresponding segmentation mask, backtrack to the three-dimensional space according to the projection relationship recorded in step S101, and extract the three-dimensional point cloud segmentation result corresponding to each image.

[0152] In practice, the N obtained masks are used to extract the point cloud index corresponding to each segmentation mask pixel according to the projection relationship obtained in step S102. The 2D segmentation result is then backtracked to the three-dimensional space to obtain the three-dimensional segmentation result corresponding to each projected image.

[0153] The segmentation results corresponding to the 10 projected images of the beam bridge in the embodiment of this invention are shown below. Figure 7 .

[0154] S104. Perform "hard voting" on the multiple segmentation point cloud indices obtained in step S103 to obtain preliminary point cloud segmentation results;

[0155] In practice, the N sets of 3D segmentation indices obtained in step S103 are read sequentially. Then, a "hard voting" method is used to filter the N sets of indices, selecting the indices that appear more than k times as the initial segmentation results of the 3D point cloud. The number of filtering times k is N / 2 by default, but can also be set according to user needs.

[0156] In this invention, for the point cloud of bridge piers of a beam bridge, the number of screening iterations k is set to 4, 5, and 6 respectively. The segmentation results corresponding to different number of screening iterations k are shown in [the figure]. Figure 8 .

[0157] S105. Based on the geometric features and spatial relationships of the segmented targets, the segmented point cloud in step S104 is fitted with multiple planes using the RANSAC algorithm. The positions are then filtered and redundant noise is removed based on the planes to obtain the final bridge component segmentation results.

[0158] In practice, based on the geometric features of the segmented target, the RANSAC algorithm is used to fit multiple planes in the preliminary segmented point cloud obtained in step S104. According to the geometric features of the segmented component itself, the position is filtered by the planar spatial relationship. Statistical filtering is used to remove redundant noise. The preliminary point cloud segmentation results are then post-processed to obtain the final segmented point cloud.

[0159] In the embodiments of this invention, the hard voting results and final segmentation results of different components of beam bridges and suspension bridges are shown in the following examples. Figure 9 .

[0160] The above experimental results demonstrate that the interactive bridge point cloud semantic segmentation method based on a large visual model disclosed in this invention can achieve general bridge point cloud semantic segmentation without sample annotation learning, and the method described in this invention is indeed effective.

Claims

1. An interactive bridge point cloud semantic segmentation method based on a large visual model, characterized in that, Includes the following steps: Step 1) Obtain bridge point cloud data and project the bridge point cloud data into a sequence of N RGB images, and record the projection relationship between the bridge point cloud data and the image pixels; Step 2) Use image semantic annotation software to obtain the mask of the component to be segmented from the first image and use it as the initial cue; Step 3) Generate prompt information using a cross-perspective prompt enhancement strategy; Step 4) Use prompts and a large visual model to segment the RGB image sequence sequentially to obtain the corresponding segmentation mask sequence; Step 5) Based on the segmentation mask, backtrack the RGB image sequence to the three-dimensional space according to the projection relationship, and extract the three-dimensional point cloud index corresponding to each image to obtain N three-dimensional point cloud segmentation results; Step 6) Process the N point cloud segmentation results to obtain a preliminary segmented point cloud; Step 7) Based on the geometric features and spatial location of the bridge, the spatial location of the preliminary segmented point cloud is filtered using multi-plane spatial relationships to obtain the final bridge component segmentation results; The steps for generating prompt information using a cross-perspective prompt enhancement strategy include: S1) Calculate the mask pixels in the initial prompt. ; S2) Calculate the segmentation mask pixels of the previous image. ; S3) If Then the initial prompt and the previous segmented image will be used as the current prompt simultaneously. ; Otherwise, use the initial prompt. As the current prompt information .

2. The interactive bridge point cloud semantic segmentation method based on a large visual model according to claim 1, characterized in that, In an RGB image sequence, adjacent numbered images partially overlap.

3. The interactive bridge point cloud semantic segmentation method based on a large visual model according to claim 1, characterized in that, The steps to project bridge point cloud data into a sequence of N RGB images include: Step 1) Using the point cloud data center as the center of rotation, rotate 360° / N around the Z-axis each time; after each rotation, project the point cloud data onto the XZ plane, with the projection range being... ; , These are the upper and lower limits of the projection range on the X-axis; , These are the upper and lower limits of the projection range on the Z-axis; Step 2) Expand the projection range to the left and right respectively. Expand both upwards and downwards The final RGB image is obtained, with an image size of [size missing]. ; Step 3) Set the RGB image resolution to the set grid size v. : Step 4) Map the point cloud data to the corresponding image pixels and record the correspondence between the point cloud and the pixels, where the 3D points... Corresponding to pixels .

4. The interactive bridge point cloud semantic segmentation method based on a large visual model according to claim 3, characterized in that, When each pixel corresponds to multiple 3D points, the average RGB value of all 3D points in the same pixel is taken as the RGB value of that pixel.

5. The interactive bridge point cloud semantic segmentation method based on a large visual model according to claim 1, characterized in that, The steps for segmenting an RGB image sequence include: Step 1) Based on the segmentation bridge component type, use image semantic annotation software to annotate the segmentation targets in the first image of the RGB image sequence, obtain the corresponding segmentation mask, realize the segmentation of the first image, and use it as the initial prompt; Step 2) Generate cue information using a cross-view cue enhancement strategy, and segment the images in the image sequence using a large visual model to obtain the projected images. The segmentation mask; i = 2, 3, ..., N; Step 3) Repeat step 2) to complete the segmentation of the N RGB image sequence and obtain the corresponding segmentation mask sequence.

6. The interactive bridge point cloud semantic segmentation method based on a large visual model according to claim 1, characterized in that, Extract all point clouds corresponding to the segmentation mask sequence, and count the point clouds that appear more than k times as the initial segmentation point clouds.

7. The interactive bridge point cloud semantic segmentation method based on a large visual model according to claim 1, characterized in that, The steps for spatial location filtering of the preliminary segmented point cloud using multi-planar spatial relationships include: Step 1) Fit the preliminary segmented point cloud using a plane fitting algorithm to obtain multiple fitting planes; the plane fitting algorithm includes the RANSAC algorithm and the least squares algorithm; Step 2) Based on the geometric features and spatial location of the bridge components, the planar location is filtered using the fitted plane, and noise points are removed to obtain the final bridge component segmentation result.

8. The interactive bridge point cloud semantic segmentation method based on a large visual model according to claim 1, characterized in that, When there are multiple types of segmented bridge components, these types of segmented bridge components are sorted according to the hierarchical relationship between the components. First, the segmentation result of the bridge component corresponding to the j-th type of segmented bridge component is obtained through the interactive bridge point cloud semantic segmentation method. Then, the point cloud data corresponding to the segmented bridge component of that type is deleted, and the interactive bridge point cloud semantic segmentation method is repeated until the segmentation result of each type of segmented bridge component is obtained.

9. The interactive bridge point cloud semantic segmentation method based on a large visual model according to claim 1, characterized in that, The large visual model includes SegGPT, SAM, and SEEM models.

Images