Bridge structure modeling method based on 3D point cloud processing

Abstract

The present invention discloses a bridge structure modeling method based on three-dimensional point cloud processing, and belongs to the technical field of point cloud modeling. The present invention first obtains a complete surface image of the bridge, segments the bridge part through a segmentation network to make a mask, reconstructs a high-precision point cloud of the bridge using a motion recovery structure method and a multi-view stereo vision method, and then establishes a BIM model based on the point cloud; solves the problem of missing bridge point cloud data due to the occlusion of a ground three-dimensional laser scanner, and solves the problem of poor accuracy in obtaining bridge point cloud data and inability to collect the bottom of the bridge by an airborne laser radar; establishes a point cloud model using a bridge mask through a network segmentation method, without having to rely on manual cropping of environmental point clouds in the later stage, and generates a high-precision point cloud model and imports it into the modeling software for refined modeling; in addition, the present invention can also be migrated to point cloud modeling of other buildings and structures.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud modeling, and in particular to a bridge structure modeling method based on three-dimensional point cloud processing. Background Art

[0002] Point cloud data can reflect the true shape of an object and is widely used in the field of bridge monitoring. However, since most bridges have long spans and high piers, and the environments in which the bridges are located are complex and diverse (such as bridges across rivers, etc.), the ground-based three-dimensional laser scanner has a limited measurement distance and cannot obtain high-precision bridge point cloud data. In addition, due to occlusion and other reasons, it is impossible to obtain point cloud data on the complete surface of the bridge. The method of using airborne laser radar to scan bridges is limited by the load-bearing capacity of the carrier and the accuracy of the scanner. The quality of the bridge point cloud data obtained is often not high and cannot meet the requirements for making high-precision models. However, drones equipped with high-resolution cameras can obtain complete high-definition images of the bridge surface, remove the environment in the image through image segmentation, and only retain the bridge part. This can save subsequent point cloud preprocessing steps such as environmental point cloud cutting, and the algorithm can reconstruct a high-precision, pure point cloud model of a textured bridge, and then produce a BIM model. To this end, the present invention proposes a bridge structure modeling method based on three-dimensional point cloud processing. Summary of the invention

[0003] The technical problem to be solved by the present invention is: how to segment the various parts of the bridge structure through a segmentation network based on the acquired complete surface image of the bridge so as to reduce the amount of calculation and generate a high-precision bridge point cloud that does not require subsequent manual cropping, establish a BIM model based on the high-precision bridge point cloud, and provide a bridge structure modeling method based on three-dimensional point cloud processing.

[0004] The present invention solves the above technical problems through the following technical solutions, and the present invention comprises the following steps:

[0005] Step S1: Bridge image data acquisition

[0006] Use drones to carry cameras to collect images of various parts of the bridge structure, obtain multi-view bridge images, and create a bridge segmentation dataset;

[0007] Step S2: Segmentation model training and mask creation

[0008] The segmentation network is trained using the bridge segmentation dataset to obtain a segmentation model, and the segmentation model is used to segment the bridge part from the collected bridge image, remove the background, and create a mask detection image for the segmented bridge part;

[0009] Step S3: Feature point extraction and matching

[0010] Obtain image feature points in the mask detection image of the bridge part and match the feature points, accurately align the images, and restore the relative position and posture of each image in three-dimensional space, in preparation for sparse point cloud reconstruction through the three-dimensional coordinates of the feature points;

[0011] Step S4: Bridge sparse point cloud reconstruction

[0012] First, select the initial image to initialize the model, then use the PnP algorithm to calculate the camera pose in a picture-by-picture manner, match the two-dimensional feature points of the image to the three-dimensional coordinates, and then complete the reconstruction of the overall sparse point cloud of the bridge to obtain the overall sparse point cloud of the bridge;

[0013] Step S5: Bridge high-precision point cloud reconstruction

[0014] Based on the overall sparse point cloud of the bridge, dense point cloud reconstruction is performed to fill the gaps between the sparse point clouds to obtain a complete high-precision point cloud of the bridge;

[0015] Step S6: Bridge structure model building

[0016] The bridge structure model is made based on the established high-precision point cloud of the bridge.

[0017] Furthermore, in step S1, the process of collecting bridge image data is as follows:

[0018] Step S101: Conduct an on-site survey of the environment surrounding the bridge, plan the flight route of the drone according to the structure and construction characteristics of the bridge, prepare a data collection field operation plan, and prepare instruments and equipment and arrange personnel;

[0019] Step S102: For the bridge deck structure, a camera is mounted on the bottom of the drone, and a bridge deck image is acquired through route planning, with a heading overlap rate greater than 80% and a lateral overlap rate greater than 60%;

[0020] Step S103: For the bottom and side structures of the bridge, a camera is mounted on the top of the drone, and the drone is manually controlled. The camera faces upward to shoot images of the bottom of the bridge, and the camera tilts to shoot the side of the bridge, and the flight track covers the bottom and side of the bridge;

[0021] Step S104: For the bridge pier structure, an adsorption-type drone equipped with a camera is used. After taking off, the drone is kept horizontal and then approaches the bridge pier until it is adsorbed on the bridge pier. The complete surface of the bridge pier is photographed along a crawling route from bottom to top and then from top to bottom.

[0022] Step S105: For the remaining key structures of the bridge, a drone is used to hover at the key structures, and a high-definition camera array is used to collect images of the remaining key structures of the bridge, where the remaining key structures of the bridge include bearings, abutments, main beams, cross beams and bridge towers.

[0023] Furthermore, in step S1, the process of making the bridge segmentation dataset is as follows:

[0024] Step S111: acquiring a set number of bridge images taken by drones;

[0025] Step S112: using the LabelMe tool to mark the bridge part in the bridge image by manual segmentation;

[0026] Step S113: Expand the data set using an image enhancement method and divide it into a training set and a validation set according to a set ratio.

[0027] Furthermore, in step S2, the process of obtaining the mask detection image is as follows:

[0028] Step S21: first adjust the size of the training set image to the set size, then input it into the YOLO v8s-seg segmentation network for training, use the validation set to verify the trained model, and adjust the hyperparameters until the set segmentation accuracy is achieved, thus obtaining the segmentation model;

[0029] Step S22: Input the acquired bridge image into the segmentation model to segment the bridge part, remove the background and output a mask detection image of the bridge part.

[0030] Furthermore, in step S3, the specific processing process is as follows:

[0031] Step S31: extracting image features using a scale-invariant feature transformation matching algorithm;

[0032] Step S32: Use the Euclidean distance to calculate the detected image feature points, and perform matching calculations on the sequence images of adjacent frames obtained by shooting. During the feature point matching process, calculate the Euclidean distance of the corresponding feature points of the image, and select the feature point pair with the smallest distance as the matching point pair to align the multi-view images.

[0033] Furthermore, in step S4, the specific processing process is as follows:

[0034] Step S41: Using an incremental reconstruction method to reconstruct the sparse point cloud of the entire bridge, first select an initial image to initialize the model, then use the PnP algorithm to calculate the camera pose in a way of calculating each image one by one, and match the two-dimensional feature points of the image to the three-dimensional coordinates;

[0035] Step S42: Expand the point set using the triangulation method, optimize the camera pose and sparse point positions using bundle adjustment, and complete the camera pose solution and sparse point cloud reconstruction.

[0036] Furthermore, in the step S41, the PnP algorithm estimates the camera extrinsic parameters, i.e., the rotation matrix R and the translation vector T, i.e., the camera pose, based on the known two-dimensional image points and three-dimensional world point pairs; in the step S42, the triangulation method is based on the known camera intrinsic parameters and the camera extrinsic parameters calculated by the PnP algorithm and the corresponding points in the image, and then calculates the position of the three-dimensional world point through the geometric relationship of triangulation.

[0037] Furthermore, in step S42, the specific process of optimizing the camera pose and sparse point positions using bundle adjustment is as follows:

[0038] S421: Initial Estimates

[0039] The preliminary camera pose and 3D point coordinates are obtained through feature point matching and triangulation;

[0040] S422: Establish error function

[0041] Optimize the pose and 3D point coordinates by minimizing the reprojection error. The reprojection error refers to the difference between the projected position of a 3D point on the image plane and the actual observed position, where the 3D point is the 3D world point, i.e., a sparse point.

[0042] S423: Iterative calculation

[0043] After each iteration, the reprojection error is recalculated and the parameters are adjusted to gradually reduce the error until it converges to an optimal value. Through multiple iterations, the bundle adjustment can gradually approach the actual camera pose and three-dimensional point coordinates, thereby optimizing the camera pose and sparse point positions. The parameters include: camera extrinsics, i.e., the rotation matrix R and translation vector T of each camera; the coordinates of the three-dimensional points, i.e., the precise position of each three-dimensional point in the point cloud model.

[0044] Furthermore, in step S5, the specific processing process is as follows:

[0045] Step S51: Calculate the disparity map through stereo matching to obtain image depth information;

[0046] Step S52: Fusing the sparse point cloud with the disparity map to optimize the point cloud;

[0047] Step S53: Based on the method in step S51-step S52, the sparse point cloud is optimized according to the mask detection image of the bridge part from multiple perspectives to generate a dense point cloud, that is, a high-precision point cloud of the entire bridge is obtained;

[0048] Step S54: Use Meshlab software to map the texture of the image to the surface of the high-precision point cloud model, and export it as bridge point cloud model data with real texture in .xyz format.

[0049] Furthermore, in step S6, the bridge structure model making process is as follows:

[0050] Step S61: importing the bridge point cloud model data in .xyz format into ReCap Pro software and converting the format into .rcp format;

[0051] Step S62: Import the .rcp format point cloud into the Revit software, use the software's functional modules to operate the point cloud model, and construct a bridge structure model. For families that are not available in the software, it is necessary to create a family of such bridge structures. Finally, create a bridge BIM model for the white film model map that only contains the bridge geometry but does not contain materials or textures.

[0052] Compared with the prior art, the present invention has the following advantages: the bridge structure modeling method based on 3D point cloud processing first obtains the complete surface image of the bridge, segments the bridge part through a segmentation network to make a mask, reconstructs the high-precision point cloud of the bridge using a motion recovery structure method (feature point detection and matching, incremental reconstruction of sparse point cloud) and a multi-view stereo vision method (sparse point cloud fusion disparity map), and then establishes a BIM model based on the point cloud; solves the problem of missing bridge point cloud data caused by occlusion of ground 3D laser scanners, and solves the problem of poor accuracy of bridge point cloud data obtained by airborne laser radar and inability to collect the bottom of the bridge; establishes a point cloud model using a bridge mask through a network segmentation method, does not need to rely on manual cropping of environmental point clouds in the later stage, and generates a high-precision point cloud model and imports it into the modeling software for refined modeling; in addition, the present invention can also be migrated to the point cloud modeling of other buildings and structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of a bridge structure modeling method based on three-dimensional point cloud processing in an embodiment of the present invention;

[0054] FIG2 (a) is a diagram of a bridge deck image acquisition process in an embodiment of the present invention;

[0055] FIG2( b ) is a diagram of the bridge bottom image acquisition process in an embodiment of the present invention;

[0056] FIG2( c ) is a diagram of the bridge side image acquisition process in an embodiment of the present invention;

[0057] FIG2( d ) is a diagram of the bridge pier image acquisition process in an embodiment of the present invention;

[0058] FIG2 (e) is a diagram of the process of capturing images of the remaining key structures of the bridge in an array according to an embodiment of the present invention;

[0059] FIG3 (a) is an original image of a bridge captured by a drone in an embodiment of the present invention, wherein (1) is an original image of the bottom surface of the bridge, (2) is an original image of the pier, (3) is an original image of the bottom of the bridge, and (4) is an original image of the bridge support;

[0060] FIG3(b) is an image of an original image of a bridge captured by a drone according to an embodiment of the present invention after exposure adjustment, wherein (1) is an image of the original image of the bottom surface of the bridge after the exposure is adjusted higher, (2) is an image of the original image of the pier after the exposure is adjusted lower, (3) is an image of the original image of the bottom of the bridge after the exposure is adjusted higher, and (4) is an image of the original image of the bridge support after the exposure is adjusted lower;

[0061] FIG3 (c) is an image of the original image of the bridge captured by the drone in an embodiment of the present invention after random occlusion, wherein (1) is an image of the original image of the bottom of the bridge after random occlusion, (2) is an image of the original image of the pier after random occlusion, (3) is an image of the original image of the bottom of the bridge after random occlusion, and (4) is an image of the original image of the bridge support after random occlusion;

[0062] FIG3(d) is an image of a bridge captured by a drone in an embodiment of the present invention after being randomly rotated, wherein (1) is an image of the bridge bottom surface after being randomly rotated, (2) is an image of the bridge pier after being randomly rotated, (3) is an image of the bridge bottom after being randomly rotated, and (4) is an image of the bridge support after being randomly rotated;

[0063] Figure 4 (a) is the accuracy curve during the YOLO v8s-seg network training process;

[0064] Figure 4 (b) is the precision and recall (PR) curve during the training of the YOLO v8s-seg network;

[0065] FIG5( a ) is a mask image after the bridge deck is segmented in an embodiment of the present invention;

[0066] FIG5( b ) is a mask image after the side of the bridge is segmented in an embodiment of the present invention;

[0067] FIG5( c ) is a mask image after the bridge bottom is segmented in an embodiment of the present invention;

[0068] FIG5( d ) is a mask image after the bridge pier is segmented in an embodiment of the present invention;

[0069] Figure 6 is a schematic diagram of bridge feature detection and matching in an embodiment of the present invention;

[0070] Figure 7 This is an example diagram of the incremental reconstruction of a bridge sparse point cloud in an embodiment of the present invention;

[0071] Figure 8This is an example diagram of dense point cloud reconstruction and texture mapping of a bridge in an embodiment of the present invention;

[0072] FIG9 (a) is an example diagram of a point cloud of a partial BIM model of a bridge according to an embodiment of the present invention;

[0073] FIG9( b ) is an example diagram of a partial BIM model of a bridge in an embodiment of the present invention;

[0074] FIG9( c ) is an example diagram of a complete structural BIM model of a bridge in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0076] like Figure 1 As shown, this embodiment provides a technical solution: a bridge structure modeling method based on three-dimensional point cloud processing, comprising the following steps:

[0077] Step 1: Bridge aerial image data collection

[0078] According to the bridge image data acquisition process, the drone-mounted camera is used to collect images of each part of the bridge structure and a bridge segmentation dataset is created;

[0079] Step 2: Segmentation model training and mask creation

[0080] The segmentation network is trained to obtain a segmentation model, and the segmentation model is used to segment the bridge part from the collected bridge image, remove the background, and create a mask detection image for the segmented bridge part;

[0081] Step 3: Feature point extraction and matching

[0082] Obtaining image feature points in the mask detection image of the bridge part and matching the feature points can accurately align the images and restore the relative position and posture of each image in three-dimensional space, preparing for sparse point cloud reconstruction through the three-dimensional coordinates of the feature points;

[0083] Step 4: Bridge sparse point cloud reconstruction

[0084] First, select the initial image to initialize the model, then use the PnP algorithm to calculate the camera pose in a picture-by-picture manner, match the two-dimensional feature points of the image to the three-dimensional coordinates, and then complete the reconstruction of the overall sparse point cloud of the bridge to obtain the overall sparse point cloud of the bridge;

[0085] Step 5: High-precision point cloud reconstruction of the bridge

[0086] Based on the overall sparse point cloud of the bridge, dense point cloud reconstruction is performed to fill the gaps between the sparse point clouds to obtain a complete high-precision point cloud of the bridge;

[0087] Step 6: Modeling the bridge structure based on point cloud

[0088] The bridge structure model is made based on the established high-precision point cloud of the bridge.

[0089] In step 1, the bridge image data acquisition process is as follows:

[0090] (1) Conduct an on-site survey of the environment around the bridge, carefully observe and record the topography, geology, hydrology, ecology, etc. of the surveyed area, and take photos at the same time. Plan the flight route and site of the drone according to the structure and construction characteristics of the bridge, prepare a data collection field operation plan, and prepare instruments and equipment and arrange personnel;

[0091] (2) As shown in Figure 2 (a), a DJI M300RTK drone equipped with a P1 camera was used to take aerial photogrammetry of the bottom surface of the bridge. A parallel route covering the entire bridge was set along the direction of traffic using the route planning tool. During flight, the drone took photos from multiple angles, such as directly below, in front, behind, to the left, and to the right, to obtain multi-angle images. The heading overlap rate was greater than 80%, the lateral overlap rate was greater than 60%, and the flight speed did not exceed 1.47 m / s.

[0092] (3) As shown in Figure 2(b) and Figure 2(c), for the bottom and sides of the bridge, place the camera on top of the drone and manually control the drone. The camera faces upward to take images of the bottom of the bridge, and the camera tilts to take images of the sides of the bridge. The flight trajectory covers the bottom and sides of the bridge. The flight requirements are the same as above.

[0093] (4) As shown in Figure 2(d), for photographing bridge piers, an adsorption type drone equipped with a camera is used. Unlike ordinary drones, this adsorption type drone has two modes: flight and adsorption crawling. The unique frame design allows the drone to switch smoothly between the flight and adsorption modes. This adsorption type drone can take close-up photos on the surface of the bridge pier. After takeoff, the drone is kept horizontal and then approaches the bridge pier until it adsorbs the bridge pier. It crawls along the pier from bottom to top and then from top to bottom to photograph the complete surface of the bridge pier.

[0094] (5) As shown in Figure 2(e), for key structures of the bridge, such as the bearings, abutments, main beams, cross beams and bridge towers, a drone is used to hover over the key structures and use a high-definition camera array to collect high-resolution images of the remaining key structures of the bridge.

[0095] In step 1, the process of making the bridge segmentation dataset is as follows:

[0096] (1) Collect 500 images of bridges taken by drones;

[0097] (2) Using manual segmentation, use the LabelMe tool to label the bridge part in the bridge image;

[0098] (3) As shown in Figure 3(a), Figure 3(b), Figure 3(c), and Figure 3(d), after image enhancement methods such as random image flipping, random occlusion, rotation, cropping, color change, adding noise, and adjusting exposure, the dataset was expanded to 2000 images and divided into training and validation sets in a ratio of 9:1;

[0099] In step 2, the process of using the segmentation model to segment the bridge part from the acquired image and create a mask is as follows:

[0100] (1) First, adjust the size of the training set image to 640×640px, and then input it into the YOLO v8s-seg segmentation network for training. Use the validation set to verify the trained model and adjust the hyperparameters until a high segmentation accuracy is achieved. The parameter settings during the network training process include: the number of training steps is 1000, the block size is 16, and the basic learning rate is 0.01. The accuracy graph and PR curve graph of the network during the training process are shown in Figure 4 (a) and Figure 4 (b). The results show that the segmentation accuracy reaches 0.996, which meets the accuracy requirements.

[0101] (2) As shown in Figure 5 (a) to Figure 5 (d), the captured bridge image is input into the segmentation network to segment the bridge part, remove the background and output the Mask detection image of the bridge part.

[0102] In step three, if Figure 6 As shown, the process of detecting features in the image based on the mask of the bridge part and matching the features is as follows:

[0103] (1) Use the scale-invariant feature transform matching algorithm (SIFT) to extract image features;

[0104] (2) The detected image feature points are calculated using the Euclidean distance, and the sequence images of adjacent frames are matched. During the feature point matching process, the Euclidean distance of the corresponding feature points in the image is calculated, and the feature point pair with the smallest distance is selected as the matching point pair to align the multi-view images.

[0105] In step 4, if Figure 7 As shown in the figure, the process of camera pose solution and reconstruction of the overall sparse point cloud of the bridge is as follows:

[0106] (1) An incremental reconstruction method is used to reconstruct the sparse point cloud of the entire bridge. First, an initial image is selected to initialize the model. Then, the PnP algorithm is used to calculate the camera pose image by image, and the two-dimensional feature points of the image are matched to the three-dimensional coordinates.

[0107] (2) The triangulation method is used to expand the point set, and the bundle adjustment method is used to optimize the model's pose, complete the camera pose solution and the reconstruction of the sparse point cloud.

[0108] In the above step (1), the PnP algorithm is described as follows: Assume that there are N known three-dimensional world coordinate points , whose projections on the image plane are two-dimensional points , the three-dimensional point can be projected onto the two-dimensional image through the perspective projection model. Through the relationship between the camera's internal parameters and external parameters, the following projection equation can be written:

[0109] ;

[0110] in, It is the intrinsic parameter matrix of the camera, including focal length, principal point, etc.; is the rotation matrix, which represents the rotation relationship from the world coordinate system to the camera coordinate system; is the translation vector, which represents the translation relationship from the world coordinate system to the camera coordinate system; are the homogeneous coordinates of a three-dimensional world coordinate point.

[0111] The goal of the PnP algorithm is to estimate the camera's external parameters (i.e., rotation matrix R and translation vector T), that is, the camera's pose, based on known 2D image points and 3D world (coordinate) point pairs.

[0112] It should be noted that in real applications, points in the image may be affected by noise or false matches. To improve robustness, the RANSAC algorithm (Random Sample Consensus Algorithm) is usually used in combination with the PnP algorithm. The RANSAC algorithm randomly selects a subset from a set of possible point matches and calculates the pose estimate of the subset. Through multiple iterations, the RANSAC algorithm can select a model with the most inliers (i.e., true matching points), thereby avoiding the influence of false matches on pose estimation.

[0113] In the above step (1), the triangulation method described above: the basic principle is to use two or more cameras with known positions to observe the same object from different perspectives, and then solve the three-dimensional coordinates of the object through geometric relationships.

[0114] Suppose there are two cameras and , they respectively photographed three-dimensional points (three-dimensional world coordinate points) , the projection in the image and ,Right now is the 3D point P on the camera The projection in is the 3D point P on the camera Based on the known camera intrinsic parameters, the camera extrinsic parameters (rotation matrix R and translation vector T) calculated by the PnP algorithm, and the corresponding points in the image, the position of the three-dimensional point P can be calculated through the geometric relationship of triangulation.

[0115] Mathematical model of triangulation: Assume that the internal and external parameters of the two cameras are known, and the relationship between the two-dimensional points on the image and the camera coordinate system is known. Using the perspective projection model of the camera, the relationship between the two-dimensional image point and the three-dimensional world point can be expressed as:

[0116] ;

[0117] in, , It's a camera and camera The internal parameter matrix of , It's a camera and The rotation matrix of , It's a camera and camera The translation matrix of is a point in three-dimensional space; , It's a camera and camera Homogeneous coordinates of the two-dimensional image points in .

[0118] These projection equations represent the projection relationship of the three-dimensional point P in the camera coordinate system. According to these equations, the problem can be transformed into a process of solving the three-dimensional coordinate P.

[0119] In triangulation under two perspectives, given the internal and external parameters of two cameras and the two-dimensional points under the two camera perspectives, the position of the three-dimensional point can be restored by geometric calculation. The specific steps are as follows:

[0120] (1) Normalize 2D points: First, normalize the 2D image points and Convert to unit coordinates in the camera coordinate system (remove the camera's intrinsic parameters): , ;

[0121] (2) Reconstructing the projection line: For each image point, a projection line can be inferred from its coordinates in the camera coordinate system and the camera's external parameters. , the projection line of point P is from the camera optical center along Similarly, for the camera , the projection line of point P is from the camera optical center along Direction of rays.

[0122] (3) Calculate the 3D point: Use the intersection of the two projection lines to calculate the position of the 3D point P. The mathematical method to solve this problem is usually to minimize the reprojection error and find the intersection of the two rays in 3D space. The most common method is to use the least squares method to minimize the distance between the two projection lines through optimization methods.

[0123] The generalization of triangulation based on two views leads to multi-view triangulation: if there are more than two camera views, more projection lines can be used to accurately estimate the position of the 3D point. In this case, triangulation becomes a multi-view problem, and the method is similar to that of two views, except that the intersection of multiple projection lines needs to be processed. Multi-view triangulation is usually completed through the following steps: Normalize all image points: convert all two-dimensional image points into unit coordinates in the camera coordinate system; construct projection lines and calculate the corresponding projection lines for each view; solve the intersection points and solve the position of the 3D point by minimizing the distance of all projection lines. At this stage, the reconstructed point cloud is sparse and contains only a small number of 3D points.

[0124] In the above step (2), bundle adjustment (BA) is an algorithm that optimizes the camera pose (position and orientation) and three-dimensional point coordinates in multi-view images to improve the accuracy of the model.

[0125] Initial estimation: In the process of 3D reconstruction or multi-view reconstruction, the preliminary camera pose and 3D point coordinates are first obtained through feature point matching and triangulation.

[0126] Establishing the error function: Bundle adjustment optimizes the pose and 3D point coordinates by minimizing the reprojection error. The reprojection error is the difference between the projected position of a 3D point on the image plane and the actual observed position.

[0127] Suppose there is a three-dimensional point and camera The pose of Project to On the image plane, we get the projection point:

[0128] ;

[0129] The goal of bundle adjustment is to minimize the sum of the reprojection errors of all observation points, that is:

[0130] ;

[0131] in, is the actual observed value, It is the predicted value of the 3D point projected by the camera model.

[0132] Iterative calculation: Bundle adjustment is an iterative process. After each iteration, the reprojection error is recalculated and the parameters are adjusted. In bundle adjustment, the optimized parameters include: camera extrinsics, i.e. the rotation matrix R and translation vector T of each camera; the coordinates of the 3D points, i.e. the precise position of each 3D point in the model. The error is gradually reduced until it converges to an optimal value. Through multiple iterations, bundle adjustment can gradually approach the actual camera pose and 3D point coordinates.

[0133] In step five, if Figure 8 As shown in the figure, the process of reconstructing the high-precision point cloud of the bridge is as follows:

[0134] (1) Multi-view stereo vision (MVS) is used to reconstruct dense point clouds, thus obtaining high-precision point clouds of the entire bridge.

[0135] (2) Use Meshlab software to map the texture of the image to the surface of the three-dimensional high-precision point cloud model and export it as bridge point cloud model data with real texture in .xyz format.

[0136] The multi-view stereo vision method (MVS) specifically refers to the multi-view stereo (Sparse-to-Dense MVS) method based on sparse point clouds. This method uses sparse point clouds as a starting point to reconstruct dense point clouds. The goal of dense point cloud reconstruction is to fill the gaps between sparse point clouds to make the 3D reconstruction more accurate. The specific steps are as follows:

[0137] (1) Disparity map and depth map calculation: Calculate the disparity map through stereo matching to obtain image depth information;

[0138] (2) Combining sparse point cloud with disparity map: The sparse point cloud is combined with the disparity map to optimize the point cloud.

[0139] In step six, as shown in FIG9 (a) to FIG9 (c), the process of making a bridge structure model based on the established high-precision point cloud of the bridge is as follows:

[0140] (1) Import the .xyz format point cloud into ReCap Pro software and convert it to .rcp format;

[0141] (2) Import the .rcp format point cloud into Revit software, use functions such as slicing and distance measurement to operate the point cloud model, and construct a bridge structure model. For families that are not available in the software, such as supports, piers, bridge towers, and guardrails, it is necessary to create specific families for such bridge structures. Finally, a realistic bridge BIM model is created for the white film model map that only contains the bridge geometry but does not contain materials or textures.

[0142] In summary, the bridge structure modeling method based on three-dimensional point cloud processing proposed in the present invention first obtains the complete surface image of the bridge, segments the bridge part through a segmentation network to make a mask, utilizes the motion recovery structure method (feature point detection and matching, incremental reconstruction of sparse point cloud) and multi-view stereo vision method (sparse point cloud fusion disparity map) to reconstruct the high-precision point cloud of the bridge, and then establishes a BIM model based on the point cloud; solves the problem of missing bridge point cloud data caused by occlusion of ground three-dimensional laser scanners, and solves the problem of poor accuracy of bridge point cloud data obtained by airborne laser radar and inability to collect the bottom of the bridge; establishes a point cloud model using a bridge mask through a network segmentation method, does not need to rely on manual cropping of environmental point clouds in the later stage, and generates a high-precision point cloud model and imports it into the modeling software for refined modeling; in addition, the present invention can also be migrated to the point cloud modeling of other buildings and structures.

[0143] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A bridge structure modeling method based on three-dimensional point cloud processing, characterized in that: The following steps are involved: Step S1: Bridge image data acquisition Use drones to carry cameras to collect images of various parts of the bridge structure, obtain multi-view bridge images, and create a bridge segmentation dataset; Step S2: Segmentation model training and mask creation The segmentation network is trained using the bridge segmentation dataset to obtain a segmentation model, and the segmentation model is used to segment the bridge part from the collected bridge image, remove the background, and create a mask detection image for the segmented bridge part; Step S3: Feature point extraction and matching Obtain image feature points in the mask detection image of the bridge part and match the feature points, accurately align the images, and restore the relative position and posture of each image in three-dimensional space, in preparation for sparse point cloud reconstruction through the three-dimensional coordinates of the feature points; Step S4: Bridge sparse point cloud reconstruction First, select the initial image to initialize the model, then use the PnP algorithm to calculate the camera pose in a picture-by-picture manner, match the two-dimensional feature points of the image to the three-dimensional coordinates, and then complete the reconstruction of the overall sparse point cloud of the bridge to obtain the overall sparse point cloud of the bridge; In step S4, the specific processing process is as follows: Step S41: Using an incremental reconstruction method to reconstruct the sparse point cloud of the entire bridge, first select an initial image to initialize the model, then use the PnP algorithm to calculate the camera pose in a way of calculating each image one by one, and match the two-dimensional feature points of the image to the three-dimensional coordinates; Step S42: Expand the point set using a triangulation method, optimize the camera pose and sparse point positions using bundle adjustment, and complete the camera pose calculation and sparse point cloud reconstruction; In the step S41, the PnP algorithm estimates the camera extrinsic parameters, i.e., the rotation matrix R and the translation vector T, i.e., the camera pose, based on the known two-dimensional image points and three-dimensional world point pairs; in the step S42, the triangulation method calculates the position of the three-dimensional world point through the geometric relationship of triangulation based on the known camera intrinsic parameters and the camera extrinsic parameters calculated by the PnP algorithm and the corresponding points in the image; In step S42, the specific process of optimizing the camera pose and sparse point positions using bundle adjustment is as follows: S421: Initial Estimates The preliminary camera pose and 3D point coordinates are obtained through feature point matching and triangulation; S422: Establish error function Optimize the pose and 3D point coordinates by minimizing the reprojection error. The reprojection error refers to the difference between the projected position of a 3D point on the image plane and the actual observed position, where the 3D point is the 3D world point, i.e., a sparse point. 3D Point and camera The pose of Project to On the image plane, we get the projection point: ； The goal of bundle adjustment is to minimize the sum of the reprojection errors of all observation points, that is: ； in, is the actual observed value, It is the predicted value of the 3D point projected by the camera model; S423: Iterative calculation After each iteration, the reprojection error is recalculated and the parameters are adjusted to gradually reduce the error until it converges to an optimal value. Through multiple iterations, the bundle adjustment can gradually approach the actual camera pose and 3D point coordinates, and optimize the camera pose and sparse point positions. The parameters include: camera extrinsics, i.e., the rotation matrix R and translation vector T of each camera; the coordinates of the 3D points, i.e., the precise position of each 3D point in the point cloud model; Step S5: Bridge high-precision point cloud reconstruction Based on the overall sparse point cloud of the bridge, dense point cloud reconstruction is performed to fill the gaps between the sparse point clouds to obtain a complete high-precision point cloud of the bridge; Step S6: Bridge structure model building The bridge structure model is made based on the established high-precision point cloud of the bridge.

2. The bridge structure modeling method based on three-dimensional point cloud processing according to claim 1 is characterized in that: In step S1, the process of collecting bridge image data is as follows: Step S101: Conduct an on-site survey of the environment surrounding the bridge, plan the flight route of the drone according to the structure and construction characteristics of the bridge, prepare a data collection field operation plan, and prepare instruments and equipment and arrange personnel; Step S102: For the bridge deck structure, a camera is mounted on the bottom of the drone, and a bridge deck image is acquired through route planning, with a heading overlap rate greater than 80% and a lateral overlap rate greater than 60%; Step S103: For the bottom and side structures of the bridge, a camera is mounted on the top of the drone, and the drone is manually controlled. The camera faces upward to shoot images of the bottom of the bridge, and the camera tilts to shoot the side of the bridge, and the flight track covers the bottom and side of the bridge; Step S104: For the bridge pier structure, an adsorption-type drone equipped with a camera is used. After taking off, the drone is kept horizontal and then approaches the bridge pier until it is adsorbed on the bridge pier. The complete surface of the bridge pier is photographed along a crawling route from bottom to top and then from top to bottom. Step S105: For the remaining key structures of the bridge, a drone is used to hover at the key structures, and a high-definition camera array is used to collect images of the remaining key structures of the bridge, where the remaining key structures of the bridge include bearings, abutments, main beams, cross beams and bridge towers.

3. The bridge structure modeling method based on three-dimensional point cloud processing according to claim 1 is characterized in that: In step S1, the process of making the bridge segmentation dataset is as follows: Step S111: acquiring a set number of bridge images taken by drones; Step S112: using the LabelMe tool to mark the bridge part in the bridge image by manual segmentation; Step S113: Expand the data set using an image enhancement method and divide it into a training set and a validation set according to a set ratio.

4. The bridge structure modeling method based on three-dimensional point cloud processing according to claim 1 is characterized in that: In step S2, the process of obtaining the mask detection image is as follows: Step S21: first adjust the size of the training set image to the set size, then input it into the YOLO v8s-seg segmentation network for training, use the validation set to verify the trained model, and adjust the hyperparameters until the set segmentation accuracy is achieved, thus obtaining the segmentation model; Step S22: Input the acquired bridge image into the segmentation model to segment the bridge part, remove the background and output a mask detection image of the bridge part.

5. The bridge structure modeling method based on three-dimensional point cloud processing according to claim 4 is characterized in that: In step S3, the specific processing process is as follows: Step S31: extracting image features using a scale-invariant feature transformation matching algorithm; Step S32: Use the Euclidean distance to calculate the detected image feature points, and perform matching calculations on the sequence images of adjacent frames obtained by shooting. During the feature point matching process, calculate the Euclidean distance of the corresponding feature points of the image, and select the feature point pair with the smallest distance as the matching point pair to align the multi-view images.

6. The bridge structure modeling method based on three-dimensional point cloud processing according to claim 1 is characterized in that: In step S5, the specific processing process is as follows: Step S51: Calculate the disparity map through stereo matching to obtain image depth information; Step S52: Fusing the sparse point cloud with the disparity map to optimize the point cloud; Step S53: Based on the method in step S51-step S52, the sparse point cloud is optimized according to the mask detection image of the bridge part from multiple perspectives to generate a dense point cloud, that is, a high-precision point cloud of the entire bridge is obtained; Step S54: Use Meshlab software to map the texture of the image to the surface of the high-precision point cloud model, and export it as bridge point cloud model data with real texture in .xyz format.

7. The bridge structure modeling method based on three-dimensional point cloud processing according to claim 6 is characterized in that: In step S6, the bridge structure model making process is as follows: Step S61: importing the bridge point cloud model data in .xyz format into ReCap Pro software and converting the format into .rcp format; Step S62: Import the .rcp format point cloud into the Revit software, use the software's functional modules to operate the point cloud model, and construct a bridge structure model. For families that are not available in the software, it is necessary to create a family of such bridge structures. Finally, create a bridge BIM model for the white film model map that only contains the bridge geometry but does not contain materials or textures.

Images