Rebar-YOLOv8-seg network and depth data-based steel bar spacing measurement method for realizing three-dimensional point cloud segmentation

Abstract

Three-dimensional point cloud segmentation is realized based on the Rebar-YOLOv8-seg network and depth data, so that the steel bar spacing is measured, and the method has the advantages of low cost and high real-time performance. The Rebar-YOLOv8-seg network improves the accuracy of steel bar segmentation and reduces the calculation cost, so that the method can meet the requirements of rapid and accurate measurement of the steel bar spacing in the power transmission and transformation extension project.

Description

Technical Field

[0001] The present invention applies computer vision and deep learning technology, and proposes a method for measuring the spacing between steel bars based on the Rebar-YOLOv8-seg network and deep data to achieve three-dimensional point cloud segmentation. The goal is to meet the requirement that metal measuring tools are not allowed to be used in the construction of substation expansion projects, and to invent a method for quickly and accurately measuring the spacing between steel bars. Background Art

[0002] During the construction of the substation expansion project, the spacing between steel bars needs to be checked because the spacing between steel bars directly affects the bearing capacity of the concrete structure. Since the original basic equipment in the expansion project is energized, the construction safety regulations clearly stipulate that the use of metal measuring tools for measurement is not allowed. At present, the main method for measuring the spacing between steel bars in the expansion project is to use electric power construction personnel to hold an insulating ruler for measurement, which is time-consuming and laborious. In addition, since the insulating ruler is soft and has a certain degree of elasticity, the measurement results are greatly affected by human factors. Therefore, research on high-precision non-contact steel bar measurement methods continues to emerge.

[0003] At present, the non-contact steel bar spacing measurement methods available in the literature include laser radar scanning, photogrammetry, and indirect measurement using a depth camera to build a three-dimensional model. Among them, the laser radar scanning method requires high equipment costs and requires additional equipment maintenance. In addition, this method takes a long time to manually segment and process the three-dimensional point cloud data, and the real-time performance is not high. Although the equipment cost required for the photogrammetry method is greatly reduced, the photogrammetry method requires the acquisition of multi-angle images to form the three-dimensional structural information of the steel bar. Not only is the image processing time-consuming, but it also requires the calculation of an accurate scale to reflect the size relationship between the steel bar image and the actual steel bar. Although the image captured by the depth camera contains the three-dimensional coordinates of the steel bar to be measured, it is necessary to cooperate with image and point cloud processing technology to accurately segment the steel bar and clarify its specific location. The key to this method lies in the image and point cloud processing technology, which determines the performance of this method.

[0004] In terms of steel bar segmentation, since the project has clear requirements for real-time performance, the YOLO series network has attracted much attention due to its low computational cost and stable performance. YOLOv8-seg is a relatively representative instance segmentation network with relatively low computational cost, which has better real-time performance than the two-stage instance segmentation model. YOLOv8-seg mainly consists of three parts: backbone network, neck network and segmentation head. Among them, the backbone network is responsible for extracting basic features from the image, gradually reducing the size of the input image through a series of convolutional layers, and extracting low-level and high-level semantic features at the same time. The spatial pyramid pooling fusion module (SPPF) is used at the end to extract contextual information from images of different scales, enhancing the generalization ability of the model; the neck network is used to fuse features of different scales in order to process targets of different sizes in the image. The C2f module is combined with the upsampling module to complete the fusion of features of different scales and perform feature enhancement; the segmentation head is responsible for the final target classification, bounding box regression and segmentation mask generation.

[0005] Since the steel bars to be tested are crisscrossed in the image, it is necessary to detect all the steel bars and ensure that each steel bar is segmented completely. Currently, the YOLOv8-seg segmentation effect cannot meet the measurement requirements.

[0006] In terms of steel bar positioning measurement, it is necessary to determine the intersection of the steel bars, so it is necessary to perform straight line fitting on the steel bar point cloud data. Among the current straight line fitting methods, the random sampling consensus method (RANSAC) is a method to achieve optimal straight line fitting. The straight line fitting process is: randomly select two data points to fit a straight line, and then calculate the distance from the remaining points to the straight line. If the distance from the remaining points to the straight line is less than the preset threshold, the point is identified as an inlier. If the number of inliers in the current model exceeds the previous best model, the best model is updated, and the fitting process is repeated until the number of inliers in the model is stable or the best number of iterations is reached, and then the optimal fitting straight line model is output. This method can meet the straight line fitting requirements in the steel bar measurement process and can effectively determine the intersections of criss-crossing steel bars.

[0007] In summary, there are currently relevant technical means and a certain research basis for non-contact steel bar measurement, but there is no measurement method with low equipment cost, fully automatic measurement process, and meeting the real-time requirements of the project. Summary of the invention

[0008] The present invention proposes a method for measuring the spacing between steel bars based on the Rebar-YOLOv8-seg network and depth data to achieve three-dimensional point cloud segmentation. The required equipment cost is low, and no manual point cloud processing is required, which has strong real-time performance. The present invention can not only quickly extract steel bar point clouds, but also ensure the accuracy of point clouds. It is a new type of non-contact method for measuring the spacing between steel bars. The Rebar-YOLOv8-seg network is improved on the basis of the YOLOv8-seg network, has better steel bar instance segmentation accuracy, and reduces the computational cost.

[0009] The present invention proposes a method for measuring the spacing between steel bars based on the Rebar-YOLOv8-seg network and depth data to achieve three-dimensional point cloud segmentation. First, an RGB-D camera is used to simultaneously collect the RGB image and depth data of the steel bar to be measured, and the RGB image is input into the Rebar-YOLOv8-seg network to output the steel bar mask RGB image, determine the position of the steel bar in the image, and combine the steel bar depth data to obtain the initial three-dimensional coordinates of the steel bar to be measured, and form the initial three-dimensional point cloud of the steel bar. Then, the initial point cloud of the steel bar is statistically filtered to remove the surrounding outliers to obtain the filtered point cloud. The principal component analysis method is then used to reconstruct the coordinate system of the filtered point cloud, and the three-dimensional point cloud is converted into a two-dimensional point cloud. Finally, the random sampling consensus method (RANSAC) is used to perform straight line fitting on the two-dimensional steel bar point cloud, and the fitted straight line is regarded as the center line of each steel bar. The coordinates of all intersections of the intersecting center lines are calculated, and then the coordinates of the midpoints of the line segments between adjacent intersections are calculated. The distance between adjacent opposite midpoints is calculated as the steel bar spacing, and the steps are as follows:

[0010] Step 1: Use an RGB-D camera to simultaneously collect the RGB image and depth data of the steel bar to be tested.

[0011] Step 2: Input the RGB image into the Rebar-YOLOv8-seg network and output it as a steel bar mask RGB image. Binarize the steel bar mask image to obtain the specific position of the steel bar in the image. The specific position is represented by the pixel coordinates of the pixel points that make up the steel bar image.

[0012] The Rebar-YOLOv8-seg network is an improved network based on YOLOv8-seg. The SPPF module in the YOLOv8-seg backbone is replaced by the SPPF-LSKA module to effectively capture the long-range dependencies in the image and enhance the network's ability to extract features at different scales. The PAN structure in the YOLOv8-seg neck is replaced by the BiFPN network to reduce the complexity and computational overhead of the network and improve the network's ability to fuse multi-scale features. The YOLOv8-seg segmentation head is replaced by the lightweight shared convolution segmentation head LSCS-Head module, in which the LSCS-Head module introduces Group Norm convolution to improve the segmentation head's positioning and classification capabilities. At the same time, LSCS-Head uses Group Norm convolution, classification convolution, and bounding box regression convolution as shared convolution parts to significantly reduce the number of parameters, while maintaining accuracy, so that the segmentation head has better multi-scale perception capabilities. The Rebar-YOLOv8-seg network structure is shown in the figure below. Figure 1 shown.

[0013] Step 3: Map the pixel coordinates of the steel bar extracted in step 2 to the depth data to obtain the depth value corresponding to each pixel of the steel bar. According to the pixel coordinates and depth value of the steel bar, calculate the coordinates of the pixel of the steel bar image in the world coordinate system. The specific calculation formula is:

[0014]

[0015] In the formula, are the horizontal and vertical coordinates of the steel bar pixel points; is the world coordinate corresponding to the steel bar pixel point, where is equal to the depth value corresponding to each pixel, is the focal length of the camera; therefore, the point in the coordinate system corresponding to the steel bar pixel is expressed as Form the initial point cloud of steel bars.

[0016] Step 4: Initial point cloud of the steel bars in step 3 Perform statistical filtering to remove outliers and obtain the filtered point cloud. Indicates that the specific filtering process is:

[0017] (1) Neighborhood search: for point cloud For each point in , based on the spatial geometric relationship, search for a specified number of neighboring points. The number of neighboring points It is an adjustable parameter used to control the search range. .

[0018] (2) Distance calculation: Calculate point cloud From the current point to its Euclidean distance of the neighborhood points and calculate the average distance , the current point coordinates are , The coordinates of the field point are , the average distance The calculation formula is:

[0019]

[0020] (3) Statistical analysis: Assume that the global average distance of point cloud data follows a normal distribution. , for point cloud The average distance of all points Perform statistical analysis and calculate the mean and standard deviation , and its calculation formula is:

[0021]

[0022]

[0023] in, It is a point cloud The total number of points contained in .

[0024] (4) Outlier judgment and removal: For point clouds For each point in If it exceeds the global mean Plus , then the point is considered as an outlier and deleted. In this method, The specific judgment conditions are as follows:

[0025]

[0026] in It is a preset parameter used to control the sensitivity of identifying outliers.

[0027] Step 5: Use principal component analysis to analyze the point cloud filtered in step 4. Reconstruct the coordinate system and transform the 3D point cloud Convert to 2D point cloud .

[0028] Among them, Contains The specific process of reconstructing the coordinate system is:

[0029] (1) Point cloud The coordinates are arranged into a Matrix with 3 rows and 3 columns .

[0030] (2) Calculate The mean of each column is obtained by subtracting the mean of the column from all values ​​in the column to obtain the matrix .

[0031] (3) Construct the covariance matrix. The specific formula is:

[0032]

[0033] Among them, the matrix For the matrix The transpose of .

[0034] (4) Covariance matrix Perform eigenvalue decomposition, calculate the eigenvalue and sort it from large to small to get the eigenvalue , , …, and its corresponding eigenvector , , …, ;Will The corresponding eigenvector As the first principal component, The corresponding eigenvector as the second principal component.

[0035] According to the principal component analysis results, the three-dimensional point cloud Convert to 2D point cloud , the calculation formula is:

[0036]

[0037] Among them, the eigenvector .

[0038] Step 6: Use the random sampling consensus method RANSAC to analyze the two-dimensional steel point cloud in step 5. Perform straight line fitting, and the fitted straight line is equivalent to the center line of each steel bar.

[0039] Step 7: Calculate the coordinates of all the intersection points of the intersecting straight lines in step 6, and then find the coordinates of the midpoints of the line segments between adjacent intersection points. The distance between the opposite midpoints is calculated as the steel bar spacing, such as Figure 2 shown.

[0040] The RGB-D camera is an electronic device that can simultaneously acquire RGB images and depth data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is the model structure diagram of Rebar-YOLOv8-seg.

[0042] Figure 2 It is a schematic diagram of the steel bar straight line fitting results and the intersection points, midpoints, and steel bar spacing. DETAILED DESCRIPTION

[0043] The specific implementation of the present invention is described by taking a set of data from a steel bar spacing measurement experiment in Northeast Forestry University as an example.

[0044] Experimental Equipment The RGB-D camera used in the experiment is an OAK-D-Pro camera, and the resolution of RGB images and depth data is set to 1280×720.

[0045] The number of steel bars used in this experiment is 8, all with a diameter of 10 mm. The number of horizontal and vertical steel bars is 4, respectively, arranged in a grid. The height of the camera plane from the ground is 70 cm. This method is used to measure the spacing of 18 groups of steel bars at the same time.

[0046] To avoid experimental contingency, the present invention conducted 50 steel bar spacing measurement experiments, calculated the measurement average of 18 groups of spacing and compared it with the true value, and obtained the average absolute error of the measured value to be 1.98 mm and the average relative error to be 1.76%. The total time required from data collection to finally obtaining the 18 groups of steel bar spacing measurement values ​​was about 4.6 seconds.

[0047] The present invention combines the image segmentation model Rebar-YOLOv8-seg and depth data, and proposes a steel bar spacing measurement method based on three-dimensional point cloud segmentation. This method can realize the simultaneous measurement of multiple groups of steel bar spacings, effectively improving the measurement efficiency; the measurement error is small and the time is short, which meets the requirements of high precision and good real-time performance in engineering.

Claims

1. A method for measuring the spacing between steel bars based on three-dimensional point cloud segmentation using the Rebar-YOLOv8-seg network and depth data, characterized in that: First, an RGB-D camera is used to simultaneously collect the RGB image and depth data of the steel bar to be tested, and the RGB image is input into the Rebar-YOLOv8-seg network to output the steel bar mask RGB image, determine the position of the steel bar in the image, and combine the steel bar depth data to obtain the initial three-dimensional coordinates of the steel bar to be tested, forming the initial three-dimensional point cloud of the steel bar; then the initial point cloud of the steel bar is statistically filtered to remove the surrounding outliers to obtain the filtered point cloud; the principal component analysis method is then used to reconstruct the coordinate system of the filtered point cloud, and the three-dimensional point cloud is converted into a two-dimensional point cloud; finally, the random sampling consensus method (RANSAC) is used to perform straight line fitting on the two-dimensional steel bar point cloud, and the fitted straight line is regarded as the center line of each steel bar; the coordinates of all intersection points of the intersecting center lines are calculated, and then the coordinates of the midpoints of the line segments between adjacent intersection points are calculated, and the distance between adjacent opposite midpoints is calculated as the steel bar spacing, and the steps are as follows: Step 1: Use an RGB-D camera to simultaneously collect RGB images and depth data of the steel bar to be tested; Step 2: Input the RGB image into the Rebar-YOLOv8-seg network, and output it as a steel bar mask RGB image. Binarize the steel bar mask image to obtain the specific position of the steel bar in the image. The specific position is represented by the pixel coordinates of the pixel points that make up the steel bar image. Step 3: Map the pixel coordinates of the steel bar extracted in step 2 to the depth data to obtain the depth value corresponding to each pixel of the steel bar. According to the pixel coordinates and depth value of the steel bar, calculate the coordinates of the pixel of the steel bar image in the world coordinate system. The specific calculation formula is: In the formula, are the horizontal and vertical coordinates of the steel bar pixel points; is the world coordinate corresponding to the steel bar pixel point, where Equal to the depth value corresponding to each pixel, is the focal length of the camera; therefore, the point in the coordinate system corresponding to the steel bar pixel is expressed as Forming the initial point cloud of steel bars; Step 4: Initial point cloud of the steel bars in step 3 Perform statistical filtering to remove outliers and obtain the filtered point cloud. Indicates that the specific filtering process is: (1) Neighborhood search: for point cloud For each point in , based on the spatial geometric relationship, search for a specified number of neighboring points. The number of neighboring points It is an adjustable parameter used to control the search range. ; (2) Distance calculation: Calculate point cloud From the current point to its Euclidean distance of the neighborhood points and calculate the average distance , the current point coordinates are , The coordinates of the field point are , the average distance The calculation formula is: (3) Statistical analysis: Assume that the global average distance of point cloud data follows a normal distribution. , for point cloud The average distance of all points Perform statistical analysis and calculate the mean and standard deviation , and its calculation formula is: in, It is a point cloud The total number of points included in (4) Outlier judgment and removal: For point clouds For each point in If it exceeds the global mean Plus , then the point is considered as an outlier and deleted. In this method, The specific judgment conditions are as follows: in is a preset parameter used to control the sensitivity of identifying outliers; Step 5: Use principal component analysis to analyze the point cloud filtered in step 4. Reconstruct the coordinate system and transform the 3D point cloud Convert to 2D point cloud ; Among them, Contains The specific process of reconstructing the coordinate system is: (1) Point cloud The coordinates are arranged into a Matrix with 3 rows and 3 columns ; (2) Calculate The mean of each column is obtained by subtracting the mean of the column from all values ​​in the column to obtain the matrix ; (3) Construct the covariance matrix. The specific formula is: Among them, the matrix For the matrix The transpose of (4) Covariance matrix Perform eigenvalue decomposition, calculate the eigenvalue and sort it from large to small to get the eigenvalue , , …, and its corresponding eigenvector , , …, ;Will The corresponding eigenvector As the first principal component, The corresponding eigenvector as the second principal component; According to the principal component analysis results, the three-dimensional point cloud Convert to 2D point cloud , the calculation formula is: Among them, the eigenvector Step 6: Use the random sampling consensus method RANSAC to analyze the two-dimensional steel point cloud in step 5. Perform straight line fitting, the fitted straight line is equivalent to the center line of each steel bar; Step 7: Calculate the coordinates of all intersection points of the intersecting straight lines in step 6, then find the coordinates of the midpoints of the line segments between adjacent intersection points, and calculate the distance between the opposite midpoints as the steel bar spacing.

2. The method for measuring the spacing between steel bars based on the Rebar-YOLOv8-seg network and depth data to achieve three-dimensional point cloud segmentation as described in claim 1, characterized in that: In step 2, the Rebar-YOLOv8-seg network is an improved network based on YOLOv8-seg. The SPPF module in the YOLOv8-seg backbone is replaced by the SPPF-LSKA module to effectively capture the long-range dependencies in the image and enhance the network's feature extraction capabilities at different scales. The PAN structure in the neck part of YOLOv8-seg is replaced by the BiFPN network to reduce the complexity and computational overhead of the network and improve the network's ability to fuse multi-scale features. The segmentation head of YOLOv8-seg is replaced by the lightweight shared convolutional segmentation head LSCS-Head module, in which the LSCS-Head module introduces Group Norm convolution to improve the positioning and classification capabilities of the segmentation head. At the same time, LSCS-Head uses Group Norm convolution, classification convolution, and bounding box regression convolution as shared convolution parts to significantly reduce the amount of parameters, while maintaining accuracy, so that the segmentation head has better multi-scale perception capabilities.

3. The method for measuring the spacing between steel bars based on the Rebar-YOLOv8-seg network and depth data to achieve three-dimensional point cloud segmentation as described in claim 1, characterized in that: The RGB-D camera is an electronic device that can simultaneously acquire RGB images and depth data.

Images