A crack three-dimensional path planning method based on binocular vision and skeleton refinement

Abstract

This invention discloses a 3D path planning method for road cracks based on binocular vision and skeleton refinement, relating to the fields of intelligent engineering machinery and computer vision technology. It simultaneously acquires RGB images and depth point cloud data of the road surface using a binocular depth camera, employs an instance segmentation model to perform semantic segmentation of road cracks and outputs a multi-instance probability mask; after linear interpolation for size restoration, binarization, bitwise OR merging, and morphological dilation processing, it repairs minor breaks and internal voids in the cracks; it uses an iterative morphological refinement algorithm to extract the center skeleton line of the crack with a single pixel width, and simplifies the path by removing noise through Euclidean arc length filtering and equal-interval downsampling; it indexes the pixel coordinates of the 2D skeleton to the depth point cloud to obtain 3D spatial coordinates and generate control commands for robotic arm crack filling. This invention enables precise coordinate mapping, providing stable and reliable precise path guidance for robotic arm crack filling, meeting the needs of automated, precise, and efficient maintenance of road cracks in various scenarios.

Description

Technical Field

[0001] This invention relates to the fields of intelligent engineering machinery and computer vision technology, and in particular to a crack three-dimensional path planning method based on binocular vision and skeleton refinement. Background Technology

[0002] With the continuous development of transportation infrastructure construction and road network operation and maintenance, pavement cracks, as the most typical early-stage defects in road engineering, have made efficient and precise automated maintenance a core development need in the industry. Traditional pavement crack repair mainly relies on manual construction, using manually driven maintenance vehicles and manually operated spraying equipment to complete the crack filling operation. This method generally suffers from low construction efficiency, high labor intensity, and high operating costs. Furthermore, it poses safety hazards in scenarios such as high-grade highways and expressways, and is difficult to adapt to the requirements of large-scale and standardized road maintenance operations. Against this backdrop, intelligent pavement repair vehicles equipped with automated robotic arms are gradually becoming the mainstream development direction of pavement maintenance equipment. The intelligent crack detection and automated path planning technology of the vehicle-mounted robotic arm are the core keys to determining the accuracy, efficiency, and stability of crack filling operations.

[0003] Current technologies for road crack detection and path planning using vehicle-mounted robotic arms still suffer from several insurmountable technical shortcomings: First, the path extraction accuracy is insufficient. Traditional edge detection algorithms or conventional semantic segmentation methods can only obtain the outer contour of the crack, generating a working path that is the edge line on both sides of the crack. The robotic arm working along the edge path will cause a large waste of crack filling material and cannot achieve precise sealing of the crack center, making it difficult to meet the maintenance requirements. Second, three-dimensional spatial information is lacking. Existing solutions mostly generate two-dimensional working paths based on monocular RGB images, without obtaining crack depth and spatial pose information. This makes it impossible to adapt to the relative distance changes caused by vehicle vibration and road surface undulations during operation, and problems such as the robotic arm nozzle colliding with the road surface and unbalanced spraying spacing are prone to occur, resulting in poor operational stability. Third, environmental robustness and path continuity are weak. Road surface oil stains, shadows, and texture interference can easily cause false crack detection. Moreover, natural cracks are mostly distributed discontinuously, resulting in a broken and discontinuous working path that cannot support the robotic arm to complete continuous and smooth crack filling actions.

[0004] In summary, existing pavement crack detection and path planning technologies generally suffer from limitations such as 2D path redundancy, lack of 3D spatial information, and poor adaptability to complex pavement environments. There is an urgent need to design a 3D path planning method that integrates binocular depth vision and skeleton refinement algorithms to achieve accurate mapping from 2D pixel coordinates to 3D spatial coordinates. This would provide stable and reliable accurate path guidance for robotic arm crack filling in complex unstructured pavement scenarios, thus meeting the precise, efficient, and stable crack filling operation requirements of intelligent pavement repair vehicles. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a three-dimensional crack path planning method based on binocular vision and skeleton refinement. By introducing morphological skeleton refinement technology and combining binocular depth information, it achieves accurate mapping from two-dimensional pixel coordinates to three-dimensional spatial coordinates, providing stable and reliable precise path guidance for robotic arm crack filling.

[0006] The technical solution adopted by this invention to solve its technical problem is: a crack 3D path planning method based on binocular vision and skeleton refinement, comprising the following steps:

[0007] S1. Data Acquisition: Simultaneously acquire RGB image data and aligned depth point cloud data of the road surface area using a binocular depth camera;

[0008] S2. Crack semantic segmentation: Input the RGB image into a pre-trained deep instance segmentation model and output the original probability mask set of multiple crack instances on the road surface;

[0009] S3. Mask preprocessing: For each instance mask, linear interpolation size restoration and fixed threshold binarization transformation are performed sequentially. The masks are then merged into a global crack total mask through bitwise OR operation, and morphological dilation is performed to obtain the preprocessed crack mask.

[0010] S4. Crack skeleton extraction: An iterative morphological thinning algorithm is used to peel off the edge pixels of the preprocessed crack mask layer by layer to obtain the center skeleton line of the crack with a single pixel width.

[0011] S5. Path optimization: Extract the contour point set of the skeleton line of the crack center with a single pixel width and complete the path sorting. Calculate the physical arc length of the contour based on Euclidean distance. Filter out noisy contours according to a fixed arc length threshold. Perform equal-interval downsampling on the effective contour point set with a fixed step size to obtain a sparse two-dimensional path point set.

[0012] S6. 3D Coordinate Mapping and Command Generation: Index the pixel coordinates of the sparse 2D path point set to the aligned depth point cloud data, obtain the corresponding 3D spatial coordinates, and generate control commands for the vehicle-mounted robotic arm crack sealing operation according to the path sequence.

[0013] Furthermore, in step S1, a ZED series binocular depth camera is used. The binocular depth camera is fixed directly above the base of the road repair vehicle's robotic arm, with the lens pointing vertically towards the ground. The installation height is 1m above the ground. The horizontal field of view of the binocular depth camera covers a road area with a width of not less than 2.2m, and the relative error of the depth measurement is ≤2%. The binocular depth camera acquires RGB images with a resolution of 1920×1080 at a data acquisition frame rate of 30fps, and calculates the depth map based on the parallax principle to generate aligned depth point cloud data.

[0014] Furthermore, step S2 specifically includes the following steps:

[0015] S2.1 Construct a YOLO series instance segmentation model, using the CSPDarknet backbone feature extraction network as the model feature extraction structure, and perform transfer learning training on a dedicated dataset for road cracks.

[0016] S2.2. Extract the Alpha channel from the BGRA format image output by the binocular depth camera to obtain the BGR image, and then rearrange the channels of the BGR image to convert it into an RGB format image, which is used as input data for the YOLO series instance segmentation model.

[0017] Furthermore, step S3 specifically includes the following steps:

[0018] S3.1 Initialize a single-channel image with all zeros of the same size as the original image as the total mask. ;

[0019] S3.2, Use linear interpolation to mask each instance. Enlarge the image to the original size and binarize it with a fixed threshold of 0.5. Areas with pixel values ​​≥ 0.5 are marked as the crack foreground, and areas with pixel values ​​< 0.5 are marked as the background.

[0020] S3.3. Merge the binary instance masks into the total mask using a bitwise OR operation. In Chinese, the mask merging formula is: ;

[0021] S3.4, Adopt All-1 matrix structure elements Morphological dilation is performed on the total mask to obtain the preprocessed crack mask; the morphological dilation formula is:

[0022] In the formula, For the crack mask after preprocessing, This is a morphological dilation operation. for All-1 matrix structure elements.

[0023] Furthermore, step S4 specifically includes the following steps:

[0024] S4.1, Using cross-shaped structural elements Execute the iterative morphological thinning algorithm;

[0025] S4.2 Initialize the skeleton diagram For an all-zero image, use the preprocessed crack mask. Assign to temporary image Enter the iterative loop and repeat the iteration until a temporary image is obtained. No effective crack pixels are output, and the final single-pixel width crack center skeleton line S is output.

[0026] The formula for skeleton extraction in a single iteration is:

[0027]

[0028] In the formula, For morphological erosion operations, E is a cross-shaped structuring element. For temporary images, This is the image after erosion. The image after the opening operation. S represents the skeleton features of the current layer, and S represents the overall skeleton diagram.

[0029] Furthermore, step S5 specifically includes the following steps:

[0030] S5.1. Use OpenCV's contour search algorithm to extract the contour point set of the center skeleton line S of the single-pixel width crack, and sort it continuously from beginning to end according to the path direction.

[0031] S5.2 Calculate the physical arc length of the contour by accumulating the Euclidean distances of adjacent pixels. The formula for calculating arc length is: ;

[0032] In the formula, N is the total number of points on a single profile. , These are the coordinates of two adjacent pixels on the skeleton path;

[0033] S5.3 Set a fixed arc length threshold T=20 to filter out physical arc lengths. The tiny noise profile;

[0034] S5.4 Perform equal-interval downsampling on the retained valid contour point set, set a fixed downsampling step size Step=5, and extract a path key point every 5 contour points to obtain a sparse two-dimensional path point set. .

[0035] Furthermore, step S6 specifically includes the following steps:

[0036] S6.1 Traversing a Sparse Two-Dimensional Path Point Set Each pixel coordinate The formula for obtaining the 3D coordinates is as follows: Align the point cloud matrix of the depth point cloud data with the camera calibration parameters and the parallax principle index.

[0037] ;

[0038] For three-dimensional spatial coordinates, Two-dimensional pixel coordinates, To align depth point cloud data, Millimeter-level three-dimensional spatial coordinates;

[0039] S6.2, Obtain the coordinates Save the commands sequentially to generate control instructions for the vehicle-mounted robotic arm's crack sealing operation.

[0040] The beneficial effects of this invention are:

[0041] 1. This invention uses an iterative morphological refinement algorithm to uniformly process road surface cracks of various widths and shapes into a single-pixel-width crack center skeleton line. This solves the problem that traditional detection methods can only extract the outer contour of the crack and form redundant paths on both sides. It can guide the vehicle-mounted robotic arm to strictly carry out crack filling operations along the crack center, significantly improving the crack filling and sealing effect, while effectively avoiding the ineffective waste of crack filling material.

[0042] 2. This invention relies on the aligned depth point cloud data collected by a binocular camera to achieve precise mapping from two-dimensional pixel coordinates to millimeter-level three-dimensional spatial coordinates. It can adapt in real time to changes in working distance caused by road surface undulations and vehicle vibration, effectively avoiding problems such as the robotic arm nozzle colliding with the road surface and the imbalance of spraying spacing, and significantly improving the positioning accuracy and operational stability of the operation.

[0043] 3. This invention can effectively repair tiny breaks in cracks and fill internal voids through morphological expansion processing. Combined with arc length filtering, it can accurately remove interference noise such as road stains, shadows, and textures, solving the problem of discontinuous distribution of natural cracks and the problem of false detection caused by environmental factors, which leads to broken operation paths, and ensuring that the generated crack filling path is continuous and smooth.

[0044] 4. Balancing detection accuracy and real-time calculation, it is adaptable to high-speed mobile operation scenarios. The entire process is automated without human intervention, reducing the intensity of manual labor and the safety risks of on-site road operations. It can be widely applied to automated crack maintenance operations on various types of road surfaces such as highways, municipal roads, and rural roads. Attached Figure Description

[0045] Figure 1 This is an overall flowchart of the crack three-dimensional path planning method proposed in this invention.

[0046] Figure 2 This is a flowchart of the morphological skeleton extraction algorithm in this invention.

[0047] Figure 3 This is the final skeleton diagram of the morphological skeleton extraction algorithm in this invention.

[0048] Figure 4 This is a comparison diagram of the crack centerline of the present invention.

[0049] Figure 5 This is a comparison image of the crack border of the present invention.

[0050] Figure 6 This is a visualization of the path generated by the vehicle-mounted robotic arm according to the present invention. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the accompanying drawings.

[0052] This invention discloses a crack 3D path planning method based on binocular vision and skeleton refinement.

[0053] A crack 3D path planning method based on binocular vision and skeleton refinement includes the following steps:

[0054] S1. Data Acquisition: RGB image data and aligned depth point cloud data of the road surface area are simultaneously acquired using a binocular depth camera. This step uses dedicated hardware to acquire road surface images and depth data, completing system calibration and providing accurate raw data for subsequent processing. Details are as follows:

[0055] S1.1. A ZED 2i binocular depth camera is used as the core data acquisition device. It is rigidly mounted on the rear of the road repair vehicle, directly above the base of the vehicle-mounted robotic arm, at a height of 1m above the ground. The camera lens is perpendicular to the road surface, ensuring that the camera's optical axis is perpendicular to the road surface plane. This installation method effectively reduces the impact of vehicle vibration on the relative position of the camera and robotic arm, ensuring coordinate mapping stability. At a 1m installation height, the camera's horizontal field of view can cover a road area with a width of not less than 2.2 meters, covering the single-lane operation range. The relative error of depth measurement is ≤2%, meeting the millimeter-level accuracy requirements for crack sealing operations.

[0056] Spatial coordinate transformation calibration is performed using an eye-on-hand calibration method. Images and end-effector coordinates of the robotic arm are acquired under different poses through a calibration board. The homogeneous transformation matrix Hand-Eye Matrix from the camera coordinate system to the robotic arm base coordinate system is obtained, achieving accurate transformation between the camera coordinate system and the robotic arm base coordinate system. At the same time, camera intrinsic parameter calibration and lens distortion correction are performed to eliminate image distortion errors caused by wide-angle distortion and ensure that the mapping between two-dimensional pixel coordinates and three-dimensional spatial coordinates is unbiased.

[0057] S1.2. Set the acquisition parameters using the ZED SDK API: image resolution is 1920×1080, data acquisition frame rate is 30fps, coordinate unit is set to millimeters, coordinate system is the image coordinate system, and depth mode is set to performance-priority mode. This mode balances depth calculation accuracy and computational load, ensuring real-time system operation. Simultaneously acquire left-eye RGB images of the road surface using the camera. Calculate the depth map based on the principle of binocular parallax, generating aligned depth point cloud data that corresponds one-to-one with the pixels of the RGB image. Simultaneously output the RGB image, depth map, and 3D point cloud data.

[0058] S2. Crack Semantic Segmentation: Input the RGB image into a pre-trained deep instance segmentation model, and output the original probability mask set of multiple crack instances on the road surface. This step uses a deep learning model to detect and segment road surface cracks, outputting crack instance masks to achieve accurate differentiation between cracks and the background. Specifically, it includes the following steps:

[0059] S2.1 A YOLOv11 instance segmentation model was constructed as the detection network, with the CSPDarknet backbone feature extraction network as the model's feature extraction structure. Pre-trained weights from the COCO public dataset were loaded, and fine-tuning was performed on a self-built road surface crack dataset using transfer learning. This self-built dataset contains 9000 images of road surface cracks from different scenarios, including highways, municipal roads, and rural roads, covering all types of cracks such as transverse cracks, longitudinal cracks, mesh cracks, and block cracks. It also includes complex interference scenes such as oil stains, shadows, tire marks, and road textures. All images were annotated with polygons according to instance segmentation annotation standards. The model training hyperparameters were set as follows: initial learning rate 0.001, batch size 8, training epochs 100, and a cosine annealing learning rate decay strategy to give the model strong generalization ability and adaptability to complex scenes.

[0060] S2.2 During the acquisition process, the original image of the left eye and the corresponding point cloud data are acquired in real time. The original output of the ZED2i binocular depth camera is a 4-channel BGRA format image by default. To adapt to the OpenCV image processing library, the alpha transparency channel needs to be removed to obtain a standard 3-channel BGR image for local display and logging. Since the YOLOv11 instance segmentation model is based on the RGB color space during training, the BGR image needs to be channel rearranged to convert it into an RGB format image to adapt to the input requirements of the YOLOv11 instance segmentation model. The preprocessed RGB image is input into the model, and after inference, the original probability mask set of multiple crack instances on the road surface is output, completing the accurate segmentation of the crack region.

[0061] S3. Mask Preprocessing: For each instance mask, linear interpolation for size restoration and fixed-threshold binarization are performed sequentially. These are then merged into a global crack mask via bitwise OR operations. Morphological dilation is then applied to the global mask using fixed structuring elements to repair crack breaks and fill internal voids, resulting in the preprocessed crack mask. The specific method is as follows:

[0062] S3.1 Initialize a single-channel image with all zeros of the same size as the original image as the total mask. In the image, a pixel value of 0 represents the background, and 255 represents the foreground area of ​​the crack.

[0063] S3.2, Traverse each original detection mask output by the model. Linear interpolation is used to enlarge the image to the original size, and binarization is performed with a fixed threshold of 0.5: areas with pixel values ​​≥ 0.5 are marked as crack foreground, and areas with pixel values ​​< 0.5 are marked as background;

[0064] S3.3. Merge all binarized instance masks into the total mask using a bitwise OR operation. In Chinese, the mask merging formula is: In the formula, Resize represents the linear interpolation amplification operation.

[0065] S3.4, Adopt All-1 matrix structure elements Morphological dilation is performed on the total mask to obtain the preprocessed crack mask. The morphological dilation formula is:

[0066] ;

[0067] In the formula, For the crack mask after preprocessing, This is a morphological dilation operation. for A matrix structure element consisting entirely of 1s. Morphological dilation can connect tiny breakpoints in cracks, fill internal voids, improve mask connectivity, solve the problem of discontinuous distribution of natural cracks, and provide continuous and complete input data for subsequent skeleton extraction.

[0068] S4. Crack Skeleton Extraction: An iterative morphological thinning algorithm is used to peel away edge pixels layer by layer from the preprocessed crack mask to obtain the single-pixel-width crack center skeleton line. This step is the core process, using an iterative morphological thinning algorithm to extract the single-pixel-width crack center skeleton line S, effectively solving the path redundancy problem of traditional methods that only extract the outer contour of the crack. Specifically, it includes the following steps:

[0069] S4.1, Using cross-shaped structural elements By performing an iterative morphological thinning algorithm, this structural element can better preserve line connectivity;

[0070] S4.2 Initialize the skeleton diagram For an all-zero image, use the preprocessed crack mask. Assign to temporary image The process enters an iterative loop, where each iteration peels off a layer of edge pixels using the difference between erosion and opening operations; this iteration is repeated until a temporary image is reached. If there are no valid crack pixels, the final output is a single-pixel-width crack center skeleton line S. The technical principle is as follows: the edge of the crack area is shrunken by erosion operation, the small protrusions in the eroded area are removed by opening operation, and the skeleton features located at the crack center that were removed in this iteration are extracted by set difference operation. Finally, a single-pixel-width crack center line is retained, so that the robotic arm can work along the crack center without path back and forth and without waste of crack filling glue.

[0071] The skeleton extraction formula for a single iteration is as follows:

[0072]

[0073] In the formula, For morphological erosion operations, E is a cross-shaped structuring element. For temporary images, This is the image after erosion. The image after the opening operation. S represents the skeleton features of the current layer, and S represents the overall skeleton diagram.

[0074] S5. Path Optimization: Extract the contour point set of the single-pixel width crack center skeleton line and sort the paths. Calculate the physical arc length of the contour based on Euclidean distance. Filter out noisy contours using a fixed arc length threshold. Perform equal-interval downsampling on the effective contour point set with a fixed step size to obtain a sparse two-dimensional path point set. This step is used for noise filtering and sparsification of the crack center skeleton line to ensure continuous and smooth paths and reduce the system's computational load. Specifically, it includes the following steps:

[0075] S5.1. The contour point set of the center skeleton line S of the single-pixel width crack is extracted using the contour search algorithm of OpenCV, and the contour point set is sorted continuously from beginning to end according to the path direction to avoid the problem of backtracking in the operation path of the robotic arm.

[0076] S5.2 Calculate the physical arc length of the contour by accumulating the Euclidean distances of adjacent pixels. The formula for calculating arc length is: ;

[0077] In the formula, N is the total number of points on a single profile. , These are the coordinates of two adjacent pixels on the skeleton path;

[0078] S5.3 Set a fixed arc length threshold T=20 to filter out physical arc lengths. The subtle noise profiles eliminate false detection interference caused by road surface stains and shadows.

[0079] S5.4 Perform equal-interval downsampling on the retained valid contour point set, set a fixed downsampling step size Step=5, and extract a path key point every 5 contour points to obtain a sparse two-dimensional path point set. This reduces the number of path points while ensuring path accuracy, thereby improving system computational efficiency.

[0080] S6. 3D Coordinate Mapping and Command Generation: Index the pixel coordinates of the sparse 2D path point set to the aligned depth point cloud data, obtain the corresponding 3D spatial coordinates using the 3D coordinate acquisition formula, and generate control commands for the vehicle-mounted robotic arm's crack sealing operation according to the path sequence; specifically including the following steps:

[0081] S6.1 Traversing a Sparse Two-Dimensional Path Point Set Each pixel coordinate Using the calibration parameters and parallax principle of a binocular camera, the point cloud matrix of the depth point cloud data is indexed and aligned through the ZED SDK interface to obtain the high-precision three-dimensional coordinates of the pixel in the camera coordinate system.

[0082] The formula for obtaining 3D coordinates is:

[0083] ;

[0084] For three-dimensional spatial coordinates, Two-dimensional pixel coordinates, To align depth point cloud data, Millimeter-level three-dimensional spatial coordinates;

[0085] S6.2, Obtain the coordinates Save the data sequentially to generate an instruction file in the format Crack_ID, Segment_ID, Point_ID, X, Y, Z. This file will guide the vehicle-mounted robotic arm to perform precise crack-filling operations along the crack centerline. Crack_ID represents the crack number, Segment_ID represents the segment number within the same crack segment, and Point_ID represents the sequential number of the path points.

[0086] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A crack 3D path planning method based on binocular vision and skeleton refinement, characterized in that: Includes the following steps: S1. Data Acquisition: Simultaneously acquire RGB image data and aligned depth point cloud data of the road surface area using a binocular depth camera; S2. Crack semantic segmentation: Input the RGB image into a pre-trained deep instance segmentation model and output the original probability mask set of multiple crack instances on the road surface; S3. Mask preprocessing: For each instance mask, linear interpolation size restoration and fixed threshold binarization transformation are performed sequentially. The masks are then merged into a global crack total mask through bitwise OR operation, and morphological dilation is performed to obtain the preprocessed crack mask. S4. Crack skeleton extraction: An iterative morphological thinning algorithm is used to peel off the edge pixels of the preprocessed crack mask layer by layer to obtain the center skeleton line of the crack with a single pixel width. S5. Path optimization: Extract the contour point set of the skeleton line of the crack center with a single pixel width and complete the path sorting. Calculate the physical arc length of the contour based on Euclidean distance. Filter out noisy contours according to a fixed arc length threshold. Perform equal-interval downsampling on the effective contour point set with a fixed step size to obtain a sparse two-dimensional path point set. S6. 3D Coordinate Mapping and Command Generation: Index the pixel coordinates of the sparse 2D path point set to the aligned depth point cloud data, obtain the corresponding 3D spatial coordinates, and generate control commands for the vehicle-mounted robotic arm crack sealing operation according to the path sequence.

2. The crack 3D path planning method based on binocular vision and skeleton refinement according to claim 1, characterized in that: In step S1, a ZED series binocular depth camera is used. The binocular depth camera is fixed directly above the base of the road repair vehicle's robotic arm, with the lens pointing vertically towards the ground. The installation height is 1m above the ground. The horizontal field of view of the binocular depth camera covers a road area with a width of not less than 2.2m, and the relative error of the depth measurement is ≤2%. The binocular depth camera acquires RGB images with a resolution of 1920×1080 at a data acquisition frame rate of 30fps, and calculates the depth map based on the parallax principle to generate aligned depth point cloud data.

3. The crack 3D path planning method based on binocular vision and skeleton refinement according to claim 1, characterized in that: Step S2 specifically includes the following steps: S2.1 Construct a YOLO series instance segmentation model, using the CSPDarknet backbone feature extraction network as the model feature extraction structure, and perform transfer learning training on a dedicated dataset for road cracks. S2.

2. Extract the Alpha channel from the BGRA format image output by the binocular depth camera to obtain the BGR image, and then rearrange the channels of the BGR image to convert it into an RGB format image, which is used as input data for the YOLO series instance segmentation model.

4. The crack 3D path planning method based on binocular vision and skeleton refinement according to claim 3, characterized in that: Step S3 specifically includes the following steps: S3.1 Initialize a single-channel image with all zeros of the same size as the original image as the total mask. ; S3.2, Use linear interpolation to mask each instance. Enlarge the image to the original size and binarize it with a fixed threshold of 0.

5. Areas with pixel values ​​≥ 0.5 are marked as the crack foreground, and areas with pixel values ​​< 0.5 are marked as the background. S3.

3. Merge the binary instance masks into the total mask using a bitwise OR operation. In Chinese, the mask merging formula is: ; S3.4, Adopt All-1 matrix structure elements Morphological dilation is performed on the total mask to obtain the preprocessed crack mask; the morphological dilation formula is: In the formula, For the crack mask after preprocessing, This is a morphological dilation operation. for All-1 matrix structure elements.

5. The crack 3D path planning method based on binocular vision and skeleton refinement according to claim 4, characterized in that: Step S4 Specifically, the following steps are included: S4.1, Using cross-shaped structural elements Execute the iterative morphological thinning algorithm; S4.2 Initialize the skeleton diagram For an all-zero image, use the preprocessed crack mask. Assign to temporary image Enter the iterative loop and repeat the iteration until a temporary image is obtained. If there are no effective crack pixels, output the final single-pixel width crack center skeleton line S; The formula for skeleton extraction in a single iteration is: In the formula, For morphological erosion operation, E is a cross-shaped structuring element. For temporary images, This is the image after erosion. The image after the opening operation. S represents the skeleton features of the current layer, and S represents the overall skeleton diagram.

6. The crack 3D path planning method based on binocular vision and skeleton refinement according to claim 5, characterized in that: Step S5 specifically includes the following steps: S5.

1. Use OpenCV's contour search algorithm to extract the contour point set of the center skeleton line S of the single-pixel width crack, and sort it continuously from beginning to end according to the path direction. S5.2 Calculate the physical arc length of the contour by accumulating the Euclidean distances of adjacent pixels. The formula for calculating arc length is: ; In the formula, N is the total number of points on a single profile. , These are the coordinates of two adjacent pixels on the skeleton path; S5.3 Set a fixed arc length threshold T=20 to filter out physical arc lengths. The tiny noise profile; S5.4 Perform equal-interval downsampling on the retained valid contour point set, set a fixed downsampling step size Step=5, and extract a path key point every 5 contour points to obtain a sparse two-dimensional path point set. .

7. The crack 3D path planning method based on binocular vision and skeleton refinement according to claim 6, characterized in that: Step S6 specifically includes the following steps: S6.1 Traversing a Sparse Two-Dimensional Path Point Set Each pixel coordinate The formula for obtaining the 3D coordinates is as follows: Align the point cloud matrix of the depth point cloud data with the camera calibration parameters and the parallax principle index. ； For three-dimensional spatial coordinates, Two-dimensional pixel coordinates, To align depth point cloud data, Millimeter-level three-dimensional spatial coordinates; S6.2, Obtain the coordinates Save the commands sequentially to generate control instructions for the vehicle-mounted robotic arm's crack sealing operation.

Images