Building block pavement disease detection method and equipment based on RGB-D image

Abstract

The invention relates to a building block pavement disease detection method and equipment based on an RGB-D image, and the method comprises the steps: collecting a pavement color image and a depth image, inputting a pre-constructed semantic segmentation model, and obtaining a segmentation result of a building block and a thin and narrow seam; based on the depth image, mapping a segmentation result of the building block and the thin and narrow seam to a three-dimensional point cloud space; performing plane fitting on the point cloud data by adopting a random sampling consistency algorithm, extracting a point cloud contour, and calculating horizontal displacement between adjacent building blocks; respectively fitting the point clouds of the adjacent building blocks as independent planes by adopting a random sampling consistency algorithm, and calculating the vertical displacement between the adjacent building blocks; and based on the horizontal displacement and the vertical displacement between the adjacent building blocks, generating quantitative data of horizontal displacement and vertical displacement of the building block pavement of the target road section, marking disease positions exceeding a preset limit value, and completing disease detection of the building block pavement. Compared with the prior art, the method realizes accurate quantification of horizontal and vertical displacement between the building blocks of the building block pavement.

Description

Technical Field

[0001] This invention relates to the fields of non-destructive testing and computer vision technology in road engineering, and in particular to a method and equipment for detecting defects in masonry pavement based on RGB-D images. Background Technology

[0002] Due to external factors such as underground construction disturbance, foundation settlement, and environmental loads, block pavements are prone to horizontal and vertical displacement, affecting structural stability and service performance. Pavement distress detection is a crucial step in pavement maintenance. The detection results directly determine the maintenance mileage and strategies; therefore, an efficient and accurate monitoring method is urgently needed.

[0003] Currently, the condition assessment of block pavements still mainly relies on manual inspections, such as visually identifying joint damage and measuring displacement with a ruler. These methods suffer from low efficiency, limited accuracy, and strong subjectivity, making it difficult to meet the needs of rapid and objective monitoring of large-scale pavement systems.

[0004] Chinese patent application CN119579508A discloses a method for detecting and evaluating defects in railway tunnel lining based on anchorless instance segmentation. This method trains a deep learning model to obtain segmented mask images, and then calculates the minimum length of various defects based on these mask images to assess the tunnel's deterioration level. However, this method relies solely on the collected pixel images for defect detection and evaluation, resulting in insufficient accuracy and a limited range of calculated results. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method and device for detecting block pavement defects based on RGB-D images.

[0006] The objective of this invention can be achieved through the following technical solutions: A method for detecting defects in masonry pavement based on RGB-D images, the method comprising: RGB-D camera is used to acquire road surface color images and depth images and preprocess them. The preprocessed color images are then input into a pre-constructed semantic segmentation model to obtain the segmentation results of blocks and narrow joints. Based on the depth image, the segmentation results of the blocks and narrow seams are mapped to a three-dimensional point cloud space to obtain point cloud data; The point cloud data is fitted to a plane using a random sampling consistency algorithm. The point cloud contour is obtained by a concave polygon boundary extraction algorithm. The boundary line segments of adjacent blocks are obtained by a least squares straight line fitting method. N points are uniformly sampled along each boundary line segment, and the horizontal displacement between adjacent blocks is calculated based on the sampled points. The point clouds of adjacent blocks are fitted as independent planes using the random sampling consensus algorithm. The vertical distances from the left point cloud to the right plane and from the right point cloud to the left plane are calculated. The average distance between the two sides is taken as the vertical displacement between adjacent blocks. Based on the horizontal and vertical displacements between adjacent blocks, quantitative data of the horizontal and vertical displacements of the block pavement in the target road section are generated, and the locations of defects exceeding preset limits are marked, thus completing the defect detection of the block pavement.

[0007] Furthermore, the RGB-D camera is mounted on a data acquisition device that can move slowly along the urban block road surface, and continuously acquires color and depth images based on a preset acquisition frequency and acquisition duration while the data acquisition device is moving slowly along the urban block road surface.

[0008] Furthermore, the semantic segmentation model is constructed by combining a Transformer encoder and a U-Net decoder.

[0009] Furthermore, the semantic segmentation model is pre-trained based on a data training set after data augmentation processing. The data training set includes an image set covering the facades of ancient walls and newly built buildings, as well as an image set of masonry pavement collected from multiple perspectives and scenes. The data augmentation processing includes flipping and brightness adjustment. In the data training set, all images are labeled using annotation tools.

[0010] Furthermore, the process of labeling the images in the training data set using annotation tools includes: The images in the training set are labeled using LabelMe to obtain a binary mask for each image. In the binary mask, 0 represents a seam and 255 represents a block. The Canny algorithm is used to perform edge detection on the binary mask to extract the boundary line between the seam and the block. With the boundary line as the center, the boundary line is expanded to both sides by a preset number of pixels to obtain the candidate box of the transition area. Using a depth-based transition calculation formula, the transition values ​​of image pixels in the candidate box of the transition region are calculated, and a grayscale mask including the binary mask and the transition values ​​is generated by combining the original binary mask. Output the image and its corresponding grayscale mask to complete the annotation.

[0011] Furthermore, during the pre-training of the semantic segmentation model, a hybrid loss function is used as the main convergence metric. This hybrid loss function includes the intersection-union ratio loss and the mean squared error loss for adaptive continuous transition value regression. When the semantic segmentation model segments the preprocessed color image, it outputs a grayscale mask and compares the transition values ​​of the image pixels with an adaptive threshold to determine whether the image pixels are classified as seams or blocks, thus obtaining the segmentation results for blocks and narrow seams. The adaptive threshold is calculated based on the acquired road surface depth image.

[0012] Furthermore, the process of calculating the horizontal displacement between adjacent blocks based on sampling points includes: N points are uniformly sampled along each boundary line segment. Based on the adjacency relationship between the boundary line segments, the shortest distance from each sampled point on each line segment to its adjacent line segment, and the shortest distance from each sampled point on the adjacent line segment to its corresponding line segment are calculated to obtain a first set of shortest distances and a second set of shortest distances. Based on the first set of shortest distances and the second set of shortest distances, the average distance from all sampled points on each line segment to its adjacent line segment and the average distance from all sampled points on the adjacent line segment to its corresponding line segment are calculated. The average of the two average distances is then taken again to obtain the horizontal displacement between adjacent blocks.

[0013] Furthermore, when performing planar fitting on the point cloud data using the aforementioned random sampling consistency algorithm, only the point cloud data corresponding to the blocks and joints are retained, while irrelevant interference points are eliminated.

[0014] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the block pavement defect detection method based on RGB-D images as described above.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the block pavement defect detection method based on RGB-D images as described above.

[0016] Compared with the prior art, the beneficial effects of the present invention include: 1. This invention proposes a method for detecting defects in brick pavement based on road surface color images and depth images. It enables precise quantification of horizontal and vertical displacements between brick blocks in brick pavement. Compared to existing high-cost LiDAR systems and 2D image processing methods, the application of an RGB-D camera in this method significantly reduces system deployment costs and environmental constraints, while also improving the real-time performance and scalability of field applications. This method demonstrates great potential and application value, especially in situations where complex instruments cannot be deployed on-site or where rapid response is required. This invention maps the brick segmentation results to a point cloud space based on depth images, then fits the point cloud data using the Random Sample Consensus (RANSAC) algorithm, effectively eliminating the influence of 3D disturbances on boundary extraction. A concave polygon boundary extraction algorithm is used to obtain the point cloud contour, and the boundary line segments are obtained through least-squares line fitting, adapting to irregular brick boundaries and capturing narrow joint ranges more accurately than traditional boundary extraction methods. This invention quantifies the horizontal and vertical displacement of brick defects based on three-dimensional point clouds, filling the gap in two-dimensional image technology that cannot distinguish three-dimensional defects, avoiding misjudgment and missed detection. For the boundary features of different paving patterns such as herringbone, woven, and extended, the horizontal displacement can be accurately quantified through line fitting and sampling calculation, with no scene limitations and strong adaptability.

[0017] 2. This invention segments masonry blocks and narrow seams using a pre-built semantic segmentation model. The training set of the semantic segmentation model not only includes a set of masonry pavement images collected from multiple perspectives and scenes, but also includes an image set covering ancient walls and the facades of newly built buildings. The wall images cover diverse brick textures, arrangement structures, and light and shadow variations, which can enrich the model's feature learning and improve its generalization ability. At the same time, the addition of the wall dataset also reduces the cost of collecting and labeling pavement-specific datasets.

[0018] 3. This invention utilizes a lightweight road surface information acquisition device equipped with an RGB-D camera, which can travel slowly along urban block roads, enabling continuous acquisition and real-time transmission of color-depth images. It achieves large-scale, high-efficiency data acquisition without manual intervention, significantly reducing inspection costs and meeting the needs of daily and rapid inspection of urban roads.

[0019] 4. When annotating images in the training set, this invention first uniformly annotates binary masks, then combines Canny edge detection to extract boundary lines, calculates transition values ​​based on depth distance and generates grayscale masks, and integrates binary hard segmentation and gradient transition information. This not only preserves the clear target region division, but also restores the physical features of boundary gradation in real scenes, providing richer feature support for model learning.

[0020] 5. In this invention, when training the semantic segmentation model using the training set, the intersection-over-union (IoU) loss and mean squared error (MSE) loss are fused to balance the dual requirements of boundary alignment and transition value regression: the IoU loss ensures accurate segmentation of the core regions of blocks and seams, while the MSE loss optimizes the fitting of continuous values ​​in the transition region, avoiding boundary ambiguity or unnatural transitions caused by a single loss function. For the task of segmenting narrow seams, the hybrid loss function can guide the model to focus on both the integrity of the target region and the details of the boundary, and also improves the subsequent displacement quantization.

[0021] 6. When using the semantic segmentation model, this invention outputs a grayscale mask and compares the transition values ​​of image pixels with an adaptive threshold to obtain the segmentation results of blocks and narrow joints. The adaptive threshold is calculated based on the acquired road surface depth image, rather than a fixed value, which can dynamically adapt to the differences in image features under different scenarios, avoid missegmentation caused by a one-size-fits-all approach, reduce the problem of surface texture confusion in three-dimensional defects that is prone to occur in two-dimensional image segmentation, and reduce errors for subsequent horizontal / vertical displacement quantization.

[0022] 7. This invention employs a semantic segmentation model that integrates a Transformer encoder and a U-Net decoder, achieving outstanding segmentation results for blocks and narrow joints in masonry pavements.

[0023] 8. The proposed method for detecting defects in masonry pavement shows that, in actual experiments, the measurement error is generally controlled within 0.8 mm. Furthermore, field experimental results demonstrate that, regardless of whether the displacement is low (2 mm) or large (10 mm), the proposed method maintains measurement errors within 0.7 mm in both horizontal and vertical directions, with standard deviations within 1.1 mm, exhibiting good measurement stability and repeatability. Field verification confirms the method's high accuracy, with an RMSE below 0.40 mm and a MAPE below 9%. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method of the present invention.

[0025] Figure 2 This is a schematic diagram of the data acquisition device of the present invention; Figure 3 This is a schematic diagram of the displacement measurement method of the present invention; Figure 4 This is a comparison chart of prediction results in an embodiment of the present invention; Figure 5 This is a schematic diagram of the detection area in an embodiment of the present invention; Figure 6 This is a normal distribution diagram of the measured displacement in an embodiment of the present invention; Figure 7This is a comparison diagram of the actual displacement and measured displacement of different block pavements in the embodiments of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] Example 1 This embodiment discloses a method for detecting defects in masonry pavement based on RGB-D images. The method is as follows: Figure 1 As shown, the process includes steps S1-S5, each described in detail below: Step S1: Use an RGB-D camera to acquire color and depth images of the road surface and perform preprocessing. Input the preprocessed color images into a pre-built semantic segmentation model to obtain the segmentation results of the blocks and narrow joints.

[0028] RGB-D camera equipped with such Figure 2 The acquisition device shown can move slowly along the urban block road surface, and continuously acquires color and depth images while the acquisition device is moving slowly along the urban block road surface based on a preset acquisition frequency and acquisition duration.

[0029] In practical use, it can slowly move along the street to collect ground images while simultaneously detecting road surface defects.

[0030] Preprocessing specifically includes: The synchronized color and depth images are uniformly cropped and scaled to a preset pixel size to adapt to the input requirements of the subsequent semantic segmentation model; Depth images are smoothed to remove abnormal depth values ​​caused by environmental interference; color images are lightly denoised to preserve block texture and seam features while reducing the impact of shooting noise.

[0031] The semantic segmentation model is constructed by combining a Transformer encoder and a U-Net decoder. It has outstanding global modeling capabilities and moderate multi-scale information processing, making it suitable for high-precision segmentation applications that require the fusion of global and local information.

[0032] The semantic segmentation model is pre-trained on a data training set that has undergone data augmentation. The data training set includes an image set covering the facades of ancient walls and newly built buildings, as well as an image set of masonry pavement collected from multiple perspectives and scenes. The data augmentation process includes flipping and brightness adjustment. In the data training set, all images are labeled using annotation tools.

[0033] Specifically, in this embodiment, the image set covering both ancient walls and the facades of newly constructed buildings includes 353 512×512 pixel images of exposed wall surfaces, used for model pre-training to learn diverse brick textures and structural features. The multi-view, multi-scene image set of masonry pavement includes 120 masonry pavement images acquired from multiple scenes and perspectives, such as motor vehicle lanes and sidewalks, using smartphones and RGB-D cameras, used to fine-tune the pre-trained model to improve its segmentation performance of pavement blocks and joints.

[0034] To ensure consistency and accuracy of annotations, all images are manually annotated using LabelMe, with a uniform pixel value of 0 representing a seam and 255 representing a block.

[0035] To further enhance the model's generalization ability and robustness, data augmentation was performed on both datasets. Augmentation techniques used included flipping and brightness adjustment. After augmentation, the image set covering ancient walls and the facades of newly built buildings was expanded to 1412 images, and the image set of masonry pavement collected from multiple perspectives and scenes was expanded to 480 images, effectively enhancing the diversity and scale of the training data.

[0036] In this embodiment, the following settings are used for model training: The training period is 100 epochs, the batch size is 16, the input image size is 512×512 pixels, and the intersection-over-union (IoU) loss is used as the main convergence metric.

[0037] Step S2: Based on the depth image, the segmentation results of the blocks and narrow seams are mapped to the three-dimensional point cloud space to obtain point cloud data.

[0038] Step S3: The Random Sample Consensus (RANSAC) algorithm is used to perform planar fitting on the point cloud data. The point cloud contour is obtained by the concave polygon boundary extraction algorithm. The boundary line segments of adjacent blocks are obtained by the least squares line fitting method. N points are uniformly sampled along each boundary line segment, and the horizontal displacement between adjacent blocks is calculated based on the sampled points.

[0039] The process of calculating the horizontal displacement between adjacent blocks based on sampling points includes: N points are uniformly sampled along each boundary line segment. Based on the adjacency relationship between the boundary line segments, the shortest distance from each sampled point on each line segment to its adjacent line segment, and the shortest distance from each sampled point on an adjacent line segment to its corresponding line segment are calculated to obtain a first set of shortest distances and a second set of shortest distances. Based on the first set of shortest distances and the second set of shortest distances, the average distance from all sampled points on each line segment to its adjacent line segment and the average distance from all sampled points on an adjacent line segment to its corresponding line segment are calculated. The average of the two average distances is then taken again to obtain the horizontal displacement between adjacent blocks.

[0040] Step S4: Use the random sampling consensus algorithm to fit the point clouds of adjacent blocks as independent planes, calculate the vertical distance from the left point cloud to the right plane and the right point cloud to the left plane, and take the average distance on both sides as the vertical displacement between adjacent blocks.

[0041] The effectiveness and robustness of the random sample consensus algorithm are affected by parameters such as the number of samples, the number of iterations, and the threshold. When using the random sample consensus algorithm to perform planar fitting on point cloud data, only the point cloud data corresponding to blocks and joints are retained, while irrelevant and interfering points are removed.

[0042] Specifically, in this embodiment, the maximum number of iterations of the random sampling consensus algorithm is set to 10,000, and the distance threshold is set to 0.002 meters to ensure the accuracy and stability of planar extraction.

[0043] The planar parameters are calculated from the point cloud using the following formula: Where a, b, c, and d represent plane coefficients.

[0044] Typical defects in block joints include excessive joint width and misalignment. The former manifests as excessive horizontal displacement between blocks, while the latter mainly involves abnormal vertical displacement. This embodiment proposes precise quantification methods for both horizontal and vertical displacements, the basic principles of which are as follows: Figure 3 As shown.

[0045] To quantify horizontal displacement, a random sampling consensus algorithm is first used to fit the point cloud plane, and the point cloud is then projected onto this plane to eliminate the influence of 3D perturbation on boundary extraction. Subsequently, a concave polygon boundary extraction algorithm (Concave Hull, Alpha=0.1) is used to obtain the point cloud contour, and the boundary line segment is obtained through a least-squares line fitting method, denoted as... and And sample N points evenly along each line segment, denoted as . and Its coordinates can be calculated using the following formula: in, and Indicates the endpoints of the first line segment. and This indicates the endpoint of the second line segment.

[0046] Next, the shortest distance from each sampling point to the other line segment is calculated using the following formula. and : in, and These are the proportional parameters on the line segment, with values ​​of 0 and 1 corresponding to the start and end points of the line segment.

[0047] Ultimately, horizontal displacement By average width on both sides and Calculation of the average value: To achieve accurate quantification of the vertical displacement of the block pavement, this embodiment, for example, proposes a vertical displacement calculation method based on plane fitting in step S4. By fitting the three-dimensional point clouds of two selected blocks as a plane, the vertical distance between the point clouds on both sides is calculated. As shown in the following formula, the vertical distance between any point on the plane can be obtained. Distance to its opposite plane .

[0048] For quantization of vertical displacement, the point clouds of the blocks on both sides are fitted as planes, and the distance from the point cloud to the opposite plane is calculated. An arbitrary point on a plane... vertical distance to another plane It can be obtained using the following formula: To comprehensively assess the vertical displacement, the average distance between the two sides was used as the final estimate. : in, This represents the average distance from the left point cloud to the right fitted plane. This represents the average distance from the right-side point cloud to the left-side fitted plane. and These represent the number of point clouds on the left and right sides, respectively.

[0049] Step S5: Based on the horizontal and vertical displacements between adjacent blocks, generate quantitative data of the horizontal and vertical displacements of the block pavement in the target road section, and mark the locations of defects that exceed the preset limits, thus completing the defect detection of the block pavement.

[0050] Example 2 This embodiment is based on the RGB-D image-based block pavement defect detection method disclosed in Embodiment 1 above, and provides a practical application example and accuracy demonstration.

[0051] The following settings are used for model training in this embodiment.

[0052] The training cycle is 100 epochs, the batch size is 16, and the input image size is 512×512 pixels. All models use default training parameters and environment configurations to ensure sufficient learning of data features.

[0053] To evaluate the performance of different network architectures in the block pavement segmentation task, this embodiment selects three models—U-Net, DeepLabv3+, and the semantic segmentation model proposed in this method—for comparative experiments.

[0054] Under complex imaging conditions, this embodiment systematically evaluated the performance of three semantic segmentation models in different interference scenarios, specifically including three typical working conditions: scenes with clear textures, scenes with shadow effects, and scenes with severe pollution.

[0055] The performance of the three models shows a significant gradient difference, and the prediction mask comparison results are as follows: Figure 4 As shown in Table 1 below, the performance of the three models exhibits a clear gradient difference. The evaluation metrics used are Intersection over Union (IoU), Precision, Recall, and F1 score. Detailed numerical results are shown in Table 1 below.

[0056] Table 1 Performance comparison of different models The stability and measurement accuracy of the RGB-D camera were systematically evaluated through preliminary tests.

[0057] During the experiment, the depth camera was fixed on an adjustable bracket at a height of 0.75 meters and kept strictly perpendicular to the ground. To ensure the consistency of data acquisition, the camera's default parameter settings were used, including a resolution of 1280×720 pixels, a frame rate of no less than 30fps, and a duration of no less than 5 seconds for each acquisition.

[0058] This embodiment uses high-density polystyrene foam material to simulate the horizontal and vertical displacement of the block pavement. Data is collected through pre-tests under different preset working conditions. The block size is uniformly set as 100mm × 200mm rectangular bricks, covering three typical paving patterns: extended, woven, and herringbone. The arrangement of various blocks and joint numbering are as follows: Figure 5 As shown.

[0059] To evaluate the impact of inherent camera errors on the measurement results, this embodiment selects herringbone block pavement as the research object and conducts repeatability tests under typical conditions of 2 mm horizontal displacement and 0 mm vertical displacement. Each frame of data acquired by the camera is considered an independent measurement, and 100 independent repeat tests are performed under this condition to verify the stability of the proposed method. Statistical results are as follows: Figure 6 As shown, both the horizontal and vertical displacements of the joints exhibit a normal distribution. Table 2 presents the measurement results of the joint displacements in herringbone block pavements; Tables 3 and 4 correspond to the measurement data for woven and extended block pavements, respectively.

[0060] Table 2. Calculation results of displacement in herringbone arrangement Table 3. Calculation results of displacement for woven arrangement Table 4. Calculation results of displacement for extended arrangement This embodiment also systematically verified the displacement measurement under three block pavement laying processes—herringbone, woven, and extended—through field experiments.

[0061] Experimental results show that using an RGB-D camera can effectively acquire high-quality images of block pavement and generate point cloud data with good initial structure, laying a good foundation for subsequent accurate quantitative analysis of horizontal and vertical displacements and helping to reduce the impact of unnecessary systematic errors on experimental results.

[0062] The results of joint displacement measurements for three different road sections are shown in Table 5. Figure 7 The study demonstrates the fitting effect between the measured displacement values ​​and the actual values ​​for 20 joints in each of three road sections. The measured values ​​for both horizontal and vertical displacements show high consistency with the actual values, and the goodness of fit (R0) is high. 2 All values ​​were 0.99. For horizontal displacement, the RMSE was 0.30 mm, MAPE was 6.21%, and MFE was 0.16 mm; for vertical displacement, the RMSE was 0.28 mm, MAPE was 8.73%, and MFE was 0.10 mm.

[0063] Table 5 Displacement characteristics of joints on different road surfaces Example 3 This embodiment is based on the RGB-D image-based block pavement defect detection method disclosed in Embodiment 1 above. It discloses a block pavement defect detection method that uses a special annotation method to annotate the training set when constructing the data training set of the semantic segmentation model, and uses a hybrid loss function as the main convergence index to train the semantic segmentation model.

[0064] The process of labeling images in the training set using annotation tools includes: LabelMe is used to label the images in the training set and obtain the binary mask of each image. In the binary mask, 0 represents a seam and 255 represents a block. The Canny algorithm is used to perform edge detection on the binary mask to extract the boundary lines between the seams and the blocks. The boundary lines are then expanded to both sides by a preset number of pixels to obtain candidate boxes for the transition region. Using a depth-based transition calculation formula, image pixels within the candidate transition region are calculated. Transition value And combine the original binary mask to generate a grayscale mask that includes the binary mask and the transition value; Output the image and its corresponding grayscale mask to complete the annotation.

[0065] Specifically, the principle behind the depth-based transition calculation formula for calculating the transition values ​​of image pixels within the candidate transition region is based on the height difference between the seam and the block in the depth image. The depth value of the transition region lies between these two values, and the transition value is obtained through depth normalization. Its specific expression is: in, This represents the average depth of the seam area. This represents the average depth of the block area. For transition region pixels The depth value.

[0066] In response to changes in the training data set, the training method for semantic segmentation models also needs to be changed accordingly.

[0067] The semantic segmentation model uses a hybrid loss function as the main convergence metric during pre-training.

[0068] The hybrid loss function includes the crossover ratio loss and the mean squared error loss for adaptive continuous transition value regression, and its specific expression is as follows: in, For mixed loss, To lose weight, To compare the losses, To accommodate the mean squared error loss of continuous transition value regression, To predict transition values ​​for the model, To indicate the true value, and This refers to the image size.

[0069] Mean squared error loss ensures numerical fitting in the transition region, while cross-union ratio loss avoids boundary blurring, adapting to the segmentation requirements of narrow seams.

[0070] The training parameters used during training were also adjusted, specifically: the learning rate was changed from 0.001 for binary classification to 0.0005 to avoid oscillations during regression training; and the training period was extended to 120 epochs to ensure that the model learns continuous transition features.

[0071] When the semantic segmentation model segments the preprocessed color image, it outputs a grayscale mask and compares the transition values ​​of the image pixels with an adaptive threshold to determine whether the image pixels are classified as seams or blocks, thus obtaining the segmentation results of blocks and narrow seams; the adaptive threshold is calculated based on the acquired road surface depth image.

[0072] Adaptive threshold setting essentially involves splitting transition regions in a grayscale image. The process of calculating and obtaining this threshold based on the acquired road surface depth image includes: Obtain the grayscale mask image output by the semantic segmentation model; Perform pixel value histogram statistics on the grayscale mask image, record the frequency of occurrence of pixels in the transition area of ​​the mask from 1 to 254, and identify the double peaks and valleys in the histogram. The double peaks are specifically the low pixel value peaks near the seams and the high pixel value peaks near the blocks, and the valleys are specifically the lowest pixel values ​​between the double peaks. Calculate the statistical characteristics of the grayscale image: mean, median, and 75th percentile of pixel values, as a reference for threshold adjustment; The Otsu thresholding method (maximum inter-class variance method) is used to calculate the initial threshold that maximizes the variance between seam class pixels (0 + low transition value) and block class pixels (255 + high transition value). The specific method is as follows: iterate through the pixel values ​​of the pixels with a mask of 1-254, calculate the inter-class variance when each value is used as a threshold, and select the pixel value with the largest variance as the initial threshold to ensure the optimal separation between the two types of regions. From the original depth image, extract the depth samples corresponding to pixels with a mask of 0 (seams) in the grayscale mask image, calculate the average depth of the seams, extract the depth samples corresponding to pixels with a mask of 255 (blocks), and calculate the average depth of the blocks; find the abrupt change region (i.e., the actual physical boundary) between the average depth of the seams and the average depth of the blocks in the depth image, map it to the grayscale mask image, and record the transition value range of this region, if the initial threshold... If the value does not fall within the transition range, it is corrected to the middle value of that range to ensure that the binarized boundary is consistent with the actual 3D physical boundary and to avoid 2D texture interference. If the initial threshold... If it falls within the transition value range, it is used as the threshold. Output.

[0073] The final output is still a binary segmentation map, which is directly connected to subsequent processes such as RANSAC plane fitting and Concave Hull boundary extraction for horizontal / vertical displacement quantization.

[0074] Example 4 Based on Embodiment 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned method for detecting block pavement defects based on RGB-D images.

[0075] At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the above-mentioned method for detecting block pavement defects based on RGB-D images. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0076] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0077] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0078] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting defects in masonry pavement based on RGB-D images, characterized in that, The method includes: RGB-D camera is used to acquire road surface color images and depth images and preprocess them. The preprocessed color images are then input into a pre-constructed semantic segmentation model to obtain the segmentation results of blocks and narrow joints. Based on the depth image, the segmentation results of the blocks and narrow seams are mapped to a three-dimensional point cloud space to obtain point cloud data; The point cloud data is fitted to a plane using a random sampling consistency algorithm. The point cloud contour is obtained by a concave polygon boundary extraction algorithm. The boundary line segments of adjacent blocks are obtained by a least squares straight line fitting method. N points are uniformly sampled along each boundary line segment, and the horizontal displacement between adjacent blocks is calculated based on the sampled points. The point clouds of adjacent blocks are fitted as independent planes using the random sampling consensus algorithm. The vertical distances from the left point cloud to the right plane and from the right point cloud to the left plane are calculated. The average distance between the two sides is taken as the vertical displacement between adjacent blocks. Based on the horizontal and vertical displacements between adjacent blocks, quantitative data of the horizontal and vertical displacements of the block pavement in the target road section are generated, and the locations of defects exceeding preset limits are marked, thus completing the defect detection of the block pavement.

2. The method for detecting block pavement defects based on RGB-D images according to claim 1, characterized in that, The RGB-D camera is mounted on a data acquisition device that can move slowly along the urban block road surface. Based on a preset acquisition frequency and acquisition duration, the data acquisition device continuously acquires color and depth images while moving slowly along the urban block road surface.

3. The method for detecting block pavement defects based on RGB-D images according to claim 1, characterized in that, The semantic segmentation model is constructed by combining a Transformer encoder and a U-Net decoder.

4. The method for detecting block pavement defects based on RGB-D images according to claim 1, characterized in that, The semantic segmentation model is pre-trained on a data training set after data augmentation. The data training set includes an image set covering the facades of ancient walls and newly built buildings, as well as an image set of masonry pavement collected from multiple perspectives and scenes. The data augmentation process includes flipping and brightness adjustment. In the data training set, all images are labeled using annotation tools.

5. The method for detecting block pavement defects based on RGB-D images according to claim 4, characterized in that, The process of labeling images in the training set using annotation tools includes: The images in the training set are labeled using LabelMe to obtain a binary mask for each image. In the binary mask, 0 represents a seam and 255 represents a block. The Canny algorithm is used to perform edge detection on the binary mask to extract the boundary line between the seam and the block. With the boundary line as the center, the boundary line is expanded to both sides by a preset number of pixels to obtain the candidate box of the transition area. Using a depth-based transition calculation formula, the transition values ​​of image pixels in the candidate box of the transition region are calculated, and a grayscale mask including the binary mask and the transition values ​​is generated by combining the original binary mask. Output the image and its corresponding grayscale mask to complete the annotation.

6. The method for detecting block pavement defects based on RGB-D images according to claim 5, characterized in that, The semantic segmentation model uses a hybrid loss function as the main convergence metric during pre-training. The hybrid loss function includes the intersection-union ratio loss and the mean squared error loss for adaptive continuous transition value regression. When segmenting the preprocessed color image, the semantic segmentation model outputs a grayscale mask and compares the transition values ​​of the image pixels with an adaptive threshold to determine whether the image pixels are classified as seams or blocks, thus obtaining the segmentation results for blocks and narrow seams. The adaptive threshold is calculated based on the acquired road surface depth image.

7. The method for detecting block pavement defects based on RGB-D images according to claim 1, characterized in that, The process of calculating the horizontal displacement between adjacent blocks based on sampling points includes: N points are uniformly sampled along each boundary line segment. Based on the adjacency relationship between the boundary line segments, the shortest distance from each sampled point on each line segment to its adjacent line segment, and the shortest distance from each sampled point on the adjacent line segment to its corresponding line segment are calculated to obtain a first set of shortest distances and a second set of shortest distances. Based on the first set of shortest distances and the second set of shortest distances, the average distance from all sampled points on each line segment to its adjacent line segment and the average distance from all sampled points on the adjacent line segment to its corresponding line segment are calculated. The average of the two average distances is then taken again to obtain the horizontal displacement between adjacent blocks.

8. The method for detecting block pavement defects based on RGB-D images according to claim 1, characterized in that, When performing planar fitting on point cloud data using the aforementioned random sampling consistency algorithm, only the point cloud data corresponding to the blocks and joints are retained, while irrelevant interference points are removed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the block pavement defect detection method based on RGB-D images as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the block pavement defect detection method based on RGB-D images as described in any one of claims 1-8.

Images