Method and system for detecting size of pavement patching area based on vehicle-mounted equipment

Abstract

The invention provides a pavement patching area size detection method and system based on vehicle-mounted equipment, and relates to the technical field of civil engineering, and the method comprises the steps: obtaining an inclined pavement image collected by the vehicle-mounted equipment; using the instance segmentation model to carry out target segmentation of the repairing area on the inclined pavement image to obtain a pixel-level segmentation map; correcting the pixel-level segmented image in combination with camera equipment parameters and image depth information; and based on the corrected pixel-level segmentation image, carrying out geometric measurement and calculation on the length and the width of the repaired area to obtain the size of the repaired area. According to the method, full-process automation from target detection, pixel-level segmentation to geometric dimension measurement and calculation is realized, low efficiency and subjectivity of traditional manual patrol are avoided, and the efficiency and scientificity of road surface detection and evaluation are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of civil engineering, in particular to a method and system for detecting the size of a road repair area based on a vehicle-mounted device. BACKGROUND

[0002] In economically developed areas, the construction of expressway networks has reached a considerable scale. However, one of the main challenges currently faced is how to efficiently assess the technical condition of these roads, and to improve the scientificity and accuracy of the assessment process, it is urgent to adopt scientific and academic methods; expressways mostly use asphalt pavement, and professional road maintenance departments can repair cracks in a timely manner, therefore, the main concern of expressway pavement diseases is focused on repair treatment; however, due to the limitations of current road repair technology, the repaired asphalt pavement is difficult to completely restore to the original state, therefore, the size of the repair area becomes an important indicator for assessing the technical condition of expressways.

[0003] Traditional manual detection methods for the size of the repair area are inefficient and subjective, in order to meet the current demand for rapid and accurate assessment of the technical condition of expressways, it is necessary to propose an algorithm that can quickly detect the size of the repair area.

[0004] In view of this problem, Liu et al. proposed a two-stage crack segmentation method, first using an improved YOLOv3 to detect cracks on the road surface, and then using an improved U-Net to segment the detected cracks. Zhang et al. used a transformer to segment road diseases, and proposed a new method for evaluating the technical condition of the road surface. Geng et al. proposed a YOLO-based road disease detection method, designed a special dynamic downsampling module to improve the efficiency of feature extraction for cracks. Wang et al. constructed a segmentation dataset for road diseases, containing 13 classes and 5633 labels, and evaluated the performance of various semantic segmentation models on this dataset. Tao et al. proposed a new method for detecting cement pavement diseases based on weak supervision, based on the image class given by the classification branch to generate a roughly appropriate target frame. Wang et al. proposed a single-stage instance segmentation method that simultaneously performs local semantic segmentation and global semantic segmentation. Ye et al. proposed a road crack instance segmentation method based on an improved YOLOv7.

[0005] For the size measurement problem of road surface diseases, Shan et al. finely annotated 500 road surface crack images collected by a UAV and proposed a corresponding method for measuring the area, length and width of road surface cracks. Rakshitha et al. proposed introducing a set of advanced U-Net architecture for accurate pixel-level crack detection and quantification, and proposed a disease length and width measurement method based on crack skeleton. Pan et al. proposed a segmentation and quantification method based on 3D information for road surface potholes, and based on the detected depth, the area and volume of the pothole can be calculated. Wang et al. proposed a lightweight segmentation method for asphalt pavement cracks, and based on previous research, a length and width measurement method based on crack skeleton was proposed, and a corresponding crack grading evaluation index was proposed.

[0006] At present, there have been a large number of segmentation and quantitative analysis researches on road surface cracks, but the research focusing on road repair is still relatively weak. In fact, the repair area is significantly different in width and morphology from natural cracks, which is mainly affected by the repair process and disease type; therefore, the existing automatic detection and size calculation of road repair areas still have deficiencies in performance and accuracy. SUMMARY

[0007] The present disclosure proposes a road repair area size detection method and system based on a vehicle-mounted device to solve the above problems, which realizes the full-process automation from target detection, pixel-level segmentation to geometric size calculation, avoids the inefficiency and subjectivity of traditional manual inspection, and significantly improves the efficiency and scientificity of road detection and evaluation.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions: A road repair area size detection method based on a vehicle-mounted device, comprising: obtaining an inclined road surface image collected by a vehicle-mounted device; performing target segmentation of the repair area on the inclined road surface image by using an instance segmentation model to obtain a pixel-level segmentation map; correcting the pixel-level segmentation map in combination with camera device parameters and image depth information; based on the corrected pixel-level segmentation map, performing geometric calculation of the length and width of the repair area to obtain the size of the repair area.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions: A road repair area size detection system based on a vehicle-mounted device, comprising: an image acquisition module configured to obtain an inclined road surface image collected by a vehicle-mounted device; a target segmentation module configured to perform target segmentation of the repair area on the inclined road surface image by using an instance segmentation model to obtain a pixel-level segmentation map; An image correction module is configured to correct the pixel-level segmentation map in combination with the camera device parameters and the image depth information. A size measurement module is configured to perform geometric measurement of the length and width of the repair area based on the corrected pixel-level segmentation map to obtain the size of the repair area.

[0010] According to some embodiments, the present disclosure adopts the technical solutions as follows: A computer program product comprising a computer program, which, when executed by a processor, implements the road repair area size detection method based on a vehicle-mounted device.

[0011] According to some embodiments, the present disclosure adopts the technical solutions as follows: A non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implements the road repair area size detection method based on a vehicle-mounted device.

[0012] According to some embodiments, the present disclosure adopts the technical solutions as follows: An electronic device comprising a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory to make the electronic device execute the road repair area size detection method based on a vehicle-mounted device.

[0013] Compared with the prior art, the present disclosure has the following beneficial effects: The present disclosure proposes a road repair area size detection method and system based on a vehicle-mounted device, which realizes full-process automation from target detection, pixel-level segmentation to geometric size measurement, avoids the inefficiency and subjectivity of traditional manual inspection, significantly improves the efficiency and scientificity of road detection and evaluation, and is embodied in the following points: (1) Segmentation refinement: three functionally complementary branches are introduced for pixel-level instance segmentation, each branch undertakes different tasks, and cooperates to complete the identification, classification and spatial scale estimation of the repair area, effectively improving the segmentation accuracy of the repair area.

[0014] (2) Image correction: for the inclined road images collected by the vehicle-mounted device, the conversion from the pixel coordinate system to the world coordinate system is realized in combination with the camera device parameters and the image depth information, the image is corrected, and high-quality basic data is provided for subsequent size measurement.

[0015] (3) The size calculation method is efficient and practical: the proposed calculation algorithm for the length and maximum width of the strip-shaped repair area uses center line fitting combined with curve integral to calculate the length, and uses the maximum inscribed circle method to determine the width. The calculation process is efficient and the result is accurate, which can meet the actual road maintenance evaluation requirements. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which form a part of this disclosure, are used to provide further understanding of the disclosure, and the illustrative embodiments of the disclosure and their description serve to explain the disclosure. They do not constitute improper limitations on the disclosure.

[0017] Figure 1 The method flowchart of Example 1; Figure 2 The example segmentation model structure diagram of Example 1; Figure 3 The device image acquisition schematic diagram of Example 1; Figure 4 The repair area size measurement diagram of Example 1. DETAILED DESCRIPTION

[0018] The disclosure will be further described below in conjunction with the drawings and examples.

[0019] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the disclosure belongs.

[0020] It should be noted that the terms used herein are only for the purpose of describing the specific embodiments and are not intended to limit the exemplary embodiments according to the disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "comprise" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or combination thereof.

[0021] Example 1 In an embodiment of the disclosure, a vehicle-mounted device-based road surface repair area size detection method is provided, comprising: Step S0: obtaining the inclined road surface image collected by the vehicle-mounted device; Step S1: using an instance segmentation model to perform target segmentation of the repair area on the inclined road surface image to obtain a pixel-level segmentation map; Step S2: correcting the pixel-level segmentation map in combination with the camera device parameters and image depth information; Step S3: based on the corrected pixel-level segmentation map, performing geometric measurement of the length and width of the repair area to obtain the size of the repair area.

[0022] As an embodiment, a road surface repair area size detection method based on a vehicle-mounted device of the present disclosure first uses a single-stage segmentation technique to achieve accurate identification of the repair area, then corrects the image in combination with the camera device parameters for the tilted road surface image collected by the vehicle-mounted device; finally, the maximum inscribed circle diameter is used to determine the maximum width of the repair area, and the least squares method is used to fit the center line of the strip-shaped repair for calculating the length of the repair area. The specific implementation process is as follows: Step S1: using an instance segmentation model to perform target segmentation of the repair area of the tilted road surface image to obtain a pixel-level segmentation map.

[0023] This step uses a single-stage instance segmentation method to accurately identify the road surface repair area. This method integrates the detection and segmentation tasks into a unified network framework, achieving fast positioning and pixel-level segmentation of the repair area. Through end-to-end training, the model can effectively capture the morphological features of the repair area, improving the accuracy and real-time performance of the segmentation, and meeting the needs of online detection of vehicle-mounted devices. This segmentation result provides an accurate basis for subsequent size calculation.

[0024] To achieve high-precision detection and size estimation of the road surface repair area, this method constructs a multi-task network structure containing three functionally complementary branches, i.e., an instance segmentation model for single-stage instance segmentation; each branch undertakes different tasks and collaborates to complete the identification, classification, and spatial scale estimation of the repair area. The specific design is as follows: The instance segmentation model, as shown in Figure 2 includes a feature extraction backbone network and a three-branch network. The feature extraction backbone network uses networks such as SegNet and ResNet to extract image features, and the three-branch network performs target segmentation of the repair area based on the extracted image features for coarse positioning and fine correction. The three branches are described as follows: Branch 1: global semantic segmentation branch This branch is mainly responsible for pixel-level semantic segmentation of the repair area in the input image. To improve the model's segmentation ability for small areas, fuzzy boundaries, and other difficult samples, Focal Loss is used to suppress the dominant role of easy-to-classify samples, and Dice Loss is used to enhance the handling of unbalanced classes and the ability to depict boundary regions. Through end-to-end training, branch 1 can generate a fine-grained segmentation map, providing a basis for accurate contour extraction of the repair area.

[0025] Branch 2: classification confidence prediction branch This branch aims to make a binary classification decision for each pixel point, i.e., whether it is the center point of the repair area, and output the corresponding confidence score. This mechanism is consistent with the target center point detection idea in the YOLO series. To improve the model's discrimination ability in the areas of slight repair, wear and tear, and fuzzy boundaries, this branch uses Focal Loss optimization to effectively alleviate the imbalance between positive and negative samples. The predicted center point confidence can be used as a screening basis in the post-processing stage, which helps to eliminate false detection areas and enhance the reliability of overall detection.

[0026] Branch 3: Distance and size regression branch This branch aims to achieve accurate quantification of the size of the repair area, specifically by predicting the boundary box offset of the center point , i.e., the distance from the center point to the boundaries in the four directions. This design can effectively construct the complete boundary box of the target. Considering that the offset is a continuous value and requires high prediction accuracy, this branch uses L1Loss as the regression loss function to improve the stability and robustness of the prediction.

[0027] To improve the accuracy and stability of target positioning, this model combines the outputs of branch 2 and branch 3. First, based on the center point classification result (branch 2), the candidate points of the potential repair area are determined, and then the boundary box offset corresponding to each center point is predicted by the boundary box regression branch (branch 3) , to construct a complete target framework. Finally, using the pixel-level segmentation map output by branch 1, a more accurate target contour is extracted within each predicted frame, playing the role of "rough positioning + fine correction", further eliminating false detections and improving boundary fitting effects.

[0028] Step S2: Correct the pixel-level segmentation map by combining camera device parameters and image depth information.

[0029] This step addresses the image tilt problem caused by device rotation around the x-axis. First, coordinate conversion between the camera coordinate system and the world coordinate system is achieved based on the rotation matrix. Then, by mapping the pixel coordinate system to the image coordinate system and the camera coordinate system in sequence, combining the camera internal and external parameters and depth information, the position of the target point in the three-dimensional world coordinate system is accurately restored, providing a spatial coordinate basis for subsequent size calculation. The specific steps are as follows: (1) Image depth information construction As shown in Figure 2 , due to the installation angle of the device, the device will rotate around the x-axis, which needs to be converted from the camera coordinate system to the world coordinate system. The following is the rotation matrix for rotation around the x-axis :

[0030] According to the rotation matrix, the target point coordinates can be converted from the world coordinate system to the camera coordinate system, and the specific formula is as follows:

[0031] In the formula, represents the horizontal coordinate of the target point in the camera coordinate system, represents the vertical coordinate of the target point in the camera coordinate system, is the depth coordinate of the target point in the camera coordinate system, represents the horizontal coordinate of the target point in the world coordinate system, represents the vertical coordinate of the target point in the world coordinate system, is the depth coordinate of the target point in the world coordinate system.

[0032] From the above two formulas, the final conversion from the world coordinate system to the camera coordinate system is obtained as follows:

[0033] At the same time, the specific formula for converting the target point coordinates from the camera coordinate system to the world coordinate system is as follows:

[0034]

[0035] In the known precondition, can be transformed into the form represented by and , as follows:

[0036] (2) Image world coordinate system restoration The image is converted from the pixel coordinate system to the image coordinate system, the camera coordinate system and the world coordinate system in turn, and finally the coordinates of the pixel-level segmentation map target point in the world coordinate system are calculated.

[0037] The formula for converting between the pixel coordinate system and the image coordinate system is:

[0038]

[0039] In the formula, represents the horizontal coordinate of the target point in the pixel coordinate system, represents the vertical coordinate of the target point in the pixel coordinate system, , represents the scaling factor of the image in the horizontal and vertical directions, represents the horizontal coordinate of the target point in the image coordinate system, represents the longitudinal coordinate of the target point in the image coordinate system, represents the maximum pixel width of the image, represents the maximum pixel length of the image.

[0040] The formula for converting the image coordinate system to the camera coordinate system is:

[0041]

[0042] In the formula, represents the horizontal coordinate of the target point in the camera coordinate system, represents the vertical coordinate of the target point in the camera coordinate system, , represents the focal length of the camera on the x and y axes, represents the optical center of the camera, is the depth coordinate of the target point in the camera coordinate system.

[0043] Referring to formula (6) and formula (9), the coordinates of the target point in the camera coordinate system can be converted to the following form:

[0044] According to the inferred coordinates of the target point in the camera coordinate system , and and formula (4), the coordinates of the target point in the world coordinate system , and can be inferred, and the target coordinates can be converted accordingly.

[0045] Step S3: Based on the corrected pixel-level segmentation map, the length and width of the repair area are geometrically measured to obtain the size of the repair area.

[0046] Based on the corrected pixel-level segmentation result, first, for the strip-shaped repair area, the least squares method is used to fit the center line thereof, and then the actual length thereof is calculated. Subsequently, the maximum inscribed circle of the repair area is fitted to obtain the maximum width thereof. Through the quantification of these two key dimensions, a comprehensive evaluation of the size of the repair area is realized, which provides a scientific basis for subsequent road condition determination and maintenance planning. The specific method is as follows: S3-1: Length calculation method.

[0047] For the strip-shaped pavement repair area, since it usually has a certain width, if the dense pixel points are directly fitted with a regression curve, it will result in high computational overhead. Therefore, the embodiment proposes an efficient alternative method: (1) AsFigure 3 drawing the center line of the region based on the extracted repair region boundary; (2) as shown in (d) of Figure 3 selecting the pixel points in the center line region, and performing curve fitting on the pixel points by using the least square method; (3) integrating the curve obtained by fitting, so as to obtain the accurate length of the strip-shaped repair region.

[0048] The method not only reduces the calculation complexity, but also ensures the accuracy of the length calculation.

[0049] S3-2: width calculation method.

[0050] The maximum width of the repair region is obtained by fitting the maximum inscribed circle of the repair region: (1) as shown in (e) of Figure 3 traversing each pixel point in the repair region, taking the point as the center, drawing the maximum circle inscribed in the contour boundary, and recording the diameter of each inscribed circle; (2) as shown in (f) of Figure 3 comparing the diameters of all circles, and selecting the maximum diameter, which is the maximum width of the repair region.

[0051] The method effectively measures the maximum transverse expansion range of the strip-shaped repair region in the two-dimensional image by using the exhaustion method combined with geometric inscribed analysis, and has high adaptability and accuracy.

[0052] Embodiment 2 In an embodiment of the present disclosure, a vehicle-mounted device-based road surface repair region size detection system is provided, comprising: an image acquisition module configured to acquire an inclined road surface image collected by a vehicle-mounted device; a target segmentation module configured to perform target segmentation of the repair region on the inclined road surface image by using an instance segmentation model to obtain a pixel-level segmentation map; an image correction module configured to correct the pixel-level segmentation map in combination with camera device parameters and image depth information; a size calculation module configured to perform geometric calculation of the length and width of the repair region based on the corrected pixel-level segmentation map to obtain the size of the repair region.

[0053] Embodiment 3 In an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the vehicle-mounted device-based road surface repair region size detection method.

[0054] Embodiment 4 In an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the method for detecting a size of a road surface repair area based on a vehicle-mounted device.

[0055] Embodiment 5 In an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device implements the method for detecting a size of a road surface repair area based on a vehicle-mounted device.

[0056] The present disclosure is described with reference to the flowcharts and / or block diagrams of the method, device (system), and computer program product according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The means for implementing the functions specified in one or more flows and / or blocks.

[0057] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The steps for implementing the functions specified in one or more flows and / or blocks.

[0058] Although the specific embodiments of the present disclosure are described above with reference to the accompanying drawings, the present disclosure is not limited to the specific embodiments. It should be understood by those skilled in the art that various modifications or changes can be made to the technical solutions of the present disclosure without creative labor, and these modifications or changes are still within the protection scope of the present disclosure.

Claims

1. A method for detecting the size of a road repair area based on a vehicle-mounted device, characterized in that: include: Obtaining inclined road surface images collected by vehicle-mounted equipment; The instance segmentation model is used to segment the target area of ​​the inclined road image and obtain a pixel-level segmentation map. Combining camera equipment parameters and image depth information, the pixel-level segmentation map is corrected; Based on the corrected pixel-level segmentation map, the length and width of the patched area are geometrically measured to obtain the size of the patched area.

2. A method for detecting the size of a road repair area based on a vehicle-mounted device according to claim 1, characterized in that: The instance segmentation model adopts a multi-task network structure with three functionally complementary branches, which are: The global semantic segmentation branch performs pixel-level semantic segmentation on the patched area in the input image; The classification confidence prediction branch performs a binary classification on each pixel to determine whether it is the center point of the patch area and outputs the corresponding confidence score; The distance and size regression branch predicts the bounding box offset of the patch area relative to the center point, and obtains the distance from the center point to the boundary in the four directions of up, down, left, and right.

3. A method for detecting the size of a road repair area based on a vehicle-mounted device according to claim 1, characterized in that: The correction of the pixel-level segmentation map is performed by converting the pixel coordinate system and the world coordinate system, and calculating the coordinates of the target point of the pixel-level segmentation map in the world coordinate system.

4. A method for detecting the size of a road repair area based on a vehicle-mounted device as claimed in claim 3, characterized in that: The coordinates of the target point of the pixel-level segmentation map in the world coordinate system are calculated as follows: Based on the rotation matrix, calculate the coordinate transformation between the camera coordinate system and the world coordinate system; The pixel coordinate system of the pixel-level segmentation map is mapped to the image coordinate system and the camera coordinate system in sequence. Combined with the internal and external parameters of the camera and the depth information, the coordinates of the target point in the three-dimensional world coordinate system are restored by using the coordinate conversion between the camera coordinate system and the world coordinate system.

5. The method for detecting the size of a road repair area based on a vehicle-mounted device according to claim 1, wherein: The geometric measurement of the length is to use the least square method to fit the center line of the repair area to calculate the actual length.

6. A method for detecting the size of a road repair area based on a vehicle-mounted device as claimed in claim 1, characterized in that: The geometric measurement of the width is to obtain the maximum width of the repair area by fitting the maximum inscribed circle of the repair area.

7. A road repair area size detection system based on vehicle-mounted equipment, characterized in that: include: The image acquisition module is configured to: acquire an image of an inclined road surface collected by an onboard device; The target segmentation module is configured to: use the instance segmentation model to segment the target of the patch area of ​​the inclined road image to obtain a pixel-level segmentation map; The image correction module is configured to: correct the pixel-level segmentation map by combining camera device parameters and image depth information; The size measurement module is configured to: perform geometric measurement of the length and width of the patching area based on the corrected pixel-level segmentation map to obtain the size of the patching area.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for detecting the size of a road repair area based on a vehicle-mounted device according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the method for detecting the size of a road repair area based on a vehicle-mounted device as described in any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement a road repair area size detection method based on a vehicle-mounted device as described in any one of claims 1 to 6.

Images