Road disease detection and account updating method, device and equipment and storage medium

Abstract

The application discloses a road disease detection and account updating method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a to-be-detected road image and position information of a road region corresponding to the to-be-detected image; extracting a marking mask image of the to-be-detected road image and performing skeletonization and topological repair processing on the marking mask image to determine a marking center line; determining a plurality of candidate to-be-updated account images from a traffic marking account based on the position information of the to-be-detected road and position information corresponding to each account image in the traffic marking account; and determining an account updating strategy based on the comprehensive similarity between the to-be-detected road image and the candidate to-be-updated account image. The application realizes intelligent, real-time and fine maintenance and management of road markings and associated facility accounts.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, equipment and storage medium for road defect detection and record updating. Background Technology

[0002] With the rapid development of intelligent transportation and digital maintenance systems, traffic safety facilities such as road markings, raised pavement markers, and facade markings play a fundamental role in road traffic safety, order control, and asset management. In recent years, highway and urban road management departments in various regions have gradually introduced video inspection and vehicle-mounted sensing equipment to replace traditional manual inspection methods, aiming to achieve intelligent identification and dynamic maintenance of facility status.

[0003] In related technologies, relying solely on image feature methods lacks a unified data standard and geometric description system, making it difficult to handle issues such as broken markings, residual old markings, or non-standard settings in complex environments. Furthermore, the ledger updates lack spatial indexing and change detection mechanisms, making it difficult to automatically align and dynamically maintain inspection results with the asset database. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, equipment, storage medium, and program product for road defect detection and record updating, aiming to solve the technical problem of low reliability in related technologies for road defect detection and record updating.

[0005] To achieve the above objectives, this application provides a method for road defect detection and record updating, the method comprising:

[0006] Obtain the image of the road to be detected and the location information of the road area corresponding to the image;

[0007] Extract the road marking mask image from the image of the road to be detected, and perform skeletonization and topology repair on the road marking mask image to determine the center line of the road marking;

[0008] Based on the location information of the road to be detected and the location information corresponding to each image in the traffic marking ledger, multiple candidate images to be updated are determined from the traffic marking ledger.

[0009] The ledger update strategy is determined based on the comprehensive similarity between the road image to be detected and the candidate ledger image to be updated. The comprehensive similarity is determined based on the difference in paving mask features, the distance between paving center lines, location information, and semantic consistency score between the road image to be detected and the ledger image to be updated.

[0010] In one embodiment, the steps of extracting the road marking mask image from the image of the road to be detected and performing skeletonization and topological repair processing on the road marking mask image to determine the center line of the road marking include:

[0011] Edge detection is performed on the road image to be detected to obtain a binary edge image of the road image to be detected;

[0012] A Hough transform is performed on a binary edge image to extract a set of line segments related to traffic markings. The set of line segments includes the number of line segments, the average line segment length, and the directional variance.

[0013] Based on the set of line segments, the complexity of traffic markings in the road image to be detected is determined;

[0014] If the complexity of traffic markings is low, the Zhang-Suen thinning algorithm is used to perform skeletonization processing on the marking mask image to extract the marking skeleton;

[0015] If the traffic markings are complex, the Guo-Hall algorithm is used to skeletonize the marking mask image and extract the marking skeleton.

[0016] Perform topology repair on the road marking skeleton to determine the center line of the road marking.

[0017] In one embodiment, the step of performing topology repair on the road marking skeleton and determining the road marking centerline includes:

[0018] Traverse the gradation skeleton and identify all discontinuous breakpoints;

[0019] Define a sliding local repair window. Within the local repair window area, use the shortest path algorithm and / or morphological dilation operation to connect the line segments at both ends of the breakpoint to obtain continuous lines.

[0020] Based on a preset branch length threshold, burrs in continuous markings are identified and removed.

[0021] After removing the burrs, the marking skeleton is smoothed to determine the center line of the marking.

[0022] In one embodiment, the step of determining the ledger update strategy based on the comprehensive similarity between the road image to be detected and the candidate ledger images to be updated includes:

[0023] Within the road marking buffer zone of the road image to be inspected, the defect area is determined, and the defect type within the defect area is identified.

[0024] Based on the disease severity calculation model, the severity of disease types is determined;

[0025] The traffic marking register will be updated based on the severity of the damage and overall similarity.

[0026] In one embodiment, the steps of updating the traffic marking ledger based on the severity of the damage and overall similarity include:

[0027] Based on the cross-union ratio of the road marking masks between the road image to be detected and the candidate ledger images to be updated, the differences in the marking mask features are determined.

[0028] The distance between the road centerline in the road image to be detected and the road centerline in the candidate ledger image to be updated is determined;

[0029] The comprehensive similarity score is obtained by weighting and combining the differences in tracing mask features, the distance between tracing centerlines, the differences in positional information, and semantic consistency scores.

[0030] If the overall similarity between the road image to be detected and the candidate ledger image to be updated is not less than the similarity threshold and the degree of change in the severity of the damage of the road image to be detected compared with the severity of the damage of the candidate ledger image to be updated is not less than the preset threshold, then the candidate ledger image to be updated with the highest overall similarity will be replaced with the road image to be detected.

[0031] Otherwise, add the road images to be detected to the traffic marking register.

[0032] In one embodiment, the step of determining multiple candidate images to be updated from the traffic marking ledger based on the location information of the road to be detected and the location information corresponding to each image in the traffic marking ledger includes:

[0033] Construct an R-tree spatial index based on the minimum outer rectangle of each ledger image;

[0034] Based on the spatial coordinates of the road to be detected, a spatial query window is defined, and an overlap query operation is performed to obtain an initial set of candidate images.

[0035] The initial candidate image set is converged based on the name information of the road to be detected, lane number, and overlap of the mileage marker intervals. Images that are spatially adjacent but logically discontinuous are removed to obtain candidate ledger images to be updated.

[0036] In one embodiment, after the step of determining the ledger update strategy, the method further includes the step of:

[0037] Based on the model prediction results generated during the process of disease identification and ledger matching, the uncertainty of the output object-level prediction is evaluated.

[0038] If the uncertainty of the object-level prediction exceeds the uncertainty threshold, or if there is a conflict with the historical ledger records, a manual review task will be generated and output.

[0039] Secondly, to achieve the above objectives, this application further provides a road defect detection and record-keeping device, the device comprising:

[0040] The image acquisition module is used to acquire the image of the road to be detected and the location information of the corresponding inspection area;

[0041] The feature extraction module is used to extract the road marking mask image of the road image to be detected and to perform skeletonization and topological repair processing on the road marking mask image to determine the center line of the road marking;

[0042] The candidate ledger determination module is used to determine multiple candidate ledger images to be updated from the traffic marking ledger based on the location information of the road to be detected and the location information corresponding to each ledger image in the traffic marking ledger.

[0043] The ledger update module is used to determine the ledger update strategy based on the comprehensive similarity between the road image to be detected and the candidate ledger image to be updated. The comprehensive similarity is determined based on the difference in paving mask features, the distance between paving center lines, location information and semantic consistency score between the road image to be detected and the ledger image to be updated.

[0044] Thirdly, to achieve the above objectives, this application further provides a road defect detection and record-keeping update device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the road defect detection and record-keeping update method described above.

[0045] Fourthly, to achieve the above objectives, this application further provides a storage medium, which is a computer-readable storage medium, storing a computer program on the storage medium. When the computer program is executed by a processor, it implements the steps of the above-described road defect detection and ledger update method.

[0046] One or more technical solutions proposed in this application have at least the following technical effects:

[0047] By introducing a road marking mask skeletonization and topology repair mechanism, a unified and stable geometric representation is constructed, effectively mitigating the unreliable recognition problems caused by road marking breaks, residual interference, and non-standard settings in complex road environments. Simultaneously, a spatial index is constructed based on location information, and multi-dimensional similarity indicators such as road marking feature differences, centerline distance, and semantic consistency are integrated to achieve accurate matching and change determination between inspection images and existing records, avoiding erroneous and missed updates caused by relying solely on single image features or coarse-grained spatial matching. Furthermore, by combining damage severity assessment and prediction uncertainty analysis, automatic record updates are supported, and manual review can be triggered when results conflict or lack credibility, thereby significantly improving the accuracy, reliability, and engineering usability of road damage detection and traffic marking record updates. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the overall process of an embodiment of the road defect detection and record updating method of this application.

[0051] Figure 2 This is a flowchart illustrating the implementation details of the road defect detection and record-keeping method of this application.

[0052] Figure 3 This is a schematic diagram of the process for extracting the center line of a gradation line according to an embodiment of this application.

[0053] Figure 4 This is a schematic diagram of the ledger matching process in an embodiment of this application.

[0054] Figure 5 This is a schematic diagram of the road defect detection and record updating device for this application.

[0055] Figure 6 This is a schematic diagram of the road defect detection and record-keeping equipment used in this application.

[0056] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0057] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0058] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0059] The main solution of this application embodiment is: extract the center line of the road marking through skeletonization and topology repair, determine the type of disease and quantify its severity within the buffer zone, use R-tree index to achieve efficient spatial matching, and calculate the comprehensive similarity based on mask difference, distance of the road marking center line, positioning deviation and semantic consistency, thereby intelligently judging the update or addition of road diseases.

[0060] Specifically, this application provides a method for road defect detection and record updating, referring to... Figure 1 and Figure 2 , Figure 1 This is a schematic diagram of the overall process of an embodiment of the road defect detection and record updating method of this application. Figure 2A flowchart illustrating the implementation details of the road defect detection and record-keeping method of this application.

[0061] In this embodiment, the road defect detection and record update method includes steps S10 to S40:

[0062] Step S10: Obtain the image of the road to be detected and the location information of the road area corresponding to the image to be detected.

[0063] It should be noted that the road image to be detected can be obtained from the road video stream captured in real time by the vehicle-mounted integrated system through the front or top-down camera, while simultaneously recording the positioning and attitude information of the GNSS and IMU sensors to confirm the location information of the corresponding road area.

[0064] Furthermore, to ensure the stability of data processing, the images of the roads to be detected can be preprocessed. For example, operations such as image stabilization, noise reduction, exposure and white balance normalization, glare / rain and fog suppression, and motion blur compensation can be performed on video frames to improve image quality and ensure temporal continuity, thereby providing clear and stable input for subsequent defect detection.

[0065] Specifically, optical flow is used for image stabilization, with a window size of 15×15 and a maximum displacement threshold of 10 pixels. Gaussian filtering is used for noise reduction, with a kernel size of 5×5 and a standard deviation of 1.5. HDR processing is used for exposure, fusing 3-5 images with different exposure times of 1 / 60s, 1 / 125s, and 1 / 250s. Dark channel prior algorithm is used for dehazing, with a window size of 15×15 and an atmospheric light value estimation window size of 0.1 times the image size. Motion blur compensation uses Wiener filtering for deblurring, with a point spread function (PSF) length estimated to be 5-10 pixels.

[0066] Step S20: Extract the road marking mask image from the image of the road to be detected and perform skeletonization and topology repair processing on the road marking mask image to determine the center line of the road marking.

[0067] Specifically, the grading mask image can be extracted using a semantic segmentation model or an instance segmentation model. Then, the grading mask image is refined and topologically repaired using a skeletonization algorithm, redundant pixels are deleted, and the skeleton structure of the object is preserved. Finally, a skeleton image with a single pixel width is generated to achieve the extraction of the grading center line.

[0068] Understandably, after acquiring high-quality images, this step aims to address a core cognitive deepening problem: how to transform the unstructured pixel information "seen" by the deep learning model into structured geometric features that conform to engineering definitions and can be used for precise measurement and analysis. This is a crucial step that distinguishes it from traditional disease detection methods and lays the foundation for all subsequent quantitative analyses.

[0069] In one feasible implementation, step S20 includes steps A10 to A60:

[0070] Step A10: Perform edge detection on the road image to be detected to obtain a binary edge image of the road image to be detected.

[0071] Step A20: Perform Hough transform on the binary edge image to extract the set of line segments related to traffic markings in the binary edge image; the set of line segments includes the number of line segments, the average line segment length, and the direction variance.

[0072] Step A30: Determine the complexity of traffic markings in the road image to be detected based on the set of line segments.

[0073] Step A40: If the complexity of the traffic markings is low, the Zhang-Suen thinning algorithm is used to perform skeletonization processing on the marking mask image to extract the marking skeleton.

[0074] In step A50, if the traffic markings are complex, the Guo-Hall algorithm is used to perform skeletonization processing on the marking mask image to extract the marking skeleton.

[0075] Step A60: Perform topology repair on the road marking skeleton to determine the center line of the road marking.

[0076] Step A60 includes steps A61 to A64:

[0077] Step A61: Traverse the gradation skeleton and identify all discontinuous breakpoints;

[0078] Step A62: Determine a sliding local repair window. Within the local repair window area, use the shortest path algorithm and / or morphological dilation operation to connect the line segments at both ends of the breakpoint to obtain continuous lines.

[0079] Step A63: Based on the preset branch length threshold, identify and remove burrs in continuous markings.

[0080] Step A64: Smooth the deburred marking skeleton and determine the center line of the marking.

[0081] Specifically, when marking lines are extracted using semantic segmentation models (such as DeepLabV3+ and PSPNet) or instance segmentation models (such as MaskR-CNN and YOLACT), the resulting pixel mask is a discrete, unstructured "pixel cluster." While it visually identifies the marking line region, it cannot be directly used to accurately calculate key geometric parameters such as the length, curvature, and continuous break distance of the marking line. Therefore, referring to... Figure 3 This implementation introduces subsequent skeletonization and topology repair processing:

[0082] First, by predicting the complexity of the datum line, the optimal algorithm is adaptively selected, ensuring the robustness of skeleton extraction.

[0083] For example, the specific steps for predicting the complexity of the marking line include:

[0084] Edge detection is performed on the road image to be detected using the Canny edge detection algorithm (low threshold set to 50, high threshold set to 150, Sobel kernel size 3×3) or the Sobel operator to generate a binary edge image. This step focuses on the region of interest (ROI) of the road area (e.g., the lower half of the image to avoid interference from irrelevant background).

[0085] Apply a Hough transform to the edge image (e.g., a probabilistic Hough transform with an accumulator threshold of 50, a minimum segment length of 20 pixels, and a maximum segment gap of 5 pixels) to detect and extract the set of line segments associated with the markings, including the number of line segments. Average line segment length and directional variance (The directional variance is obtained by calculating the standard deviation of the angles of all line segments.)

[0086] The computational complexity metric can be expressed as:

[0087] ;

[0088] This indicator reflects the fragmentation and directional diversity of the markings: simple markings (such as solid or dashed lines) typically have fewer segments, longer lengths, and consistent directions. Small; complex markings (such as arrows, directional text, or cross markings) have many short line segments with diverse directions. big).

[0089] If complexity index Less than the preset threshold (For example Based on empirical calibration of typical road image datasets, such as straight line markings on urban roads. Arrow markings If the line complexity is low, the Zhang-Suen thinning algorithm is selected to ensure fast processing and good connectivity; otherwise, if the line complexity is high, the Guo-Hall algorithm is selected to better preserve the geometric features of complex shapes and avoid distortion.

[0090] Among them, the preset threshold The method can be calibrated and validated using offline datasets (such as image sets containing simple lane line vs. complex arrow samples). This prediction method has low computational cost (edge ​​detection). Hough transform , (in pixels), it improves the robustness of skeleton extraction in complex road environments, while not relying on the mask output of the semantic segmentation model, thus avoiding the propagation of segmentation errors.

[0091] Experiments show that this prediction mechanism improves the skeleton integrity rate by more than 12% in complex marking scenarios (such as arrows and numbers), and its overall processing efficiency is comparable to that of non-adaptive methods.

[0092] For low-complexity datum mask images, the Zhang-Suen thinning algorithm is used to iteratively remove boundary pixels to maintain the topological structure and connectivity of the datum. The Zhang-Suen thinning algorithm is suitable for continuous and simple datum lines.

[0093] The Zhang-Suen thinning algorithm terminates its iterations when the difference in the number of pixels deleted between two consecutive iterations is less than 0.1% of the total number of pixels, or when the maximum number of iterations is reached (recommended value 100). This algorithm is suitable for processing continuous lines and can maintain the connectivity and endpoint positions of the lines.

[0094] For complex gradation mask images, the Guo-Hall thinning algorithm employs a template matching method, which yields better results in processing gradations with complex shapes such as arrows and characters, and better preserves their geometric features. Specifically, the Guo-Hall algorithm uses 8-neighborhood template matching, and the template library contains 256 possible neighborhood configurations. This algorithm is suitable for processing complex gradations, such as arrows and directional markers, and can better preserve the geometric features of the gradations.

[0095] Furthermore, due to factors such as lighting, occlusion, and dirt in real-world scenes, as well as potential errors in the segmentation model itself, the extracted original skeleton often contains defects such as breakpoints, burrs (redundant branches), and jagged edges. These defects do not conform to the geometric continuity and smoothness that road markings, as physical entities, should possess. Therefore, topology repair is necessary, and the process includes:

[0096] Connectivity check: Traverse the skeleton and identify all discontinuous breakpoints. Breakpoint traversal can be implemented using algorithms such as depth-first search / breadth-first search. For example, the breakpoint determination threshold is 3 pixels; that is, when the distance between two connected components is less than 3 pixels, it is determined to be a breakpoint.

[0097] Breakpoint Repair: Within local regions, shortest path algorithms (such as Dijkstra's algorithm) and / or morphological dilation operations are used to intelligently connect breaks caused by minor segmentation errors, restoring the continuity of the tracing lines. For example, a branch length threshold is set to 5% of the total tracing line length. When a branch length is less than 5% of the total tracing line length, it is considered a redundant branch and deleted. Short "glitches" caused by image noise or false edge detection are removed, while the main centerline is preserved. The shortest path algorithm has a search radius of 10 pixels, and the kernel size of the morphological operation is 3×3.

[0098] Geometric smoothing: Gaussian filtering and other methods are used to smooth the repaired centerline, eliminating jagged edges and making it more consistent with the smooth curves in actual engineering. For example, Gaussian filtering is used for smoothing, with a kernel size of 5×5 and a standard deviation of 1.0.

[0099] Understandably, through the complete process of "segmentation-skeletonization-topology repair," this implementation successfully transforms the raw output of computer vision (pixel mask) into a high-quality, standardized, vectorized geometric primitive—the "marking centerline." This centerline possesses a precise mathematical definition and a complete topological structure; it is not only a visual representation but also computable and analyzable engineering data.

[0100] Step S30: Based on the location information of the road to be detected and the location information corresponding to each image in the traffic marking ledger, multiple candidate images to be updated are determined from the traffic marking ledger.

[0101] Specifically, after achieving "identification" and "quantification" in a single detection, the next core problem this embodiment aims to solve is how to effectively correlate the current detection results with massive amounts of historical record data. This is a key bottleneck in achieving facility lifecycle management, because inefficient and inaccurate matching will render any cross-time status tracking and analysis impossible. Therefore, this embodiment introduces a candidate record determination method consisting of two steps: "coarse screening" and "fine screening."

[0102] Specifically, refer to Figure 4 To improve the efficiency of updating the traffic markings, the coarse screening process requires retrieving candidate images from the traffic markings register to reduce the range of subsequent matching updates and improve efficiency.

[0103] In one feasible implementation, step S30 includes steps AB10 to B30:

[0104] Step B10: Construct an R-tree spatial index based on the minimum outer rectangle of each ledger image.

[0105] Step B20: Based on the spatial coordinates of the road to be detected, define a spatial query window, perform an overlap query operation, and obtain an initial candidate image set.

[0106] Step B30: Based on the name information, lane number, and overlap of the mileage marker intervals of the road to be detected, the initial candidate image set is converged, and spatially adjacent but logically discontinuous images are removed to obtain candidate ledger images to be updated.

[0107] Specifically, this implementation first reconstructs the previously disorganized ledger data, which required sequential comparison, using a spatially hierarchical R-tree. As a highly optimized spatial data structure, the R-tree organizes two-dimensional spatial information (the geographical location range of the marked lines) into a tree-like hierarchy, making the efficiency of range queries directly proportional to the logarithm of the total data volume, rather than linearly related. This fundamentally solves the problem of slow system response as the ledger data scales.

[0108] Subsequently, through a two-step strategy of "rapid screening using R-tree space" plus "refined selection based on logical information," the system can narrow down candidate matching objects from a database of millions of records to just a few dozen or even a handful within milliseconds. Experiments show that after using R-tree indexing, the average number of candidates decreased from 128 to 23, and the matching latency decreased from 38.2ms / object to 12.6ms / object, improving efficiency by approximately three times. This not only significantly reduces the computational burden of subsequent fine matching but, more importantly, provides technical feasibility for achieving high-frequency, near real-time dynamic tracking and management of road infrastructure assets, making it an indispensable first step in building a dynamic lifecycle management system.

[0109] For example, based on the geographic location information of each image or object in the traffic marking ledger (such as WGS84 coordinates, road ID, lane number, and kilometer marker intervals), an efficient spatial index structure—the R-tree—is constructed. The R-tree can organize the storage objects according to the minimum bounding rectangle (MBR) of spatial regions, supporting fast range queries and neighborhood retrieval.

[0110] Specifically, it is recommended to set the minimum node capacity M_min of the R-tree to 2 and the maximum node capacity M_max to 50. When the node capacity exceeds M_max, a node splitting strategy is adopted. Two seed nodes are selected, and the remaining nodes are assigned to the nearest seed nodes. This process is repeated until all nodes are assigned. This strategy can effectively balance the depth of the index tree and the node utilization rate. Experiments show that using this parameter setting, the query efficiency of the R-tree index is more than 100 times better than that of linear search.

[0111] This index structure enables the rapid identification of ledger entries geographically close to the road area to be detected from massive ledger data, significantly reducing the computational complexity and retrieval time required for subsequent matching and improving the overall system efficiency. Furthermore, the R-tree index has dynamic update capabilities, automatically maintaining spatial hierarchical relationships as ledger content changes, thus ensuring the long-term scalability and stability of the index structure.

[0112] Specifically, using the GNSS positioning information and road reference lines corresponding to the road images to be detected as a query window, a spatial overlap search is performed in the R-tree to quickly filter out the ledger images that intersect or are adjacent to the area to be detected in terms of spatial range. Subsequently, combined with constraints such as road mileage marker intervals, lane numbers, and driving directions, the preliminary results are converged a second time to exclude spatially adjacent but logically discontinuous objects, forming a concise candidate ledger set. In this way, the system not only ensures the spatial relevance of the matching data but also avoids redundant calculations caused by full comparison, effectively improving the real-time performance and accuracy of the ledger update process, thereby achieving rapid response and accurate updates to changes in road markings.

[0113] Step S40: Determine the ledger update strategy based on the comprehensive similarity between the road image to be detected and the candidate ledger image to be updated; the comprehensive similarity is determined based on the difference in paving mask features, the distance between paving center lines, location information and semantic consistency score between the road image to be detected and the ledger image to be updated.

[0114] After the initial screening, a more detailed screening is needed to determine which ledger images require updating. This detailed screening aims to address another core challenge in lifecycle management: how to accurately determine if "this one" is the same as "that one" in history. Matching solely based on location (GPS) or appearance (traditional image features) is prone to errors in complex urban environments with dense road markings, tall buildings obstructing view, and variable lighting, leading to misattributions, chaotic ledger records, and failed lifecycle tracking. Therefore, this embodiment innovatively proposes a multi-dimensional comprehensive similarity calculation method. It decomposes the matching problem into a comprehensive consideration of four dimensions: geometric shape, linear structure, spatial location, and logical attributes, thereby achieving unprecedented matching robustness.

[0115] Specifically, in a feasible implementation, step S40 includes steps C10 to C30:

[0116] Step C10: Determine the defect area within the road marking buffer zone of the road image to be detected, and determine the defect type within the defect area.

[0117] Step C20: Based on the disease severity calculation model, determine the disease severity of the disease type.

[0118] Step C30: Update the traffic marking ledger based on the severity of the damage and the overall similarity.

[0119] The types of road defects include missing road markings, old road markings not being removed, and / or non-standard road marking installation. The severity of defects in missing road markings is determined based on the road marking coverage loss rate, maximum interruption length, and edge roughness. The severity of defects in old road markings not being removed is determined based on the brightness contrast attenuation metric. The severity of defects in non-standard road marking installation is determined based on the compliance with regulations.

[0120] Step C30 includes steps D10~D0:

[0121] Step D10: Determine the differences in paving mask features based on the cross-union ratio of the paving mask between the road image to be detected and the candidate ledger image to be updated.

[0122] Step D20: Determine the distance between the road centerline in the road image to be detected and the road centerline in the candidate ledger image to be updated;

[0123] Step D30 involves weighting and combining the differences in tracing mask features, the distance between tracing centerlines, the differences in positional information, and the semantic consistency scores to obtain the comprehensive similarity score.

[0124] Step D40: If the overall similarity between the road image to be detected and the candidate ledger image to be updated is not less than the similarity threshold and the degree of change of the severity of the damage in the road image to be detected compared with the severity of the damage in the candidate ledger image to be updated is not less than the preset threshold, then the candidate ledger image to be updated with the highest overall similarity is replaced with the road image to be detected.

[0125] Step D50, otherwise, add the road image to be detected to the traffic marking register.

[0126] The core of this implementation method is to calculate a comprehensive similarity score S_match, which integrates information from the following four dimensions:

[0127] Geometric similarity (difference in datum mask features) is obtained by calculating the intersection-over-union (IoU) of the datum masks between the image to be detected and the candidate ledger images. This dimension primarily answers whether the overall shape and coverage of the two datum lines are similar. IoU is sensitive to large-scale morphological changes such as wear and patching of the datum lines and is the basis for determining identity.

[0128] Linear structure similarity (based on grading centerline distance): Based on the extracted structured grading centerlines, the grading centerline distance can be determined by calculating the Fréchet distance or Hausdorff distance between the two centerlines. This dimension answers whether the orientation and twisting shape of the two lines are consistent. Compared to mask IoU, grading centerline distance is more sensitive to small displacements, rotations, or local deformations of the grading. It focuses on the intrinsic structure of the grading as a "line," effectively distinguishing gradings that are close in location but have different orientations, greatly improving the recognition in dense areas.

[0129] Spatial location similarity (based on GNSS distance): This can be achieved by calculating the Haversine distance between the GNSS positioning coordinates of the two entities. This dimension answers whether they are geographically close enough. This is a fundamental constraint for matching, and this application ensures high-precision positioning input by fusing IMU data and post-processing.

[0130] Logical attribute similarity (based on semantic consistency score): compares high-level semantic information such as the type of markings (e.g., solid / dashed lines), color, and lane they belong to. This dimension answers whether their identity information is consistent. Semantic information provides strong logical constraints for matching, effectively eliminating markings that are coincidentally similar in space and shape but are fundamentally different.

[0131] Specifically, after extracting the road marking mask image and the road marking centerline, the first step is to perform defect area detection within the road marking buffer zone. In this embodiment, road defect types include missing road markings, old road markings not being removed, and / or improperly installed road markings.

[0132] The determination of defect areas combines characteristics such as holes, breaks, abrupt width changes, and brightness differences in the pavement marking mask to identify types of defects, including missing pavement markings, uncleared old markings, and non-standard pavement marking installation. Missing pavement markings are determined based on characteristics such as mask holes, break lengths, and abrupt width changes. Uncleared old markings are determined by combining historical map sheet differences, directional consistency, and brightness thresholds. Non-standard installation is addressed by using a rule base to verify width, spacing, arrow positions, and pedestrian crossing strips. Raised pavement markers and vertical markings are determined to be missing or non-standard by checking the required grid points and spacing / tilt thresholds.

[0133] Next, based on the severity calculation model, various types of defects are quantified. For defects in road markings, a comprehensive score is calculated based on the coverage loss rate, the proportion of the maximum interruption length, and the edge roughness index. For old road markings that have not been removed, attenuation analysis is performed using a brightness contrast attenuation metric. For non-standard settings, a rule base is used to check the compliance of width, spacing, arrow position, and pedestrian crossing strip spacing, and a rule compliance score is output based on the inverse index of rule compliance. Based on the comprehensive analysis of various defect types, a defect severity calculation model is confirmed, which can be expressed as:

[0134]

[0135] in: The weights of each indicator can be adjusted according to the scenario, and the sum of all weights is 1.

[0136] R loss =A missing / A total Missing area A missing Total area A total It mainly reflects the overall wear and tear of the road marking materials and is the basis for assessing whether the road marking functions (such as nighttime reflectivity and rainy night markings) meet the standards.

[0137] Maximum interrupt length Total length of the markings The proportion. The deeper meaning of this indicator lies in assessing whether the continuity of the lane marking guidance function has been disrupted. A short and wide wear ( R loss A higher fracture might be less desirable than a thin but long one. R gap (Higher) lines pose a greater risk to the driver's visual guidance. This is especially true for solid lines. R gap This value indicates a serious driving safety hazard.

[0138] It can be calculated using methods such as contour curvature variance or fractal dimension. It quantifies the "roughness" of the road marking edges, which not only affects the aesthetics of the city, but more importantly, blurry edges reduce the visibility of the road markings under complex lighting conditions (such as backlighting and wet, reflective surfaces), and is an early sign of road marking quality deterioration.

[0139] : Indicates that the brightness of the new markings is calculated. L new brightness compared to old markingsL old Differences between ( R lum = 1 - L old / L new This is used to assess the severity of old road markings that have not been removed. It is directly related to the reflectivity and contrast of the road markings; too low a contrast will render the road markings ineffective at night or in bad weather.

[0140] : An inverse indicator of rule compliance, where To assign a score to a rule, for example, the degree of irregularity in the markings' geometric dimensions, spacing, and position can be quantified by comparing them with a rule base K containing more than 20 rules, thus obtaining a rule score. It reflects the engineering quality and compliance of the road marking construction; non-compliant road markings may convey incorrect or misleading information to drivers.

[0141] Understandably, through the above multi-dimensional quantitative assessment model, this application transforms the traditional, experience-based, vague, and inconsistent disease assessment in maintenance work into an automated, precise, and standardized scientific assessment. The output severity score... S sev It can be directly used for prioritizing maintenance work orders, accurately budgeting maintenance costs, and conducting macro-analysis and prediction of the health status of regional road networks, thereby realizing intelligent and scientific road maintenance decision-making.

[0142] Subsequently, the comprehensive similarity between the road image to be detected and the candidate ledger images is calculated. This process includes calculating the intersection-over-union (IoU) of the road mask between the two images to reflect geometric differences; calculating the Fréchet or Hausdorff distance between their centerlines to reflect the degree of line deviation; and weighting and combining the differences in road mask features, the distance between the road centerlines, the GNSS positioning deviation, and the semantic consistency score to form the comprehensive similarity.

[0143] Specifically, the cross-union ratio (CUI) of the tracing mask between the image to be detected and the candidate ledger images is first calculated to reflect geometric differences. The CUI is calculated as follows:

[0144]

[0145] in, M t As a marker mask for the image to be detected, M lrepresents the line mask for the candidate ledger images, where ∩ denotes the intersection and ∪ denotes the union. A larger IoU value indicates a more similar geometric shape, and the value ranges from [0,1].

[0146] Then, the distance metric between the center line of the grading line in the image to be detected and the center line of the grading line in the candidate ledger image is calculated to reflect the degree of line offset.

[0147] In one example, the distance between the center lines of the road markings is calculated using the Fréchet distance. The Fréchet distance takes into account the directionality of the curves and is more suitable for evaluating the similarity of the center lines. The specific calculation method is as follows:

[0148] For two center lines C t and C l The distances are calculated using dynamic programming, with n and m points respectively. The distance matrix D is defined as follows: D i express C t The first i points and C l The Friesian distance between the first j points. Its recurrence relation is:

[0149]

[0150] in, Let be the Euclidean distance between the two points. The final Fréchet distance is D. n The algorithm has a time complexity of O(n×m) and a space complexity of O(n×m). For long marking lines, segmented calculation or sampling methods can be used to reduce the computational complexity.

[0151] In another example, the distance between the center lines of the markings is calculated using Hausdorff distance: Hausdorff distance calculates the maximum and minimum distance between two curves, regardless of the directionality of the curves. For two center lines... C t and C l The formula for calculating the Hausdorff distance is:

[0152] d H ( C t , C l ) = max(sup {x∈C t } inf {y∈ C l} ||xy||, sup {y∈C l } inf {x∈ C t} ||xy||)

[0153] in, d H ( C t , C l ( ) represents two center lines C t and C l The Hausdorff distance, where x represents the centerline. C t A point in the center line, where y represents the center line. C l One of the points, sup {x∈C t } inf {y∈ C l} ||xy|| indicates from C t point to C l Maximum and minimum distance h( C t , C l ), sup {y∈C l } inf {x∈ C t} ||xy|| indicates from C l point to C t Maximum and minimum distance h( C l , C t ), max(sup {x∈C t } inf {y∈ C l} ||xy||, sup {y∈C l } inf {x∈ C t} ||xy||) represents the maximum value among them.

[0154] The specific calculation steps are as follows: For C tFor each point x in the array, calculate its distance to... C l The minimum distance between all points is taken as the maximum value to obtain h( C t , C l );for C l For each point y in the array, calculate its distance to... C t The minimum distance between all points is taken as the maximum value to obtain h( C l , C t ); The Hausdorff distance is max(h( C t , C l ), h( C l , C t The time complexity of this algorithm is O(n×m), which can be optimized to O(n×log(m)) using spatial indexing structures such as KD-trees.

[0155] Then, the distance from the center line of the marking d F Distance normalization is performed, and the final distance is normalized:

[0156] d F_norm = d F / d max ,in, d F Distance from the centerline d max As a normalization constant, a value of 10 meters is recommended. For different road types, d max The distance can be dynamically adjusted: a suggested value of 5-8 meters is recommended for urban roads, and 10-15 meters for highways. The normalized distance value is used for comprehensive similarity calculation to ensure the consistency of dimensions of indicators across different dimensions.

[0157] Next, the GNSS positioning distance between the road image to be detected and the candidate ledger images is calculated to reflect spatial similarity. The calculation method is as follows:

[0158] d gps = Haversine( lat t lon t latl lon l )

[0159] in,( lat t , lon t ) represents the GPS coordinates of the image to be detected. lat l lon l ( ) represents the GPS coordinates of the candidate ledger image, and Haversine is the formula for calculating the great circle distance between two points. Distance normalization:

[0160] d gps_norm = d gps / r

[0161] in, r For the normalized radius, a value of 3-10 meters is recommended, which can be dynamically adjusted according to the road type.

[0162] Next, a semantic consistency score is used to measure whether the object type matches the attributes. Semantic consistency score S sem The following dimensions are used for calculation: consistency of object type (e.g., whether the type of marking or the type of disease are consistent), consistency of attributes (e.g., whether the attributes such as the color and material of the marking are consistent), and consistency of rule compliance (e.g., whether they comply with the same rule constraints).

[0163] The calculation formula is: S sem = w type × S type + w attr × S attr + w rule × S rule ,in, S type , S attr , S rule The scores are for consistency in type, attribute, and rule compliance, respectively. w type , w attr , w rule For the corresponding weights.

[0164] Finally, the overall similarity can be represented as follows:

[0165] in,

[0166] The weighted coefficient of the overall similarity reflects the importance of geometric, location, and semantic elements.

[0167] Based on this, a similarity threshold is set. With severity change threshold When the overall similarity and (Severity of damage to the new ledger images) Difference in the severity of damage compared to old ledger images Update the ledger when the severity change threshold is exceeded; otherwise, keep it. Add new objects when the similarity threshold is not reached.

[0168] When the overall similarity is not less than the similarity threshold and the change in the severity of the disease exceeds the set threshold, the system considers that there is a significant change in the road section and automatically replaces the original ledger object with the new detection results.

[0169] When the overall similarity or change rate does not reach the threshold, the system determines it to be a newly added or suspected newly added object and adds it to the ledger as a new entry.

[0170] Experiments show that, compared with using only IoU matching (success rate 81.67%, false match rate 18.33%), the comprehensive similarity matching method of this application can achieve a success rate of 95.0%, reduce the false match rate to 5.0%, and improve the matching accuracy by about 13.33 percentage points, which significantly improves the accuracy and reliability of ledger updates.

[0171] Furthermore, to improve the reliability of the results, the system can introduce uncertainty assessment before updating. Specifically, steps S40 is followed by steps S50-S60:

[0172] Step S50: Based on the model prediction results generated during the process of performing disease identification and ledger matching, assess the uncertainty of the output object-level prediction.

[0173] Step S60: If the uncertainty of the object-level prediction exceeds the uncertainty threshold, or if there is a conflict with the historical ledger records, then generate and output a manual review task.

[0174] Specifically, in critical areas with high reliability requirements, such as road asset management, any automated system must address the core issue of "trustworthiness." Traditional AI models are like "black boxes," with opaque decision-making processes and an inability to assess the reliability of their own results. False positives or false negatives directly impact the accuracy of maintenance decisions and may even pose safety risks. To address this challenge, this application proactively designs a complete closed loop from "uncertainty assessment" to "human review" and then to "model self-optimization," aiming to upgrade the system from a static "black box tool" into a dynamic, trustworthy, and self-evolving "learning partner."

[0175] This embodiment employs uncertainty quantification technology, allowing the model to output an accompanying "uncertainty score U" along with the detection or matching results.

[0176] In one example, using the Monte Carlo Dropout method, during the model inference (prediction) phase, this embodiment keeps the Dropout layer in the network active and performs T random forward propagations (e.g., T=30) on the same input. Since the random deactivation paths are different each time, T slightly different prediction results are obtained. The variance between these results is quantified as the model's "uncertainty" regarding the prediction. The larger the variance, the lower the "consensus" within the model regarding the result, and the less reliable the result.

[0177] In another example, the Deep Ensemble method is used to independently train M models (e.g., M=5-10) with identical structures but slightly different initializations or training data, forming an "expert committee". During inference, all models simultaneously predict the input, and the degree of divergence (variance) among the M predictions represents the uncertainty of the system.

[0178] Subsequently, the system does not blindly accept all automated results, but instead sets up intelligent "sentinels." When any of the following conditions are met, the system will automatically mark the event as high-risk and generate a structured manual review task to push to maintenance experts:

[0179] When U exceeds a threshold or conflicts with historical records, a review task is automatically generated, and the results of manual review are fed back into model training. Specifically, this manifests as follows:

[0180] High uncertainty: The uncertainty score U of the prediction result exceeds the preset threshold (e.g., U ≥ 0.3). This indicates that the model itself does not have sufficient confidence in this judgment.

[0181] Logical Conflict: There is a clear contradiction between the current detection results and historical records. This application designs a conflict detection mechanism, including:

[0182] Location conflict: The current position of the matched object deviates too much from its historical position (e.g., more than 5 meters).

[0183] Type conflict: The same object undergoes an illogical and drastic change in its disease type (such as changing directly from "minor wear" to "non-standard settings").

[0184] Severity conflict: The severity score shows an unreasonable jump (such as changing from 0.9 to 0.1 in a short period of time).

[0185] Understandably, the results of manual review go beyond correcting a single event. The "correct" or "incorrect" labels confirmed by review experts, along with corresponding images, uncertainty scores, and other data, are automatically packaged into high-quality training samples and fed back into the model's training database. The system can periodically, or after accumulating a certain number of samples, utilize this valuable expert knowledge for incremental training or retraining. This closed-loop mechanism of "detection-evaluation-review-learning" enables the technical solution of this application to possess continuous self-evolution capabilities. In practical use, the system becomes increasingly "intelligent," more "vigilant" about scenarios where errors have occurred, and its overall performance and reliability continuously improve with time and data accumulation, ultimately achieving a perfect combination of automation and high reliability.

[0186] Finally, the updated structured ledger data is output in GeoJSON or Shapefile format and pushed to the GIS platform, work order system and asset management platform to realize automated closed-loop management of road facility inspection, maintenance and rectification.

[0187] Understandably, by integrating multi-source information processing technologies such as image segmentation, geometric skeleton extraction, spatial index retrieval, and disease identification and severity quantification assessment, a fully automated system has been constructed, encompassing road disease detection and dynamic updating of the ledger. This system not only enables efficient candidate ledger retrieval based on R-tree spatial indexing, significantly reducing matching computation and improving update efficiency, but also achieves accurate identification and quantitative assessment of diseased areas through comprehensive similarity determination of marking centerlines and mask features, ensuring the accuracy and reliability of update results. Simultaneously, the combination of uncertainty assessment and a closed-loop mechanism for manual review significantly enhances the system's adaptive learning capability and long-term operational stability, thereby achieving intelligent, real-time, and refined maintenance management of road markings and related facilities.

[0188] To enable those skilled in the art to better understand this embodiment, the technical solution of this embodiment is explained and illustrated below through specific implementation examples in specific application scenarios. It should be understood that the following examples are only used to explain this application and are not intended to limit the scope of protection of the claims of this application.

[0189] This example proposes an end-to-end integrated solution that integrates vehicle-mounted imaging, GNSS / IMU, edge computing, communication, and verification modules. Addressing issues such as missing pavement markings, uncleared old markings, improper installation, damaged raised pavement markers, and non-standard facade markings, the system achieves a fully automated closed-loop process of "detection – assessment – ​​location – matching – update – linkage." Through multi-source sensor collaboration, centerline / geometric topology constraints, rule compliance reasoning, multi-indicator severity scoring, and cross-temporal and spatial ledger matching, the system achieves accurate identification, quantitative assessment, and dynamic ledger maintenance.

[0190] Specifically, refer to Figure 4 The method flow in this example is as follows:

[0191] Data Acquisition and Preprocessing:

[0192] The acquisition module simultaneously acquires video streams from the vehicle-mounted camera, GNSS / IMU positioning data, and vehicle attitude information;

[0193] The preprocessing module performs image stabilization, noise reduction, exposure / white balance normalization, glare / rain / fog suppression, motion blur compensation, and timing alignment.

[0194] Road markings and facility inspection:

[0195] Use semantic / instance segmentation to output line masks The centerline is obtained through skeleton refinement (such as Zhang–Suen / Guo–Hall). And perform topology repair;

[0196] Defect areas are identified within the road marking buffer zone; raised pavement markers and facade markings are identified and classified through target detection.

[0197] Disease type determination:

[0198] Marking defects are identified based on characteristics such as mask voids, break lengths, and abrupt width changes.

[0199] The old markings were not cleared, and a joint judgment was made based on historical map sheet differences, directional consistency, and brightness thresholds.

[0200] Setting incorrect rules for calling the K-validation rule base to check width, spacing, arrow position, pedestrian crossing strips, etc.

[0201] Raised pavement signs and facade markings are identified as missing or non-standard by using the required grid points and spacing / tilt angle thresholds. This is based on the rule base. Includes: lane width, length / spacing of dashed and solid line segments, number of pedestrian crossing strips, distance from arrows / guide markers to stop lines or intersections, spacing / missing rate threshold of raised pavement markers, and spacing and tilt tolerance of vertical markers.

[0202] Spatial mapping and indexing:

[0203] Using the external road reference line, the defect objects are mapped to geographic line segments / points / areas, and the WGS84 coordinates are output with the road ID, lane, and mileage marker interval attached.

[0204] Construct spatial indexes such as R trees to support fast retrieval.

[0205] Ledger matching and difference update:

[0206] Set similarity threshold With severity change threshold When the overall similarity and (Severity of damage to the new ledger images) Severity of damage to old ledger images The difference is not less than the severity change threshold. Update the ledger when the similarity threshold is reached, otherwise keep it; add new objects when the similarity threshold is not reached.

[0207] Uncertainty assessment and review closed loop:

[0208] Output pixel / object level uncertainty using methods such as MC Dropout / Deep Ensemble ;

[0209] when When the threshold is exceeded or there is a conflict with the historical record, a review task is automatically generated, and the results of the manual review are fed back into the model training.

[0210] Output and platform integration:

[0211] Structured records (GeoJSON / Shapefile) and evidence image slices are generated and pushed to GIS, work order system and asset management platform to realize closed-loop scheduling of inspection and rectification.

[0212] The implementation results of this example are shown in Tables 1, 2, and 3:

[0213] As can be seen from Table 1, the comprehensive similarity calculation matching method (IoU+GNSS+) of this example scheme D is as follows. d F + S semThe matching success rate reached 95.0%, and the false matching rate dropped to 5.0%. Compared with Scheme A, which only uses IoU (matching success rate 81.67%, false matching rate 18.33%), the matching success rate increased by 13.33 percentage points and the false matching rate decreased by 13.33 percentage points, significantly improving the accuracy and reliability of ledger matching. Compared with the lack of a ledger update mechanism in Reference Document 1, this application achieves efficient cross-temporal and spatial ledger matching, providing important technical support for road facility management.

[0214] As shown in Table 2, using R-tree retrieval can effectively reduce the number of candidate ledger images. The average number of candidates decreased from 128 to 23, a reduction of about 82%. The matching latency decreased from 38.2ms / object to 12.6ms / object, an efficiency improvement of about 3 times. This effectively solves the matching efficiency problem when the ledger data volume is large, and realizes real-time ledger updates.

[0215] As shown in Table 3, when the uncertainty threshold is set to 0.3, 53 samples were reviewed, with a manual review hit rate of 50.94% and a false negative rate of 10.0%, effectively identifying high-risk results and conducting manual reviews. When the uncertainty threshold is set to 0.4, 9 samples were reviewed, with a manual review hit rate of 55.56%, but a relatively high false negative rate of 83.33%. Considering both the hit rate and false negative rate, it is recommended that the uncertainty threshold be set between 0.3 and 0.35, which can effectively control the false negative rate while ensuring the review hit rate.

[0216] Table 1: Comparison of Intertemporal Matched Ablation

[0217]

[0218] Table 2. Acceleration effect of tree candidate set convergence

[0219]

[0220] Table 3 Sensitivity to Uncertainty Trigger Threshold

[0221]

[0222] It should be noted that all the examples above are only for understanding this application and do not constitute a limitation on the road defect detection and ledger update method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0223] This application also provides a road defect detection and record-keeping device; please refer to... Figure 5 The road defect detection and record-keeping device includes:

[0224] The image acquisition module 10 is used to acquire the image of the road to be detected and the location information of the inspection area corresponding to the image to be detected.

[0225] The feature extraction module 20 is used to extract the road marking mask image of the road image to be detected and to perform skeletonization and topological repair processing on the road marking mask image to determine the center line of the road marking.

[0226] The candidate ledger determination module 30 is used to determine multiple candidate ledger images to be updated from the traffic marking ledger based on the location information of the road to be detected and the location information corresponding to each ledger image in the traffic marking ledger.

[0227] The ledger update module 40 is used to determine the ledger update strategy based on the comprehensive similarity between the road image to be detected and the candidate ledger image to be updated. The comprehensive similarity is determined based on the difference in paving mask features, the distance between the paving center lines, the location information, and the semantic consistency score between the road image to be detected and the ledger image to be updated.

[0228] The road defect detection and record updating device provided in this application, employing the road defect detection and record updating method described in the above embodiments, can solve the technical problem of low reliability in related technologies for road defect detection and record updating. Compared with related technologies, the beneficial effects of the road defect detection and record updating device provided in this application are the same as those of the road defect detection and record updating method described in the above embodiments, and other technical features in the road defect detection and record updating device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0229] This application provides a road defect detection and record updating device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the road defect detection and record updating method described in the above embodiment.

[0230] The following is for reference. Figure 6The diagram illustrates a structural schematic of a road defect detection and record-keeping update device suitable for implementing embodiments of this application. The road defect detection and record-keeping update device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The road defect detection and record updating device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0231] like Figure 6 As shown, the road defect detection and record-keeping update device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 (ROM) or a program loaded from a storage device 1003 into a random access memory 1004 (RAM). The random access memory 1004 also stores various programs and data required for the operation of the road defect detection and record-keeping update device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 (I / O interface) is also connected to the bus 1005. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the road damage detection and record-keeping update equipment to exchange data wirelessly or via wired communication with other devices. Although the figure shows road damage detection and record-keeping update equipment with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0232] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0233] The road defect detection and record-keeping update equipment provided in this application, employing the road defect detection and record-keeping update method described in the above embodiments, can solve the technical problem of low reliability in related technologies for road defect detection and record-keeping updates. Compared with related technologies, the beneficial effects of the road defect detection and record-keeping update equipment provided in this application are the same as those of the road defect detection and record-keeping update method described in the above embodiments, and other technical features of this road defect detection and record-keeping update equipment are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0234] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0235] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0236] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the road defect detection and ledger update method described in the above embodiments.

[0237] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0238] The aforementioned computer-readable storage medium may be included in the road defect detection and record-keeping equipment; or it may exist independently and not be assembled into the road defect detection and record-keeping equipment.

[0239] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the road defect detection and record-keeping update device, the road defect detection and record-keeping update device: acquires the image of the road to be detected and the location information of the road area corresponding to the image; extracts the road marking mask image from the image of the road to be detected and performs skeletonization and topological repair processing on the road marking mask image to determine the center line of the road marking; based on the location information of the road to be detected and the location information corresponding to each record image in the traffic marking record, determines multiple candidate record images to be updated from the traffic marking record; and determines a record-keeping update strategy based on the comprehensive similarity between the image of the road to be detected and the candidate record images to be updated.

[0240] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0241] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0242] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0243] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described road defect detection and ledger update method, thereby solving the technical problem of low reliability in related technologies for road defect detection and ledger update. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the road defect detection and ledger update method provided in the above embodiments, and will not be repeated here.

[0244] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the road defect detection and ledger update method described above.

[0245] The computer program product provided in this application can solve the technical problem of low reliability in road defect detection and record updating in related technologies. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the road defect detection and record updating methods provided in the above embodiments, and will not be repeated here.

[0246] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for detecting road defects and updating records, characterized in that, The method includes: Obtain the image of the road to be detected and the location information of the road area corresponding to the image of the road to be detected; Extract the road marking mask image from the image of the road to be detected, and perform skeletonization and topology repair processing on the road marking mask image to determine the center line of the road marking; Based on the location information of the road to be detected and the location information corresponding to each image in the traffic marking ledger, multiple candidate images to be updated are determined from the traffic marking ledger. A ledger update strategy is determined based on the comprehensive similarity between the road image to be detected and the candidate ledger image to be updated; the comprehensive similarity is determined based on the difference in paving mask features, the distance between paving centerlines, positional information, and semantic consistency score between the road image to be detected and the ledger image to be updated. The step of determining the ledger update strategy based on the comprehensive similarity between the road image to be detected and the candidate ledger image to be updated includes: Within the lane marking buffer zone of the road image to be detected, the defect area is determined, and the defect type within the defect area is identified. The severity of the disease type is determined based on the disease severity calculation model; The traffic marking ledger is updated based on the severity of the damage and the overall similarity. The step of updating the traffic marking ledger based on the severity of the damage and the overall similarity includes: Based on the intersection-union ratio of the road marking masks between the road image to be detected and the candidate ledger image to be updated, the feature differences of the road marking masks are determined. The distance between the road centerline in the road image to be detected and the road centerline in the candidate ledger image to be updated is determined based on the distance metric between them. The comprehensive similarity score is obtained by weighting and combining the differences in the grading mask features, the distance between the grading center lines, the differences in positional information, and the semantic consistency score. If the overall similarity between the road image to be detected and the candidate ledger image to be updated is not less than the similarity threshold and the degree of change in the severity of the damage of the road image to be detected compared to the severity of the damage of the candidate ledger image to be updated is not less than the preset threshold, then the candidate ledger image to be updated with the highest overall similarity will be replaced with the road image to be detected. Otherwise, the image of the road to be detected will be added to the traffic marking register.

2. The method for road defect detection and record updating as described in claim 1, characterized in that, The steps of extracting the road marking mask image from the road image to be detected, performing skeletonization and topology repair on the road marking mask image, and determining the center line of the road marking include: Edge detection is performed on the road image to be detected to obtain a binary edge image of the road image to be detected; The binary edge image is subjected to Hough transform to extract the set of line segments related to traffic markings in the binary edge image; the set of line segments includes the number of line segments, the average line segment length, and the direction variance; Based on the set of line segments, determine the complexity of traffic markings in the road image to be detected; If the complexity of the traffic markings is low, the Zhang-Suen thinning algorithm is used to perform skeletonization processing on the marking mask image to extract the marking skeleton; If the traffic markings are complex, the Guo-Hall algorithm is used to perform skeletonization processing on the marking mask image to extract the marking skeleton. The topology repair process is performed on the road marking skeleton to determine the center line of the road marking.

3. The method for road defect detection and record updating as described in claim 2, characterized in that, The step of performing topology repair processing on the road marking skeleton and determining the road marking centerline includes: Traverse the gradation skeleton and identify all discontinuous breakpoints; Define a sliding local repair window. Within the local repair window area, use the shortest path algorithm and / or morphological dilation operation to connect the line segments at both ends of the breakpoint to obtain continuous lines. Based on a preset branch length threshold, burrs in the continuous markings are identified and removed; The deburred marking skeleton is smoothed to determine the center line of the marking.

4. The method for road defect detection and record updating as described in claim 1, characterized in that, The step of determining multiple candidate images to be updated from the traffic marking ledger based on the location information of the road to be detected and the location information corresponding to each image in the traffic marking ledger includes: Based on the minimum outer rectangle of each ledger image, construct an R-tree spatial index; Based on the spatial coordinates of the road to be detected, a spatial query window is defined, and an overlap query operation is performed to obtain an initial candidate image set. Based on the name information, lane number, and overlap of the mileage marker intervals of the road to be detected, the initial candidate image set is converged, and images that are spatially adjacent but logically discontinuous are removed to obtain the candidate ledger images to be updated.

5. The method for road defect detection and record updating as described in claim 1, characterized in that, Following the step of determining the ledger update strategy, the following step is also included: Based on the model prediction results generated during the process of disease identification and ledger matching, the uncertainty of the output object-level prediction is evaluated. If the uncertainty of the object-level prediction exceeds the uncertainty threshold, or if there is a conflict with the historical ledger records, a manual review task will be generated and output.

6. A road defect detection and record-keeping device, characterized in that, The device includes: The image acquisition module is used to acquire the image of the road to be detected and the location information of the inspection area corresponding to the image of the road to be detected; The feature extraction module is used to extract the road marking mask image of the road image to be detected and perform skeletonization and topological repair processing on the road marking mask image to determine the center line of the road marking; The candidate ledger determination module is used to determine multiple candidate ledger images to be updated from the traffic marking ledger based on the location information of the road to be detected and the location information corresponding to each ledger image in the traffic marking ledger. The ledger update module is used to determine the ledger update strategy based on the comprehensive similarity between the road image to be detected and the candidate ledger image to be updated; the comprehensive similarity is determined based on the difference in paving mask features, the distance between paving center lines, positional information and semantic consistency score between the road image to be detected and the ledger image to be updated. The ledger update module is also used for: Within the lane marking buffer zone of the road image to be detected, the defect area is determined, and the defect type within the defect area is identified. The severity of the disease type is determined based on the disease severity calculation model; The traffic marking ledger is updated based on the severity of the damage and the overall similarity. The ledger update module is also used for: Based on the intersection-union ratio of the road marking masks between the road image to be detected and the candidate ledger image to be updated, the feature differences of the road marking masks are determined. The distance between the road centerline in the road image to be detected and the road centerline in the candidate ledger image to be updated is determined based on the distance metric between them. The comprehensive similarity score is obtained by weighting and combining the differences in the grading mask features, the distance between the grading center lines, the differences in positional information, and the semantic consistency score. If the overall similarity between the road image to be detected and the candidate ledger image to be updated is not less than the similarity threshold and the degree of change in the severity of the damage of the road image to be detected compared to the severity of the damage of the candidate ledger image to be updated is not less than the preset threshold, then the candidate ledger image to be updated with the highest overall similarity will be replaced with the road image to be detected. Otherwise, the image of the road to be detected will be added to the traffic marking register.

7. A road defect detection and record-keeping device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the road defect detection and ledger update method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the road defect detection and ledger update method as described in any one of claims 1 to 5.

Images