A method, system and equipment for intelligent inspection and early warning of road defects

Abstract

This application provides a method, system, and device for intelligent inspection and early warning of road defects. The method includes: performing timestamp alignment and filtering on acquired road images and LiDAR point cloud data to determine image sequences and point cloud sequences; performing visual and LiDAR fusion processing based on the image sequences and point cloud sequences to determine the position, attitude, and latest road and defect map of the inspection vehicle; performing road defect target detection based on the latest road and defect map, image sequences, and point cloud sequences to determine road defect information; performing temporal modeling and state updating on the defect targets to determine the current severity and rate of change of the defect targets; predicting the development trend based on the rate of change to determine the future development trend of the defect targets; and generating early warning information for the defect targets and sending it to an external terminal if the current severity and future development trend exceed a preset threshold. The above scheme can improve the accuracy of intelligent inspection and early warning of road defects.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation infrastructure maintenance and automated inspection technology, specifically to a method, system and equipment for intelligent inspection and early warning of road defects. Background Technology

[0002] With the large-scale construction of highways, urban expressways, and dedicated smart logistics channels, road traffic volume has increased exponentially. Heavy-duty vehicles, frequent braking, and extreme weather conditions have collectively accelerated the generation and spread of road surface defects such as cracks, potholes, ruts, and corner chipping. According to statistics from the Ministry of Transport, over 40% of the sections of trunk highways in my country have surface cracks of varying degrees, and the average annual growth rate of potholes exceeds 15%. If these defects are not addressed promptly, they can reduce driving comfort and increase energy consumption, or even lead to tire blowouts, rollovers, or mass traffic accidents, resulting in enormous loss of life, property damage, and social costs.

[0003] Current maintenance mainly relies on two types of equipment: manual visual inspection and single high-definition camera or 3D laser scanning vehicle. Manual inspection requires closing some lanes, is inefficient and has a large subjective error. Pure vision systems are acceptable during the day, but are prone to misdetection at night, in backlight, rain or snow and cannot quantify pothole depth. Pure laser systems have high 3D accuracy, but cannot identify small cracks, resulting in poor accuracy in road defect inspection. Summary of the Invention

[0004] This application aims to provide a method, system, and device for intelligent inspection and early warning of road defects, which can improve the accuracy of intelligent inspection and early warning of road defects.

[0005] The technical solution of this invention is implemented as follows:

[0006] In a first aspect, embodiments of this application provide an intelligent inspection and early warning method for road defects, the method comprising:

[0007] During the patrol vehicle's journey, road images and lidar point cloud data are acquired; and the road images and lidar point cloud data are subjected to timestamp alignment and filtering processes respectively to determine the image sequence and point cloud sequence.

[0008] Based on the image sequence and the point cloud sequence, visual and laser fusion processing is performed to determine the position, attitude, and latest road and damage map of the inspection vehicle;

[0009] Based on the latest road and defect map, the image sequence, and the point cloud sequence, road surface defect targets are detected to determine road surface defect information; wherein, the road surface defect information includes the type, location, and feature parameters of the defect targets;

[0010] Based on the acquired historical inspection data, the disease targets are time-series modeled and their status updated to determine the current severity and rate of change of the disease targets; and based on the rate of change, the development trend is predicted to determine the future development trend of the disease targets.

[0011] If the current severity and the future development trend meet preset conditions greater than preset threshold conditions, then an early warning message for the target road defect is generated; and the early warning message is sent to an external terminal to realize intelligent inspection and early warning of road defects.

[0012] Secondly, embodiments of this application provide an intelligent road defect inspection and early warning system, comprising: a data acquisition module, an adaptive perception module, a map construction module, a defect detection module, a defect evolution modeling module, and an active early warning module, wherein...

[0013] The acquisition module is used to acquire road images and lidar point cloud data;

[0014] The adaptive perception module is used to acquire the road image and the lidar point cloud data during the inspection vehicle's operation; and to perform timestamp alignment and filtering on the road image and lidar point cloud data respectively to determine the image sequence and point cloud sequence.

[0015] The map building module is used to perform visual and laser fusion processing based on the image sequence and the point cloud sequence to determine the position, attitude, and latest road and defect map of the inspection vehicle.

[0016] The road damage detection module is used to detect road damage targets and determine road damage information based on the latest road and damage map, the image sequence, and the point cloud sequence; wherein, the road damage information includes the type, location, and feature parameters of the damage targets;

[0017] The disease evolution modeling module is used to perform time-series modeling and status updates on the disease target based on the acquired historical inspection data, determine the current severity and rate of change of the disease target, and predict the development trend based on the rate of change to determine the future development trend of the disease target.

[0018] The active early warning module is used to generate early warning information for the road defect target if the current severity and the future development trend exceed a preset threshold condition; and to send the early warning information to an external terminal to realize intelligent inspection and early warning of road defects.

[0019] Thirdly, embodiments of this application provide an intelligent road defect inspection and early warning device, which includes: a processor and a memory; wherein,

[0020] The memory is used to store computer programs;

[0021] The processor is configured to call and run the computer program from the memory to perform the method as described in the first aspect.

[0022] This application provides a method, system, and device for intelligent inspection and early warning of road defects. The method includes: acquiring road images and LiDAR point cloud data during the operation of an inspection vehicle; performing timestamp alignment and filtering on the road images and LiDAR point cloud data respectively to determine an image sequence and a point cloud sequence; performing visual and LiDAR fusion processing based on the image sequence and the point cloud sequence to determine the position, attitude, and latest road and defect map of the inspection vehicle; and performing road defect target detection based on the latest road and defect map, the image sequence, and the point cloud sequence. The system identifies road surface defects, including the type, location, and characteristic parameters of the defects. Based on acquired historical inspection data, it performs time-series modeling and state updates on the defects to determine their current severity and rate of change. Based on the rate of change, it predicts future trends to determine the future development trend of the defects. If the current severity and future development trend meet preset conditions exceeding preset thresholds, it generates early warning information for the defects and sends this information to an external terminal, achieving intelligent road defect inspection and early warning. In this scheme, by performing timestamp alignment and filtering on road images and LiDAR point cloud data respectively, image sequences and point cloud sequences are determined, improving the accuracy of the image and point cloud data. Visual and LiDAR fusion processing is then applied to the image and point cloud sequences to determine the position, attitude, and latest road and defect map of the inspection vehicle, enabling centimeter-level positioning and real-time 3D map updates in GNSS-free scenarios. Based on the latest road and defect maps, image sequences, and point cloud sequences, pavement defect targets are detected to determine pavement defect information. This process considers both texture and morphology, reducing missed and false detections, thereby improving the accuracy and reliability of intelligent road defect inspection. Based on acquired historical inspection data, time-series modeling, state updates, and development trend prediction are performed on defect targets to determine their future development trends. If the current severity and future development trend meet preset conditions exceeding preset thresholds, a warning message for the defect target is generated and sent to an external terminal, realizing intelligent road defect inspection and early warning. This not only improves the accuracy and reliability of intelligent road defect inspection but also enhances road safety. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0025] Figure 1 A schematic flowchart of an optional road defect intelligent inspection and early warning method provided in an embodiment of this application;

[0026] Figure 2 A schematic diagram of the framework of an intelligent road defect inspection and early warning system provided in this application embodiment;

[0027] Figure 3 This application provides a schematic diagram of the sensor arrangement for an inspection vehicle.

[0028] Figure 4 An optional road defect probability heatmap is provided for an embodiment of this application for a road defect intelligent inspection and early warning method.

[0029] Figure 5 An optional point cloud height difference heatmap is provided for an intelligent road defect inspection and early warning method according to an embodiment of this application.

[0030] Figure 6 An optional three-dimensional point cloud partial cross-sectional view of a road defect intelligent inspection and early warning method provided in an embodiment of this application;

[0031] Figure 7 A schematic diagram of an optional road defect database interface provided in an embodiment of this application for a road defect intelligent inspection and early warning method;

[0032] Figure 8 An optional road damage severity trend curve is provided for an embodiment of this application of an intelligent road damage inspection and early warning method.

[0033] Figure 9 A schematic diagram of an optional crack detection frame for an intelligent road defect inspection and early warning method provided in this application embodiment;

[0034] Figure 10This is a schematic diagram of the structure of an intelligent road defect inspection and early warning system provided in an embodiment of this application;

[0035] Figure 11 This is a schematic diagram of the structure of an intelligent road defect inspection and early warning device provided in an embodiment of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0038] In the following description, references to "some embodiments," "this embodiment," "this application embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0039] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Related road defect inspection technologies generally face problems such as insufficient real-time performance, limited positioning accuracy, limited sensing methods, lack of defect evolution prediction, and inability to form closed-loop management: most rely on offline calculations, making it impossible to provide real-time risk feedback during vehicle operation; defect coordinates depend on GNSS, and errors in tunnel or urban canyon environments can reach the meter level; pure visual methods are affected by lighting and shadows, while pure lasers are difficult to detect fine cracks, and the parameters of both types of sensors are mostly fixed values, making it difficult to adapt to nighttime and high-speed driving conditions; most systems only provide the current defect size, cannot assess the remaining safe life, and cannot proactively issue warnings to maintenance departments or road users, resulting in delayed maintenance decisions and increased accident risks.

[0041] This application aims to provide an intelligent inspection and early warning method for road defects. Through adaptive perception control, deep fusion positioning and defect identification of vision and point cloud, state space trend prediction, and threshold-based proactive early warning, it achieves centimeter-level high-precision positioning without GNSS, millimeter-level defect measurement, and hourly-level risk prediction. When the severity of a defect is detected to have exceeded or is expected to exceed a safety threshold, the system can proactively push early warning information to the maintenance platform, vehicle terminal, and roadside devices in real time, thereby constructing a closed loop of the entire process of "collection-positioning-detection-prediction-early warning". This significantly improves the timeliness and foresight of road maintenance, ensures driving safety, and reduces maintenance costs.

[0042] This application provides an intelligent inspection and early warning method for road defects. Figure 1 This is an optional flowchart illustrating an intelligent road defect inspection and early warning method provided in an embodiment of this application, which will be combined with... Figure 1 The steps shown are explained.

[0043] S101. During the inspection vehicle's operation, acquire road images and lidar point cloud data; and perform timestamp alignment and filtering on the road images and lidar point cloud data respectively to determine the image sequence and point cloud sequence.

[0044] In some embodiments of this application, the intelligent road defect inspection and early warning method is applicable to scenarios involving intelligent road defect inspection.

[0045] In some embodiments of this application, the implementing entity of the intelligent road defect inspection and early warning method is an intelligent road defect inspection and early warning device. The intelligent road defect inspection and early warning device can be a terminal device.

[0046] In some embodiments of this application, before conducting intelligent inspection and early warning of road defects, the inspection vehicle first starts the inspection system, initializes the high-definition camera, lidar sensor and parameters of each module, and establishes a blank defect file database.

[0047] In some embodiments of this application, during the operation of the inspection vehicle, road images and lidar point cloud data are collected by a high-definition camera and a lidar sensor. The road images and lidar point cloud data are then processed by timestamp alignment and filtering to determine the image sequence and point cloud sequence.

[0048] For example, during the inspection vehicle's operation, high-definition cameras and LiDAR continuously acquire road surface images and corresponding LiDAR point cloud data streams. The adaptive perception module monitors environmental and operating condition parameters in real time, optimizing the acquisition process and performing data preprocessing: for example, adjusting the camera exposure time T based on lighting conditions, such as turning on / adjusting supplementary lighting or HDR mode. e If according to the formula Dynamic adjustment Where L is the current ambient light intensity, L0 is the reference light intensity, To prevent extremely small values ​​from being divided by zero and to ensure image clarity, the lidar frame rate f is adjusted according to the vehicle speed v, for example, to satisfy... (Where d0 is the predetermined spatial resolution distance) to ensure sufficient overlap between adjacent point clouds. Multiple data streams are timestamped and filtered to remove sensor noise, resulting in high-quality and synchronized image and point cloud sequences.

[0049] S102. Based on image sequences and point cloud sequences, perform visual and laser fusion processing to determine the position, attitude, and latest road and defect maps of the inspection vehicle.

[0050] In some embodiments of this application, feature extraction is performed based on image sequences using visual processing techniques to determine road surface texture information; wherein, the road surface texture information includes road surface texture feature points or feature lines; feature extraction is performed based on point cloud sequences using laser processing techniques to determine terrain feature information; wherein, the terrain feature information includes: terrain contours or planar features; based on the road surface texture information and terrain feature information, feature matching, point cloud matching, and fusion positioning are performed to determine the position and attitude of the inspection vehicle; based on the attitude of the inspection vehicle, the map is updated to determine the latest road and defect map.

[0051] In some embodiments of this application, based on road surface texture information, feature matching is performed using visual odometry to determine the displacement information of the inspection vehicle; wherein, the displacement information is the movement displacement of the inspection vehicle relative to the road surface; based on terrain feature information, point cloud matching is performed using laser odometry to determine the pose change information of the inspection vehicle; based on the displacement information and pose change information, fusion positioning is performed using fusion odometry to determine the position and attitude of the inspection vehicle.

[0052] For example, the preprocessed synchronous data is input into the SLAM map building module to execute the vision-laser fusion localization and map update algorithm. Specifically, the visual odometry method infers the vehicle's motion based on road features matched from adjacent image frames, the laser odometry method calculates the pose change based on the spatial registration of adjacent point cloud frames, and the fusion odometry method obtains a high-precision vehicle pose T by combining visual and laser estimation through Kalman filtering or nonlinear optimization. k Then, the point cloud data acquired in the current pose is transformed and overlaid into the global map coordinate system to update the road environment map. If a loop closure is detected (e.g., by identifying similar features of previous road segments), global optimization is performed to reduce accumulated errors. This step outputs the current vehicle's precise location and the latest updated road and defect map.

[0053] S103. Based on the latest road and disease maps, image sequences, and point cloud sequences, perform pavement disease target detection to determine pavement disease information; among which, pavement disease information includes the type, location, and characteristic parameters of the disease targets.

[0054] In some embodiments of this application, a road surface defect target detection is performed on an image sequence using a preset image processing algorithm or a trained deep learning model to determine the target image region; wherein, the target image region is an image region suspected of being a defect; ground contour analysis is performed on a point cloud sequence to determine abnormal ground regions; wherein, abnormal ground regions are regions with abnormal height or abrupt changes in shape; spatial matching and result fusion are performed on the target image region and the abnormal ground region to determine the defect target; based on the latest road and defect map, the type of defect target, the location of the defect target, and the characteristic parameters of the defect target are determined.

[0055] For example, while updating the road and defect maps, the defect detection module analyzes the currently acquired image and point cloud data to identify new defects or update information on existing defects. Pre-set algorithms or models are applied to the image data to detect anomalies: for cracks, image grayscale thresholding or deep learning semantic segmentation is used to extract connected regions of crack pixels; for potholes, shadow contours or pothole shapes are detected in the image. Next, the suspected defect areas in the image coordinate system are mapped to the point cloud data using camera-LiDAR extrinsic parameters, and a corresponding 3D point cloud subset of that area is extracted to calculate its height difference features, volume features, etc. For example, for a suspected pothole area, the plane equation ax + by + cz + d = 0 of the surrounding normal road surface is fitted, and the distance from each point in the area to this plane is calculated. If the distance value of a continuous region exceeds a threshold, it is determined to be an actual pit, and its maximum depth and range are calculated. For cracks, the existence of cracks is confirmed by combining the abrupt change in the reflection intensity of the region in the point cloud, and the crack depth is estimated (if there is slight subsidence). Each disease target obtained by disease detection and identification is updated in the system: if it is a newly discovered disease for the first time, it is recorded in the database; if it is a previously existing disease, its latest feature parameter value y is updated. k .

[0056] S104. Based on the acquired historical inspection data, perform time-series modeling and status updates on the disease targets to determine the current severity and rate of change of the disease targets; and based on the rate of change, predict the development trend to determine the future development trend of the disease targets.

[0057] In some embodiments of this application, based on the acquired historical inspection data, time-series modeling of the disease target is performed to determine the model parameters; based on the model parameters, the state of the disease target is updated through a prediction calibration algorithm to determine the current severity and rate of change of the disease target.

[0058] For example, the model parameters for the corresponding disease are updated based on the data obtained from the current inspection (i.e., historical inspection data). For each recorded disease, the latest feature observation value output from the previous step is used as z. k The state estimate is updated using a prediction-correction algorithm. For example, using Kalman filtering: first, a prediction is made based on the previous state (obtaining the predicted severity), and then the new observation z is combined. k The correction is then obtained. Simultaneously, the new observations are appended to the historical sequence of the disease for offline analysis or adaptive model adjustment. If the disappearance of a disease is detected (e.g., cracks are repaired and not detected again for several consecutive times), its state can be marked as terminated in the model.

[0059] S105. If the current severity and future development trend meet the preset conditions, generate early warning information for the road defects; and send the early warning information to external terminals to realize intelligent inspection and early warning of road defects.

[0060] In some embodiments of this application, if the current severity level is greater than a preset threshold, an early warning message for the disease target is generated; or, based on future development trends, the remaining time to reach the preset threshold is determined, and if the remaining time is less than the early warning lead time, an early warning message for the disease target is generated.

[0061] For example, the status of all diseases updated in each inspection cycle is evaluated, and a pre-set rule is used to determine whether an early warning needs to be triggered. For each disease, its current severity y is compared. k With safety threshold Y crit If y k >Y crit If it is, it is marked as requiring immediate warning. Furthermore, its development trend is assessed based on the evolutionary model, and the remaining time t expected to reach the threshold is calculated. c If t c Shorter than the early warning lead time T warnIf a condition is detected, a trend warning is triggered. The system can set different thresholds and lead times for different types of defects. For example, a warning is triggered if the width of a bridge deck crack exceeds 5mm, or if the depth of a pothole reaches 3cm. Alternatively, an early warning is triggered if the crack is expected to expand to 5mm within a week, prompting maintenance. For defects that do not meet any warning conditions, the system only records the information without triggering an alarm, and the process proceeds to the next step. The warning determination process is performed periodically after each inspection or during the inspection to promptly capture changes in risk. When a defect triggers a warning condition, the system generates a warning notification containing detailed information. This notification includes: the location of the defect (absolute coordinates or road references can be used, such as distance from a road sign), type, current status parameters, predicted development, and recommended measures. The warning information is sent to a designated receiving terminal via the communication module. For example, it can be sent to the server of the highway maintenance management center so that relevant personnel can dispatch maintenance as soon as possible; if it is intended for road users, it can be sent to the road monitoring system to issue speed limits or warning information. For real-time vehicle warnings, a warning can be displayed on the driver's cab screen or an audio alert can be made to the driver that the road conditions ahead are poor. If the inspection vehicle is an unmanned patrol vehicle, the warning results can also be uploaded to the cloud and displayed on variable message signs along the road to remind other vehicles to give way. After the warning is issued, the next cycle of the inspection process begins.

[0062] Understandably, by performing timestamp alignment and filtering on road images and LiDAR point cloud data respectively, the accuracy of image and point cloud sequences can be improved. Visual and LiDAR fusion processing of image and point cloud sequences determines the position, attitude, and latest road and defect maps of the inspection vehicle, enabling centimeter-level positioning and real-time 3D map updates in GNSS-free scenarios. Based on the latest road and defect maps, image sequences, and point cloud sequences, road defect target detection is performed to determine road defect information. This process considers both texture and shape, reducing missed and false detections, thereby improving the accuracy and reliability of intelligent road defect inspection. Based on acquired historical inspection data, time-series modeling, state updates, and development trend prediction are performed on defect targets to determine their future development trends. If the current severity and future development trend meet preset conditions exceeding preset thresholds, a warning message for the defect target is generated and sent to an external terminal, realizing intelligent road defect inspection and early warning. This improves road safety while enhancing the accuracy and reliability of intelligent road defect inspection.

[0063] In some embodiments of this application, S102 can be implemented by S201-S204, as follows:

[0064] S201. Based on the image sequence, feature extraction is performed using visual processing techniques to determine the road surface texture information; wherein, the road surface texture information includes road surface texture feature points or feature lines.

[0065] S202. Based on the point cloud sequence, feature extraction is performed using laser processing technology to determine terrain feature information; wherein, terrain feature information includes: terrain outline or planar features.

[0066] S203. Based on road surface texture information and terrain feature information, feature matching, point cloud matching and fusion positioning are performed to determine the position and attitude of the inspection vehicle.

[0067] S204. Based on the posture of the inspection vehicle, update the map to determine the latest road and defect map.

[0068] For example, using preprocessed image and point cloud data, environmental features are extracted and the pose of the inspection vehicle is estimated to dynamically construct a map of the road environment and road defects. On one hand, visual SLAM technology is used to extract road texture feature points or feature lines (such as crack edges, road markings, etc.) from the image sequence, and the vehicle's motion displacement relative to the road surface is calculated through feature matching. On the other hand, laser SLAM technology is used to extract terrain contours or planar features (such as flat areas of the road surface, protrusion contours, etc.) from continuous point clouds, and the vehicle's pose change is calculated through point cloud matching. The SLAM map construction module includes a visual odometry unit and a laser odometry unit, which fuses the information from visual odometry and laser odometry. For example, by using an extended Kalman filter (EKF) or factor graph optimization method, visual feature observation and point cloud scanning matching are unified for calculation to obtain a more accurate and robust pose estimation.

[0069] In some embodiments of this application, S203 can be implemented by S2031, S2032, and S2033, as follows:

[0070] S2031. Based on road surface texture information, feature matching is performed using the visual odometer method to determine the displacement information of the inspection vehicle; wherein, the displacement information is the movement displacement of the inspection vehicle relative to the road surface.

[0071] S2032. Based on terrain feature information, point cloud matching is performed using laser odometry to determine the pose change information of the inspection vehicle.

[0072] S2033. Based on displacement information and pose change information, fusion positioning is performed using the fusion odometer calculation method to determine the position and attitude of the inspection vehicle.

[0073] In some embodiments of this application, S103 can be implemented by S1031-S1034, as follows:

[0074] S1031. Using a preset image processing algorithm or a trained deep learning model, perform road surface defect target detection on the image sequence to determine the target image region; wherein, the target image region is the image region suspected of being a defect.

[0075] S1032. Perform ground contour analysis on the point cloud sequence to identify abnormal ground areas; where abnormal ground areas are areas with height anomalies or abrupt changes in morphology.

[0076] S1033. Perform spatial matching and result fusion on the target image area and abnormal ground area to determine the disease target.

[0077] S1034. Based on the latest road and disease maps, determine the type of disease target, the location of the disease target, and the characteristic parameters of the target disease.

[0078] For example, based on images (i.e., image sequences) and point cloud data (i.e., point cloud sequences), as well as pose and map information provided by the SLAM module (i.e., the location and attitude of the inspection vehicle, the latest road and damage maps), a method combining image processing and 3D analysis is used to detect abnormal areas on the road surface. First, the image recognition submodule uses algorithms to detect suspected damage areas in the images. For example, edge detection and morphological methods are used to extract thin cracks, threshold segmentation is used to identify pothole shadow areas, or a pre-trained deep learning model (such as a convolutional neural network) is used to identify different types of road damage. Second, the suspected damage areas identified in the images are mapped to the LiDAR point cloud, and the 3D height change information at the corresponding location is obtained through coordinate transformation, thereby verifying whether there are real geometric depressions or protrusions in the area. For example, for suspected pothole locations, the height difference of the point cloud in that area relative to the surrounding normal road surface is calculated; if the negative height deviation exceeds a threshold, it is confirmed as a pothole. For cracks, changes in point cloud density or surface reflectivity can be analyzed as auxiliary criteria. By combining visual and point cloud information, misjudgments from a single information source can be effectively avoided, such as distinguishing shadows from actual potholes. The disease detection module outputs a list of disease targets, which includes the location coordinates of each identified disease (which can be represented by SLAM map coordinates or geographic coordinates), type (e.g., cracks, potholes, subsidence, etc.), and preliminary quantified feature parameters (e.g., crack length and width, pothole area and depth, etc.).

[0079] In some embodiments of this application, S104 can be implemented by S1041 and S1042, as follows:

[0080] S1041. Based on the acquired historical inspection data, perform time-series modeling of the disease targets and determine the model parameters.

[0081] S1042. Based on the model parameters, the status of the disease target is updated through a prediction calibration algorithm to determine the current severity and rate of change of the disease target.

[0082] In some embodiments of this application, S105 can be implemented by S1051 and S1052, as follows:

[0083] S1051. If the current severity or future development trend exceeds the preset threshold, an early warning message for the disease target will be generated.

[0084] In some embodiments of this application, if the current severity level is greater than a preset threshold, an early warning message for the disease target is generated; or, based on future development trends, the remaining time to reach the preset threshold is determined, and if the remaining time is less than the early warning lead time, an early warning message for the disease target is generated.

[0085] S1052. Send the early warning information to the external terminal to realize intelligent inspection and early warning of road defects.

[0086] For example, the warning conditions are determined based on the current state of the disease (i.e., the current severity) and the predicted future development trend. When the severity of the disease exceeds a preset threshold or is expected to exceed the preset threshold within a predetermined time, a warning message is generated and sent to an external terminal.

[0087] In some embodiments of this application, the method further includes:

[0088] Store road surface defect information in the defect archive;

[0089] Continue to conduct intelligent road defect inspections and obtain the latest road surface defect information;

[0090] The damage records are updated based on the latest road surface damage information.

[0091] In some embodiments of this application, inspections are repeatedly performed according to preset routes or time intervals to achieve continuous monitoring and dynamic early warning of road defects. The system can increase the inspection frequency for key road sections as needed and promptly review and verify newly detected defects. Through multiple cycles, the map and defect database are gradually improved, enhancing the accuracy of model predictions. The entire inspection process can be performed periodically without interrupting traffic, forming a long-term road health monitoring mechanism.

[0092] This application also provides a method for intelligent inspection and early warning of road defects, which specifically includes the following steps:

[0093] S1. Start the inspection system, initialize the high-definition camera, lidar sensor and parameters of each module, and establish a blank disease record database.

[0094] This includes initializing the parameters and status of each module, including loading the startup and calibration parameters of sensors such as high-definition cameras and lidar, initializing the SLAM algorithm (e.g., setting the initial pose and uncertainty), initializing the disease record database (reading historical records or creating a blank list), and detecting the connection of the communication module, to confirm that each sensor is working properly and that the time synchronization mechanism is effective.

[0095] S2. During the inspection vehicle's operation, synchronous road images and lidar point cloud data are collected, the environment and driving status are detected, sensor settings are adaptively adjusted, and multi-source data are synchronized and filtered to obtain optimized image sequences and point cloud sequences.

[0096] S3. Perform visual-laser fusion SLAM processing on image sequences and point cloud sequences, continuously calculate the position and attitude of the inspection vehicle, and update the map information of road environment and defects in real time.

[0097] The preprocessed synchronous data is input into the SLAM map building module to execute the vision-laser fusion localization and map update algorithm. Specifically, the visual odometry method infers the vehicle's motion based on road features matched from adjacent image frames, the laser odometry method calculates pose changes based on spatial registration of adjacent point cloud frames, and the fusion odometry method obtains a high-precision vehicle pose T by combining visual and laser estimation through Kalman filtering or nonlinear optimization. k Then, the point cloud data acquired in the current pose is transformed and overlaid into the global map coordinate system to update the road environment map. If a loop closure is detected (e.g., by identifying similar features of previous road segments), global optimization is performed to reduce accumulated errors. This step outputs the current vehicle's precise location and the latest updated road and defect map.

[0098] S4. Using the latest acquired images and point cloud data, detect and identify road surface defects, including locating the defects in the map, determining the type of defects and quantifying their feature dimensions, and recording the identification results as defects file entries.

[0099] S5. Associate the currently identified disease characteristic data with historical data, perform status estimation and update for each disease, obtain its current severity and rate of change, and predict its future development trend based on the rate of change.

[0100] The disease evolution modeling module is invoked to update the model parameters for the corresponding disease based on the data obtained from the current inspection. For each recorded disease, the latest feature observation value output from the previous step is used as z. k The state estimate is updated using a prediction-correction algorithm. For example, when applying Kalman filtering: first predict based on the previous state. (Predicted severity) ), and then combined with the new observation z k Correction obtained Simultaneously, the latest observations are appended to the historical sequence of the disease for offline analysis or adaptive adjustment of the model. If the disappearance of a disease is detected (e.g., cracks are repaired and not detected again for several consecutive times), its state can be marked as terminated in the model.

[0101] S6. Compare the current severity and predicted future development trend with the preset threshold conditions to determine whether to trigger an early warning. When it is found that the current severity of the disease has exceeded the safety threshold, or is expected to exceed the safety threshold within a predetermined time, mark the disease as requiring an early warning.

[0102] The proactive early warning module assesses the status of all diseases updated in each inspection cycle and determines whether an early warning needs to be triggered according to pre-set rules. For each disease, it compares its current severity y. k With safety threshold Y crit If y k >Y crit If it is, it is marked as requiring immediate warning. Furthermore, its development trend is assessed based on the evolutionary model, and the remaining time t expected to reach the threshold is calculated. c If t c Shorter than the early warning lead time T warn If a condition is detected, a trend warning will be triggered. The system can set different thresholds and lead times for different types of defects. For example, a warning will be issued if the width of a bridge deck crack exceeds 5mm, if the depth of a pothole reaches 3cm, or if a crack is expected to expand to 5mm within a week, an early warning will be issued to prompt maintenance. For defects that do not meet any warning conditions, the system will only record the issue without triggering an alarm, and the process will proceed to the next step. The warning determination process is conducted periodically after each inspection or during the inspection to promptly capture changes in risk.

[0103] S7. For diseases marked as requiring warning, generate warning information including disease location, type, current status and treatment suggestions, and send the warning information to the remote maintenance management platform or nearby vehicle and road warning devices via wireless communication.

[0104] When a road defect triggers the warning conditions, the system generates a warning notification containing detailed information. This notification includes: the location of the defect (using absolute coordinates or road references, such as distance from a road sign), type, current status parameters, predicted development, and recommended measures. The warning information is then sent to a designated receiving terminal via the communication module. For example, it can be sent to the server of the highway maintenance management center so that relevant personnel can dispatch repairs as soon as possible; if intended for road users, it can be sent to the road monitoring system to issue speed limits or warnings. For real-time vehicle-mounted warnings, a pop-up warning or audible alert can be displayed on the driver's cab screen to indicate poor road conditions ahead. If the inspection vehicle is an unmanned inspection vehicle, the warning results can also be uploaded to the cloud and displayed on variable message signs along the road to remind other vehicles to take evasive action. After issuing the warning, the next cycle of the inspection process begins.

[0105] S8. Repeat steps S2 to S7 to continuously inspect the road. In subsequent inspections, update existing defect files, add new defect files, and mark the status of repaired or disappeared defects to remove them, thereby forming a closed-loop mechanism for road defect monitoring and early warning.

[0106] The above steps form a complete inspection cycle, which can be repeatedly executed according to preset inspection routes or time intervals to achieve continuous monitoring and dynamic early warning of road defects. The system can increase the inspection frequency for key road sections as needed and promptly review and verify newly detected defects. Through multiple cycles, the map and defect database are gradually improved, enhancing the accuracy of model predictions. The entire inspection process can be executed periodically without interrupting traffic, forming a long-term road health monitoring mechanism.

[0107] This application provides an intelligent road defect inspection and early warning system, such as... Figure 2 As shown, the system includes: at least one high-definition camera and one lidar for acquiring image data and 3D point cloud data of the road surface. If necessary, it may also include an inertial measurement unit (IMU) and a GNSS (Global Positioning System) module. The high-definition camera acquires image information of the road surface, and the lidar scans and acquires 3D point cloud data of the road surface. The installation positions and attitudes of the high-definition camera and lidar are calibrated to ensure that their coordinate systems are aligned with the same spatial reference coordinate system. Optionally, the IMU and GNSS provide auxiliary pose estimation and global positioning information. Through the above settings, image frames and corresponding timestamped point cloud frame data can be simultaneously acquired during vehicle operation.

[0108] The central processing unit connects to the high-definition camera and LiDAR, and runs the following modules:

[0109] The adaptive perception module is used to receive image data and point cloud data and perform preprocessing, including monitoring ambient light and vehicle status, adjusting the exposure of the high-definition camera and the scanning frequency parameters of the LiDAR in real time based on the monitoring results, and outputting high-quality images and point clouds that are aligned and synchronized.

[0110] The adaptive perception module, through intelligent control algorithms of the environmental perception unit and sensor perception unit, adaptively adjusts sensor operating parameters based on external conditions and driving status, thereby ensuring data quality. For example, it detects the current light intensity to adjust camera exposure time and gain, ensuring clear images are acquired even in high-contrast or nighttime environments; it monitors vehicle speed to dynamically adjust the LiDAR scanning frequency and point cloud sampling density, ensuring sufficiently dense ground point clouds are obtained even at high speeds. The adaptive perception module also includes a data synchronization and filtering unit, which performs time alignment and noise reduction on data from different sensors. For instance, it uses timestamps to synchronize image frames and point cloud frames, and employs filtering algorithms to remove image noise and outliers from the point cloud, improving the robustness of subsequent processing. After processing by this module, it outputs corrected and optimized image and point cloud data, providing high-quality input for subsequent SLAM mapping and defect detection.

[0111] The SLAM map building module is used to perform simultaneous localization and map building by fusing vision and LiDAR based on preprocessed image data and point cloud data, and to estimate the position and attitude of the inspection vehicle in real time and generate a map of the road environment.

[0112] This module utilizes preprocessed image and point cloud data to extract environmental features and estimate the pose of the inspection vehicle, dynamically constructing a map of the road environment and road defects. On one hand, visual SLAM technology is used to extract road texture feature points or lines (such as crack edges and road markings) from image sequences, and the vehicle's displacement relative to the road surface is calculated through feature matching. On the other hand, laser SLAM technology is used to extract terrain contours or planar features (such as flat road areas and protrusion contours) from continuous point clouds, and the vehicle's pose change is calculated through point cloud matching. The SLAM map construction module includes a visual odometry unit and a laser odometry unit, fusing information from visual and laser odometry. For example, extended Kalman filters (EKF) or factor graph optimization methods are used to unify visual feature observations and point cloud scanning matching for more accurate and robust pose estimation. The fused positioning estimation formula is as follows:

[0113]

[0114] in, This represents the state estimate of the fused localization at time k, including vehicle pose and other state variables. This indicates a combination of observations from high-definition cameras and lidar. For predictive observation models, The filter gain matrix is ​​calculated considering the uncertainties of both visual and laser observations. Through the aforementioned fusion positioning estimation, the precise position and attitude of the inspection vehicle in the map coordinate system can be obtained in real time. Simultaneously, the SLAM map building module uses the estimated pose to accumulate the point clouds collected at each time step into a unified coordinate system, generating a 3D map representation of the road and its defects. For example, the vehicle coordinate pose Tk is used for coordinate transformation, and the i-th point cloud collected by the laser radar at time k is... Transform to the initial reference coordinate system: The accumulated 3D point cloud map can be refined into an elevation grid map or digital elevation model of the road surface to display subtle undulations and defect locations. This module can also include a loop closure detection submodule, which identifies previously constructed map features and performs closed-loop optimization when inspection vehicles repeatedly pass through the same road segment, correcting accumulated errors and continuously improving map accuracy.

[0115] The road damage detection module analyzes images and point clouds to identify road damage targets and outputs the type, location, and characteristic parameters of the damage. The module includes:

[0116] The image recognition submodule, based on a preset image processing algorithm or a trained deep learning model, detects image regions suspected of being diseased from image data;

[0117] The point cloud analysis submodule performs ground contour analysis on point cloud data to extract areas with height anomalies or abrupt changes in shape.

[0118] The decision fusion submodule performs spatial matching and result fusion on the suspected disease information obtained from the image recognition submodule and the point cloud analysis submodule, eliminates false alarms, and outputs the confirmed disease targets along with their corresponding spatial coordinates and size parameters.

[0119] This module receives image and point cloud data output from the adaptive perception module, as well as pose and map information provided by the SLAM module. It employs a combination of image processing and 3D analysis to detect abnormal areas on the road surface. First, the image recognition submodule uses algorithms to detect suspected damage areas in the image. For example, it uses edge detection and morphological methods to extract thin cracks, uses threshold segmentation to identify pothole shadow areas, or uses pre-trained deep learning models (such as convolutional neural networks) to identify different types of road damage. Second, the suspected damage areas identified in the image are mapped to the LiDAR point cloud. Coordinate transformation is used to obtain the 3D height change information at the corresponding location, thereby verifying whether there are actual geometric depressions or protrusions in the area. For example, for suspected pothole locations, the height difference of the point cloud in that area relative to the surrounding normal road surface is calculated. If the negative height deviation exceeds a threshold, it is confirmed as a pothole. For cracks, changes in point cloud density or surface reflectivity can be analyzed as auxiliary criteria. By combining visual and point cloud information, misjudgments from a single information source can be effectively avoided, such as distinguishing shadows from actual potholes. The disease detection module outputs a list of disease targets, which includes the location coordinates of each identified disease (which can be represented by SLAM map coordinates or geographic coordinates), type (e.g., cracks, pits, subsidence, etc.), and preliminary quantified feature parameters (e.g., crack length and width, pit area and depth, etc.).

[0120] The disease evolution modeling module is used to perform time-series modeling and status updates of disease targets, estimate the severity and rate of change of diseases based on data from previous inspections, and predict their future development trends.

[0121] Once each disease is identified, a record is created in the system and updated with data from subsequent inspections. This module models the evolution of the disease using a time-series modeling algorithm: it organizes the disease characteristic data obtained from a single inspection into a sequence according to time order to analyze its changing patterns. This invention preferably employs a state-space-based time-series model to recursively estimate and predict the severity and development rate of the disease. The state of a particular disease is defined as a vector.

[0122]

[0123] in Indicates the severity of the disease. The linear state-space model represents the rate of disease change.

[0124]

[0125] in Here is the state transition matrix. The time interval between two consecutive inspections. For process noise, Represents the observation matrix. Representing observation noise, the state components satisfy a recurrence relation: .

[0126] This model allows the use of Kalman filtering or other estimation algorithms to estimate new observations z obtained from each inspection. k Recursively estimate the current disease status s k (Includes filtered disease severity y) k Estimated value and disease change rate v k It can predict the state at future moments. For example, it can predict the state based on an estimated current state. Severity of disease over time ,in Indicates the future Predict severity in real time. Indicates the predicted time interval. This represents an estimate of the acceleration of disease evolution. If it is necessary to consider the scenario of accelerated disease expansion, it can be extended to a second-order model (where the state includes an acceleration term) or external influencing factors (such as traffic flow, temperature, etc.) can be introduced as part of the state to perform more complex modeling of the evolution. The disease evolution modeling module also maintains a disease archive database, recording the historical detection data, model parameters, and prediction results for each disease, in order to analyze its evolution trend longitudinally.

[0127] The proactive early warning module is used to determine early warning conditions based on the current state of the disease and the predicted future development trend. When the severity of the disease exceeds a preset threshold or is expected to exceed the preset threshold within a predetermined time, it generates early warning information and sends it to an external terminal.

[0128] The module's judgment unit is configured with several early warning trigger conditions, including two types: threshold exceedance early warning and trend prediction early warning. First, if the measured parameters of a disease exceed a safety threshold during the latest inspection (e.g., pit depth exceeds a certain value, crack density exceeds specification limits), an early warning is immediately generated. Second, based on the evolution model's prediction results, if the severity of a disease is expected to develop to a dangerous level within a predetermined time ΔT (e.g., within the next week or several days), an early warning is generated in advance. The proactive early warning module acquires the current severity y of each disease. k and the rate of change of disease v k Combined with the preset warning threshold Y crit and time threshold T warn Make judgments, such as calculating the expected failure time. If t c <T warnThis triggers a warning signal. The information generation unit generates a warning signal, which can take the form of: sending a fault alarm to the remote maintenance management center (including fault location, type, current status, and expected trend) so that maintenance personnel can arrange repairs in a timely manner; for the inspection vehicle itself, it can provide a warning via onboard display or voice prompts about serious faults ahead; if this system is integrated into an autonomous vehicle or road vehicle-to-infrastructure (V2I) system, it can send the warning information to nearby vehicles or roadside units, triggering safety strategies such as speed limits and detours. The active warning module also includes an information communication unit that transmits warning data externally via a wireless network (such as 4G / 5G communication or dedicated short-range communication DSRC) to ensure the real-time nature and wide coverage of the warning.

[0129] The road defect inspection and early warning system provided in this embodiment mainly includes hardware units on a vehicle platform and corresponding software function modules.

[0130] The inspection vehicle platform is equipped with high-definition cameras and LiDAR, which are connected to the onboard industrial control computer (central processing unit) via a data bus. The high-definition cameras are industrial cameras with at least 2 megapixels, equipped with a global shutter to avoid motion distortion, and the lens field of view covers the entire lane width. The LiDAR is preferably a 360° rotating scanning 3D LiDAR (such as a 16-line or 32-line LiDAR), and its installation angle can be adjusted to fully cover the road surface in front of the vehicle. The vehicle is also equipped with an IMU (Inertial Measurement Unit) and a GNSS antenna to provide auxiliary positioning information and time synchronization. The central processing unit is equipped with a high-performance CPU and GPU to run SLAM algorithms and deep learning models, ensuring real-time processing performance.

[0131] like Figure 3 The diagram shows the sensor layout of the inspection vehicle. A 360° lidar is installed on the cylindrical roof, high-definition cameras are installed at the front and rear of the vehicle, and an IMU / GNSS is installed inside the vehicle.

[0132] In terms of software architecture, the central processing unit deploys the following functional modules:

[0133] The adaptive perception module includes a light perception submodule (reading ambient light sensor data or average image brightness to determine lighting conditions), a camera control submodule (calling the camera SDK interface to adjust parameters such as exposure, gain, and frame rate), a lidar control submodule (adjusting lidar rotation frequency, point cloud filtering resolution, etc.), and a data synchronization and filtering submodule (performing time synchronization, distortion correction, and noise reduction on the raw data streams from the camera and lidar). In this embodiment, the light perception submodule calculates the brightness histogram of the most recent image frame every 0.5 seconds and determines whether the current image is too dark or too bright based on a preset threshold. If it is too dark, the camera control submodule is triggered to increase the exposure time or turn on the LED fill light; if it is too bright, the exposure is reduced to prevent overexposure. Similarly, when the vehicle speed is detected to increase to over 80 km / h, the lidar control submodule increases the lidar scanning frequency from 10 Hz to 20 Hz and reduces the voxel size of the point cloud filter to ensure sampling density. After processing by this module, the corrected image frame and the time-matched point cloud data frame are output.

[0134] The SLAM map building module includes a visual odometry submodule, a laser odometry submodule, and a graph optimization fusion submodule. The visual odometry submodule uses an ORB feature extraction and matching algorithm to estimate the relative pose change of the camera from adjacent image frames, obtaining the visual odometry increment ΔTv. The laser odometry submodule uses a point cloud registration algorithm (such as NDT or ICP) to calculate the optimal rigid body transformation ΔTl between consecutive point cloud frames, obtaining the laser odometry increment. The graph optimization fusion submodule constructs the accumulated visual and laser constraints over a period of time into a graph optimization problem. Within a local keyframe window, it solves for the optimal pose of each keyframe by minimizing reprojection error and point cloud matching error, thus fusing visual and laser information. To improve robustness, this embodiment also utilizes short-term pose changes provided by the IMU as priors to assist the constraint optimization process. Finally, the SLAM map building module outputs the vehicle's current pose estimate T at a frequency of 10Hz. wb (Transformation matrix from world coordinate system to vehicle coordinate system), and an incrementally updated dense point cloud map M. The point cloud map M is maintained in raster storage format; each newly added frame of point cloud is processed through T... wb After projection, the data is incorporated into M, and the height values ​​of the corresponding grid cells are updated using a weighted average, thus forming a three-dimensional elevation map of the road surface that expands as the road travels.

[0135] The road defect detection module comprises an image detection submodule and a point cloud analysis submodule, with their results fused in the decision unit. The image detection submodule runs a lightweight convolutional neural network model on the GPU to identify road defect regions in images. During training, the model has been optimized to learn from samples of various defects such as cracks, potholes, and patches, enabling it to infer the pixel mask of the defect within 20ms on a single frame. In real-time operation, for each acquired image frame, the image detection submodule outputs a heatmap of the same size showing the probability of defects, such as... Figure 4 As shown, the color bars represent confidence levels, ranging from 0 to 1. Simultaneously, the point cloud analysis submodule acquires synchronized laser point cloud frames and performs rapid geometric analysis on the CPU: for example, projecting the point cloud onto the ground plane to generate a height difference matrix, detecting low-lying connected regions as pit candidates; differentiating the height matrix to obtain slope changes, detecting sharply descending linear features as crack / edge candidates. Figure 5 The image shows a point cloud height difference heatmap. The gray-blue to yellow gradient maps the height difference ΔHeight. The central blue depression is approximately -0.04 m, while the surrounding yellow high areas represent normal road surfaces. The image demonstrates the three-dimensional quantitative representation of potholes in the point cloud: the depressed areas show significantly lower values. Figure 6 The image shows a partial cross-sectional view of the 3D point cloud, revealing the true concave morphology of the point cloud, with the central depression significantly lower than the surrounding area. The decision unit spatially aligns and jointly judges candidate results from the image and point cloud: for anomalies at the same location simultaneously marked as abnormal in both the image and point cloud, the confidence level is increased and it is confirmed as a genuine disease; if detected only on one side of the image or point cloud, its feature intensity determines whether it should be considered a disease or await confirmation in the next frame. For example, if a shaded area is highly likely to be identified as a pit in the image, but no obvious height anomaly is observed in the point cloud, it is judged as a false alarm and not recorded as a disease. The output of the disease detection module is a structured list of disease information. Each record includes: disease ID, spatial location (represented using SLAM map coordinates), type (crack / pit / …), size parameters (such as length, width, depth), and confidence level. This information is sent to the disease evolution modeling module and used to mark disease locations on map M (for easy visualization).

[0136] The disease evolution modeling module includes a disease data management submodule, a state estimation submodule, and a trend prediction submodule. The data management submodule maintains a list of disease records, each recording all historical observation data and timestamps for the disease since its initial discovery. When a new disease list is received from the disease detection module, the module first compares the disease location to determine if it's an updated observation from an existing record: if the location is within a certain tolerance range and the type is consistent, it belongs to the same disease ID and a new observation is added; otherwise, a new disease record is created. Then, the state estimation submodule performs a Kalman filter update (according to the aforementioned state-space model) on each disease to obtain the latest state estimate for that disease. For example, for a new observation z k (If the current crack width is observed to be 4.5 mm), use the previous cycle to predict y k| Calculate the Kalman gain K using k-1 and the estimated uncertainty. k The update yields the corrected y. k (Filtering out the effects of single-observation noise) and updated disease change rate v k Next, the trend prediction submodule calculates the disease severity prediction for several future time points (such as t after 1 week or 1 month) based on the updated state estimate. And evaluate when the threshold Y is reached at time t. crit The probability. If the forecast indicates that the disease is likely to become severe within the set outlook period, a "Forecast Warning Pending" flag is set. All updated disease statuses and forecast results are stored in the disease archive list for future reference, such as... Figure 7 The image shown is a schematic diagram of the disease database interface.

[0137] like Figure 8 The figure shows the trend curve of disease severity. Figure 8 The X-axis represents time, the Y-axis represents crack width, the crosses indicate historical observations, the smooth curve represents the predicted trend, and the dashed line represents the disease severity threshold of 10 mm. The intersection of the curve and the threshold indicates the expected time t for exceeding the threshold. c .

[0138] The proactive early warning module includes a threshold judgment submodule, an early warning information generation submodule, and a communication and dissemination submodule. The threshold judgment submodule scans the current status y of all diseases. k and predicted state For any disease, an early warning is required if any of the following conditions are met: (a) The immediate threshold is exceeded: y k >Y crit (b) Prediction threshold exceeds limit: there exists t < T warn make >Y critIf the conditions are met, the system enters the early warning information generation submodule to generate an early warning message for the defect. The message includes: defect ID and location description (which can be converted to road name + station number or GPS coordinates), defect type, current status (e.g., "pothole current depth 5cm"), development trend (e.g., "at the current speed, it is expected to reach 10cm in 3 days"), and recommended measures (e.g., "Immediate repair recommended"). To facilitate reading by different users, the early warning message is categorized and classified, for example, using professional terminology for maintenance managers and concise prompts for drivers. The communication dissemination submodule is responsible for sending the message through appropriate channels: In this embodiment, if the inspection vehicle has a mobile internet connection, the detailed early warning is uploaded to the maintenance cloud platform via a 4G network, and simultaneously, short-range communication triggers electronic variable signs along the road to display "Uneven road surface ahead, slow down." If the vehicle is driven, warning lights illuminate in the driver's cab and a voice announcement is made. For defects that do not trigger an early warning, the threshold judgment submodule does not process them. The communication dissemination submodule can be integrated with existing traffic information systems to ensure that early warning messages reach relevant personnel and equipment in a timely manner.

[0139] This embodiment further elaborates on the workflow of the system described in Embodiment 1. Before the inspection task begins, the operator sets the inspection route or area in the system, as well as the types of diseases that need to be monitored and the corresponding threshold parameters. Then, the operation is carried out according to the following steps:

[0140] S11. Power on the system on the inspection vehicle. Each sensor begins collecting data but does not record it immediately. The system loads prior parameters (camera intrinsics, camera-radar extrinsic matrix, IMU noise parameters, initial SLAM map, etc.) and performs self-tests on the camera, radar, IMU, and communication modules to ensure proper connection. After the self-test passes, the system enters inspection mode.

[0141] S12. The vehicle travels along the predetermined route, and the system enters a loop process. The adaptive perception module continuously monitors the environment and adjusts when external conditions exceed the normal range. For example, when entering a tunnel and causing a sudden drop in light, the system automatically increases the camera exposure to twice its original value and turns on the supplementary lighting within one second to ensure sufficient image brightness; after exiting the tunnel and restoring lighting, the exposure is restored to normal to prevent overexposure. In highway scenarios, when the vehicle speed exceeds 100 km / h, to prevent missing minor defects, the system increases the camera frame rate from 30fps to 60fps to ensure that each meter of road surface is captured at least once. The LiDAR always operates at 20Hz to provide high-frequency point clouds, and all collected data frames enter the subsequent processing pipeline through shared memory.

[0142] The S13 SLAM module receives data streams from sensors and processes them in parallel using a multi-threaded approach: one thread handles visual odometry, another handles laser odometry, and the main thread fuses the two and maintains the global map. Whenever the visual thread obtains a new keyframe pose, the laser thread is immediately triggered to register and optimize the point cloud at the corresponding moment. In urban environments with numerous buildings, laser odometry may degrade (lacking planar features), but the rich texture features provided by visual odometry can maintain localization; conversely, in textureless road sections at night, laser point clouds ensure that localization does not diverge. Through the complementarity of the two odometry methods, the main thread ultimately obtains a stable localization output. Map updates take into account vehicle bumps and displacements; IMU data fusion makes point cloud projection more accurate, avoiding false fluctuations caused by vehicle suspension movement.

[0143] S14. As the vehicle moves forward, the system completes image + point cloud defect detection approximately every 50 milliseconds (corresponding to a detection frequency of 20Hz). When the vehicle passes a section of road with obvious cracks, the image detection submodule can draw the crack outline on the frame. Multiple frames accumulate to form a continuous crack shape, such as... Figure 9 The diagram shows a crack detection frame. Because the crack width is smaller than the point spacing, it may be difficult to detect directly using laser point clouds. However, by stitching together multiple point cloud frames, a slightly sunken strip-shaped area on the road surface can be observed. Based on this, the system determines that a longitudinal crack approximately 5m long has appeared. This crack is assigned ID=CRK001 and its initial width of approximately 3mm is recorded, with its location at map coordinates (125.0m, 3.5m) (relative to a reference point). If the crack extends further as the vehicle continues, the system will merge the subsequent portions into the same ID. For example, when a vehicle approaches a pothole, the laser radar point cloud first detects a sudden 4cm drop in road surface elevation and an area of ​​approximately 0.2㎡ about 30m ahead. Simultaneously, a black shadow shape appears in the camera image. Matching these two elements confirms it as a pothole, assigning it ID=PIT007, and recording a depth of 4cm and a diameter of 0.5m. This demonstrates that the system can detect and quantify new defects in a single pass.

[0144] S15. After each inspection route is completed (or at a set time interval, such as every 10 minutes), the system inputs all detected disease information into the evolution modeling module for batch updates. For the newly discovered CRK001 crack, which has no previous historical records, the data management submodule creates a new entry and stores its first observation: time t0, width y0 = 3mm. Since a single observation cannot estimate the trend, the initial state v0 can be set to 0, or a small value can be given based on experience. For the PIT007 pit, a new record is also created (depth 4cm). If there are already some old diseases in the disease database at this time (such as the previously existing crack CRK000, with a width of 5mm in the previous cycle and 5.5mm measured in this cycle), then for these existing entries, the state estimation submodule will update their state by combining the new and old observations (for example, the growth v of CRK000 is estimated to be +0.5mm / cycle), and all updated states and prediction results are stored for later use.

[0145] S16. The system performs an early warning judgment based on the updated list of disease statuses. Continuing the example above, assume the thresholds are set as follows: crack width threshold 10mm, pit depth threshold 50mm; lead time T warn =7 days. For the new crack CRK001, the current width is 3mm, far below 10mm, and it does not exceed the limit either currently or in the short term, so no warning will be triggered, and it will only be recorded normally. For the old crack CRK000, the current width of 5.5mm is also below the threshold, but its growth trend v≈0.5mm / cycle (if the inspection cycle is 7 days, then it is about 0.5mm / week), and it is estimated that it may reach 10mm in about 9 weeks. If the management department believes that 9 weeks is sufficient to arrange maintenance in a timely manner, then no warning is needed immediately; however, the system can issue a warning when the estimated remaining time is less than the warning lead time. For the pit PIT007, the depth of 4cm has not exceeded the 5cm threshold, so no warning will be issued for the time being. However, if a certain defect meets the conditions, such as another previously existing pit PIT005, which was previously 4.0cm and is now measured at 5.2cm, then it exceeds the threshold of 50mm, and the system will immediately generate a warning for this pit. For example, if a subsidence point SUB002 is measured to have a subsidence of 20mm in this instance, which is below the threshold of 30mm, but the model predicts that it may reach 35mm in the next inspection (about 7 days later), then the system will also generate an early warning for SUB002 because it will exceed the standard within the lead time. This step ensures that warnings are only issued for the defects that truly need attention, avoiding frequent alarms that may cause interference.

[0146] S17. For each road defect requiring a warning, the system organizes the warning content and sends it out via wireless network. In this embodiment, PIT005 triggers an immediate threshold warning, and the system generates the message: "Warning: A serious pothole (5.2cm deep) has been found in the right lane 2.3km ahead. Please slow down and repair it as soon as possible." This message is sent to the maintenance center backend via the 5G network and simultaneously notifies the inspection personnel through the vehicle-mounted equipment. If the inspection task is performed by an autonomous vehicle, the system will instruct the vehicle to stop in a safe area, and the backend will decide whether to dispatch a maintenance team. For the predictive warning SUB002, the system message may be: "Reminder: Road subsidence has occurred at K55+300. The subsidence is expected to exceed 3cm in one week. Please arrange preventive maintenance." This message is only sent to the maintenance management platform and is not released to the public. Through the above processing, relevant personnel can obtain information on the development of road defects in a timely manner and take corresponding measures. After the warning is sent, the cyclical inspection process ends or the next round of inspection is repeated.

[0147] S18. After receiving an early warning, maintenance personnel can dispatch personnel for inspection and repair. After the repair is completed, the status of the defect can be updated and marked through the system (for example, marking PIT005 as repaired in the background software). If the system does not detect the defect during the next inspection, the early warning loop is confirmed to be closed; if it is still detected, the repair may not be thorough or the information may be incorrect, and the system will continue to alarm. This feedback mechanism ensures the effectiveness of the early warning and closed-loop management.

[0148] This application integrates high-definition cameras, LiDAR, and IMUs onto inspection vehicles to construct a multi-source perception unit. Visual odometry and LiDAR are fused in a tightly coupled SLAM manner within a central processing platform, achieving centimeter-level autonomous positioning and real-time 3D road mapping. Leveraging this high-precision spatial reference, the system can deeply align image texture features with point cloud geometry, simultaneously quantifying key parameters such as crack width and pothole depth, significantly reducing missed and false detection rates, and assigning precise coordinates to each defect for convenient later maintenance. For scenarios with variable environments and fluctuating vehicle speeds, the adaptive perception module adjusts the high-definition camera exposure, LiDAR frame rate, and point cloud filtering density in real time, ensuring high signal-to-noise ratio data output even under nighttime, backlight, and high-speed driving conditions. Meanwhile, the defect detection module uses a dual-channel discrimination strategy to match and verify deep learning image segmentation results with point cloud anomaly areas, taking into account both texture cracks and three-dimensional potholes, achieving one-stop intelligent identification of multiple types of defects. Based on the detection results, the disease evolution modeling module continuously tracks the status of each disease, introduces an acceleration-adaptive time series model to estimate its development rate and predict the remaining safe life, and the proactive early warning module automatically pushes graded early warning information according to the dual judgment strategy of threshold and lead time. This "discovery-prediction-early warning" closed loop transforms maintenance from passive emergency repair to proactive intervention, which not only improves road safety but also significantly saves operation and maintenance costs.

[0149] In this embodiment, adaptive multi-source perception is employed: the system senses illumination and vehicle speed in real time and dynamically adjusts parameters such as exposure and laser scanning frequency to ensure high signal-to-noise ratio images and point clouds are acquired under various environments. Visual-LiDAR SLAM fusion is achieved through visual odometry, laser odometry, and IMU factor map optimization, enabling centimeter-level positioning and real-time 3D map updates in GNSS-free scenarios. Dual-domain disease identification is performed by jointly verifying the image heatmap generated by the convolutional network with the point cloud elevation difference threshold, taking into account both texture and shape to reduce missed and false detections. A disease state space model is used, with x = [y, v]. T A Kalman filter recursion is established to predict disease severity and rate of change online and adaptively correct the covariance. Dual-threshold proactive early warning: when y k ≥ Y crit or (Y) crit –y k ) / v k ≤ T warn Immediately push multi-level early warning information to the cloud, roads, or vehicle terminals. Disease record closed loop: The database continuously records observed, estimated, and predicted curves, and supports writing back maintenance results, forming a closed loop for the entire process of detection-decision-review.

[0150] Based on the intelligent road defect inspection and early warning method of the above embodiments, this application also provides an intelligent road defect inspection and early warning system, such as... Figure 10 As shown, Figure 10 This is a schematic diagram of the structure of a road defect intelligent inspection and early warning system provided in an embodiment of this application. The road defect intelligent inspection and early warning system 10 includes: a data acquisition module 1001, an adaptive perception module 1002, a map construction module 1003, a defect detection module 1004, a defect evolution modeling module 1005, and an active early warning module 1006.

[0151] Acquisition module 1001 is used to acquire road images and lidar point cloud data;

[0152] The adaptive perception module 1002 is used to acquire the road image and the lidar point cloud data during the inspection vehicle's operation; and to perform timestamp alignment processing and filtering processing on the road image and lidar point cloud data respectively to determine the image sequence and point cloud sequence.

[0153] The map building module 1003 is used to perform visual and laser fusion processing based on the image sequence and the point cloud sequence to determine the position, attitude and latest road and defect map of the inspection vehicle;

[0154] The road damage detection module 1004 is used to detect road damage targets and determine road damage information based on the latest road and damage map, the image sequence, and the point cloud sequence; wherein, the road damage information includes the type, location, and feature parameters of the damage targets;

[0155] The disease evolution modeling module 1005 is used to perform time-series modeling and status updates on the disease target based on the acquired historical inspection data, determine the current severity and rate of change of the disease target, and predict the development trend based on the rate of change to determine the future development trend of the disease target.

[0156] The active early warning module 1006 is used to generate early warning information for the road disease target if the current severity and the future development trend meet preset conditions; and send the early warning information to an external terminal to realize intelligent inspection and early warning of road diseases.

[0157] Based on the intelligent road defect inspection and early warning method described in the above embodiments, this application also provides an intelligent road defect inspection and early warning device, such as... Figure 11 As shown, Figure 11 This is a schematic diagram of the structure of a road defect intelligent inspection and early warning device provided in an embodiment of this application. The road defect intelligent inspection and early warning device 11 includes a processor 1101 and a memory 1102. The memory 1102 is used to store computer programs; the processor 1101 is used to call and run the computer programs from the memory to execute the road defect intelligent inspection and early warning method as described in the above embodiment.

[0158] In the embodiments of this application, the processor 1101 described above can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and the embodiments of this application do not specifically limit it.

[0159] Furthermore, in the embodiments of this application, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0160] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0162] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0163] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0164] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0165] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0166] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0167] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0168] The above description is merely an embodiment of this application, but the protection scope of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A road disease intelligent inspection and early warning method, characterized in that, The method comprises: During the driving of the inspection vehicle, road images and laser radar point cloud data are acquired, and the road images and the laser radar point cloud data are respectively subjected to timestamp alignment processing and filtering processing to determine an image sequence and a point cloud sequence; Based on the image sequence and the point cloud sequence, visual and laser fusion processing is performed to determine the position, attitude and latest road and disease map of the inspection vehicle; Based on the latest road and disease map, the image sequence and the point cloud sequence, road disease target detection is performed to determine road disease information; wherein the road disease information comprises the type, position and characteristic parameters of the disease target; Based on the acquired historical inspection data, time series modeling and state updating are performed on the disease target to determine the current severity and change rate of the disease target; and according to the change rate, development trend prediction is performed to determine the future development trend of the disease target; If the current severity and the future development trend meet preset conditions, early warning information of the disease target is generated; and the early warning information is sent to an external terminal to realize intelligent inspection and early warning of road diseases; Wherein, based on the latest road and disease map, the image sequence and the point cloud sequence, road disease target detection is performed to determine road disease information, comprising: Through a preset image processing algorithm or a trained deep learning model, road disease target detection is performed on the image sequence to determine a target image region; wherein the target image region is an image region suspected of disease; Ground contour analysis is performed on the point cloud sequence to determine an abnormal ground region; wherein the abnormal ground region is a region with abnormal height or abrupt change in appearance; Spatial matching and result fusion are performed on the target image region and the abnormal ground region to determine the disease target; Based on the latest road and disease map, the type of the disease target, the position of the disease target and the characteristic parameters of the disease target are determined; Wherein, based on the acquired historical inspection data, time series modeling and state updating are performed on the disease target to determine the current severity and change rate of the disease target, comprising: Based on the acquired historical inspection data, time series modeling is performed on the disease target to determine model parameters; Based on the model parameters, the state of the disease target is updated by a prediction-correction algorithm to determine the current severity and change rate of the disease target, comprising: calling a disease evolution modeling module to update the model parameters of the corresponding disease based on the data obtained in the current inspection; for each recorded disease, taking the latest feature observation value output in the last step as z k , updating the state estimation thereof by a prediction-correction algorithm , s k represents the current disease state, represents the current disease severity, represents the current disease change rate; applying Kalman filtering: first, according to the last state prediction , the predicted severity is obtained k , and then the new observation z is corrected to obtain ; wherein, s k-1 represents the last disease state, is a state transition matrix, is the interval between adjacent two inspections, y k+1 represents the next time disease severity; z k is the disease feature observation value at the kth inspection, s k|k is the optimal estimation of the disease state obtained after combining z k at the kth inspection; at the same time, the new observation value is added to the historical sequence of the disease for offline analysis or adaptive adjustment of the model, and if a disease disappears, the state termination is marked in the model.

2. The method of claim 1, wherein, Based on the image sequence and the point cloud sequence, visual and laser fusion processing is performed to determine the position, attitude and latest road and disease map of the inspection vehicle, comprising: Based on the image sequence, feature extraction is performed through visual processing technology to determine road texture information; wherein the road texture information comprises road texture feature points or feature lines; Based on the point cloud sequence, feature extraction is performed through laser processing technology to determine terrain feature information; wherein the terrain feature information comprises terrain contour or plane feature; Based on the road texture information and the terrain feature information, feature matching, point cloud matching and fusion positioning are performed to determine the position of the inspection vehicle and the attitude of the inspection vehicle; Based on the attitude of the inspection vehicle, map updating is performed to determine the latest road and disease map.

3. The method of claim 2, wherein, The feature matching, point cloud matching and fusion positioning are performed based on the road surface texture information and the terrain feature information to determine the position of the inspection vehicle and the attitude of the inspection vehicle, including: Based on the road surface texture information, feature matching is performed by a visual odometry calculation method to determine displacement information of the inspection vehicle; wherein the displacement information is the movement displacement of the inspection vehicle relative to the road surface; Based on the terrain feature information, point cloud matching is performed by a laser odometry calculation method to determine pose change information of the inspection vehicle; Based on the displacement information and the pose change information, fusion positioning is performed by a fusion odometry calculation method to determine the position of the inspection vehicle and the attitude of the inspection vehicle.

4. The method of claim 1, wherein, If the current severity and the future development trend meet a preset condition, an early warning information of the disease target is generated; And the early warning information is sent to an external terminal to realize intelligent inspection and early warning of road diseases, including: If the current severity or the future development trend is greater than a preset threshold, an early warning information of the disease target is generated; The early warning information is sent to an external terminal to realize intelligent inspection and early warning of road diseases.

5. The method of claim 4, wherein, If the current severity or the future development trend is greater than a preset threshold, an early warning information of the disease target is generated, including: If the current severity is greater than the preset threshold, an early warning information of the disease target is generated; or, Based on the future development trend, a remaining time to reach the preset threshold is determined, and if the remaining time is less than an early warning advance period, an early warning information of the disease target is generated.

6. The method of claim 1, wherein, The method further includes: Storing the road disease information to a disease archive; Continuing to perform intelligent inspection of road diseases to obtain the latest road disease information; Updating the disease archive through the latest road disease information.

7. A road disease intelligent inspection and early warning system, characterized in that, Including: A collection module, an adaptive perception module, a map construction module, a disease detection module, a disease evolution modeling module and an active early warning module, wherein, The collection module is configured to collect road images and laser radar point cloud data; The adaptive perception module is configured to acquire the road images and the laser radar point cloud data during the driving of the inspection vehicle, and perform timestamp alignment processing and filtering processing on the road images and the laser radar point cloud data respectively to determine image sequences and point cloud sequences; The map construction module is configured to perform visual and laser fusion processing based on the image sequences and the point cloud sequences to determine the position, attitude and the latest road and disease map of the inspection vehicle; The disease detection module is configured to perform road disease target detection based on the latest road and disease map, the image sequences and the point cloud sequences to determine road disease information; wherein the road disease information includes the type, position and characteristic parameters of the disease target; The disease evolution modeling module is configured to perform time series modeling and state updating on the disease target based on the acquired historical inspection data to determine the current severity and the change rate of the disease target; and perform development trend prediction according to the change rate to determine the future development trend of the disease target; The active early warning module is configured to generate early warning information of the disease target if the current severity and the future development trend meet preset conditions, and send the early warning information to an external terminal to realize intelligent inspection early warning of road diseases. The disease detection module is further configured to perform road disease target detection on the image sequence by using a preset image processing algorithm or a trained deep learning model to determine a target image region; the target image region is a suspected disease image region; perform ground contour analysis on the point cloud sequence to determine an abnormal ground region; the abnormal ground region is a region with abnormal height or abrupt change in appearance; perform spatial matching and result fusion on the target image region and the abnormal ground region to determine the disease target; determine a type of the disease target, a position of the disease target, and a characteristic parameter of the disease target based on the latest road and disease map. The disease evolution modeling module is further configured to perform time series modeling on the disease target based on the obtained historical inspection data, determine model parameters, perform state updating on the disease target by a prediction-correction algorithm based on the model parameters, and determine a current severity and a change rate of the disease target, including: calling the disease evolution modeling module to update model parameters of a corresponding disease based on data obtained in current inspection; for each recorded disease, taking a latest feature observation value output in a previous step as z k updating a state estimation thereof by a prediction-correction algorithm , s k represents a current disease state, represents a current disease severity, represents a current disease change rate; applying Kalman filtering: first, predicting a predicted severity according to a previous state prediction , and then correcting to obtain by combining a new observation z k ; wherein, represents a current disease state predicted according to a disease state at a previous moment; s k-1 represents a disease state at a previous moment, is a state transition matrix, is a time interval between adjacent two inspections, y k+1 represents a disease severity at a next moment; z k is a disease feature observation value at the kth inspection, s k|k is an optimal estimation of a disease state obtained after combining z k at the kth inspection; meanwhile, a new observation value is added to a historical sequence of the disease for offline analysis or adaptive adjustment of the model, and if a disease disappears, a state termination is marked in the model.

8. A road disease intelligent inspection and early warning device, characterized in that, comprising: a processor and a memory, the memory is configured to store a computer program; the processor is configured to call and run the computer program from the memory to execute the method of any one of claims 1 to 6.

Images