A method and system for real-time monitoring of road damage

Abstract

This invention belongs to the technical field of road surface information detection, and discloses a real-time road damage monitoring method and system. The method includes: acquiring road surface images captured by an onboard camera in real time and identifying road surface damage information; performing a storage operation for road surface damage information that meets spatiotemporal constraints; the storage operation includes: updating the constraint time based on the current vehicle speed and the maximum distance between the road surface damage and the vehicle; storing road surface damage information in the current road surface image that meets the spatial constraints into the storage space; and not allowing the storage of road surface damage information from new road surface images within the constraint time until the constraint time is reached, after which the storage operation is re-executed; if the distance between the currently requested damage location and each already stored damage location is greater than or equal to the drift distance, then the spatial constraints are deemed met. Based on spatiotemporal constraints, the burden on computing power platforms and storage space can be reduced, which is more conducive to the promotion of this road monitoring method.

Description

Technical Field

[0001] This invention belongs to the technical field of road surface information detection, and more specifically, relates to a method and system for real-time monitoring of road damage. Background Technology

[0002] As the lifeblood of the national economy, highways have seen their total mileage and maintenance costs increase year by year with the advancement of national infrastructure construction. Traditional road condition detection mainly relies on manual labor or dedicated LiDAR monitoring vehicles, which are inefficient, costly, and unable to achieve real-time monitoring. With the development of autonomous driving technology, the demand for high-precision, real-time monitoring of road conditions is also increasing. Therefore, it is urgent to develop an efficient, real-time, and intelligent road damage monitoring system.

[0003] Currently, there are some studies on real-time road damage monitoring systems. For example, Chinese patent application (publication number: CN220796043U, title: Road Inspection System) discloses a road inspection system, including: a vehicle-mounted inspection device for inspecting road conditions and simultaneously collecting inspection video streams; an edge computing platform for receiving the inspection video streams from the vehicle-mounted inspection device and intelligently recognizing the inspection video streams to obtain rendered framed video streams and formatted result data; and a road inspection platform for receiving rendered framed video streams and formatted result data from the edge computing platform. This patent application can effectively solve the needs of real-time display of inspection operations, synchronization of raw inspection data, and real-time storage and synchronous uploading in high-throughput concurrent scenarios when data bandwidth is limited.

[0004] However, because vehicle-mounted inspection equipment performs real-time inspections, it continuously collects and processes a large amount of information, which places extremely high demands on computing speed and storage capacity, making it difficult to promote in practical applications. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method and system for real-time monitoring of road damage, the purpose of which is to reduce the computational and storage burden and make road monitoring easier to promote.

[0006] To achieve the above objectives, the present invention provides a method for real-time monitoring of road damage, comprising:

[0007] The system acquires road surface images captured by an onboard camera in real time and identifies road surface damage information at various locations within the images. It then performs a storage operation on the road surface damage information that meets spatiotemporal constraints. This storage operation includes: updating the constraint time based on the current vehicle speed and the maximum distance between the furthest road surface damage in the current road surface image and the vehicle (the constraint time is the current maximum distance divided by the current vehicle speed); storing the road surface damage information in the current road surface image that meets the spatial constraints into the storage space; and not allowing the storage of road surface damage information from new road surface images within the constraint time until the constraint time is reached, after which the storage operation is re-executed. If the distance between the damage location of the road surface damage information currently requesting storage and the damage location of each road surface damage information already stored is greater than or equal to a preset positioning drift distance, then the road surface damage information currently requesting storage is determined to meet the spatial constraints.

[0008] The stored road damage information is used for real-time analysis and monitoring of road damage.

[0009] In an optional embodiment, identifying road damage information at various locations in the road image includes defining various locations of road damage in the road image using detection boxes and calculating road damage information within each detection box.

[0010] Before updating the constraint time, the distance between each detection box and the vehicle in the current road image is calculated, and the maximum distance is selected to update the constraint time. The methods for calculating the distance between the detection box and the vehicle include:

[0011] Transform the vertices of the detection box from the pixel coordinate system to the camera coordinate system;

[0012] Calculate the pixel area of ​​the detection box in the pixel coordinate system and the actual area in the camera coordinate system, respectively;

[0013] To calculate the distance between vehicles, use the formula: Distance between vehicles = (Actual area ÷ Pixel area) × Camera focal length.

[0014] In an optional embodiment, the current vehicle speed is determined before updating the constraint time. The method for determining the current vehicle speed includes:

[0015] The vehicle acceleration and angular velocity provided by the IMU and the vehicle speed provided by the GPS are obtained. Kalman filtering is used to fuse the data provided by the IMU and the data provided by the GPS to obtain the current vehicle speed used to update the constraint time.

[0016] In an optional embodiment, the method utilizes a deep learning model to identify the type of road damage at various locations in the road image, wherein the deep learning model is a trained deep learning model.

[0017] The method further includes implementing active learning of the deep learning model in the new environment, wherein the active learning process includes:

[0018] Obtain the uncertainty of each category obtained by training an existing deep learning model in an early environment;

[0019] Acquire road surface images captured by the vehicle-mounted camera in the new environment as the initial image set;

[0020] The existing deep learning model is used to classify the damage categories of each road surface image in the initial image set, so as to obtain the category and confidence level of each road surface damage in each image.

[0021] The uncertainty of the corresponding road surface image is obtained by weighting and summing the category confidence scores of each road surface damage in the road surface image, with the weights being the uncertainty scores of the corresponding categories.

[0022] Select the road surface images with the highest uncertainty and calculate their similarity. Remove the road surface images with the highest similarity from the initial image set to obtain the filtered image set.

[0023] The existing deep learning model is retrained based on the selected image set.

[0024] In an optional embodiment, the method further includes performing road damage analysis and displaying the analysis results through a visualization interface.

[0025] In an optional embodiment, the road damage includes cracks, and the method further includes performing road damage analysis, the road damage analysis including:

[0026] The system tracks changes in cracks and automatically triggers an alert when the crack area growth rate exceeds a set warning threshold.

[0027] The present invention also provides a real-time road damage monitoring system, which includes edge devices and a cloud server;

[0028] The edge device includes an onboard camera, sensors, a positioning module, an edge computing module, and a communication module that can be mounted on a vehicle. The onboard camera is used to capture road images, the sensors are used to collect vehicle driving status, the positioning module is used to locate the vehicle, and the edge computing module is used to perform preliminary processing on the data collected by the edge device and then transmit it to the cloud server through the communication module.

[0029] The cloud server is used to receive data transmitted by the edge device and perform road damage analysis and monitoring;

[0030] The edge device or the cloud server is used to execute the method described above.

[0031] The present invention also provides a computing device for real-time monitoring of road damage, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0033] The present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the method described above.

[0034] In summary, compared with the prior art, the technical solutions conceived in this invention have the following main advantages:

[0035] 1. This invention proposes a method for real-time monitoring of road damage and its application. After acquiring vehicle-mounted photos, it does not perform real-time analysis and monitoring based on each photo. Instead, it filters information through spatiotemporal constraints. The temporal constraint considers that when vehicles are traveling at slow speeds or stationary (e.g., waiting at traffic lights), they continuously capture numerous images of the same road surface damage, leading to repetitive identification by the computing platform. Introducing a temporal constraint can reduce this repetitive identification work to some extent. The spatial constraint considers that cameras typically have very high frame rates, capturing multiple images of the same road surface damage when passing it. By applying a spatial constraint, it ensures that cracks within the drift range of the positioning module are not counted multiple times, thus also reducing the repetitive identification work caused by the aforementioned reasons. Therefore, based on spatiotemporal constraints, the burden on the computing platform and storage space can be reduced, which is more conducive to the promotion of this road monitoring method.

[0036] 2. In an optional embodiment, Kalman filtering is used to fuse the data provided by the IMU and the data provided by the GPS to obtain the current vehicle speed used to update the constraint time, which can improve the accuracy of speed estimation.

[0037] 3. In an optional embodiment, when identifying the type of road damage in various locations in a road image using a deep learning model, a method for active learning of the deep learning model is also proposed to optimize the model so that it can adapt to road conditions in different regions and cities, thereby achieving accurate monitoring of road damage under different environments. Attached Figure Description

[0038] Figure 1 This is a flowchart of the steps of a real-time road damage monitoring method according to an embodiment of the present invention;

[0039] Figure 2 This is a diagram illustrating the delineation of damaged areas in an image according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram illustrating data analysis in one embodiment of the present invention;

[0041] Figure 4 This is a system block diagram of a real-time road damage monitoring system according to an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0043] Example 1

[0044] like Figure 1 The diagram shows a flowchart of a real-time road damage monitoring method according to an embodiment of the present invention, which can be executed in the order of steps S1 to S4:

[0045] Step S1: Acquire road surface images captured by the vehicle-mounted camera in real time;

[0046] Step S2: Identify road damage information at various locations in the road surface image;

[0047] Step S3: Perform the storage operation of road surface damage information that satisfies the spatiotemporal constraints;

[0048] Step S4: Perform real-time analysis and monitoring of road damage based on the stored road surface damage information.

[0049] Specifically, in step S1, a high frame rate industrial camera can be selected and installed on a regular vehicle, such as a bus, taxi, private car, or smart car. In addition, the vehicle is also equipped with sensors (such as IMU sensors), positioning modules (such as GPS locators), and edge computing modules. The sensors are used to collect the vehicle's driving status, the positioning module is used to locate the vehicle, and the edge computing module is used to perform preliminary processing on the data collected by the edge devices and then transmit it to the cloud server through the communication module.

[0050] Specifically, step S2 identifies road damage information in the road image.

[0051] A single road surface image may contain multiple road surface damages, therefore, multiple road surface damages can be identified from a single image. Road surface damage can include transverse cracks, longitudinal cracks, network cracks, potholes, alligator cracks, and repair cracks. In practice, each road surface damage can be defined using a detection box. The detection box can be rectangular, and its size varies depending on the size of the target area of ​​the road surface damage, as long as it can cover the corresponding road surface damage area. Figure 2 The image shown illustrates the delineation of damaged areas in an image according to one embodiment. In a specific embodiment, road surface damage can be directly identified using a recognition model.

[0052] Road surface damage information typically includes the location of the damage, the type of damage, etc., and can be selected according to the actual situation.

[0053] Specifically, step S3 performs the storage operation of road damage information that satisfies spatiotemporal constraints.

[0054] In traditional solutions, all acquired images are typically analyzed and monitored. In real-time acquisition scenarios, a large number of images need to be processed continuously, which places extremely high demands on the computing speed and storage capacity of the computing platform.

[0055] In this application, a spatiotemporal constraint is proposed, which stores and performs in-depth analysis only on road damage information that meets the spatiotemporal constraint. This can filter out a portion of the images, reduce the computational speed requirements of the computing platform, and reduce the storage burden.

[0056] Specifically, spatiotemporal constraints are divided into time constraints and space constraints.

[0057] The time constraint prevents the storage of new pavement damage information from being stored within a specified time after the current pavement damage information is stored. This constraint time is calculated based on the current pavement damage information and is continuously updated as new information is stored. Specifically, it requires calculating the maximum distance *d* between the furthest pavement damage in the current pavement image and the vehicle. max Based on the current maximum vehicle distance d max Calculate the constraint time t based on the current vehicle speed v, t = d max / v.

[0058] For example, suppose the last time road damage information was stored in the storage space was privious_time, and the current time is denoted as current_time. If:

[0059] current_time <previous_time+t

[0060] If the time constraint is not met, the data cannot be stored.

[0061] The spatial constraint requires that the distance between the damage location of the road damage information to be stored in the storage space and the damage location of each road damage information already stored in the storage space must be greater than or equal to the drift distance of the positioning module. In other words, when the time constraint is met, it is also necessary to further determine whether the distance between the road damage location and all road damage locations already stored in the storage space are greater than or equal to the drift distance of the positioning module. If there is a historical road damage location in the storage space whose distance to the current road damage location is less than the drift distance, then the current road damage information does not meet the spatial constraint and cannot be stored.

[0062] The drift distance of the positioning module is a known parameter; for example, the positioning module can be a GPS, and the drift distance of the GPS is d. gps_max The damage location of the road surface damage information currently requested for storage is in latitude and longitude coordinates (lat. new ,long new The location of road surface damage information already stored in the storage space can be recorded as latitude and longitude coordinates. history ,long history As long as there exists a latitude and longitude coordinate in the storage space that satisfies the following conditions as the currently requested information to be stored:

[0063] distance((lat new ,long new ),(lat history ,long history )) <d gps_max

[0064] If the road damage information requested for storage does not meet the spatial constraints, it cannot be stored; otherwise, it can be stored.

[0065] In one specific embodiment, in order to accurately estimate the current vehicle speed, the estimation process includes: acquiring the vehicle acceleration and angular velocity provided by the IMU (Inertial Measurement Unit) and the vehicle speed provided by the GPS (Global Positioning System); fusing the data provided by the IMU and the data provided by the GPS using Kalman filtering to obtain the final current vehicle speed used to update the constraint time.

[0066] Assuming the IMU provides acceleration a x a y and angular velocity ω z GPS provides speed v gps The subscripts x and y represent the components along the x-axis and y-axis, respectively.

[0067] Assuming the sampling interval at the current moment is Δt, and the acceleration a is provided by the IMU...x a y The speed can be preliminarily predicted as follows:

[0068]

[0069] In the formula, the superscript prev indicates the previous time and the superscript pred indicates the current time.

[0070] Based on the speed v provided by GPS gps and direction angle θ gps It can also be used to make a preliminary prediction of speed:

[0071]

[0072] Subsequently, the Kalman gain K is used to fuse the speed information from the IMU and GPS, and the updated speed is:

[0073]

[0074] In the formula, K is the Kalman gain, which can be calculated from the error covariance matrix.

[0075] The formula for calculating the Kalman gain K is:

[0076]

[0077] Where P pred It is the prediction error covariance matrix, R gps This is the GPS observation noise covariance. The final state covariance update is:

[0078] P new =(IK)·P pred

[0079] Ultimately, the vehicle's actual speed was:

[0080]

[0081] In one specific embodiment, before updating the constraint time t, the distance between each detection box in the current road image and the vehicle is calculated, and the constraint time is updated by selecting the maximum vehicle distance; the method for calculating the distance between the detection box and the vehicle includes:

[0082] Transform the vertices of the detection box from the pixel coordinate system to the camera coordinate system;

[0083] Calculate the pixel area of ​​the detection box in the pixel coordinate system and the actual area in the camera coordinate system, respectively;

[0084] Calculate the distance between vehicles: Distance between vehicles = (Actual area / Pixel area) * Camera focal length.

[0085] Assume the camera intrinsic parameters are:

[0086]

[0087] In the formula, f x f y c is the scaling factor of focal length f in the x and y directions. x c y The principal point coordinates are the optical center of the imaging plane.

[0088] In this embodiment, the Z-axis of the camera coordinate system represents the vehicle's travel direction, and the Y-axis represents the direction perpendicular to the road surface. Since the distance between the camera and the road surface is known, i.e., the Y-coordinate of the target (road damage) in the camera coordinate system is known, the coordinate system transformation can be achieved using the following formula:

[0089]

[0090] In the formula, X represents the X coordinate of the detected target in the camera coordinate system, Y represents the Y coordinate of the detected target in the camera coordinate system, Z represents the Z coordinate of the detected target in the camera coordinate system, u represents the horizontal coordinate of the detected target in the pixel coordinate system, and v represents the vertical coordinate of the detected target in the pixel coordinate system.

[0091] Through the above coordinate transformation, the vertices of the detection box can be converted from pixel coordinates to camera coordinates, and then the length and width of the detection box in the camera coordinate system can be calculated.

[0092] Calculate the length L and width W of the detection box in the camera coordinate system, and the final area is:

[0093] A = L × W

[0094] L and W are the length and width calculated from the vertices of the transformed bounding box.

[0095] The distance between the target and the camera is calculated based on the size of the bounding box, using the object's size A in the camera coordinate system. real With the bounding box size A in pixel coordinates bbox The proportional relationship is used to estimate the distance d between the target and the camera:

[0096]

[0097] In the formula, f is the focal length.

[0098] In this application, information filtering is achieved through step S3. The time constraint considers that when vehicles are traveling at slow speeds or stationary (e.g., waiting at traffic lights), they will continuously capture numerous images of the same road surface damage, leading to repetitive identification by the computing platform. Introducing a time constraint can reduce this repetitive identification work to some extent. The spatial constraint considers that cameras typically have very high frame rates, capturing multiple images of the same road surface damage when passing it. By applying a spatial constraint, it can be ensured that cracks within the drift range of the positioning module are not counted multiple times, thereby also reducing the repetitive identification work caused by the aforementioned reasons. Therefore, based on the spatiotemporal constraints of step S3, the burden on the computing platform and storage space can be reduced.

[0099] In step S4, further real-time analysis and monitoring are performed on the information stored in the storage space.

[0100] After the filtering in step S3, the stored information can be analyzed and monitored in real time. The type of analysis can be flexibly selected according to actual needs. For example, road damage data can be statistically analyzed, damage location and damage status can be obtained, and damage changes can be monitored.

[0101] like Figure 3 The diagram illustrates data analysis in one embodiment of the present invention. The data can be displayed through a visual interface, such as showing road damage data statistics, real-time location maps, and damage status maps. A list interface provides detailed information on all detection points, supporting filtering and searching. A detailed interface provides specific information and a location map for each detection point. Quantitative analysis of road damage data can provide decision-making suggestions to help maintenance personnel make quick decisions. It can also provide real-time road damage risk maps, offering road risk planning information for autonomous vehicles. Furthermore, it can calculate the PCI (Constant Traffic Index) for each road segment based on the analyzed data. Combining PCI, crack characteristics, and bump index, a comprehensive road damage risk score is calculated for each road segment. Based on the PCI and risk score, the damage level of road segments is classified and assessed, generating decision-making suggestions and a maintenance priority list. Users can view, edit, and manage road damage data through the user interface.

[0102] In this embodiment, the focus is on monitoring and tracking changes in cracks. When the crack area growth rate exceeds a set warning threshold, an automatic warning is triggered. Specifically, based on stored crack data, the changes in crack area are analyzed periodically to calculate the speed and trend of crack expansion. If the crack area growth rate exceeds the set warning threshold, the system automatically triggers a warning and pushes a notification to the user indicating a potential risk of road collapse.

[0103] In one embodiment, when identifying the types of road damage in various locations in a road image using a deep learning model, considering the limited adaptability of deep learning models to the environment—for example, the detection capability of domestic roads will decrease after using the European road crack dataset, and even the model's detection capability varies in different cities and climates within China—this solution also proposes a method for active learning of deep learning models. The goal of active learning is to optimize the model to adapt to road conditions in different regions and cities, establish an efficient feedback mechanism, and achieve continuous optimization of the model.

[0104] The following describes the process of active learning.

[0105] The first step is to obtain the uncertainty of each category obtained by training the existing deep learning model in the early environment.

[0106] Specifically, the deep learning model obtains the uncertainty d for each category during historical training. i k The superscript i represents the historical training round, and the subscript k represents the category index. Categories include, but are not limited to, horizontal cracks, vertical cracks, mesh cracks, pits, crazing, and repair cracks.

[0107] The second step is to acquire road surface images taken by the vehicle-mounted camera in the new environment as the initial image set. Then, the existing deep learning model is used to classify the damage categories of each road surface image in the initial image set, and the categories and confidence levels of road surface damage at each location in each road surface image are obtained.

[0108] Specifically, in another city, the equipment was deployed and used to capture original photos during road application. These photos were then input into the deep learning model for identification, resulting in the categories of road surface damage and their category confidence scores (p). j (k), where k is the category index and j is the j-th road surface damage in the image.

[0109] The third step is to perform a weighted summation of the category confidence scores for each type of road damage in the road image, with the weights being the uncertainty of the corresponding category, to obtain the uncertainty of the corresponding road image.

[0110] Specifically, the uncertainty U of the image is calculated based on the uncertainty of the category and the confidence level of the damage category at various locations in the image:

[0111]

[0112] In the formula, J represents the number of road surface damages in the image.

[0113] The fourth step is to select the road surface images with the lowest uncertainty and calculate their similarity. Then, remove the road surface images with the highest similarity from the initial image set to obtain the filtered image set.

[0114] Specifically, when vehicles are moving at high speeds, the camera uses a very high frame rate to capture the road clearly, which may generate a large number of highly similar images. These images waste computing power in the subsequent training phase and can easily lead to overfitting during active learning, so they need to be removed. In this step, by calculating the uncertainty of the images, some images with low reliability are initially identified. Then, the image similarity is further calculated to delete images with high uncertainty and high similarity, thereby improving the effectiveness of subsequent training.

[0115] In practice, the high-dimensional features of the penultimate layer of the model network can be used to calculate the image similarity.

[0116] The fifth step is to retrain the existing deep learning model based on the selected image set.

[0117] Understandably, once a valuable image set is identified, it is labeled to form a new training set, which can then be used for targeted fine-tuning of the model. After training, the model can be deployed in subsequent work.

[0118] Example 2

[0119] This invention also relates to a real-time road damage monitoring system, such as... Figure 4 The diagram shows a system block diagram of a real-time road damage monitoring system according to an embodiment of the present invention. It includes edge devices, a cloud server, and typically a user terminal. The edge devices include a vehicle-mounted camera, sensors, a positioning module, an edge computing module, and a communication module. The vehicle-mounted camera captures road surface images, the sensors collect vehicle driving status data, the positioning module locates the vehicle, and the edge computing module performs preliminary processing on the data collected by the edge devices before transmitting it to the cloud server via the communication module. The cloud server receives the data transmitted by the edge devices and performs road damage analysis and monitoring.

[0120] Specifically, the camera can be a high frame rate industrial camera to capture real-time images of the road surface for damage detection; the edge computing module serves as the core processing unit of the system, responsible for data acquisition and preliminary processing; the sensor can include an IMU sensor to acquire data during vehicle movement; the positioning module can be a GPS module to provide high-precision geographic location information; and the communication module can be a 5G communication module to achieve high-speed data transmission.

[0121] The edge device or cloud server is used to perform the information storage operation based on spatiotemporal constraints in the real-time road damage monitoring method described in Example 1.

[0122] For example, the operations described in Example 1 can be performed directly on the cloud server, including information filtering and storage, further real-time analysis, and active learning by the deep learning module. The edge computing module only performs simple data processing and transmission. In this case, the requirements for the edge computing module can be greatly reduced, but it may be difficult to respond in a timely manner in areas with poor network conditions or when the cloud server is under heavy load.

[0123] For example, the spatiotemporal constraint storage operation can be performed on the edge device, while further complex operations such as data analysis and active learning can be performed on the cloud server. In this case, a sending queue can be established on the edge device, and a historical queue can be established on the cloud server. The edge device stores information that meets the spatiotemporal constraint conditions into the sending queue, and the sending queue gradually synchronizes the information to the historical queue on the cloud server. In a specific embodiment, a lightweight recognition model can be set up on the edge device to initially identify road surface damage, while a more powerful deep learning module can be set up on the cloud server for accurate damage identification.

[0124] Example 3

[0125] The present invention also relates to a computing device for real-time monitoring of road damage, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in Embodiment 1.

[0126] The computing power device for real-time road damage monitoring can be a desktop computer, laptop, handheld computer, or cloud server, etc. In this embodiment, an AI engine and an NPU computing chip can be deployed. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor realizes the various functions of the computing power device for real-time road damage monitoring by running or executing the computer programs and / or modules stored in the memory, and by calling the data stored in the memory.

[0127] Example 4

[0128] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0129] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0130] Example 5

[0131] This invention provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described in the above embodiments of this invention.

[0132] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. It should be noted that the terms "in one embodiment," "for example," and "again" are intended to illustrate the present invention and are not intended to limit the present invention.

[0133] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A method for real-time monitoring of road damage, characterized in that, include: The system acquires road surface images captured by an onboard camera in real time and identifies road surface damage information at various locations within the images. It then performs a storage operation on the road surface damage information that meets spatiotemporal constraints. This storage operation includes: updating the constraint time based on the current vehicle speed and the maximum distance between the furthest road surface damage location in the current road surface image and the vehicle (the constraint time is the current maximum distance divided by the current vehicle speed); storing the road surface damage information in the current road surface image that meets the spatial constraints into the storage space; and not allowing the storage of road surface damage information from new road surface images within the constraint time until the constraint time is reached, after which the storage operation is re-executed. If the distance between the damage location of the road surface damage information currently requesting storage and the damage location of each road surface damage information already stored is greater than or equal to a preset positioning drift distance, then the road surface damage information currently requesting storage is determined to meet the spatial constraints. The stored road damage information is used for real-time analysis and monitoring of road damage. The step of identifying road damage information at various locations in the road image includes: defining various locations of road damage in the road image using detection boxes and calculating road damage information within each detection box; Before updating the constraint time, the distance between each detection box and the vehicle in the current road image is calculated, and the maximum distance is selected to update the constraint time. The methods for calculating the distance between the detection box and the vehicle include: Transform the vertices of the detection box from the pixel coordinate system to the camera coordinate system; Calculate the pixel area of ​​the detection box in the pixel coordinate system and the actual area in the camera coordinate system, respectively; To calculate the distance between vehicles, use the formula: Distance between vehicles = (Actual area ÷ Pixel area) × Camera focal length.

2. The real-time road damage monitoring method as described in claim 1, characterized in that, Before updating the constraint time, determine the current vehicle speed. The methods for determining the current vehicle speed include: The vehicle acceleration and angular velocity provided by the IMU and the vehicle speed provided by the GPS are obtained. Kalman filtering is used to fuse the data provided by the IMU and the data provided by the GPS to obtain the current vehicle speed used to update the constraint time.

3. The real-time road damage monitoring method as described in claim 1, characterized in that, The method further includes using a deep learning model to identify the type of road damage at various locations in the road image, wherein the deep learning model is a trained deep learning model; The method further includes actively learning the deep learning model in the new environment, the active learning process including: Obtain the uncertainty of each category obtained by training an existing deep learning model in an early environment; Acquire road surface images captured by the vehicle-mounted camera in the new environment as the initial image set; The existing deep learning model is used to classify the damage categories of each road surface image in the initial image set, so as to obtain the category and confidence level of each road surface damage in each image. The uncertainty of the corresponding road surface image is obtained by weighting and summing the category confidence scores of each road surface damage in the road surface image, with the weights being the uncertainty scores of the corresponding categories. Select the road surface images with the highest uncertainty and calculate their similarity. Remove the road surface images with the highest similarity from the initial image set to obtain the filtered image set. The existing deep learning model is retrained based on the selected image set.

4. The real-time road damage monitoring method as described in claim 1, characterized in that, The method also includes performing road damage analysis and displaying the analysis results through a visualization interface.

5. The real-time road damage monitoring method as described in claim 1, characterized in that, The road damage includes cracks, and the method further includes performing road damage analysis, the road damage analysis including: The system tracks changes in cracks and automatically triggers an alert when the crack area growth rate exceeds a set warning threshold.

6. A real-time road damage monitoring system, characterized in that, This includes edge devices and cloud servers; The edge device includes an onboard camera, sensors, a positioning module, an edge computing module, and a communication module that can be mounted on a vehicle. The onboard camera is used to capture road images, the sensors are used to collect vehicle driving status, the positioning module is used to locate the vehicle, and the edge computing module is used to perform preliminary processing on the data collected by the edge device and then transmit it to the cloud server through the communication module. The cloud server is used to receive data transmitted by edge devices and perform road damage analysis and monitoring; The edge device or the cloud server is used to perform the method as described in any one of claims 1 to 2.

7. A computing device for real-time monitoring of road damage, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 5.

Images