Road damage real-time monitoring method and monitoring system thereof

Abstract

The invention belongs to the related technical field of road surface information detection, and discloses a road damage real-time monitoring method and a monitoring system thereof, and the method comprises the steps: obtaining a road surface image shot by a vehicle-mounted camera in real time, recognizing the road surface damage information, executing the storage operation of the road surface damage information meeting the space-time constraint, and obtaining the road surface damage information; the storage operation comprises the following steps: updating constraint time according to the current vehicle speed and the maximum vehicle distance of the road surface damage distance, storing the road surface damage information meeting the spatial constraint condition in the current road surface image into a storage space, and not allowing the storage of the road surface damage information of a new road surface image within the constraint time; the storage operation is executed again until the constraint time is reached; and if the distance between the damage position currently applied to be stored and each stored damage position is greater than or equal to the drift distance, determining that a spatial constraint condition is met. Based on space-time constraints, the burden of a computing power platform and a storage space can be reduced, and popularization of the road monitoring method is facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field related to road surface information detection, and more specifically, relates to a real-time monitoring method for road damage and a monitoring system thereof. Background Art

[0002] As the main artery of the national economy, the total mileage and maintenance costs of highways are increasing year by year with the advancement of national infrastructure construction. Traditional road condition detection mainly relies on manual or dedicated laser radar monitoring vehicles. This method is inefficient, costly, and cannot achieve real-time monitoring. With the development of unmanned driving technology, the demand for high-precision and real-time monitoring of road conditions is also increasing. Therefore, it is urgent to propose an efficient, real-time, and intelligent road damage real-time monitoring system.

[0003] At present, there have been some studies on real-time monitoring systems for road damage, such as the Chinese patent application (publication number: CN220796043U, name: Road Inspection System) discloses a road inspection system, including: vehicle-mounted inspection equipment, used to inspect road conditions and synchronously collect inspection video streams; edge computing platform, used to receive inspection video streams from vehicle-mounted inspection equipment and intelligently identify inspection video streams to obtain rendered framed video streams and formatted result data; road inspection platform, used to receive rendered framed video streams and formatted result data from edge computing platform. This patent application can effectively solve the needs of real-time display of inspection operations, synchronization of inspection raw data, and real-time storage and synchronous upload in high-throughput concurrent scenarios when data bandwidth is limited.

[0004] However, since the vehicle-mounted inspection equipment performs real-time inspections, it will continuously collect and process a large amount of information, which has extremely high requirements for computing speed and storage capacity, and is difficult to promote in actual applications. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a real-time monitoring method for road damage and a monitoring system thereof, which aims to reduce the computing and storage burden and make road monitoring easier to promote.

[0006] To achieve the above object, the present invention provides a method for real-time monitoring of road damage, which comprises:

[0007] Acquire the road image taken by the vehicle-mounted camera in real time and identify the road damage information at each location in the road image, and perform a storage operation of the road damage information that meets the time and space constraints, the storage operation includes: updating the constraint time according to the current vehicle speed and the maximum vehicle distance from the farthest road damage in the current road image to the vehicle, the constraint time is the current maximum vehicle distance divided by the current vehicle speed, storing the road damage information in the current road image that meets the spatial constraint conditions in the storage space, and not allowing the road damage information of the new road image to be stored within the constraint time, until the storage operation is re-executed after the constraint time is reached; if the distance between the damage position of the road damage information currently applied to be stored in the storage space and the damage position of each road damage information already stored in the storage space is greater than or equal to the preset positioning drift distance, then it is determined that the road damage information currently applied to be stored in the storage space meets the spatial constraint conditions;

[0008] The stored road damage information is used for real-time analysis and monitoring of road damage.

[0009] In an optional embodiment, the identifying the road damage information at each location in the road image includes framing the road damage at each location in the road image by means of a detection frame and calculating the road damage information in each detection frame;

[0010] Before updating the constraint time, the distance between each detection frame and the vehicle in the current road image is calculated, and the maximum distance is selected to update the constraint time; the method for calculating the distance between the detection frame and the vehicle includes:

[0011] Convert each vertex of the detection frame from the pixel coordinate system to the camera coordinate system;

[0012] Calculate the pixel area of ​​the detection frame in the pixel coordinate system and the actual area in the camera coordinate system respectively;

[0013] Calculate the distance between vehicles: distance between vehicles = (actual area ÷ pixel area) × camera focal length.

[0014] In an optional embodiment, before updating the constraint time, the current vehicle speed is first determined, and the method for determining the current vehicle speed includes:

[0015] The vehicle acceleration and angular velocity provided by the IMU and the vehicle speed provided by the GPS are obtained, and the data provided by the IMU and the GPS are fused using the Kalman filter to obtain the current vehicle speed that is ultimately used to update the constraint time.

[0016] In an optional embodiment, the method uses a deep learning model to identify the category of road damage at various locations in the road surface image, and the deep learning model is a trained deep learning model;

[0017] The method further includes implementing active learning of the deep learning model in a new environment, wherein the process of active learning includes:

[0018] Obtain the uncertainty of each category obtained by training the existing deep learning model in a previous environment;

[0019] Obtain road images taken by the vehicle-mounted camera in the new environment as the initial image set;

[0020] Using an existing deep learning model to classify the damage category of each road surface image in the initial image set, and obtain the category and confidence of the road surface damage at each location in each road surface image;

[0021] Perform weighted summation of the confidence of each category of road damage in the road surface image, with the weight being the uncertainty of the corresponding category, and obtain the uncertainty of the corresponding road surface image;

[0022] Selecting multiple road surface images with the highest uncertainty and calculating image similarity, removing multiple road surface images with the highest similarity from the initial image set to obtain a filtered image set;

[0023] The existing deep learning model is trained again based on the filtered image set.

[0024] In an optional embodiment, the method further includes performing a road damage analysis and presenting the analysis results through a visual interface.

[0025] In an optional embodiment, the road damage includes cracks, and the method further includes performing a road damage analysis, wherein the road damage analysis includes:

[0026] The changes in cracks are tracked, and when the crack area growth rate exceeds the set warning threshold, an early warning is automatically triggered.

[0027] The present invention also provides a road damage real-time monitoring system, which includes an edge device and a cloud server;

[0028] The edge device includes an on-board camera, a sensor, a positioning module, an edge computing module and a communication module that can be loaded on the vehicle; the on-board camera is used to capture road images, the sensor is used to collect vehicle driving status, the positioning module is used to locate the vehicle, and the edge computing module is used to perform preliminary processing on the data collected by the edge device and transmit it to the cloud server through the communication module;

[0029] The cloud server is used to receive data transmitted by the edge device and perform road damage analysis and monitoring;

[0030] Wherein, the edge device or the cloud server is used to execute the method as described above.

[0031] The present invention also provides a computing device for implementing real-time monitoring of road damage, including a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method described above when executing the computer program.

[0032] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0033] The present invention also provides a computer program product, comprising a computer program or instructions, which implement the steps of the method described above when executed by a processor.

[0034] In general, compared with the prior art, the above technical solutions conceived by the present invention mainly have the following beneficial effects:

[0035] 1. The present invention proposes a method for real-time monitoring of road damage and its application. After obtaining the on-board photos, it does not perform real-time analysis and monitoring based on each photo, but screens the information through time and space constraints. The time constraint takes into account that when the vehicle is traveling at a slow speed or in a stationary state (such as waiting for a traffic light), a large number of images of the same road damage will be continuously taken, resulting in repeated recognition by the computing platform. The introduction of time constraints can reduce the repeated recognition work caused by the above reasons to a certain extent. The spatial constraint takes into account that the camera's shooting frame rate is usually very high. When passing the same road damage, multiple images of the road damage will be captured. By performing spatial constraints, it can be ensured that cracks within the drift range of the positioning module will not be counted multiple times, thereby reducing the repeated recognition work caused by the above reasons. Therefore, based on time and space constraints, the burden on the computing platform and storage space can be reduced, which is more conducive to the promotion of the road monitoring method.

[0036] 2. In an optional embodiment, Kalman filtering is used to fuse the data provided by the IMU and the data provided by the GPS to obtain the current vehicle speed that is ultimately used to update the constraint time, thereby improving the accuracy of speed estimation.

[0037] 3. In an optional embodiment, when a deep learning model is used to identify the categories of road damage in various locations in a road surface image, a method for implementing active learning of the deep learning model is also proposed to optimize the model so that the model can adapt to road conditions in different regions and cities, thereby achieving accurate monitoring of road damage in different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flowchart of the steps of a method for real-time monitoring of road damage in one embodiment of the present invention;

[0039] Figure 2 It is a display diagram of framing a damaged area of ​​an image in one embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of performing data analysis in one embodiment of the present invention;

[0041] Figure 4 4 is a system block diagram of a road damage real-time monitoring system in one embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0043] Example 1

[0044] like Figure 1 The flowchart of the method for real-time monitoring of road damage in one embodiment of the present invention is shown, which can be performed in the order of step S1 to step S4:

[0045] Step S1: acquiring a road surface image taken by a vehicle-mounted camera in real time;

[0046] Step S2: identifying road damage information at various locations in the road surface image;

[0047] Step S3: Execute the storage operation of the road surface damage information that meets the time and space constraints;

[0048] Step S4: Perform real-time analysis and monitoring of the road damage on the stored road damage information.

[0049] Specifically, in step S1, a high frame rate industrial camera can be selected and installed on an ordinary vehicle, such as a bus, taxi, private car or smart car. In addition, the vehicle is also equipped with sensors (such as IMU sensors), positioning modules (such as GPS locators) and edge computing modules. The sensors are used to collect the vehicle's driving status, the positioning module is used to locate the vehicle, and the edge computing module is used to perform preliminary processing on the data collected by the edge device and transmit it to the cloud server through the communication module.

[0050] Specifically, step S2 realizes the recognition of road surface damage information in the road surface image.

[0051] A road image may contain multiple road damages, so a single image may identify multiple road damages. Road damages may include transverse cracks, longitudinal cracks, mesh cracks, potholes, cracks, and repaired cracks. In specific operations, each road damage can be framed by a detection frame, which can be a rectangular detection frame. The size of the detection frame varies with the size of the road damage target area, and it only needs to cover the corresponding road damage area, such as Figure 2 The figure shows a display diagram of framing the damaged area of ​​an image in an embodiment. In a specific embodiment, the recognition model can be directly used to recognize road damage.

[0052] The road surface damage information generally includes the road surface damage location, damage type, etc., which can be selected according to the actual situation.

[0053] Specifically, step S3 executes the storage operation of the road surface damage information that meets the time and space constraints.

[0054] In traditional solutions, all collected images are generally analyzed and monitored. In the scenario of real-time collection, a large number of images need to be processed continuously, which places extremely high demands on the computing speed and storage capacity of the computing platform.

[0055] In this application, time and space constraints are proposed, and only the road damage information that meets the time and space constraints is stored and subsequently analyzed in depth. This can filter out a portion of the image, reduce the computing speed requirements of the computing platform, and reduce the storage burden.

[0056] Specifically, spatiotemporal constraints are divided into time constraints and space constraints.

[0057] The time constraint is that after the current road damage information is stored in the storage space, the road damage information of the new road image is not allowed to be stored within the constraint time, and new data is allowed to be stored only after the constraint time is reached. Moreover, the constraint time is calculated based on the current road damage information, so the constraint time is also continuously updated with the stored information. Specifically, it is necessary to calculate the maximum distance d from the farthest road damage in the current road image to the vehicle max , according to the current maximum vehicle distance d max And the current vehicle speed v calculates the constraint time t, t = d max / v.

[0058] For example, assuming that the last time the road damage information was stored in the storage space is privious_time, and the current time is current_time, if:

[0059] current_time <previous_time+t

[0060] If the time constraint is not met, the deposit cannot be made.

[0061] The spatial constraint is that the distance between the damage position of the road damage information applied to be stored in the storage space and the damage position of each road damage information stored in the storage space is greater than or equal to the drift distance of the positioning module. In other words, when the time constraint is met, it is necessary to further determine whether the distance between the road damage location and all road damage locations stored in the storage space is greater than or equal to the drift distance of the positioning module. If there is a historical road damage location in the storage space whose distance to the current road damage location is less than the drift distance, the current road damage information does not meet the spatial constraint and cannot be stored.

[0062] The drift distance of the positioning module is a known parameter. For example, the positioning module may be a GPS, and the drift distance of the GPS is d gps_max The damage location of the road damage information currently applied to be stored in the storage space is the longitude and latitude coordinates (lat new ,long new ), the damaged location of the road damage information stored in the storage space can be recorded as longitude and latitude coordinates (lat history ,long history ), as long as there is a longitude and latitude coordinate in the storage space that satisfies the longitude and latitude coordinates of the current application to store information:

[0063] distance((lat new ,long new ),(lat history ,long history )) <d gps_max

[0064] Then, the pavement damage information currently applied for storage does not meet the space constraint and cannot be stored; otherwise, it can be stored.

[0065] In a specific embodiment, in order to accurately estimate the current vehicle speed, the estimation process includes: obtaining the vehicle acceleration and angular velocity provided by the IMU (inertial measurement unit) and the vehicle speed provided by the GPS (global positioning system), and using Kalman filtering to fuse the data provided by the IMU and the data provided by the GPS to obtain the current vehicle speed that is ultimately used to update the constraint time.

[0066] Assume that the IMU provides acceleration a x 、a y and angular velocity ω z , GPS provides speed v gps , the subscripts x and y represent the components in the x-axis and y-axis directions respectively.

[0067] Assume that the sampling interval at the current moment is Δt, based on the acceleration a provided by the IMUx 、a y , the speed can be preliminarily predicted to be:

[0068]

[0069] In the formula, the superscript prev represents the previous moment, and the superscript pred represents the current moment.

[0070] Based on the speed v provided by GPS gps and direction angle θ gps , we can also make a preliminary prediction of the speed:

[0071]

[0072] Then, the Kalman gain K is used to fuse the speed information of IMU and GPS, and the updated speed is:

[0073]

[0074] Where K is the Kalman gain, which can be calculated by the error covariance matrix.

[0075] The calculation formula of Kalman gain K is:

[0076]

[0077] Where P pred is the forecast error covariance matrix, R gps is the GPS observation noise covariance, and the final state covariance is updated as:

[0078] P new =(IK)·P pred

[0079] Finally, the actual speed of the vehicle is:

[0080]

[0081] In a specific embodiment, before updating the constraint time t, the distance between each detection frame and the vehicle in the current road image is calculated, and the maximum distance is selected to update the constraint time; the method for calculating the distance between the detection frame and the vehicle includes:

[0082] Convert each vertex of the detection frame from the pixel coordinate system to the camera coordinate system;

[0083] Calculate the pixel area of ​​the detection frame in the pixel coordinate system and the actual area in the camera coordinate system respectively;

[0084] Calculate the distance between vehicles: distance between vehicles = (actual area / pixel area)*camera focal length.

[0085] Assume that the camera internal parameters are:

[0086]

[0087] In the formula, f x 、f y is the proportional coefficient of focal length f in the x and y directions, c x 、c y is the principal point coordinate, i.e. the optical center of the imaging plane.

[0088] In this embodiment, the Z axis of the camera coordinate system is the direction of vehicle travel, and the Y axis is the direction perpendicular to the road surface. The distance between the camera and the road surface is known, that is, the Y coordinate of the target (road damage) in the camera coordinate system is known, and the coordinate system conversion can be achieved by the following formula:

[0089]

[0090] Where X represents the X coordinate of the detected target in the camera coordinate system, Y represents the Y coordinate of the detected target in the camera coordinate system, Z represents the Z coordinate of the detected target in the camera coordinate system, u represents the horizontal coordinate of the detected target in the pixel coordinate system, and v represents the vertical coordinate of the detected target in the pixel coordinate system.

[0091] Through the above coordinate transformation, each vertex of the detection frame can be converted from pixel coordinates to camera coordinates, and then the length and width of the detection frame in the camera coordinate system can be calculated.

[0092] Calculate the length L and width W of the detection box in the camera coordinate system, and the final area is:

[0093] A=L×W

[0094] Where L and W are the length and width calculated from the transformed bounding box vertices.

[0095] The distance between the target and the camera is calculated based on the size of the bounding box, and the size A of the object in the camera coordinate system is used real and the bounding box size A in pixel coordinates bbox The distance d between the target and the camera is estimated by the proportional relationship:

[0096]

[0097] Where f is the focal length.

[0098] In this application, information is screened through step S3, where the time constraint takes into account that when the vehicle is traveling at a slow speed or is stationary (such as waiting for a traffic light), a large number of images of the same road damage will be continuously taken, resulting in repeated recognition by the computing platform. The introduction of time constraints can reduce the repeated recognition work caused by the above reasons to a certain extent, and the spatial constraint takes into account that the camera's shooting frame rate is usually very high, and when passing through the same road damage, multiple images of the road damage will be captured. By performing spatial constraints, it can be ensured that cracks within the drift range of the positioning module will not be counted multiple times, thereby reducing the repeated recognition work caused by the above reasons. Therefore, based on the time and space constraints of step S3, the burden on the computing platform and storage space can be reduced.

[0099] In step S4, further real-time analysis and monitoring are performed on the information stored in the storage space.

[0100] After screening in step S3, the stored information can be analyzed and monitored in real time, wherein the type of analysis of the information can be flexibly selected according to actual needs. For example, road damage data can be counted, damage location and damage status can be obtained, and damage changes can be monitored.

[0101] like Figure 3 The figure shows a schematic diagram of data analysis in an embodiment of the present invention, which can be displayed through a visual interface, for example, displaying a road damage data statistical graph, a real-time location graph, and a damage status graph, and listing the information of all detection points in detail through a list interface, supporting filtering and searching, and providing specific information and positioning maps of each detection point through a detailed interface. It is possible to provide decision-making suggestions by quantitatively analyzing road damage data to help maintenance personnel make quick decisions, and to provide real-time road damage risk maps to provide road risk planning information for unmanned vehicles. It is also possible to calculate the PCI of each road section based on the analyzed data, and to calculate the comprehensive road damage risk score of each road section by combining PCI, crack characteristics, and bump index. According to PCI and risk score, the damage level of the road section is classified and evaluated, and decision-making suggestions and maintenance priority lists are generated. Users can view, edit, and manage road damage data through the interface on the user side.

[0102] In this embodiment, the changes in cracks are monitored and tracked. When the crack area growth rate exceeds the set warning threshold, an early warning is automatically triggered. Specifically, based on the stored crack data, the changes in the crack area are regularly analyzed, and the speed and trend of crack expansion are calculated. If the crack area growth rate exceeds the set warning threshold, the system automatically triggers an early warning and pushes a prompt to the user end that there may be a risk of road collapse.

[0103] In one embodiment, when a deep learning model is used to identify the categories of road damage in various locations in a road surface image, it is considered that the deep learning model has limited adaptability to the environment. For example, after using the European road crack dataset, the detection capability of domestic roads will decrease, and even the model detection capabilities in different cities and different climatic environments in the country are different. Therefore, this solution also proposes a method for realizing active learning of the deep learning model. The goal of active learning is to optimize the model to adapt to the road conditions in different regions and cities, establish an efficient feedback mechanism, and realize continuous optimization of the model.

[0104] The following describes the process of active learning.

[0105] The first step is to obtain the uncertainty of each category obtained by training the existing deep learning model in a previous environment.

[0106] Specifically, the deep learning model obtains the uncertainty d for each category in historical training i k , the superscript i represents the historical training round, and the subscript k represents the category index. The categories include but are not limited to transverse cracks, longitudinal cracks, mesh cracks, pits, cracks, and repaired cracks.

[0107] The second step is to obtain road images taken by the vehicle-mounted camera in the new environment as the initial image set, and use the existing deep learning model to classify the damage category of each road image in the initial image set to obtain the category and confidence of the road damage at each location in each road image.

[0108] Specifically, the original photos are taken during the on-road application process of the equipment deployed in another city and input into the deep learning model for recognition, and the categories of road damage at each location and their category confidence p are obtained. j (k), k is the category index, and j is the jth road damage in the image.

[0109] In the third step, the confidence of each road surface damage in the road surface image is weighted and summed, and the weight is the uncertainty of the corresponding category to obtain the uncertainty of the corresponding road surface image.

[0110] Specifically, the uncertainty U of the image is calculated based on the uncertainty of the category and the confidence of the category of damage in each location in the image:

[0111]

[0112] Where J is the amount of road damage in the image.

[0113] In the fourth step, multiple road images with the lowest uncertainty are selected and the image similarity is calculated, and multiple road images with the highest similarity are eliminated from the initial image set to obtain a screened image set.

[0114] Specifically, when the vehicle is moving at high speed, in order to take a clear picture of the road, the camera frame rate is very high, which may generate a large number of images with extremely high similarity. On the one hand, these images will waste the computing power of the subsequent training stage, and on the other hand, they are easy to cause overfitting in the active learning process, so they need to be eliminated. In this step, by calculating the uncertainty of the image, some images with low credibility are preliminarily determined, and then the image similarity is further calculated, and images with high uncertainty and high similarity are deleted to improve the effect of subsequent training.

[0115] In specific operations, the high-dimensional features of the penultimate layer of the model network can be obtained to calculate the similarity of the images.

[0116] The fifth step is to train the existing deep learning model again based on the filtered image set.

[0117] It can be understood that after determining the valuable image set, the image set is annotated to form a new training set, which can be used for targeted fine-tuning training of the model, and after the training is completed, it can be put into subsequent deployment work.

[0118] Example 2

[0119] The present invention also relates to a road damage real-time monitoring system, such as Figure 4 The figure shows a system block diagram of a real-time road damage monitoring system in one embodiment of the present invention, which includes an edge device, a cloud server, and usually a user end. The edge device includes a vehicle-mounted camera, a sensor, a positioning module, an edge computing module, and a communication module that can be loaded on a vehicle; the vehicle-mounted camera is used to capture road images, the sensor is used to collect the vehicle's driving status, the positioning module is used to locate the vehicle, and the edge computing module is used to perform preliminary processing on the data collected by the edge device and transmit it to the cloud server through the communication module; the cloud server is used to receive the data transmitted by the edge device and perform road damage analysis and monitoring.

[0120] Specifically, the camera can be a high-frame rate industrial camera that captures road surface images in real time for damage detection; the edge computing module, as the core processing unit of the system, is responsible for data collection and preliminary processing; the sensor can include an IMU sensor to obtain data during vehicle driving; the positioning module can be a GPS module that provides high-precision geographic location information, and the communication module can be a 5G communication module to achieve high-speed data transmission.

[0121] Among them, the edge device or cloud server is used to execute the operation of storing information based on time and space constraints in the real-time monitoring method for road damage introduced in Example 1.

[0122] For example, the operations in Example 1 can be fully performed directly on the cloud server, including information screening and storage, further real-time analysis, and active learning of the deep learning module. The edge computing module only performs simple data sorting and transmission. At this time, the requirements for the edge computing module can be greatly reduced, but in places with poor network conditions or when the cloud server is under great pressure, it may be difficult to respond in time.

[0123] For another example, the edge device can also perform a storage operation with spatiotemporal constraints, and the cloud server can perform further complex operations such as data analysis and active learning. In this case, a sending queue can be established on the edge device and a historical queue can be established on the cloud server. The edge device stores information that meets the spatiotemporal constraints in the sending queue, and the sending queue gradually synchronizes the information to the historical queue of the cloud server. In a specific embodiment, a lightweight recognition model can be set on the edge device to preliminarily identify road damage, and a more powerful deep learning module can be set on the cloud server to accurately identify damage.

[0124] Example 3

[0125] The present invention also relates to a computing device for implementing real-time monitoring of road damage, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in Example 1 when executing the computer program.

[0126] The computing device for implementing real-time monitoring of road damage can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. In this embodiment, an AI engine and an NPU computing chip can be deployed. The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor implements various functions of the computing device for real-time monitoring of road damage by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory.

[0127] Example 4

[0128] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0129] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0130] Example 5

[0131] The embodiment of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the method of the above embodiment of the present invention.

[0132] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. It should be noted that "in one embodiment", "for example", "for example", etc. of the present invention are intended to illustrate the present invention, rather than to limit the present invention.

[0133] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. A method for real-time monitoring of road damage, characterized in that: include: Acquire the road image taken by the vehicle-mounted camera in real time and identify the road damage information at each location in the road image, and perform a storage operation of the road damage information that meets the time and space constraints, the storage operation includes: updating the constraint time according to the current vehicle speed and the maximum vehicle distance from the farthest road damage location in the current road image to the vehicle, the constraint time is the current maximum vehicle distance divided by the current vehicle speed, storing the road damage information in the current road image that meets the spatial constraint conditions into the storage space, and not allowing the road damage information of the new road image to be stored within the constraint time, until the storage operation is re-executed after the constraint time is reached; if the distance between the damage location of the road damage information currently applied to be stored in the storage space and the damage location of each road damage information stored in the storage space is greater than or equal to the preset positioning drift distance, then it is determined that the road damage information currently applied to be stored in the storage space meets the spatial constraint conditions; The stored road damage information is used for real-time analysis and monitoring of road damage.

2. The method for real-time monitoring of road damage according to claim 1, characterized in that: The identifying the road damage information at each location in the road image includes: framing the road damage at each location in the road image by using a detection frame and calculating the road damage information in each detection frame; Before updating the constraint time, the distance between each detection frame and the vehicle in the current road image is calculated, and the maximum distance is selected to update the constraint time; the method for calculating the distance between the detection frame and the vehicle includes: Convert each vertex of the detection frame from the pixel coordinate system to the camera coordinate system; Calculate the pixel area of ​​the detection frame in the pixel coordinate system and the actual area in the camera coordinate system respectively; Calculate the distance between vehicles: distance between vehicles = (actual area ÷ pixel area) × camera focal length.

3. The method for real-time monitoring of road damage according to claim 1, characterized in that: Before updating the constraint time, the current vehicle speed is determined first. The method for determining the current vehicle speed includes: The vehicle acceleration and angular velocity provided by the IMU and the vehicle speed provided by the GPS are obtained, and the data provided by the IMU and the GPS are fused using the Kalman filter to obtain the current vehicle speed that is ultimately used to update the constraint time.

4. The method for real-time monitoring of road damage according to claim 1, characterized in that: The method further includes identifying the category of road surface damage at various locations in the road surface image using a deep learning model, wherein the deep learning model is a trained deep learning model; The method further includes performing active learning of the deep learning model in a new environment, wherein the process of active learning includes: Obtain the uncertainty of each category obtained by training the existing deep learning model in a previous environment; Obtain road images taken by the vehicle-mounted camera in the new environment as the initial image set; Using an existing deep learning model to classify the damage category of each road surface image in the initial image set, and obtain the category and confidence of the road surface damage at each location in each road surface image; Perform weighted summation of the confidence of each category of road damage in the road surface image, with the weight being the uncertainty of the corresponding category, and obtain the uncertainty of the corresponding road surface image; Selecting multiple road surface images with the highest uncertainty and calculating image similarity, removing multiple road surface images with the highest similarity from the initial image set to obtain a filtered image set; The existing deep learning model is trained again based on the filtered image set.

5. The method for real-time monitoring of road damage according to claim 1, characterized in that: The method also includes performing a road damage analysis and presenting the analysis results through a visual interface.

6. The method for real-time monitoring of road damage according to claim 1, characterized in that: The road damage includes cracks, and the method further includes performing a road damage analysis, wherein the road damage analysis includes: The changes in cracks are tracked, and when the crack area growth rate exceeds the set warning threshold, an early warning is automatically triggered.

7. A road damage real-time monitoring system, characterized in that: Including edge devices and cloud servers; The edge device includes an on-board camera, a sensor, a positioning module, an edge computing module and a communication module that can be loaded on the vehicle; the on-board camera is used to capture road images, the sensor is used to collect vehicle driving status, the positioning module is used to locate the vehicle, and the edge computing module is used to perform preliminary processing on the data collected by the edge device and transmit it to the cloud server through the communication module; The cloud server is used to receive data transmitted by the edge device and perform road damage analysis and monitoring; Wherein, the edge device or the cloud server is used to execute the method as described in any one of claims 1 to 3.

8. A computing device for implementing real-time monitoring of road damage, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images