Road detection method and system

Abstract

A road detection method and system are disclosed. The road detection system includes an image acquisition device, an inertial measurement device, and a processor. The image acquisition device is used to acquire road images of the road that a vehicle passes through during driving. The inertial measurement device is used to acquire inertial measurement data during driving of the vehicle. The processor is used to analyze the road image to obtain a first detection result for characterizing the road surface flatness, analyze the inertial measurement data to obtain a second detection result for characterizing the road surface flatness, and combine the first detection result and the second detection result to obtain a third detection result for characterizing the road surface flatness. In this way, the shortcomings of a single detection method can be overcome and the quality of road disease detection can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of road damage detection, and in particular to a road detection method and system for detecting the smoothness of a road surface. Background Art

[0002] In urban municipal scenarios, "smooth roads, smooth water, bright lights, and stable bridges" are the main responsibilities of municipal departments. "Smooth roads" refers to daily patrols and maintenance of roads within the jurisdiction, timely discovery and repair of abnormalities such as road dents and cracks, and ultimately ensuring that the road surface is smooth. "Smooth roads" has become the primary focus of municipal departments as the most frequent and most influential work task.

[0003] In actual operation, a large number of grassroots inspectors are required to manually detect and report the road surface to find problems. This solution consumes a lot of manpower and is difficult to implement on roads with heavy traffic, which seriously affects work efficiency.

[0004] Therefore, a road detection solution is needed that can automatically detect the road flatness. Summary of the invention

[0005] A technical problem to be solved by the present disclosure is to provide a road detection solution that can automatically detect the road flatness.

[0006] According to a first aspect of the present disclosure, there is provided a road detection system, comprising: an image acquisition device for acquiring road images of a road passed by a vehicle during travel; an inertial measurement device for acquiring inertial measurement data during vehicle travel; and a processor for analyzing the road images to obtain a first detection result for characterizing the road surface smoothness, analyzing the inertial measurement data to obtain a second detection result for characterizing the road surface smoothness, and combining the first detection result and the second detection result to obtain a third detection result for characterizing the road surface smoothness.

[0007] According to a second aspect of the present disclosure, a road detection method is provided, comprising: obtaining a first detection result for characterizing the road surface smoothness based on a road image of the road passed by a vehicle during driving; obtaining a second detection result for characterizing the road surface smoothness based on inertial measurement data during vehicle driving; and obtaining a third detection result for characterizing the road surface smoothness by combining the first detection result and the second detection result.

[0008] According to a third aspect of the present disclosure, a road detection device is provided, including: a first analysis module, used to obtain a first detection result for characterizing the road surface flatness of the road based on a road image of the road passed by a vehicle during driving; a second analysis module, used to obtain a second detection result for characterizing the road surface flatness of the road based on inertial measurement data during the vehicle driving process; and a third analysis module, used to combine the first detection result and the second detection result to obtain a third detection result for characterizing the road surface flatness of the road.

[0009] According to a fourth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor executes the method described in the second aspect above.

[0010] According to a fifth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the second aspect above.

[0011] Therefore, the present invention obtains the final detection result of the road surface flatness used to characterize the road by simultaneously combining the road surface flatness detection result obtained based on road image data and the road surface flatness detection result obtained based on inertial measurement data, which can overcome the shortcomings of a single detection method and improve the quality of road disease detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0013] Figure 1 A schematic structural diagram of a road detection system according to an embodiment of the present disclosure is shown.

[0014] Figure 2 A schematic structural diagram of a road detection system according to another embodiment of the present disclosure is shown.

[0015] Figure 3 A schematic diagram of an application scenario according to an embodiment of the present disclosure is shown.

[0016] Figure 4 A schematic flow chart of a road detection method according to an embodiment of the present disclosure is shown.

[0017] Figure 5 A schematic diagram of the workflow of the road damage identification system is shown.

[0018] Figure 6A structural block diagram of a road detection device according to an embodiment of the present disclosure is shown.

[0019] Figure 7 A structural block diagram of a computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0021] Figure 1 A schematic structural diagram of a road detection system according to an embodiment of the present disclosure is shown.

[0022] See also Figure 1 The road detection system 100 includes an image acquisition device 110 , an inertial measurement device 120 and a processor 130 .

[0023] The image acquisition device 110 and the inertial measurement device 120 are arranged on the vehicle. The processor 130 can be arranged on the vehicle together with the image acquisition device 110 and the inertial measurement device 120. For example, the processor 130 can be implemented as a vehicle-mounted central control machine. The processor 130 can also be not arranged on the vehicle. For example, the processor 130 can be implemented as a server, and the server can obtain the data collected by the image acquisition device 110 and the inertial measurement device 120 through wireless communication.

[0024] The image acquisition device 110 may be any device having an image acquisition function (ie, an imaging function), such as but not limited to an image sensor. As an example, the image acquisition device 110 may be a wide-angle camera with a wide imaging range.

[0025] The image acquisition device 110 is used to acquire road images of the roads that the vehicle passes through during driving. The image acquisition device 110 can be configured to acquire road images in a fixed direction relative to the vehicle. For example, the image acquisition device 110 can be configured to acquire road images obliquely below the front of the vehicle, or to acquire road images obliquely below the rear of the vehicle in the data acquisition mode. Thus, by controlling the image acquisition device 110 to continuously acquire images during driving of the vehicle, road images of the roads that the vehicle passes through during driving can be obtained.

[0026] The inertial measurement device 120 refers to a device capable of measuring the inertia of an object. Inertia is an inherent property of an object, and is the property of an object to resist changes in its motion state.

[0027] In the present disclosure, the inertial measurement device 120 is used to measure the inertia of the vehicle during driving to collect inertial measurement data during the vehicle driving. The inertia of the vehicle refers to the property of the vehicle to resist the change of its motion state by the road disease such as uneven road surface.

[0028] Therefore, the inertial measurement data refers to data that can be used to determine the road surface smoothness of the road that the vehicle passes through during the driving process, that is, data that can directly or indirectly reflect the road surface smoothness of the road that the vehicle passes through during the driving process.

[0029] The vertical acceleration data can reflect the vibration of the vehicle during the road driving process, and the vibration of the vehicle is mainly caused by the uneven road surface. Therefore, the acceleration data can indirectly reflect the flatness of the road surface. In other words, the acceleration data can be used to judge the flatness of the road surface.

[0030] When a vehicle is driving on an uneven road, the components of gravity acceleration on different coordinate axes in a specific coordinate system (such as a spatial rectangular coordinate system with the plane where the vehicle chassis is located as the X-axis, the Y-axis as the coordinate plane, and the direction perpendicular to the vehicle chassis as the Z-axis) will change. The smoothness of the road surface can also be judged based on the changes in gravity acceleration on different coordinate axes. That is, the gravity acceleration data in a specific coordinate system can also be used to judge the smoothness of the road surface.

[0031] When a vehicle is driving on a flat road, the angle between the vehicle's direction of travel and the horizontal direction is zero or almost zero. When the road surface is concave or sloped, there will be a certain angle between the vehicle's direction of travel and the horizontal direction. The angle data between the vehicle's direction of travel and the horizontal direction during driving can also be used to determine the flatness of the road surface.

[0032] Therefore, the inertial measurement data may include, but is not limited to: vertical acceleration data during vehicle driving, gravity acceleration data during vehicle driving, and angle data between the vehicle's traveling direction and the horizontal direction during vehicle driving.

[0033] The specific structure of the inertial measurement device varies according to the type of inertial measurement data. As an example, the inertial measurement device may include, but is not limited to, one or more combinations of an acceleration sensor, a gravity sensor, and a direction sensor. Among them, the acceleration sensor is used to detect the acceleration data in the vertical direction during the vehicle's driving process; the gravity sensor is used to detect the gravity acceleration data during the vehicle's driving process; and the direction sensor is used to detect the angle data between the vehicle's driving direction and the horizontal direction during the vehicle's driving process.

[0034] The processor 130 may be connected to the image acquisition device 110 and the inertial measurement device 120 by wire or wirelessly, respectively, to obtain the road image acquired by the image acquisition device 110 and the inertial measurement data acquired by the inertial measurement device 120 .

[0035] In the process of using the image acquisition device 110 to capture road images of the roads that the vehicle passes through during driving, it is inevitable that the road image covering the entire road surface cannot be obtained due to obstruction. This causes a problem of perception loss when analyzing the road surface flatness based on the road image, that is, it is impossible to obtain the road surface flatness detection results of all roads.

[0036] The inertial measurement data collected by the inertial measurement device 120 can directly or indirectly reflect the road surface smoothness. However, compared with image data, the inertial measurement data has the problem of too small a perception domain, that is, some inertial measurement data may not be able to accurately deduce the road surface smoothness. This also causes the problem of perception loss when analyzing the road surface smoothness based on the inertial measurement data.

[0037] To this end, the present disclosure proposes that the road surface smoothness of the road can be determined by combining the road image and the inertial measurement data at the same time. That is, the processor 130 can determine the road surface smoothness of the road based on the road image and the inertial measurement data at the same time. Determining the road surface smoothness is to obtain a detection result for characterizing the road surface smoothness of the road (corresponding to the third detection result described below).

[0038] Specifically, the processor 130 can analyze the road image to obtain a first detection result for characterizing the road surface smoothness; analyze the inertial measurement data to obtain a second detection result for characterizing the road surface smoothness; and then combine the first detection result and the second detection result to obtain a third detection result for characterizing the road surface smoothness. The third detection result can be used as the final detection result.

[0039] The following is an exemplary description of the process of obtaining the first detection result, the second detection result, and the third detection result.

[0040] 1. First test result

[0041] In order to reduce the impact of non-road areas on the detection results, the processor 130 can segment the road image before analyzing the road image collected by the image acquisition device 110 to remove the non-road area in the road image. That is, the image areas in the road image after segmentation are all road areas. Among them, a machine learning model for identifying the contour of the road area in the image can be pre-trained, and the road image is segmented using the machine learning model. The machine learning model can be a segmentation network based on CNN (Convolutional Neural Networks) and FCN (Fully Convolutional Networks), such as a deep learning segmentation network Unet.

[0042] The processor 130 may analyze the segmented road image to obtain a first detection result for characterizing the road surface smoothness.

[0043] By using the image acquisition device 110 to acquire images of the road that the vehicle passes through during driving, a plurality of road images corresponding to different moments can be obtained. The processor 130 can segment each road image and analyze the segmented road images to obtain the detection results of the road surface flatness of the road area in each road image. The road areas in different road images are generally roads at different positions. Therefore, the first detection result can include a plurality of road surface flatness detection results corresponding to different road positions. For the sake of distinction, the road surface flatness detection result mentioned here can be referred to as the first road surface flatness detection result.

[0044] The first road surface roughness detection result may be numerical data used to characterize the road surface roughness of the road area in the road image, or data used to indicate the category to which the road surface in the road area in the road image belongs. The category refers to a category that can characterize the road surface roughness of the road, and may include but is not limited to potholes, cracks, flatness, bumps, and the like.

[0045] As an example, the processor 130 may use image detection technology to detect road damage on road images, and the detection content includes road cracks, potholes, bumps, etc. The image detection technology may be, but is not limited to, an object detection algorithm based on a deep neural network, such as YOLO and RCNN.

[0046] 2. Second test result

[0047] Before analyzing the inertial measurement data, the processor 130 may filter the inertial measurement data to eliminate interference. For example, the processor 130 may filter high-frequency burr noise through a Butterworth low-pass filter to prevent the influence of external factors such as thermal noise of the inertial measurement device 120.

[0048] The processor 130 may analyze the filtered inertial measurement data to obtain a second detection result for characterizing the road surface smoothness.

[0049] The inertial measurement device 120 can be used to obtain multiple inertial measurement data at different times. The processor 130 can analyze the inertial measurement data at different times to obtain multiple road surface flatness detection results. For the sake of distinction, the road surface flatness detection result mentioned here can be referred to as a second road surface flatness detection result. That is, the second detection result can include multiple second road surface flatness detection results.

[0050] The processor 130 may analyze the inertial measurement data according to the specific data type of the inertial measurement data and adopt an analysis method that matches the data type to obtain the second detection result.

[0051] Taking the inertial measurement data as acceleration data in the vertical direction as an example, after completing the filtering process, the processor 130 can divide the acceleration data into multiple acceleration data sequences, each acceleration data sequence includes one or more acceleration values; then perform feature extraction on the acceleration data sequence to obtain the feature extraction result of the acceleration data sequence; finally, input the feature extraction result into a pre-trained recognition model to obtain the prediction result output by the recognition model for characterizing the road surface flatness of the road corresponding to the acceleration data sequence.

[0052] The data length of the divided acceleration data sequence can be set according to actual conditions or empirical values. The feature extraction results may include but are not limited to one or any combination of features such as mean, variance, 1 / 4 median, 3 / 4 median, etc. The recognition model can be a classification model for predicting the category of the road surface. The category refers to the category that can characterize the road surface flatness, which may include but is not limited to potholes, cracks, flatness, bumps, etc. As an example, the recognition model can be but is not limited to a model based on a k-means clustering algorithm, a classification model based on a support vector machine (SVM), a classification model based on a Resnet classification network, and the like.

[0053] Taking the inertial measurement data as gravity acceleration data as an example, the inertial measurement device 120 may be a gravity sensor. The processor 130 may calculate the inclination angle of the vehicle relative to the horizontal plane based on the acceleration caused by gravity measured by the gravity sensor during the vehicle's driving process, and then obtain the flatness of the road surface. Among them, the gravity acceleration data collected by the gravity sensor may refer to the acceleration data of gravity acceleration on different coordinate axes in a specific coordinate system (such as a spatial rectangular coordinate system with the plane where the vehicle chassis is located as the X-axis, the coordinate plane where the Y-axis is located, and the direction perpendicular to the vehicle chassis as the Z-axis).

[0054] Taking the inertial measurement data as the angle data between the traveling direction and the horizontal direction during the vehicle's driving process as an example, the inertial measurement device 120 may be a direction sensor. The processor 130 may determine the road surface flatness of the road the vehicle is traveling through based on the angle data measured by the direction sensor during the vehicle's driving process.

[0055] 3. The third test result

[0056] The third detection result can be used as the final detection result of the road surface smoothness.

[0057] The first detection result can be used as a benchmark detection result, and the second detection result can be used to fill in the gaps in the first detection result. That is, for a road section where the first detection result is missing due to road obstruction or other reasons, the second detection result of the road section can be used as the first detection result, and finally a third detection result including the first detection result and the second detection result can be obtained.

[0058] The second detection result can also be used as a benchmark detection result, and the first detection result can be used to fill in the gaps in the second detection result. That is, for a road section where the second detection result is missing due to reasons such as the perception domain, the first detection result of the road section can be used as the second detection result, and finally a third detection result including the second detection result and the first detection result can be obtained.

[0059] As an example, the process of combining the first detection result and the second detection result to obtain the third detection result may refer to the process of merging the first detection result and the second detection result. That is, the detection results corresponding to the same road position in the first detection result and the second detection result (i.e., the first road surface flatness detection result and the second road surface flatness detection result described below) are deduplicated, and the first road surface flatness detection results and the second road surface flatness detection results corresponding to different road positions are retained.

[0060] For example, the time when the image acquisition device 110 acquires each road image and the time when the inertial measurement device 120 acquires the inertial measurement data can be recorded respectively, and then the acquisition time of the road image on which the first road surface roughness detection result depends and the acquisition time of the inertial measurement data on which the second road surface roughness detection result depends can be determined, and then the first road surface roughness detection result and the second road surface roughness detection result corresponding to the same time or the same time period can be merged to obtain a third detection result.

[0061] For example, Figure 2 As shown, the road detection system 100 may further include a positioning device 140. The positioning device 140 is used to determine the position information of the vehicle during the vehicle's travel. The positioning device 140 may be, but is not limited to, a GPS positioning device.

[0062] The processor 130 may map the first result and the second detection result back to the physical space according to the positioning information obtained by the positioning device 140, that is, determine the road position corresponding to the first road surface roughness detection result and the road position corresponding to the second road surface roughness detection result. For example, the processor 130 may determine the road position corresponding to the first road surface roughness detection result according to the acquisition time of the road image on which the calculation of the first road surface roughness detection result depends and the position information of the vehicle at the acquisition time; the processor 130 may determine the road position corresponding to the second road surface roughness detection result according to the acquisition time of the inertial measurement data on which the calculation of the second road surface roughness detection result depends and the position information of the vehicle at the acquisition time.

[0063] After mapping the first detection result and the second detection result back to the physical space respectively, the processor 130 can deduplicate the first road surface flatness detection result and the second road surface flatness detection result corresponding to the same road position, that is, retain only one road surface flatness detection result, and retain the first road surface flatness detection result and the second road surface flatness detection result corresponding to different road positions.

[0064] After obtaining the third detection result, the processor 130 may also map the third detection result to the road image to obtain a road heat map that can reflect the road surface smoothness. In this way, the road surface smoothness detection result of the road that the vehicle passes through can be visually presented. In the road heat map, different types of road surface diseases such as potholes, cracks, and bumps can be represented by different visualization parameters such as different colors and brightness.

[0065] Figure 3 A schematic diagram of an application scenario according to an embodiment of the present disclosure is shown.

[0066] like Figure 3As shown, the road detection system can be deployed on a vehicle that can travel along a predetermined direction of travel on a road to realize a vehicle-mounted road detection system.

[0067] During the driving process of the vehicle, the road detection system can use the image acquisition device and the inertial measurement device to collect the road image and inertial measurement data of the road in real time. By analyzing the collected road image and inertial measurement data, the road surface flatness detection result P of different road sections during the driving process determined based on the road image (i.e., the first detection result mentioned above) and the road surface flatness detection result S of different road sections during the driving process determined based on the inertial measurement data (i.e., the second detection result mentioned above) can be obtained respectively.

[0068] Due to problems such as road occlusion during image acquisition and too small a perceptual domain in inertial measurement, it is impossible to obtain P and S for the entire road section, that is, when detecting the road surface flatness of the entire road, there are some sections where P or S is missing. Therefore, for sections where a certain detection result is missing (such as missing P or missing S), the road surface flatness detection result R of the section can be obtained based on another detection result. For sections where both detection results are not missing, any one detection result (P or S) can be selected as the road surface flatness detection result R of the section, or the two detection results can be combined to obtain the road surface flatness detection result R of the section.

[0069] Optionally, if the two detection results for the same road section are inconsistent, the road section may be marked as a suspect road section, and the road surface smoothness of the suspect road section may be determined by manual detection.

[0070] Optionally, if both detection results for the same road section are missing, the road section may be marked as a missing road section, and the road surface smoothness of the missing road section may be determined by manual detection.

[0071] The present disclosure may also be implemented as a road detection method. Figure 4 A schematic flow chart of a road detection method according to an embodiment of the present disclosure is shown. Figure 4 The method shown can be executed by a module or device having a data processing function, such as the above Figure 1 or Figure 2 The processor 130 shown in FIG. 1 performs Figure 4 The following is a schematic description of the main steps of the road detection method, and the details involved can be found in the above description.

[0072] See also Figure 4In step S410, based on the road image of the road that the vehicle passes through during driving, a first detection result for characterizing the road surface flatness is obtained. The first detection result and its acquisition process can be referred to the above related description, which will not be repeated here.

[0073] The road image may be a road image of the road that the vehicle passes through during driving, which is acquired in real time, i.e., a real-time road image. The road image may also be a road image of the road that the vehicle passes through during driving, which is acquired previously, i.e., an offline road image.

[0074] For example, an image acquisition device disposed on a vehicle may be used to acquire a road image of a road that the vehicle passes through during driving, and the road image acquired in real time may be acquired from the image acquisition device. Alternatively, the road image acquired by the image acquisition device may also be stored in a storage device, and the road image may be acquired from the storage device. For the image acquisition device, please refer to the above description, which will not be repeated here.

[0075] In step S420, based on the inertial measurement data during the vehicle driving process, a second detection result for characterizing the road surface flatness is obtained. The second detection result and its acquisition process can be referred to the above related description, which will not be repeated here.

[0076] The inertial measurement data may be inertial measurement data collected in real time during the vehicle's driving process, i.e., real-time inertial measurement data. The inertial measurement data may also be inertial measurement data collected previously during the vehicle's driving process, i.e., offline inertial measurement data.

[0077] For example, an inertial vehicle device provided on the vehicle can be used to measure the inertia of the vehicle during driving, so as to collect inertial measurement data during the driving of the vehicle, and obtain the inertial measurement data collected in real time from the inertial measurement device. Alternatively, the inertial measurement data collected by the inertial measurement device can also be stored in a storage device, and the inertial measurement data can be obtained from the storage device. For the inertial measurement device, please refer to the relevant description above, which will not be repeated here.

[0078] Step S410 and step S420 may be performed simultaneously in any order.

[0079] In step S430, the first detection result and the second detection result are combined to obtain a third detection result for characterizing the road surface smoothness.

[0080] As an example, the first detection result can be used as a benchmark detection result, and the second detection result can be used to fill in the gaps in the first detection result. That is, for a road section where the first detection result is missing due to road obstruction or other reasons, the second detection result of the road section can be used as the first detection result, and finally a third detection result including the first detection result and the second detection result can be obtained.

[0081] As an example, the second detection result can also be used as a benchmark detection result, and the first detection result can be used to fill in the gaps in the second detection result. That is, for a road section where the second detection result is missing due to reasons such as the perception domain, the first detection result of the road section can be used as the second detection result, and finally a third detection result including the second detection result and the first detection result is obtained.

[0082] As an example, the process of combining the first detection result and the second detection result to obtain the third detection result may be a process of merging the first detection result and the second detection result. That is, the detection results corresponding to the same road position in the first detection result and the second detection result (i.e., the first road surface flatness detection result and the second road surface flatness detection result) are merged, and the first road surface flatness detection result and the second road surface flatness detection result corresponding to different road positions are retained.

[0083] The specific process of obtaining the first test result, the second test result and the third test result can be found in the above description and will not be repeated here.

[0084] The present disclosure can be implemented as a road damage recognition system for identifying potholes, cracks, bumps and other road damages. The road damage recognition system can be partially or completely mounted on a vehicle. For the structure of the road damage recognition system, please refer to the above Figure 1 , Figure 2 shown.

[0085] Figure 5 The schematic diagram of the workflow of the road damage identification system is shown. Among them, step S511, step S521 and step S531 can be performed simultaneously without any particular order. That is, after the road damage identification system is started, step S511, step S521 and step S531 can be performed simultaneously.

[0086] Steps S511 to S513 are mainly used for road damage identification based on image technology; steps S521 to S524 are mainly used for road damage identification based on inertial sensing technology; steps S531, S540, and S550 are mainly used for fusing the road damage identification results based on image technology (corresponding to the first detection result mentioned above) with the road damage identification results based on inertial sensing technology (corresponding to the second detection result mentioned above) to generate a road damage heat map that can reflect road damage such as potholes, cracks, and bumps.

[0087] The following is a brief description of the workflow of the road damage identification system.

[0088] The road hazard identification system can be installed on a municipal patrol car to realize an intelligent vehicle-mounted road hazard identification system. As an example, the road hazard identification system can include an on-vehicle wide-angle camera, an accelerometer sensor, a GPS module, and an on-vehicle central control unit.

[0089] The patrol car turns on the data collection mode during the daily patrol process. In the data collection mode, steps S511, S521 and S531 are executed. In step S511, the wide-angle camera records the image of the front of the car in real time; in step S521, the accelerometer collects the output signal of the vertical direction of the physical coordinate system in real time; in step S531, the GPS module collects the geographical location information in real time.

[0090] The vehicle-mounted central control machine processes the output of various sensors in real time. The following describes the specific processing details of each type of sensor separately.

[0091] 1) Image data

[0092] In step S512, the road may first be segmented in real time using a deep learning segmentation network Unet to eliminate the influence of non-road areas on the results.

[0093] In step S513, image detection technology (such as YOLO, RCNN) is used to detect road damage in the segmented road area, and the detection content includes road cracks, potholes, bumps, etc.

[0094] 2) Acceleration data

[0095] In step S522, firstly, a filter (such as a Butterworth low-pass filter or a Chebyshev filter) may be used to filter high-frequency burr noise to prevent the influence of external factors such as sensor thermal noise.

[0096] In step S523, after the filtering preprocessing is completed, the data is divided into frames, and features are extracted based on the data of each frame. The extracted features may include mean, variance, 1 / 4 and 3 / 4 medians, etc.

[0097] In step S524, the pre-trained classification model can finally be used to classify the above features and identify signals such as pits and cracks.

[0098] 3) Positioning data

[0099] In step S540, the image-based detection results and the acceleration-based detection results can be mapped back to the physical space based on the point information recorded in real time by the GPS module. Specifically, after a patrol is completed, the image-based detection results and the acceleration-based detection results can be geospatially merged and deduplicated.

[0100] In step S550, after mapping the GPS data to the physical space, a heat map of road damage on the patrol path can be obtained, thereby generating road maintenance tasks and realizing a closed loop of work tasks.

[0101] In summary, the road disease automatic identification system of the disclosed embodiment records information about the road surface that the municipal patrol car passes through through the vehicle camera, inertial sensor module and GPS module; and based on time series analysis, image analysis and machine learning technology, the vehicle-mounted central control machine realizes automatic analysis and calculation to obtain road disease hot spots; and finally superimposes road GIS information to realize automatic identification and automatic diagnosis of road diseases. In this way, it can effectively save municipal maintenance costs and promptly prevent the damage to driving safety caused by road diseases.

[0102] In addition, the present disclosure combines image analysis and inertial sensing technology at the same time. Even when there is a road occlusion problem, inertial sensing can be used to collect and analyze signals to complete the assessment of road damage. Compared with video analysis, inertial sensing also has the problem of too small a perception domain. Therefore, it is most appropriate to combine the two solutions and output a comprehensive assessment at the same time.

[0103] The road detection method disclosed in the present invention may also be implemented as a road detection device. Figure 6 The structure block diagram of the road detection device according to the exemplary embodiment of the present disclosure is shown. Among them, the functional units of the road detection device can be implemented by hardware, software or a combination of hardware and software that implements the principles of the present disclosure. It can be understood by those skilled in the art that Figure 6 The functional units described may be combined or divided into sub-units to implement the principles of the above invention. Therefore, the description herein may support any possible combination, division, or further limitation of the functional units described herein.

[0104] The following is a brief description of the functional units that the road detection device may have and the operations that each functional unit may perform. For the details involved, please refer to the relevant description above and will not be repeated here.

[0105] See also Figure 6 The road detection device 600 includes a first analysis module 610 , a second analysis module 620 and a third analysis module 630 .

[0106] The first analysis module 610 is used to obtain a first detection result for characterizing the smoothness of the road surface based on a road image of the road that the vehicle passes through during driving.

[0107] The second analysis module 620 is used to obtain a second detection result for characterizing the road surface smoothness of the road based on the inertial measurement data during the vehicle's driving process.

[0108] The third analysis module 630 is used to combine the first detection result and the second detection result to obtain a third detection result used to characterize the road surface flatness of the road.

[0109] Regarding the specific implementation process of the third analysis module 630 combining the first detection result and the second detection result to obtain the third detection result, please refer to the relevant description above, which will not be repeated here.

[0110] As an example, the process of the third analysis module 63 combining the first detection result and the second detection result to obtain the third detection result may refer to the process of merging the first detection result and the second detection result to remove duplicates. That is, the detection results corresponding to the same road position in the first detection result and the second detection result (i.e., the first road surface flatness detection result and the second road surface flatness detection result) are merged, and the first road surface flatness detection result and the second road surface flatness detection result corresponding to different road positions are retained.

[0111] Figure 7 A schematic diagram of the structure of a computing device that can be used to implement the above road detection method according to an embodiment of the present invention is shown.

[0112] See also Figure 7 , the computing device 700 includes a memory 710 and a processor 720 .

[0113] The processor 720 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 720 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 720 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0114] The memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 720 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 710 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0115] The memory 710 stores executable codes, and when the executable codes are processed by the processor 720 , the processor 720 can execute the road detection method mentioned above.

[0116] The road detection system, the road detection method, the road detection device, and the computing device according to the present disclosure have been described above in detail with reference to the accompanying drawings.

[0117] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0118] Alternatively, the present disclosure may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present disclosure.

[0119] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0120] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system and method according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A road detection system, comprising: An image acquisition device, used to acquire a road image of a road that a vehicle passes through during driving, wherein the road image includes an image of a front obliquely lower portion of the vehicle and an image of a rear obliquely lower portion of the vehicle; An inertial measurement device for collecting inertial measurement data while the vehicle is in motion; and a processor, configured to analyze the road image to obtain a first detection result for characterizing the road surface smoothness of the road, analyze the inertial measurement data to obtain a second detection result for characterizing the road surface smoothness of the road, and combine the first detection result and the second detection result to obtain a third detection result for characterizing the road surface smoothness of the road; The processor is further configured to filter the inertial measurement data to eliminate interference before analyzing the inertial measurement data; The processor is further configured to map the third detection result to a road image to obtain a road thermal map capable of reflecting the road surface smoothness; Among them, the first detection result includes multiple first road surface flatness detection results, and the second detection result includes multiple second road surface flatness detection results; the processor is also used to deduplicate the first road surface flatness detection results and the second road surface flatness detection results having the same road position, and retain the first road surface flatness detection results and the second road surface flatness detection results corresponding to different road positions to obtain the third detection result.

2. The road detection system according to claim 1, further comprising: The positioning device is used to determine the position information of the vehicle during its driving process. The processor is specifically configured to determine, based on the position information, a road position corresponding to the first road surface flatness detection result and a road position corresponding to the second road surface flatness detection result.

3. The road detection system according to claim 1, wherein: The processor is further configured to segment the road image before analyzing the road image to remove non-road areas in the road image.

4. The road detection system according to claim 1, wherein: The inertial measurement device includes at least one of the following: An acceleration sensor is used to detect the acceleration data in the vertical direction during the vehicle's driving process; Gravity sensor, used to detect gravity acceleration data during vehicle driving; The direction sensor is used to detect the angle data between the vehicle's traveling direction and the horizontal direction during driving.

5. The road detection system according to claim 4, wherein: The inertial measurement device includes an acceleration sensor, The processor is specifically used to divide the acceleration data into multiple acceleration data sequences, each of which includes one or more acceleration values, perform feature extraction on the acceleration data sequence to obtain feature extraction results of the acceleration data sequence, input the feature extraction results into a pre-trained recognition model, and obtain a prediction result output by the recognition model for characterizing the road surface smoothness of the road corresponding to the acceleration data sequence, wherein the prediction result is the second detection result.

6. The road detection system according to claim 5, wherein: The recognition model is a classification model for predicting the categories of road surfaces, and the categories include: potholes, cracks, flatness, and bumps.

7. A road detection method, comprising: Based on a road image of a road that the vehicle passes through during driving, a first detection result for characterizing the road surface flatness of the road is obtained, wherein the road image includes an image of a front obliquely lower portion of the vehicle and an image of a rear obliquely lower portion of the vehicle; Based on the inertial measurement data during the vehicle's driving process, a second detection result for characterizing the smoothness of the road surface of the road is obtained; Combining the first detection result and the second detection result to obtain a third detection result for characterizing the smoothness of the road surface of the road; Wherein, before obtaining a second detection result for characterizing the road surface flatness of the road based on the inertial measurement data during the vehicle driving process, the method further includes: filtering the inertial measurement data; The method further includes: mapping the third detection result to a road image to obtain a road heat map capable of reflecting the road surface flatness; Among them, the first detection result includes multiple first road surface flatness detection results, and the second detection result includes multiple second road surface flatness detection results; the first detection result and the second detection result are combined to obtain a third detection result for characterizing the road surface flatness of the road, including: deduplicating the first road surface flatness detection result and the second road surface flatness detection result having the same road position, and retaining the first road surface flatness detection result and the second road surface flatness detection result corresponding to different road positions to obtain the third detection result.

8. A road detection device, comprising: A first analysis module is used to obtain a first detection result for characterizing the flatness of the road surface of the road based on a road image of the road passed by the vehicle during driving, wherein the road image includes an image of the oblique lower part of the front of the vehicle and an image of the oblique lower part of the rear of the vehicle; A second analysis module, configured to obtain a second detection result for characterizing the smoothness of the road surface of the road based on the inertial measurement data during the vehicle's driving process; and a third analysis module, configured to combine the first detection result and the second detection result to obtain a third detection result for characterizing the road surface smoothness of the road; The second analysis module is further used to: filter the inertial measurement data before obtaining the second detection result for characterizing the road surface flatness of the road based on the inertial measurement data during the vehicle driving process; The third analysis module is further used to: map the third detection result to a road image to obtain a road heat map that can reflect the road surface flatness; Among them, the first detection result includes multiple first road surface flatness detection results, and the second detection result includes multiple second road surface flatness detection results; the third analysis module is also used to: deduplicate the first road surface flatness detection results and the second road surface flatness detection results with the same road position, and retain the first road surface flatness detection results and the second road surface flatness detection results corresponding to different road positions to obtain the third detection result.

9. A computing device comprising: processor; as well as A memory having executable codes stored therein, which, when executed by the processor, causes the processor to execute the method according to claim 7. 10 . A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to claim 7 .

Images