Highway surface defect detection method and system based on computer vision

Abstract

The present application provides a road surface defect detection method and system based on computer vision, which relates to the technical field of road defect detection. When a suspected road surface pothole area is identified in a first computer vision image, a first image feature of the suspected road surface pothole area is extracted; when a suspected road surface pothole area is identified in a second computer vision image, a second image feature of the suspected road surface pothole area is extracted; the ambient light intensity, the first image feature, the second image feature, the first distance between the suspected road surface pothole area and the vehicle at the first moment, the second distance between the suspected road surface pothole area and the vehicle at the second moment, and the image change feature of the suspected road surface pothole area at the second distance relative to the first distance are input into a road surface pothole determination model to determine a first probability that a road surface defect exists; when the first probability is greater than a probability threshold, it is determined that the road surface has a defect.

Description

Technical Field

[0001] The present application relates to the technical field of highway defect detection, and in particular to a highway surface defect detection method and system based on computer vision. Background Art

[0002] With the development of science and technology, people's requirements for driving experience are getting higher and higher, and autonomous driving has come into being. Autonomous driving can free the driver's hands and realize the automatic driving of the vehicle. For example, when the vehicle detects that there are defects on the road ahead, it will drive around the defects. Therefore, the vehicle needs to accurately detect road defects.

[0003] At present, road surface defects include potholes on the road surface, and road surface potholes include black pit areas or water pit areas. It is understandable that when a vehicle is driving, it is necessary to identify whether there are potholes on the road surface ahead at a long distance in advance. If the ambient light is dim (such as at night), the vehicle cannot accurately identify whether there are potholes on the road ahead. For example, it is easy to identify the black pit area or water pit area in front as a shadow cast by an object (signal board, tree trunk, etc.) under light (such as street lamps, moonlight), causing the vehicle to fail to bypass the pothole in time, posing a safety hazard. Summary of the invention

[0004] The present application provides a road surface defect detection method and system based on computer vision, which is used to solve the problem in the prior art that when the ambient light is dim (such as at night), the vehicle cannot accurately identify whether there is a pothole in the road ahead.

[0005] In a first aspect, the present application provides a road surface defect detection method based on computer vision, which is applied to an ECU of a vehicle in motion, and the ECU is electrically connected to an ambient light sensor and a front camera of the vehicle respectively. The method provided by the present application includes:

[0006] When the headlights of the vehicle are turned on and the ambient light intensity transmitted by the ambient light sensor is lower than a set light intensity threshold, receiving a first computer vision image in front of the vehicle transmitted from the front camera at a first moment;

[0007] When a suspected road surface pothole area is identified in the first computer vision image, extracting a first image feature of the suspected road surface pothole area, and identifying a first distance between the suspected road surface pothole area in the first computer vision image and the vehicle at a first moment, wherein the pothole area includes a black pothole area or a water puddle area;

[0008] receiving, at a second moment after a preset time period, a second computer vision image in front of the vehicle transmitted from the front camera;

[0009] When a suspected road surface pothole area is identified in the second computer vision image, extracting a second image feature of the suspected road surface pothole area in the second computer vision image, and identifying a second distance between the suspected road surface pothole area in the second computer vision image and the vehicle at a second moment;

[0010] Determine, based on the first image feature and the second image feature, an image change feature of the suspected road surface pothole area at the second distance relative to the first distance;

[0011] Inputting the ambient light intensity, the first image feature of the suspected road surface pothole area, the second image feature of the suspected road surface pothole area, the first distance, the second distance, and the image change feature into a pre-trained road surface pothole determination model to determine a first probability that there are defects on the road surface, wherein the road surface pothole determination model is obtained by inputting a training sample set constructed by multiple historical ambient light intensities, respectively corresponding first image features of historical suspected road surface pothole areas, respectively corresponding second image features of historical suspected road surface pothole areas, respectively corresponding historical first distances, respectively corresponding historical second distances, and respectively corresponding historical image change features into an initial network to be trained;

[0012] When the first probability is greater than a set probability threshold, it is determined that there are defects on the road surface.

[0013] In a possible implementation, after determining that there are defects on the road surface when the first probability is greater than a set probability threshold, the method provided by the present application further includes:

[0014] identifying contours of suspected road surface pothole areas in the second computer vision image;

[0015] generating images of standard road surface pothole areas of different pothole types under the ambient light intensity and the second distance according to the ambient light intensity, the contour of the suspected road surface pothole area in the second computer vision image, and the second distance, wherein the contour of the image of the standard road surface pothole area of ​​each pothole type is the same as the contour of the suspected road surface pothole area in the second computer vision image;

[0016] Determine the mean and covariance of the difference between the pixel points of the suspected road surface pothole area and the pixel points corresponding to the image of the standard road surface pothole area for each pothole type;

[0017] Determine the confidence level of matching the image of the standard road surface pothole area of ​​each pothole type with the suspected road surface pothole area according to the average value and covariance of the difference values ​​corresponding to each suspected road surface pothole area, the average value of the largest difference and the average value of the smallest difference among the average values ​​of the multiple difference values, and the largest covariance and the smallest covariance among the multiple covariances;

[0018] Find the second probability that there are defects on the road surface according to the highest confidence level;

[0019] The first probability is modified according to the second probability.

[0020] In a possible implementation, when the headlights of the vehicle are turned on and the ambient light intensity transmitted by the ambient light sensor is lower than a set light intensity threshold, before receiving a first computer vision image in front of the vehicle transmitted from the front camera at a first moment, the method further includes:

[0021] Constructing a training sample set according to a plurality of historical ambient light intensities, first image features of the historical suspected road surface pothole areas respectively corresponding to each other, second image features of the historical suspected road surface pothole areas respectively corresponding to each other, first historical distances respectively corresponding to each other, second historical distances respectively corresponding to each other, and change features of the historical images respectively corresponding to each other;

[0022] The training sample set is input into the network to be trained to obtain a road surface pothole determination model.

[0023] In a second aspect, the present application also provides a road surface defect detection system based on computer vision, which is applied to an ECU of a vehicle in motion, the ECU being electrically connected to an ambient light sensor and a front camera of the vehicle, respectively, and the system comprising:

[0024] an image receiving unit, configured to receive, at a first moment, a first computer vision image in front of the vehicle transmitted from the front camera when the headlights of the vehicle are turned on and the ambient light intensity transmitted by the ambient light sensor is lower than a set light intensity threshold;

[0025] a feature extraction unit, configured to extract a first image feature of the suspected road surface pothole area when a suspected road surface pothole area is identified in the first computer vision image, and identify a first distance between the suspected road surface pothole area in the first computer vision image and the vehicle at a first moment, wherein the pothole area includes a black pothole area or a water puddle area;

[0026] The image receiving unit is further configured to receive a second computer vision image in front of the vehicle transmitted from the front camera at a second moment after a preset time period;

[0027] a feature extraction unit, configured to extract a second image feature of the suspected road surface pothole area in the second computer vision image when a suspected road surface pothole area is identified in the second computer vision image, and identify a second distance between the suspected road surface pothole area in the second computer vision image and the vehicle at a second moment;

[0028] A feature determination unit, configured to determine, based on the first image feature and the second image feature, an image change feature of a suspected road surface pothole area at a second distance relative to a first distance;

[0029] a probability determination unit, for inputting ambient light intensity, a first image feature of a suspected road surface pothole area, a second image feature of a suspected road surface pothole area, a first distance, a second distance, and an image change feature into a pre-trained road surface pothole determination model to determine a first probability that a defect exists on the road surface, wherein the road surface pothole determination model is obtained by inputting a training sample set constructed by multiple historical ambient light intensities, respectively corresponding first image features of historical suspected road surface pothole areas, respectively corresponding second image features of historical suspected road surface pothole areas, respectively corresponding historical first distances, respectively corresponding historical second distances, and respectively corresponding historical image change features into an initial network to be trained;

[0030] The defect determination unit is used to determine that there are defects on the road surface when the first probability is greater than a set probability threshold.

[0031] In a possible implementation, the system provided by the present application further includes:

[0032] a contour recognition unit, configured to recognize contours of suspected road surface pothole areas in the second computer vision image;

[0033] an image generating unit, configured to generate images of standard road surface pothole areas of different pothole types under the ambient light intensity and the second distance according to the ambient light intensity, the contour of the suspected road surface pothole area in the second computer vision image, and the second distance, wherein the contour of the image of the standard road surface pothole area of ​​each pothole type is the same as the contour of the suspected road surface pothole area in the second computer vision image;

[0034] A confidence determination unit is used to determine the average value and covariance of the difference between the pixel points of the suspected road surface pothole area and the pixel points corresponding to the image of the standard road surface pothole area of ​​each pothole type; according to the average value and covariance of the difference corresponding to each suspected road surface pothole area, the average value of the largest difference and the average value of the smallest difference among the average values ​​of multiple differences, and the largest covariance and the smallest covariance among the multiple covariances, determine the confidence that the image of the standard road surface pothole area of ​​each pothole type matches the suspected road surface pothole area respectively;

[0035] In a possible implementation, the system provided by the present application further includes:

[0036] The model training unit is used to construct a training sample set based on multiple historical ambient light intensities, respectively corresponding first image features of historical suspected road surface pothole areas, respectively corresponding second image features of historical suspected road surface pothole areas, respectively corresponding first historical distances, respectively corresponding second historical distances, and respectively corresponding historical image change features; the training sample set is input into the network to be trained to obtain a road surface pothole determination model.

[0037] The present application provides a road surface defect detection method and system based on computer vision. When the vehicle lights are turned on and the ambient light intensity transmitted by the ambient light sensor is lower than the set light intensity threshold, it means that the ambient light is dark at this time. According to the existing road surface defect algorithm, the road surface defects in front cannot be accurately identified. Therefore, the first computer vision image in front of the vehicle transmitted from the front camera can be received at the first moment. It can be understood that when the lights are turned on, the distances between the puddles, black pits, and road shadows and the vehicle are different, and the degree of reflection and brightness of the puddles, black pits, and road shadows are different, that is, the image features are different, and the process of image feature change is also different. Therefore, when it is identified that there is a suspected road surface pothole area in the first computer vision image, the first image feature of the suspected road surface pothole area is extracted, and the first distance between the suspected road surface pothole area and the vehicle at the first moment in the first computer vision image is identified, wherein the pothole area includes a black pit area or a puddle area.

[0038] A second computer vision image in front of the vehicle transmitted from the front camera is received at a second moment after a preset time period; when a suspected road surface pothole area is identified in the second computer vision image, second image features of the suspected road surface pothole area in the second computer vision image are extracted, and a second distance between the suspected road surface pothole area in the second computer vision image and the vehicle at the second moment is identified, and image change features of the suspected road surface pothole area at the second distance relative to the first distance are determined based on the first image features and the second image features.

[0039] The ambient light intensity, the first image feature of the suspected road surface pothole area, the second image feature of the suspected road surface pothole area, the first distance, the second distance, and the image change feature are input into the pre-trained road surface pothole determination model to determine the first probability that there are defects on the road surface. Since the first probability of determining the existence of defects on the road surface takes into account the ambient light intensity, the first image feature of the suspected road surface pothole area, the second image feature of the suspected road surface pothole area, the first distance, the second distance, and the image change feature, it is more in line with the actual scene, so the accuracy of determining the first probability of the existence of defects on the road surface is high. When the first probability is greater than the set probability threshold, the accuracy of determining the existence of defects on the road surface is also high. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A block diagram of the circuit connections between the ECU provided in the embodiment of the present application and the ambient light sensor and the front camera;

[0042] Figure 2 A flowchart of a road surface defect detection method based on computer vision provided in an embodiment of the present application;

[0043] Figure 3 A functional module block diagram of a road surface defect detection system based on computer vision provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments made by ordinary technicians in this field under the enlightenment of the embodiments belong to the scope of protection of the present application.

[0045] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] The embodiment of the present application provides a road surface defect detection method based on computer vision, which is applied to the ECU 101 of a vehicle in a driving state. Figure 1 As shown, ECU 101 is electrically connected to the vehicle's ambient light sensor 102 and front camera 103 respectively. Figure 2 As shown, the method provided in the embodiment of the present application includes:

[0047] S201: When the headlights of the vehicle are turned on and the ambient light intensity transmitted by the ambient light sensor 102 is lower than a set light intensity threshold, a first computer vision image in front of the vehicle transmitted from the front camera 103 is received at a first moment.

[0048] S202: When a suspected road surface pothole area is identified in the first computer vision image, extract a first image feature of the suspected road surface pothole area, and identify a first distance between the suspected road surface pothole area and the vehicle at a first moment in the first computer vision image.

[0049] Among them, the pit area includes a black pit area or a water puddle area.

[0050] S203: receiving a second computer vision image in front of the vehicle transmitted from the front camera 103 at a second moment after a preset time period.

[0051] S204: When a suspected road surface pothole area is identified in the second computer vision image, a second image feature of the suspected road surface pothole area in the second computer vision image is extracted, and a second distance between the suspected road surface pothole area in the second computer vision image and the vehicle at a second moment is identified.

[0052] S205: Determine, based on the first image feature and the second image feature, image change features of the suspected road surface pothole area at the second distance relative to the first distance.

[0053] S206: Input the ambient light intensity, the first image feature of the suspected road surface pothole area, the second image feature of the suspected road surface pothole area, the first distance, the second distance, and the image change feature into a pre-trained road surface pothole determination model to determine a first probability that there are defects on the road surface.

[0054] Among them, the highway surface pothole determination model is obtained by inputting a training sample set constructed by multiple historical ambient light intensities, the corresponding first image features of historical suspected highway surface pothole areas, the corresponding second image features of historical suspected highway surface pothole areas, the corresponding first historical distances, the corresponding second historical distances, and the corresponding historical image change features into the initial network to be trained.

[0055] Illustratively, before S201, the method provided in the embodiment of the present application further includes:

[0056] A training sample set is constructed based on multiple historical ambient light intensities, the corresponding first image features of historical suspected road surface potholes, the corresponding second image features of historical suspected road surface potholes, the corresponding first historical distances, the corresponding second historical distances, and the corresponding historical image change features. The training sample set is input into the network to be trained to obtain a road surface pothole determination model.

[0057] S207: When the first probability is greater than the set probability threshold, it is determined that there are defects on the road surface.

[0058] In summary, the road surface defect detection method and system 300 based on computer vision provided by the embodiment of the present application, when the vehicle lights are turned on and the ambient light intensity transmitted by the ambient light sensor 102 is lower than the set light intensity threshold, it means that the ambient light is dark at this time, and the road surface defects in front cannot be accurately identified according to the existing road surface defect algorithm, so the first computer vision image in front of the vehicle transmitted from the front camera 103 can be received at the first moment. It can be understood that when the lights are turned on, the reflection degree and brightness degree of the puddles, black pits, and road shadows are different when the distances between the vehicle and the puddles, black pits, and road shadows are different, that is, the image features are different, and the process of image feature change is also different. Therefore, when it is identified that there is a suspected road surface pothole area in the first computer vision image, the first image feature of the suspected road surface pothole area is extracted, and the first distance between the suspected road surface pothole area and the vehicle at the first moment in the first computer vision image is identified, wherein the pothole area includes a black pit area or a puddle area.

[0059] A second computer vision image in front of the vehicle transmitted from the front camera 103 is received at a second moment after a preset time period; when a suspected road surface pothole area is identified in the second computer vision image, second image features of the suspected road surface pothole area in the second computer vision image are extracted, and a second distance between the suspected road surface pothole area in the second computer vision image and the vehicle at the second moment is identified, and image change features of the suspected road surface pothole area at the second distance relative to the first distance are determined based on the first image features and the second image features.

[0060] The ambient light intensity, the first image feature of the suspected road surface pothole area, the second image feature of the suspected road surface pothole area, the first distance, the second distance, and the image change feature are input into the pre-trained road surface pothole determination model to determine the first probability that there are defects on the road surface. Since the first probability of determining the existence of defects on the road surface takes into account the ambient light intensity, the first image feature of the suspected road surface pothole area, the second image feature of the suspected road surface pothole area, the first distance, the second distance, and the image change feature, it is more in line with the actual scene, so the accuracy of determining the first probability of the existence of defects on the road surface is high. When the first probability is greater than the set probability threshold, the accuracy of determining the existence of defects on the road surface is also high.

[0061] In a possible implementation manner, after S206, the method provided in the embodiment of the present application further includes:

[0062] Step 1: Identify the contours of suspected road surface pothole areas in the second computer vision image.

[0063] Step 2: Generate images of standard road surface pothole areas of different pothole types under ambient light intensity and the second distance according to the ambient light intensity, the contour of the suspected road surface pothole area in the second computer vision image, and the second distance.

[0064] The contour of the image of the standard road surface pothole area of ​​each pothole type is the same as the contour of the suspected road surface pothole area in the second computer vision image. It can be understood that the image of the standard road surface pothole area is an image of the road surface pothole area simulated under the ambient light intensity and the second distance, and has high reliability.

[0065] Step 3: Determine the mean and covariance of the difference between the pixel points of the suspected road surface pothole area and the pixel points corresponding to the image of the standard road surface pothole area of ​​each pothole type.

[0066] Step 4: Determine the confidence level of the images of the standard highway surface pothole areas of each pothole type matching the suspected highway surface pothole areas based on the average value and covariance of the differences corresponding to each suspected highway surface pothole area, the average value of the largest difference and the average value of the smallest difference among the average values ​​of multiple differences, and the largest covariance and the smallest covariance among the multiple covariances.

[0067] Exemplarily, step 4 can be specifically implemented as follows: using the formula:

[0068] ;

[0069] The confidence level of the matching between the image of the standard road surface pothole area of ​​each pothole type and the suspected road surface pothole area is determined.

[0070] in, is the average value of the difference values ​​corresponding to the images of the pothole areas on the standard road surface of each pothole type, is the covariance of the images of the pothole areas on the standard road surface for each pothole type, is the average of the maximum differences among the averages of multiple differences, is the average of the minimum differences among the averages of multiple differences, is the largest covariance among multiple covariances, is the smallest covariance among multiple covariances, For confidence.

[0071] Step 5: Based on the highest confidence level, find the second probability that there are defects on the road surface.

[0072] Step 6: Modify the first probability according to the second probability.

[0073] Exemplarily, step 6 can be specifically implemented as follows: According to the formula , the first probability is corrected, where is the corrected first probability, is the first probability before correction, is the second probability, is the first weighting coefficient, is the second weighting coefficient.

[0074] Based on the above steps 1 to 6, it can be seen that since the image of the standard road surface pothole area is an image of the road surface pothole area under the ambient light intensity and the second distance, the reliability of comparing the suspected road surface pothole area with the standard road surface pothole area to determine whether the suspected road surface pothole area is a real road surface pothole area is high. It can be understood that through the above method, the accuracy of the confidence level of determining the match between the suspected road surface pothole area and the standard road surface pothole area is high, so the higher the second probability of the existence of defects on the road surface, the higher the accuracy of the first probability corrected according to the second probability.

[0075] See also Figure 3 The embodiment of the present application also provides a road surface defect detection system 300 based on computer vision, which is applied to the ECU 101 of a vehicle in a driving state. The ECU 101 is electrically connected to the vehicle's ambient light sensor 102 and the front camera 103. Specifically, the system 300 includes an image receiving unit 301, a feature extraction unit 302, a feature determination unit 303, a probability determination unit 304, and a defect determination unit 305305, wherein:

[0076] The image receiving unit 301 is used to receive a first computer vision image in front of the vehicle transmitted from the front camera 103 at a first moment when the vehicle lights are turned on and the ambient light intensity transmitted by the ambient light sensor 102 is lower than a set light intensity threshold.

[0077] The feature extraction unit 302 is used to extract the first image feature of the suspected road surface pothole area when a suspected road surface pothole area is identified in the first computer vision image, and to identify a first distance between the suspected road surface pothole area and the vehicle at a first moment in the first computer vision image, wherein the pothole area includes a black pothole area or a water puddle area.

[0078] The image receiving unit 301 is further configured to receive a second computer vision image in front of the vehicle transmitted from the front camera 103 at a second moment after a preset time period.

[0079] The feature extraction unit 302 is used to extract the second image features of the suspected road surface pothole area in the second computer vision image when a suspected road surface pothole area is identified in the second computer vision image, and to identify a second distance between the suspected road surface pothole area in the second computer vision image and the vehicle at a second moment.

[0080] The feature determination unit 303 is used to determine the image change feature of the suspected road surface pothole area at the second distance relative to the first distance according to the first image feature and the second image feature.

[0081] The probability determination unit 304 is used to input the ambient light intensity, the first image feature of the suspected road surface pothole area, the second image feature of the suspected road surface pothole area, the first distance, the second distance, and the image change feature into a pre-trained road surface pothole determination model to determine a first probability that there are defects on the road surface.

[0082] Among them, the highway surface pothole determination model is obtained by inputting a training sample set constructed by multiple historical ambient light intensities, the corresponding first image features of historical suspected highway surface pothole areas, the corresponding second image features of historical suspected highway surface pothole areas, the corresponding first historical distances, the corresponding second historical distances, and the corresponding historical image change features into the initial network to be trained.

[0083] In a possible implementation, the system 300 provided in the present application further includes:

[0084] The model training unit is used to construct a training sample set based on multiple historical ambient light intensities, respectively corresponding first image features of historical suspected road surface pothole areas, respectively corresponding second image features of historical suspected road surface pothole areas, respectively corresponding first historical distances, respectively corresponding second historical distances, and respectively corresponding historical image change features; the training sample set is input into the network to be trained to obtain a road surface pothole determination model.

[0085] The defect determination unit 305305 is used to determine that there are defects on the road surface when the first probability is greater than a set probability threshold.

[0086] In a possible implementation, the system 300 provided in the present application further includes:

[0087] The contour recognition unit is used to recognize the contour of the suspected road surface pothole area in the second computer vision image.

[0088] An image generating unit is used to generate images of standard road surface pothole areas of different pothole types under ambient light intensity and a second distance according to ambient light intensity, a contour of a suspected road surface pothole area in a second computer vision image, and a second distance, wherein the contour of the image of the standard road surface pothole area of ​​each pothole type is the same as the contour of the suspected road surface pothole area in the second computer vision image.

[0089] The confidence determination unit is used to determine the average value and covariance of the difference between the pixel points of the suspected road surface pothole area and the pixel points corresponding to the image of the standard road surface pothole area of ​​each pothole type; according to the average value and covariance of the difference corresponding to each suspected road surface pothole area, the average value of the largest difference and the average value of the smallest difference among the average values ​​of multiple differences, and the largest covariance and the smallest covariance among multiple covariances, determine the confidence that the image of the standard road surface pothole area of ​​each pothole type matches the suspected road surface pothole area.

[0090] In a possible implementation, the confidence determination unit is specifically configured to

[0091] Use the formula: ;

[0092] Determine the confidence level of the matching between the image of the standard road surface pothole area of ​​each pothole type and the suspected road surface pothole area;

[0093] in, is the average value of the difference values ​​corresponding to the images of the pothole areas on the standard road surface of each pothole type, is the covariance of the images of the pothole areas on the standard road surface for each pothole type, is the average of the maximum differences among the averages of multiple differences, is the average of the minimum differences among the averages of multiple differences, is the largest covariance among multiple covariances, is the smallest covariance among multiple covariances, For confidence.

[0094] In a possible implementation, the probability correction unit is used to calculate , the first probability is corrected, where is the corrected first probability, is the first probability before correction, is the second probability, is the first weighting coefficient, is the second weighting coefficient.

[0095] The data search unit is used to search for a second probability that there are defects on the road surface according to the highest confidence level.

[0096] The probability correction unit is used to correct the first probability according to the second probability.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A road surface defect detection method based on computer vision, characterized in that: An ECU applied to a vehicle in a driving state, wherein the ECU is electrically connected to an ambient light sensor and a front camera of the vehicle respectively, and the method comprises: When the headlights of the vehicle are turned on and the ambient light intensity transmitted by the ambient light sensor is lower than a set light intensity threshold, receiving a first computer vision image in front of the vehicle transmitted from the front camera at a first moment; When a suspected road surface pothole area is identified in the first computer vision image, extracting a first image feature of the suspected road surface pothole area, and identifying a first distance between the suspected road surface pothole area in the first computer vision image and the vehicle at the first moment, wherein the pothole area includes a black pothole area or a water pothole area; receiving, at a second moment after a preset time period, a second computer vision image in front of the vehicle transmitted from the front camera; When a suspected road surface pothole area is identified in the second computer vision image, extracting a second image feature of the suspected road surface pothole area in the second computer vision image, and identifying a second distance between the suspected road surface pothole area in the second computer vision image and the vehicle at the second moment; Determine, based on the first image feature and the second image feature, an image change feature of the suspected road surface pothole area at the second distance relative to the first distance; Inputting the ambient light intensity, the first image feature of the suspected road surface pothole area, the second image feature of the suspected road surface pothole area, the first distance, the second distance, and the image change feature into a pre-trained road surface pothole determination model to determine a first probability that there are defects on the road surface, wherein the road surface pothole determination model is obtained by inputting a training sample set constructed by multiple historical ambient light intensities, respectively corresponding first image features of historical suspected road surface pothole areas, respectively corresponding second image features of historical suspected road surface pothole areas, respectively corresponding historical first distances, respectively corresponding historical second distances, and respectively corresponding historical image change features into an initial network to be trained; When the first probability is greater than a set probability threshold, it is determined that there are defects on the road surface; After determining that there are defects on the road surface when the first probability is greater than a set probability threshold, the method further includes: identifying the contour of a suspected road surface pothole area in the second computer vision image; generating images of standard road surface pothole areas of different pothole types under the ambient light intensity and the second distance according to the ambient light intensity, the contour of the suspected road surface pothole area in the second computer vision image, and the second distance, wherein the contour of the image of the standard road surface pothole area of ​​each pothole type is the same as the contour of the suspected road surface pothole area in the second computer vision image; Determine the average value and covariance of the difference between the pixel points of the suspected road surface pothole area and the pixel points corresponding to the image of the standard road surface pothole area of ​​each pothole type; Determine the confidence that the image of the standard road surface pothole area of ​​each pothole type matches the suspected road surface pothole area respectively according to the average value and covariance of the difference values ​​corresponding to each suspected road surface pothole area, the average value of the largest difference value and the average value of the smallest difference value among the average values ​​of the multiple difference values, and the largest covariance and the smallest covariance among the multiple covariances; Find the second probability that there are defects on the road surface according to the highest confidence level; The first probability is modified according to the second probability.

2. The method according to claim 1, characterized in that: When the headlights of the vehicle are turned on and the ambient light intensity transmitted by the ambient light sensor is lower than a set light intensity threshold, before receiving a first computer vision image in front of the vehicle transmitted from the front camera at a first moment, the method further includes: Constructing a training sample set according to the multiple historical ambient light intensities, the first image features of the historical suspected road surface pothole areas, the second image features of the historical suspected road surface pothole areas, the first distances, the second distances, and the image change features; The training sample set is input into the network to be trained to obtain the road surface pothole determination model.

3. A road surface defect detection system based on computer vision, characterized in that: An ECU applied to a vehicle in a driving state, wherein the ECU is electrically connected to an ambient light sensor and a front camera of the vehicle respectively, and the system comprises: an image receiving unit, configured to receive, at a first moment, a first computer vision image in front of the vehicle transmitted from the front camera when the headlights of the vehicle are turned on and the ambient light intensity transmitted by the ambient light sensor is lower than a set light intensity threshold; a feature extraction unit, configured to extract a first image feature of the suspected road surface pothole area when a suspected road surface pothole area is identified in the first computer vision image, and identify a first distance between the suspected road surface pothole area in the first computer vision image and the vehicle at the first moment, wherein the pothole area includes a black pit area or a water puddle area; The image receiving unit is further configured to receive a second computer vision image in front of the vehicle transmitted from the front camera at a second moment after a preset time period; The feature extraction unit is configured to extract a second image feature of the suspected road surface pothole area in the second computer vision image when a suspected road surface pothole area is identified in the second computer vision image, and identify a second distance between the suspected road surface pothole area in the second computer vision image and the vehicle at the second moment; a feature determination unit, configured to determine, based on the first image feature and the second image feature, an image change feature of the suspected road surface pothole area at the second distance relative to the first distance; a probability determination unit, for inputting the ambient light intensity, the first image feature of the suspected road surface pothole area, the second image feature of the suspected road surface pothole area, the first distance, the second distance, and the image change feature into a pre-trained road surface pothole determination model to determine a first probability that a defect exists on the road surface, wherein the road surface pothole determination model is obtained by inputting a training sample set constructed by multiple historical ambient light intensities, respectively corresponding first image features of historical suspected road surface pothole areas, respectively corresponding second image features of historical suspected road surface pothole areas, respectively corresponding historical first distances, respectively corresponding historical second distances, and respectively corresponding historical image change features into an initial network to be trained; a defect determination unit, configured to determine that a defect exists on the road surface when the first probability is greater than a set probability threshold; The system further comprises: a contour recognition unit, configured to recognize the contour of a suspected road surface pothole area in the second computer vision image; an image generating unit, configured to generate images of standard road surface pothole areas of different pothole types under the ambient light intensity and the second distance according to the ambient light intensity, the contour of the suspected road surface pothole area in the second computer vision image, and the second distance, wherein the contour of the image of the standard road surface pothole area of ​​each pothole type is the same as the contour of the suspected road surface pothole area in the second computer vision image; A confidence determination unit is used to determine the average value and covariance of the difference between the pixel points of the suspected road surface pothole area and the pixel points corresponding to the image of the standard road surface pothole area of ​​each pothole type; according to the average value and covariance of the difference corresponding to each suspected road surface pothole area, the average value of the largest difference and the average value of the smallest difference among the average values ​​of multiple differences, and the largest covariance and the smallest covariance among the multiple covariances, determine the confidence that the image of the standard road surface pothole area of ​​each pothole type matches the suspected road surface pothole area respectively; A data search unit, used for searching the second probability that there are defects on the road surface according to the highest confidence level; A probability correction unit is used to correct the first probability according to the second probability.

4. The system according to claim 3, characterized in that The system further comprises: The model training unit is used to construct a training sample set according to the multiple historical ambient light intensities, the first image features of the corresponding historical suspected road surface pothole areas, the second image features of the corresponding historical suspected road surface pothole areas, the corresponding historical first distances, the corresponding historical second distances, and the corresponding historical image change features; the training sample set is input into the network to be trained to obtain the road surface pothole determination model.

Images