A pothole detection method and device based on an autonomous vehicle

Abstract

The application discloses a kind of based on automatic driving vehicle's road surface pit detection method and device, it is related to automatic driving technical field.The specific embodiment of the method includes: based on the camera and laser radar carried by automatic driving vehicle, the image and point cloud of road surface are acquired;The image of the road surface is input into trained deep learning model, and first pit detection result is obtained;From the point cloud of the road surface, point cloud located on road section is extracted;Wherein, the road section is perpendicular to the driving direction of the automatic driving vehicle;According to the elevation value of the point cloud on the road section, second pit detection result is determined;According to the first pit detection result and the second pit detection result, road surface pit detection result is determined.The embodiment can improve the accuracy of pit detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a road pit detection method and device based on an automatic driving vehicle. BACKGROUND

[0002] A pit is a bowl-shaped depression on a road surface caused by local filling of a roadbed or road surface or by driving of a vehicle. The pit is a common road defect and can affect driving safety. Therefore, accurate detection of the pit is of great significance to improving driving safety.

[0003] The prior art processes a road surface image collected by a camera through a convolutional neural network to detect whether the road surface has a pit.

[0004] However, the image is easily affected by light changes, resulting in reduced accuracy of the detection result. SUMMARY

[0005] Therefore, the embodiments of the present application provide a road pit detection method and device based on an automatic driving vehicle, which can improve the accuracy of the pit detection result.

[0006] In a first aspect, the embodiments of the present application provide a road pit detection method based on an automatic driving vehicle, comprising:

[0007] obtaining an image and point cloud of a road surface based on a camera and a laser radar carried by the automatic driving vehicle;

[0008] inputting the image of the road surface into a trained deep learning model to obtain a first pit detection result;

[0009] extracting point cloud located on a road section from the point cloud of the road surface, wherein the road section is perpendicular to a driving direction of the automatic driving vehicle;

[0010] determining a second pit detection result according to an elevation value of the point cloud on the road section;

[0011] determining a road pit detection result according to the first pit detection result and the second pit detection result.

[0012] In a second aspect, the embodiments of the present application provide a road pit detection device based on an automatic driving vehicle, comprising:

[0013] a collection module configured to obtain an image and point cloud of a road surface based on a camera and a laser radar carried by the automatic driving vehicle;

[0014] an image detection module configured to input the image of the road surface into a trained deep learning model to obtain a first pit detection result;

[0015] a point cloud detection module, configured to extract point clouds located on a road section from the point clouds of the road surface; wherein the road section is perpendicular to the driving direction of the autonomous vehicle; and determine a second pothole detection result according to the height values of the point clouds on the road section;

[0016] a fusion module, configured to determine a road surface pothole detection result according to the first pothole detection result and the second pothole detection result.

[0017] In a third aspect, an electronic device is provided, comprising:

[0018] one or more processors;

[0019] a storage device for storing one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any of the above embodiments.

[0021] In a fourth aspect, a computer readable medium is provided, which stores a computer program, and the program is executed by a processor to implement the method of any of the above embodiments.

[0022] An embodiment of the above invention has the following advantages or beneficial effects: The pothole is detected based on the camera and the laser radar carried by the autonomous vehicle itself, without the need to additionally set a detection device, thereby saving the pothole detection cost. The first pothole detection result obtained based on the image is fused with the second pothole detection result obtained based on the point cloud to obtain the final road surface pothole detection result. Since the point cloud is not affected by the light change condition, the embodiment of the present application can effectively make up for the defects of the existing method and improve the detection result accuracy. Since the pothole has a height difference with the road surface and non-road areas such as street lamps, the embodiment of the present application can accurately identify the pothole on the road surface by the height values of the point clouds on the road section, thereby further improving the accuracy of the pothole detection.

[0023] The further effects of the above non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application. Among them:

[0025] Figure 1 is a flowchart of a road surface pothole detection method based on an autonomous vehicle provided by an embodiment of the present application;

[0026] Figure 2is a schematic diagram of a road section provided by an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of a road section showing a filter window provided by an embodiment of the present application;

[0028] Figure 4 is a flowchart of a road pit detection method based on an autonomous vehicle provided by another embodiment of the present application;

[0029] Figure 5 is a schematic diagram of a road pit detection device based on an autonomous vehicle provided by another embodiment of the present application;

[0030] Figure 6 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0031] Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to assist in understanding, which should be considered in their context only. Therefore, those of ordinary skill in the art should realize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] The pit on the road surface is recognized by a convolutional neural network, which is easily affected by light. In order to make up for the defects of the prior art and improve the robustness of pit detection, the embodiments of the present application combine point cloud recognition with image recognition.

[0033] As shown in Figure 1 An embodiment of the present application provides a road pit detection method based on an autonomous vehicle, which comprises:

[0034] Step 101: Based on the camera and laser radar carried by the autonomous vehicle, the image and point cloud of the road surface are obtained.

[0035] The image of the road surface is obtained by the camera, and the point cloud of the road surface is obtained by the laser radar. The embodiments of the present application do not limit the types of the camera and the laser radar.

[0036] Step 102: The image of the road surface is input into a trained deep learning model to obtain a first pit detection result.

[0037] The deep learning model can be trained by any deep neural network with target detection function.

[0038] Step 103: extracting point clouds on the road section from the point clouds of the road surface; wherein the road section is perpendicular to the driving direction of the autonomous vehicle.

[0039] The scanning points obtained by the laser radar are arranged on the target according to the scanning lines, and considering that there is a difference in the elevation values of the pit and groove and the flat road surface and the non-road target such as the street lamp, the embodiment of the present application extracts the point clouds on the road section from the point clouds of the road surface based on the driving direction of the autonomous vehicle. In order to obtain more accurate elevation values, the road section is perpendicular to the driving direction of the autonomous vehicle, and the specific position of the road section can be adjusted according to the actual situation. Specifically, the road section can be determined according to the position of the laser radar, for example, the laser radar can be located on the road section; the road section can also be determined according to the position of the autonomous vehicle, for example, the road section is located a certain distance in front of the autonomous vehicle. As the autonomous vehicle travels, the road section changes accordingly, so the embodiment of the present application can detect the road pit and groove in real time.

[0040] As shown in FIG. 2, it is the point clouds on the road section extracted, and from the figure, it can be seen that the elevation values of the point clouds of the non-road area on both sides change greatly, and the elevation values of the point clouds of the road area in the middle change less, which is approximately a horizontal straight line. Figure 2

[0041] Step 104: determining the second pit and groove detection result according to the elevation values of the point clouds on the road section.

[0042] By utilizing the difference in the elevation values of the point clouds of different areas on the road section, the pit and groove on the road surface can be identified.

[0043] Step 105: determining the road pit and groove detection result according to the first pit and groove detection result and the second pit and groove detection result.

[0044] Based on the camera and the laser radar carried by the autonomous vehicle, the pit and groove can be detected without the need to additionally set a detection device, which can save the pit and groove detection cost. The first pit and groove detection result obtained based on the image is fused with the second pit and groove detection result obtained based on the point clouds to obtain the final road pit and groove detection result. Since the point clouds are not affected by the light change condition, the embodiment of the present application can effectively make up for the defects of the existing method and improve the accuracy of the detection result. Since there is an elevation difference between the pit and groove and the non-road area such as the road surface and the street lamp, the embodiment of the present application can accurately identify the pit and groove on the road surface by the elevation values of the point clouds on the road section, further improving the accuracy of the pit and groove detection.

[0045] In an embodiment of the present application, the deep learning model comprises: Faster-RCNN.

[0046] ​The candidate region is generated by an RPN (Region Proposal Network), which can improve the efficiency and accuracy of the pit and groove detection.

[0047] The training process of the Faster-RCNN is as follows:

[0048] S1: The ImageNet model pre-trained by the ImageNet dataset is used to initialize the shared convolutional layer, and the RPN is trained based on the road surface image labeled with pits and grooves to obtain the candidate frame.

[0049] The shared convolutional layer is a feature extraction network shared by the RPN and the Fast-RCNN.

[0050] S2: The shared convolutional layer is initialized using the ImageNet model, and the Fast-RCNN is trained using the candidate frame obtained in S1.

[0051] S3: The learning rate of the shared convolutional layer of the RPN and the Fast-RCNN is set to 0, and the RPN trained in S1 is retrained using the Fast-RCNN trained in S2.

[0052] S4: The Fast-RCNN is retrained using the candidate frame obtained in S3, and the Faster-RCNN is obtained.

[0053] In actual application scenarios, S3 and S4 can be iterated multiple times to further optimize the Faster-RCNN. The image of the road surface is input into the trained Faster-RCNN to obtain the pit and groove detection result of the road surface.

[0054] The deep learning model can also be a convolutional neural network, a Fast-RCNN, etc.

[0055] In an embodiment of the present application, the second pit and groove detection result is determined according to the elevation values of the point cloud on the road section, comprising:

[0056] According to the elevation values of the point cloud on the road section, the point cloud on the road section is divided into road point cloud and non-road point cloud;

[0057] According to the elevation values of the road point cloud, the second pit and groove detection result is determined.

[0058] The embodiment of the present application first divides the point cloud on the road section into road point cloud and non-road point cloud, and then identifies the pit and groove according to the road point cloud, which can reduce the calculation amount of pit and groove detection and improve the detection efficiency.

[0059] In an actual application scenario, the point cloud on the road section can also be divided into pit point cloud, planar point cloud and non-road point cloud according to the elevation values of the point cloud on the road section. Among them, the pit point cloud and the planar point cloud constitute the aforementioned road point cloud. For example, the point cloud on the road section is divided according to the interval in which the elevation value of the point cloud is located.

[0060] In an embodiment of the present application, the point cloud on the road section is divided into road point cloud and non-road point cloud according to the elevation values of the point cloud on the road section, comprising:

[0061] According to the preset first size and the elevation values of the point cloud on the road section, the maximum elevation difference of the point cloud in the first window, the maximum elevation difference of the point cloud in the second window and the maximum elevation difference of the point cloud in the third window are determined;

[0062] When the maximum elevation difference of the point cloud in the first window is less than the preset minimum elevation difference between the road point cloud and the non-road point cloud, the maximum elevation difference of the point cloud in the second window is less than the preset maximum elevation difference of the road point cloud in a single window, and the maximum elevation difference of the point cloud in the third window is less than the maximum elevation difference of the road point cloud in a single window, the point cloud in the first window, the second window and the third window is determined as road point cloud;

[0063] Among them, the first window, the second window and the third window are continuously arranged along the edge of the point cloud distribution on the road section to the center, and move along the edge of the point cloud distribution to the center.

[0064] As shown in the following formula: Figure 3 The first size can be determined by the user in advance according to the density of the point cloud.

[0065] When the filter window is located in the non-road area, the elevation values of the points in the three filter windows change obviously; when the filter window contains both road points and other ground object points, such as when the window 1 is located in the non-road area, the window 2 is located in the road boundary, and the window 3 is located in the road area, the elevation values of the points in the window 1 and the window 2 change obviously, and the elevation values of the points in the window 3 change gently; when the window 1 is located in the road boundary, and the window 2 and the window 3 are located in the road area, the elevation values of the points in the window 1 change obviously, and the elevation values of the points in the window 2 and the window 3 change gently; when the three filter windows are located in the road area, the elevation values of the points in the three filter windows change gently.

[0066] Considering the above differences in the elevation values of the points in different areas, the embodiment of the present application identifies the category of the point cloud through the maximum elevation difference of the point cloud in the filter window. For example, the maximum elevation difference of the point cloud in the first window is △ H1, the maximum elevation difference of the point cloud in the second window is△ H2 and the maximum height difference of the point cloud in the third window is △ H3. The minimum height difference between the road point cloud and the non-road point cloud is △ H s The maximum height difference of the road point cloud in a single window is △ H n If △ H1< △ H s , △ H2< △ H n , △ H3< △ H n , the point cloud in the first window, the second window and the third window are all road point clouds, that is, the three filtering windows are located in the road area.

[0067] The minimum height difference between the road point cloud and the non-road point cloud is used to measure whether the window is a road point cloud according to the difference between the height values of the road point cloud and the non-road point cloud; the maximum height difference of the road point cloud in a single window is used to measure whether the window is a road point cloud according to the relatively gentle change of the height value of the road point cloud. The minimum height difference between the road point cloud and the non-road point cloud and the maximum height difference of the road point cloud in a single window can be adjusted according to the actual situation of the road.

[0068] The three filtering windows move along the distribution edge to the distribution center, that is, from the non-road area to the road area, and each movement performs the above road point cloud and non-road point cloud division process.

[0069] In actual application scenarios, if the point cloud in the three filtering windows is all road point cloud, the movement can be paused. For example, in Figure 3 , the three filtering windows first move from the left edge to the distribution center, and when the point cloud in the three filtering windows is all road point cloud, the movement is stopped; then, the three filtering windows move from the right edge to the distribution center, and when the point cloud in the three filtering windows is all road point cloud, the movement is stopped; the point cloud between the two stop positions is road point cloud, and it is not necessary to identify whether it is road point cloud by moving the window. Of course, the filtering window can also move from the left edge and the right edge to the distribution center respectively.

[0070] Considering that there are non-road areas, road boundaries and road areas on the road section, the embodiment of the application sets three filtering windows, which can accurately identify the type of point cloud in the filtering window through the height value of the point in the filtering window, and identify the road point cloud on the road section through the movement of the filtering window. In actual application scenarios, other number of filtering windows can also be set, such as only setting window 1 and window 2, or increasing window 4.

[0071] In an embodiment of the present application, the second pit detection result is determined according to the elevation values of the road point cloud, including:

[0072] According to the preset second size and the elevation values of the road point cloud, the minimum elevation difference of the point cloud in the fourth window, the minimum elevation difference of the point cloud in the fifth window, and the minimum elevation difference of the point cloud in the sixth window are determined;

[0073] When the minimum elevation difference of the point cloud in the fourth window is greater than the preset minimum elevation difference of the planar point cloud and the pit point cloud, the minimum elevation difference of the point cloud in the fifth window is greater than the preset minimum elevation difference of the pit point cloud in a single window, and the minimum elevation difference of the point cloud in the sixth window is greater than the minimum elevation difference of the pit point cloud in a single window, the point cloud in the fourth window, the fifth window and the sixth window is determined as the pit point cloud;

[0074] Wherein, the planar point cloud and the pit point cloud constitute the road point cloud, and the fourth window, the fifth window and the sixth window are continuously arranged along the edge of the road point cloud distribution to the center and move along the edge of the road point cloud distribution to the center.

[0075] After dividing the point cloud on the road section into road point cloud and non-road point cloud, the embodiment of the present application further identifies the pit point cloud from the road point cloud by using a similar division method.

[0076] Specifically, three continuous filtering windows are sequentially arranged on the road section, which are the fourth window, the fifth window and the sixth window. The second size can be determined by the user in advance according to the density of the point cloud.

[0077] The road area can be divided into a planar area, a pit boundary and a pit area, the elevation values of the points in the planar area change relatively gently, and the elevation values of the points in the pit boundary and the pit area change greatly. The point cloud in the planar area is the planar point cloud, the point cloud in the pit area is the pit point cloud, and the point cloud in the pit boundary includes the planar point cloud and the pit point cloud.

[0078] Considering that the change amplitude of the elevation values of the points in the pit area is greater than that of the points in the planar area, the embodiment of the present application identifies the pit point cloud by the minimum elevation difference of the point cloud in the filtering window. For example, the minimum elevation difference of the point cloud in the fourth window is △ H4, the minimum elevation difference of the point cloud in the fifth window is △ H5, and the minimum elevation difference of the point cloud in the sixth window is △ H6. The minimum elevation difference of the planar point cloud and the pit point cloud is △ H p , and the minimum elevation difference of the pit point cloud in a single window is △ H m If △ H4> △ H p ,△ H5> △ H m 、 △ H6> △ H m If the fourth window, the fifth window and the sixth window are all located in the pit area, the point clouds in the fourth window, the fifth window and the sixth window are all pit point clouds.

[0079] The minimum elevation difference between the plane point cloud and the pit point cloud is used to measure whether the window is a pit point cloud according to the difference between the elevation values of the plane point cloud and the pit point cloud; and the minimum elevation difference of the pit point cloud in a single window is used to measure whether the window is a pit point cloud according to the large variation range of the elevation values of the pit point cloud. The minimum elevation difference between the plane point cloud and the pit point cloud and the minimum elevation difference of the pit point cloud in a single window can be adjusted according to the actual situation of the road.

[0080] The three filtering windows move along the edge of the road point cloud distribution to the center of the distribution, and each movement performs the above plane point cloud and pit point cloud division process. For example, the three filtering windows first move from the left edge to the center of the distribution until the center of the distribution; then, the three filtering windows move from the right edge to the center of the distribution until the center of the distribution.

[0081] Considering that there are plane areas, pit boundaries and pit areas on the road section, the embodiment of the present application sets three filtering windows, which can accurately identify the type of point cloud in the filtering window through the elevation values of the points in the filtering window, and identify the pit point cloud in the road point cloud through the movement of the filtering window. In actual application scenarios, other numbers of filtering windows can also be set.

[0082] In an embodiment of the present application, the road pit detection result is determined according to the first pit detection result and the second pit detection result, comprising:

[0083] For any point on the road surface: if the current point belongs to the non-pit in the first pit detection result and the second pit detection result, it is determined that the current point belongs to the non-pit in the road pit detection result, otherwise, it is determined that the current point belongs to the pit in the road pit detection result.

[0084] In order to improve the driving safety of the autonomous vehicle, the embodiment of the present application gives priority to the pit when fusing the detection results, that is, if any pit detection result identifies that the current point belongs to the pit, it is determined that the current point belongs to the pit.

[0085] In actual application scenarios, the two pit detection results can also be fused in other manners. For example, according to the confidence factor of the first pit detection result and the confidence factor of the second pit detection result, the road pit detection result is determined. The confidence factor of the first pit detection result can be the prediction accuracy of the deep learning model, which changes with the number of detections. The initial prediction accuracy can be determined by a test set, or can be determined by a user through an empirical value. The confidence factor of the second pit detection result can be the pit detection accuracy, which changes with the number of detections. The initial pit detection accuracy can be determined by pit detection on a point cloud of a road sample, or can be set by a user through an empirical value.

[0086] For example, the confidence factor of the first pit detection result is a%, and the confidence factor of the second pit detection result is b%. When a% is greater than or equal to 90% or b% is greater than or equal to 90%, the pit detection result corresponding to the larger confidence factor is taken as the road pit detection result. When a% is less than 90% and b% is less than 90%, the road pit detection result is that the road has no pit.

[0087] In an embodiment of the present application, the method further comprises:

[0088] obtaining a road coordinate in an inertial measurement unit coordinate system based on an inertial measurement unit carried by the autonomous vehicle;

[0089] determining a pit coordinate in the IMU coordinate system according to the road coordinate, the pit coordinate in the radar coordinate system and / or the pit coordinate in the camera coordinate system in the road pit detection result.

[0090] In order to facilitate the use of subsequent path planning modules, the road pit detection result is converted by the embodiment of the present application. The first pit detection result includes the pit coordinate in the camera coordinate system, the second pit detection result includes the pit coordinate in the radar coordinate system, and the road pit detection result obtained by fusion includes the pit coordinate in the radar coordinate system and / or the pit coordinate in the camera coordinate system.

[0091] As shown in Figure 4 The embodiment of the present application provides a road pit detection method based on an autonomous vehicle, which comprises:

[0092] Step 401: based on the camera and the laser radar carried by the autonomous vehicle, the image and the point cloud of the road are obtained.

[0093] Step 402: inputting the image of the road into the trained Faster-RCNN to obtain the first pit detection result.

[0094] Step 403: Extract the point cloud located on the road section from the point cloud of the road surface; wherein the road section is perpendicular to the driving direction of the autonomous vehicle.

[0095] Step 404: Based on the preset first dimension and the elevation values ​​of the point cloud on the road cross section, determine the maximum elevation difference of the point cloud in the first window, the maximum elevation difference of the point cloud in the second window, and the maximum elevation difference of the point cloud in the third window.

[0096] Step 405: When the maximum elevation difference of the point cloud in the first window is less than the preset minimum elevation difference between road point cloud and non-road point cloud, the maximum elevation difference of the point cloud in the second window is less than the preset maximum elevation difference of road point cloud in a single window, and the maximum elevation difference of the point cloud in the third window is less than the maximum elevation difference of road point cloud in a single window, the point clouds in the first window, second window and third window are determined to be road point clouds.

[0097] Step 406: Based on the preset second dimension and the elevation value of the road point cloud, determine the minimum elevation difference of the point cloud in the fourth window, the minimum elevation difference of the point cloud in the fifth window, and the minimum elevation difference of the point cloud in the sixth window.

[0098] Step 407: When the minimum elevation difference of the point cloud in the fourth window is greater than the preset minimum elevation difference between the planar point cloud and the pit point cloud, the minimum elevation difference of the point cloud in the fifth window is greater than the preset minimum elevation difference of the pit point cloud in a single window, and the minimum elevation difference of the point cloud in the sixth window is greater than the minimum elevation difference of the pit point cloud in a single window, the point clouds in the fourth, fifth, and sixth windows are determined to be pit point clouds; the second pit detection result includes pit point clouds.

[0099] Step 408: For any point on the road surface: If the current point is not a pothole in both the first pothole detection result and the second pothole detection result, then the current point is determined to be not a pothole in the road surface pothole detection result; otherwise, the current point is determined to be a pothole in the road surface pothole detection result.

[0100] Step 409: Obtain the road surface coordinates in the IMU coordinate system based on the inertial measurement unit carried by the autonomous vehicle.

[0101] Step 410: Determine the pothole coordinates in the IMU coordinate system based on the road surface coordinates, the pothole coordinates in the radar coordinate system and / or the pothole coordinates in the camera coordinate system from the road surface pothole detection results.

[0102] This invention combines image recognition and point cloud recognition to improve the accuracy of pit detection results.

[0103] like Figure 5 As shown, this embodiment of the invention provides a road surface pothole detection device based on an autonomous vehicle, comprising:

[0104] The collection module 501 is configured to acquire images and point clouds of a road surface based on a camera and a laser radar carried by an autonomous vehicle;

[0105] The image detection module 502 is configured to input the images of the road surface into a trained deep learning model to obtain a first pothole detection result.

[0106] The point cloud detection module 503 is configured to extract point clouds located on a road section from the point clouds of the road surface, wherein the road section is perpendicular to the driving direction of the autonomous vehicle; and determine a second pothole detection result according to the elevation values of the point clouds on the road section.

[0107] The fusion module 504 is configured to determine a road surface pothole detection result according to the first pothole detection result and the second pothole detection result.

[0108] In an embodiment of the present application, the deep learning model comprises a Faster-RCNN.

[0109] In an embodiment of the present application, the point cloud detection module 503 is configured to divide the point clouds on the road section into road point clouds and non-road point clouds according to the elevation values of the point clouds on the road section; and determine the second pothole detection result according to the elevation values of the road point clouds.

[0110] In an embodiment of the present application, the point cloud detection module 503 is configured to determine a maximum elevation difference of point clouds in a first window, a maximum elevation difference of point clouds in a second window and a maximum elevation difference of point clouds in a third window according to a preset first size and the elevation values of the point clouds on the road section; and determine the point clouds in the first window, the second window and the third window as road point clouds when the maximum elevation difference of the point clouds in the first window is less than a preset minimum elevation difference between road point clouds and non-road point clouds, the maximum elevation difference of the point clouds in the second window is less than a preset maximum elevation difference of road point clouds in a single window, and the maximum elevation difference of the point clouds in the third window is less than the maximum elevation difference of road point clouds in a single window; wherein the first window, the second window and the third window are continuously arranged along the edges of the point cloud distribution to the center of the road section and move along the edges of the point cloud distribution to the center.

[0111] In one embodiment of the present application, the point cloud detection module 503 is configured to determine the minimum elevation difference of the point cloud in the fourth window, the minimum elevation difference of the point cloud in the fifth window, and the minimum elevation difference of the point cloud in the sixth window according to the preset second size and the elevation values of the road point cloud; when the minimum elevation difference of the point cloud in the fourth window is greater than the preset minimum elevation difference of the planar point cloud and the pothole point cloud, the minimum elevation difference of the point cloud in the fifth window is greater than the preset minimum elevation difference of the pothole point cloud in a single window, and the minimum elevation difference of the point cloud in the sixth window is greater than the minimum elevation difference of the pothole point cloud in a single window, the point cloud in the fourth window, the fifth window and the sixth window is determined as the pothole point cloud; wherein the planar point cloud and the pothole point cloud constitute the road point cloud, and the fourth window, the fifth window and the sixth window are sequentially arranged along the edge of the road point cloud distribution to the center and move along the edge of the road point cloud distribution to the center.

[0112] In one embodiment of the present application, the fusion module 504 is configured to determine that the current point belongs to the non-pothole in the road pothole detection result if the current point belongs to the non-pothole in both the first pothole detection result and the second pothole detection result, and otherwise, determine that the current point belongs to the pothole in the road pothole detection result.

[0113] In one embodiment of the present application, the fusion module 504 is configured to obtain the road coordinates in the IMU coordinate system based on the inertial measurement unit carried by the autonomous vehicle; and determine the pothole coordinates in the IMU coordinate system according to the road coordinates, the pothole coordinates in the radar coordinate system and / or the pothole coordinates in the camera coordinate system in the road pothole detection result.

[0114] An electronic device is provided in an embodiment of the present application, comprising:

[0115] one or more processors;

[0116] a storage device configured to store one or more programs,

[0117] When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any of the above embodiments.

[0118] A computer readable medium is provided in an embodiment of the present application, which stores a computer program, and the program is executed by a processor to implement the method of any of the above embodiments.

[0119] Reference will now be made to the following description Figure 6 which shows the structure of a computer system 600 suitable for use in implementing the terminal device of an embodiment of the present application. Figure 6 The terminal device shown is merely an example and should not impose any limitation on the function and use range of the embodiments of the present application.

[0120] As Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601 which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage section 608. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0121] Connected to the I / O interface 605 are an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a display device such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable recording medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read therefrom is installed into the storage section 608 as necessary.

[0122] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable recording medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the system of the present disclosure are performed.

[0123] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0124] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0125] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The described modules can also be arranged in a processor, for example, a processor can be described as including a sending module, an obtaining module, a determining module and a first processing module. In some cases, the names of these modules do not constitute a limitation on the modules themselves, for example, the sending module can also be described as "a module that sends a picture obtaining request to a connected server".

[0126] The specific embodiments described above do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for detecting potholes based on an autonomous vehicle, characterized by, The method comprises the following steps: obtaining the image and point cloud of the road surface based on the camera and laser radar carried by the autonomous vehicle; inputting the image of the road surface into the trained deep learning model to obtain a first pothole detection result; extracting the point cloud on the road section from the point cloud of the road surface; wherein the road section is perpendicular to the driving direction of the autonomous vehicle; dividing the point cloud on the road section into road point cloud and non-road point cloud according to the height value of the point cloud on the road section; determining the minimum height difference of the point cloud in the fourth window, the minimum height difference of the point cloud in the fifth window and the minimum height difference of the point cloud in the sixth window according to the preset second size and the height value of the road point cloud; when the minimum height difference of the point cloud in the fourth window is greater than the preset minimum height difference of the planar point cloud and the pothole point cloud, the minimum height difference of the point cloud in the fifth window is greater than the preset minimum height difference of the pothole point cloud in a single window, and the minimum height difference of the point cloud in the sixth window is greater than the minimum height difference of the pothole point cloud in the single window, it is determined that the point cloud in the fourth window, the fifth window and the sixth window is the pothole point cloud, so as to obtain a second pothole detection result based on the pothole point cloud; wherein the planar point cloud and the pothole point cloud constitute the road point cloud, and the fourth window, the fifth window and the sixth window are continuously arranged along the edge of the road point cloud distribution to the center and move along the edge of the road point cloud distribution to the center; for any point on the road surface: if the current point belongs to non-pothole in the first pothole detection result and the second pothole detection result, it is determined that the current point belongs to non-pothole in the road pothole detection result, otherwise, it is determined that the current point belongs to pothole in the road pothole detection result.

2. The method of claim 1, wherein: the deep learning model comprises Faster-RCNN.

3. The method of claim 1, wherein: dividing the point cloud on the road section into road point cloud and non-road point cloud according to the height value of the point cloud on the road section comprises: determining the maximum height difference of the point cloud in the first window, the maximum height difference of the point cloud in the second window and the maximum height difference of the point cloud in the third window according to the preset first size and the height value of the point cloud on the road section; when the maximum height difference of the point cloud in the first window is less than the preset minimum height difference of the road point cloud and the non-road point cloud, the maximum height difference of the point cloud in the second window is less than the preset maximum height difference of the road point cloud in a single window, and the maximum height difference of the point cloud in the third window is less than the maximum height difference of the road point cloud in the single window, it is determined that the point cloud in the first window, the second window and the third window is the road point cloud; wherein the first window, the second window and the third window are continuously arranged along the edge of the point cloud distribution on the road section to the center and move along the edge of the point cloud distribution to the center.

4. The method of claim 1, wherein, further comprising: obtaining the road surface coordinates in the IMU coordinate system based on the inertial measurement unit carried by the autonomous vehicle; According to the road surface coordinates, the pit coordinates in the radar coordinate system and / or the pit coordinates in the camera coordinate system in the pit detection result of the road surface, pit coordinates in the IMU coordinate system are determined.

5. A pothole detection device based on an autonomous vehicle, characterized by, Comprise: The acquisition module is configured to obtain images and point clouds of the road surface based on the camera and the laser radar carried by the autonomous vehicle; The image detection module is configured to input the images of the road surface into a trained deep learning model to obtain a first pit detection result; The point cloud detection module is configured to extract point clouds located on a road section from the point clouds of the road surface; wherein the road section is perpendicular to the driving direction of the autonomous vehicle; according to the elevation values of the point clouds on the road section, the point clouds on the road section are divided into road point clouds and non-road point clouds; according to a preset second size and the elevation values of the road point clouds, the minimum elevation difference of the point clouds in the fourth window, the minimum elevation difference of the point clouds in the fifth window and the minimum elevation difference of the point clouds in the sixth window are determined; when the minimum elevation difference of the point clouds in the fourth window is greater than the preset minimum elevation difference of the planar point clouds and the pit point clouds, the minimum elevation difference of the point clouds in the fifth window is greater than the preset minimum elevation difference of the pit point clouds in a single window, and the minimum elevation difference of the point clouds in the sixth window is greater than the minimum elevation difference of the pit point clouds in the single window, the point clouds in the fourth window, the fifth window and the sixth window are determined as pit point clouds to obtain a second pit detection result based on the pit point clouds; wherein the planar point clouds and the pit point clouds constitute the road point clouds, the fourth window, the fifth window and the sixth window are continuously arranged along the edge of the road point clouds to the center and move along the edge of the road point clouds to the center; The fusion module is configured to determine that the current point belongs to a non-pit in the pit detection result of the road surface if the current point belongs to a non-pit in both the first pit detection result and the second pit detection result, otherwise, it is determined that the current point belongs to a pit in the pit detection result of the road surface.

6. An electronic device, comprising: Comprise: One or more processors; Storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-4.

7. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method of any one of claims 1-4.

Images