Pit detection method, electronic device, and storage medium

Abstract

Embodiments of the present application relate to the field of vehicle driving safety, and disclose a pit detection method, an electronic device and a storage medium. In the present application, pit images on a vehicle travel route are collected; three-dimensional information of pits on the vehicle travel route is generated according to the pit images, and the depth of the pits is determined from the three-dimensional information; and an alarm information is sent out in the case that the depth exceeds a preset threshold. According to the pit images, the three-dimensional information of the pits that can be observed at the current vehicle position is analyzed, and the three-dimensional information of the pits that cannot be observed at the current vehicle position is predicted, the depth of the pits is determined according to the obtained three-dimensional information of the pits, and the driving personnel can be timely and accurately reminded to avoid in the case that the depth exceeds the threshold. The complete three-dimensional information of the pits analyzed by using the collected pit images is used to determine the depth of the pits, so that the depth detection of the pits is more accurate, the range of the pits that can be detected is wider, and the early warning effect is better.

Description

Technical Field

[0001] This invention relates to the field of vehicle driving safety, and in particular to a pothole detection method, electronic device, and storage medium. Background Technology

[0002] During vehicle operation, uneven roads can cause driving difficulties. If a vehicle encounters a deep pothole, it may stall and become unable to move, potentially endangering the safety of the occupants. To predict potholes along the driving route, distance sensors are typically installed in vehicles to determine the presence of potholes ahead by monitoring changes in distance along the road.

[0003] The inventors discovered that conventional methods for detecting potholes have at least the following problems: pothole detection using distance sensors is affected by the distance between the vehicle and the pothole. When the vehicle is far from the pothole, the depth of the pothole cannot be accurately predicted. When the vehicle is close to the pothole, it cannot be guaranteed that the driver can safely avoid it. Therefore, a safer and more accurate method for detecting potholes is needed. Summary of the Invention

[0004] The purpose of this invention is to provide a pothole detection method, electronic device, and storage medium that can predict the depth of potholes more timely and accurately, have a wider detection range, and improve vehicle driving safety.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a pothole detection method, comprising: acquiring pothole images along a vehicle's travel route; generating three-dimensional information of the potholes along the vehicle's travel route based on the pothole images, and determining the depth of the potholes using the three-dimensional information; and issuing an alarm message when the depth exceeds a preset threshold.

[0006] Embodiments of the present invention also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described pit detection method.

[0007] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described pit detection method.

[0008] Compared to existing technologies, this invention involves acquiring images of potholes along the vehicle's path during driving, generating three-dimensional information about these potholes based on the images, and determining their depth using this information. This application obtains the three-dimensional information of potholes by analyzing the acquired pothole images. Specifically, it can analyze both the observable three-dimensional information of potholes at the current vehicle position and predict the three-dimensional information of potholes that are not observable at the current vehicle position. Based on this obtained three-dimensional information, the depth of the pothole can be determined, and if the depth exceeds a threshold, it can promptly and accurately alert the driver to avoid it. By using the complete three-dimensional information of the potholes analyzed from the acquired images to determine their depth, pothole depth detection becomes more accurate, the detected pothole area is wider, and the warning effect is better.

[0009] In addition, images of potholes along the vehicle's route are acquired, including: using the vehicle's front-facing camera to capture images and identifying whether potholes exist in the captured images; if so, image segmentation is performed on the captured images to separate the areas containing potholes; edge extraction is then performed on the images containing pothole areas to obtain the pothole images. Preprocessing the images after capture by the camera, filtering out parts that are clearly not pothole images, and focusing only on the areas containing potholes for subsequent analysis and processing, maximizes the accuracy of the generated 3D pothole information and improves the efficiency of 3D information generation.

[0010] In addition, before performing image segmentation on the captured image, the process also includes denoising the image. This removes noise from the image, making the subsequent segmentation of the pitted image more accurate.

[0011] In addition, image segmentation is performed on the captured images to segment out images containing pothole regions. This includes selecting a first target region in the captured images that matches the range of water surface ripples, and selecting a second target region in the captured images that matches the range of potholes on the ground. The intersection of the first and second target regions is taken as the image containing the pothole region. Potholes on the vehicle's travel path may be puddles, so it is necessary to consider the case where the surface of the pothole in the captured images is water, to ensure the accuracy of image recognition.

[0012] Additionally, selecting a second target region from the captured image that matches the area of ​​the pothole includes: determining any two target points at the edge of the pothole in the captured image, wherein the distance between the two target points is greater than a preset threshold; and defining a circular area with the two target points as its diameter as the second target region. To simplify the determination of the second target region, the approximate outline of the second target region can be determined directly after determining the two target points at the edge of the pothole.

[0013] In addition, after taking pictures using the vehicle's front-facing camera, the process also includes: determining the position of the marker in the captured image; generating three-dimensional information of the potholes along the vehicle's travel route based on the pothole image, and determining the depth of the potholes based on the three-dimensional information, including: determining the depth of the potholes based on the difference between the depth information of the marker in the pothole image and the depth information of the pothole area.

[0014] In addition, the process of generating three-dimensional information of potholes along the vehicle's travel route based on pothole images includes: inputting pothole images into a BP neural network model; and reconstructing the three-dimensional information of pothole images using the BP neural network model.

[0015] In addition, after determining the depth of the pit using 3D information, the process also includes uploading the pit's depth to a shared platform. Attached Figure Description

[0016] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0017] Figure 1 This is a flowchart of a pothole detection method according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram illustrating the determination of target points in the second target region according to an embodiment of the present invention;

[0020] Figure 4 This is a flowchart of a method for determining the depth of a pit according to an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of the marker in an image captured according to an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of the present invention to enable the reader to better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0024] The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.

[0025] The first embodiment of the present invention relates to a pothole detection method, comprising: acquiring pothole images along a vehicle's travel route; generating three-dimensional information of the potholes along the vehicle's travel route based on the pothole images, and determining the depth of the potholes using the three-dimensional information; and issuing an alarm message when the depth exceeds a preset threshold. This method can predict the depth of potholes more promptly and accurately, has a wider detection range, and improves vehicle driving safety. The implementation details of the pothole detection method of this embodiment are described below. These details are provided for ease of understanding and are not essential for implementing this solution.

[0026] The pit detection method in this embodiment is as follows: Figure 1 As shown, it includes:

[0027] Step 101: Collect images of potholes along the vehicle's route.

[0028] Specifically, during vehicle movement, an image of the area in front of the vehicle is captured by a front-facing camera, which can be an infrared camera. To ensure the accuracy of subsequent detection, radar or sonar sensors can also be added in front of the vehicle to assist in detecting the depth of the pit when the water level is high.

[0029] Step 102: Generate three-dimensional information of potholes along the vehicle's travel route based on the pothole image, and determine the depth of the potholes using the three-dimensional information.

[0030] Specifically, the 3D information of a current pothole image can be determined by the correspondence between a large number of pothole images and their 3D information pre-stored in a database. Alternatively, the 3D information can be determined by the depth information of each pixel in the pothole image. Furthermore, the 3D information can be determined by examining pothole images taken from two different locations by the vehicle, and by considering the differences between the images and the locations where they were taken. After determining the 3D information, the maximum vertical distance from the bottom of the pothole to the road surface is taken as the depth of the pothole.

[0031] Step 103: Issue an alarm message if the depth exceeds a preset threshold.

[0032] Specifically, the system predicts driving risks based on the depth of potholes and provides drivers with suggestions for safe driving speeds or route changes. If the pothole depth exceeds a preset threshold, it can automatically unlock the doors and open the sunroof to help occupants exit the vehicle more efficiently. It can also play safety warnings to the driver via speakers and display safety information on the screen. Furthermore, after determining the pothole depth using 3D information, it can upload road monitoring data, including at least the pothole depth, to a cloud computing platform. This data can then be used for vehicle-to-infrastructure (V2I) communication within the urban network, sharing relevant road information and improving driving safety.

[0033] Compared to existing technologies, this invention involves acquiring images of potholes along the vehicle's path during driving, generating three-dimensional information about these potholes based on the images, and determining their depth using this information. This application obtains the three-dimensional information of potholes by analyzing the acquired pothole images. Specifically, it can analyze both the observable three-dimensional information of potholes at the current vehicle position and predict the three-dimensional information of potholes that are not observable at the current vehicle position. Based on this obtained three-dimensional information, the depth of the pothole can be determined, and if the depth exceeds a threshold, it can promptly and accurately alert the driver to avoid it. By using the complete three-dimensional information of the potholes analyzed from the acquired images to determine their depth, pothole depth detection becomes more accurate, the detected pothole area is wider, and the warning effect is better.

[0034] In this embodiment of the application, after capturing images of the vehicle's travel route using the front-facing camera, the captured images can be processed. The processing method for the captured images is as follows: Figure 2 As shown, it includes:

[0035] Step 201: Denoise the captured image.

[0036] Specifically, filtering is used to remove obvious noise from the image, thus preprocessing the image data, filtering out redundant data, and reducing the amount of data to be processed later.

[0037] Step 202: Perform image segmentation on the captured image to segment out the image containing the pit area.

[0038] Step 203: Extract the edges of the image containing the pit area to obtain the pit image.

[0039] Specifically, grayscale histogram equalization can be performed on the captured image, transforming the grayscale distribution of the current image into an image with a wider range and more uniform grayscale distribution through a transformation function. In other words, it can be described as modifying the histogram of the current image to be roughly uniformly distributed across the entire grayscale range, expanding the dynamic range of the image, enhancing the image contrast, and thus more accurately determining the contours of the pit areas, achieving pit area segmentation. After grayscale histogram equalization, vertical edge detection can be used to filter the image to obtain the depth image of the pit area.

[0040] Alternatively, edge values ​​can be determined using the local window edge value calculation method. The calculated edge values ​​are then used to adjust the pulse output of the pulse-coupled neural network, improving the adaptability and accuracy of edge extraction. The local window edge value calculation method is as follows: the pre-processed image is input into the pulse-coupled neural network, where model parameters are pre-set. The image's grayscale values ​​are used as external stimuli for the network neurons. All neurons are initialized to 0. In the first iteration, the neuron's internal activation equals the external stimulus, and the threshold of all neurons decays from its initial value. An iteration is completed when the threshold of a neuron decays to less than or equal to its corresponding internal activation. Multiple iterations can improve the quality of image edge detection. In practice, a suitable number of iterations can be pre-set, and edge extraction of the pitted area image is completed after meeting the required number of iterations. Furthermore, before edge extraction, to enhance image contrast, the image can be binarized to obtain a binary image, which is then input into the pulse-coupled neural network.

[0041] Furthermore, the potholes along the route may be puddles. Since the image features of water surfaces and ground surfaces differ, both scenarios need to be considered during image processing. A first target region matching the water surface fluctuation range is selected from the captured images, and a second target region matching the ground pothole range is selected from the captured images. The intersection of the first and second target regions is taken as the image containing the pothole region. Specifically, after selecting the first target region matching the water surface fluctuation range, it is grayscale processed. Based on the brightness distribution of different regions in the image, a local threshold is calculated. Different thresholds are adaptively determined for different regions of the image. Edge detection is performed based on the determined thresholds, and the edges are smoothed and isolated points are removed to obtain the processed first target region. Similarly, after selecting the second target region, it is grayscale processed. Edge detection is performed based on the grayscale gradient information, and the gradient image is filtered and isolated points are removed to obtain the processed second target region. Finally, the intersection of the processed first and second target regions is used to obtain the image containing the pothole region.

[0042] Additionally, the second target region can be determined as follows: Any two target points located at the edge of the pit in the captured image are identified, where the distance between the two target points is greater than a preset threshold; a circular region with the diameter of the two target points is then defined as the second target region. The selection of target points is as follows... Figure 3 As shown, the dashed line represents the pit area image after edge processing. Any two points are selected at the position of the dashed line as target points. The distance between the two selected target points is approximately between 0.5 and 1.2 meters. If the two selected target points are too close, it may cause the range of the second target area to be missing. In addition, considering the common pit size, it is not easy to have excessively large pits in daily scenes. Therefore, the distance between the two target points is not often too far.

[0043] In this embodiment of the application, generating three-dimensional information of potholes along the vehicle's travel route based on pothole images and determining the depth of the potholes from the three-dimensional information can be achieved in the following ways:

[0044] One implementation method is to determine the depth of the pit using the depth information of the pit image, as follows: Figure 4 As shown, it includes:

[0045] Step 401: Determine the position of the marker in the captured image.

[0046] Step 402: Determine the depth of the pit based on the difference between the depth information of the marker in the pit image and the depth information of the pit area.

[0047] Specifically, after taking the image, such as Figure 5 As shown, a topological map of the image is constructed by combining perspective projection. Figure 5 The horizontal plane representing the road surface is used as the reference plane, and the direction perpendicular to the reference plane is selected as the reference direction. The position of a pre-set marker is determined in the image. The marker can be a common, infrequently moving object on the road, such as a traffic light or sign. PP' in the image represents the position of the marker. A relatively easy-to-locate and detect corner point P' is determined in the topology map. Point P is the intersection of a straight line passing through P' and parallel to the reference direction with the reference plane. The distance from PP' represents the depth information Z of the marker. Additionally, the depth information of the front-facing camera Qc is Z0. The depth of the pothole is determined by the difference between the depth information of the marker and the depth information of the pothole area in the pothole image, or by the difference between the depth information of the marker and the depth information of the water surface in the pothole. Furthermore, when determining the pothole depth using the marker, it is necessary not only to cut out the pothole area but also to preserve the area where the marker is located.

[0048] Another approach is to use a backpropagation (BP) neural network model to predict the depth of potholes. TensorFlow is used to implement distributed training of the neural network on the samples, recording the mapping position of each sample (i.e., the corresponding neuron), the connection weights of each neuron, and finding the Euclidean distance between each neuron and all samples mapped to it. Based on the optimized connection weights of the BP neural network, these parameters from the first-stage training are then further optimized using gradient descent. The parameters obtained after both optimizations are used as the final parameters to predict the road water depth and level.

[0049] Additionally, the depth of the pit can be determined through measurement, including configuring camera parameters. Figure 3 The image at point A includes any point B on the edge of the pit. The image at point A includes the depth of the pit, and the image at point B includes the depth of the pit. The pit image is captured three times, and the sensor data at each capture is saved. The captured images are preprocessed and corrected. The camera is then calibrated using multiple sets of sensors to establish the spatial coordinate correspondence between the scene and the image, thereby reconstructing the three-dimensional information of the image and calculating the depth of the pit based on the three-dimensional information.

[0050] In addition, if the water level in the pothole is not at the same level as the road surface, the water level and the distance between the water level and the road surface can be detected separately, and the total depth of the pothole can be determined by summing them up. By using different algorithm rules to calculate the depth corresponding to different media areas, the accuracy of the calculation can be improved.

[0051] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0052] This invention also relates to an electronic device, such as... Figure 6 As shown, it includes at least one processor 601; and a memory 602 communicatively connected to at least one processor 601; wherein the memory 602 stores instructions executable by at least one processor 601, the instructions being executed by at least one processor 601 to enable at least one processor 601 to perform the pit detection method in the above embodiments.

[0053] The memory 602 and processor 601 are connected via a bus, which may include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 601 and memory 602 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 601 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 601.

[0054] Processor 601 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 602 can be used to store data used by processor 601 during operation.

[0055] This invention also relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described method embodiments.

[0056] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0057] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of the present invention.

Claims

1. A method for detecting potholes, characterized in that, include: Collect images of potholes along the vehicle's route; Based on the pothole image, generate three-dimensional information of the potholes along the vehicle's travel route, and determine the depth of the potholes using the three-dimensional information; An alarm message will be issued if the depth exceeds a preset threshold. The images of potholes along the vehicle's route include: The vehicle's front-facing camera is used to capture images, and the presence of potholes in the captured images is identified. If such a target area exists, select a first target area in the captured image that matches the range of water surface ripples, and select a second target area in the captured image that matches the range of ground potholes. The intersection of the first target region and the second target region is taken as the image containing the pit region; The pit image is obtained by edge extraction of the image containing the pit area.

2. The pothole detection method according to claim 1, characterized in that, Before performing image segmentation on the captured image, the method further includes: The captured images are then subjected to noise reduction processing.

3. The pothole detection method according to claim 1, characterized in that, The selection of the second target area in the captured image that matches the range of the ground potholes includes: Determine any two target points at the edge of the pit in the captured image, wherein the distance between the two target points is greater than a preset threshold; The circular region with the diameter of the two target points is taken as the second target region.

4. The pothole detection method according to any one of claims 1 to 3, characterized in that, Following the capture using the vehicle's front-facing camera, the following is also included: Determine the position of the marker in the captured image; The step of generating three-dimensional information of potholes along the vehicle's travel route based on the pothole image, and determining the depth of the potholes using the three-dimensional information, includes: The depth of the pit is determined based on the difference between the depth information of the marker in the pit image and the depth information of the pit region.

5. The pothole detection method according to any one of claims 1 to 3, characterized in that, The step of generating three-dimensional information of potholes along the vehicle's travel route based on the pothole image includes: The image of the potholes is input into a BP neural network model; The 3D information of the pothole image is reconstructed using the BP neural network model.

6. The pothole detection method according to any one of claims 1 to 3, characterized in that, After determining the depth of the pit using the three-dimensional information, the method further includes: Upload the depth of the pit to the sharing platform.

7. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the pit detection method as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the pit detection method according to any one of claims 1 to 6.

Images