Method and device for recognizing road surface well lid and reminding driver

Abstract

The invention provides a method and device for recognizing a road surface well lid and reminding a driver, and belongs to the technical field of well lid recognition. The method for recognizing the road surface well lid and reminding the driver comprises the following steps that S1, road image data and three-dimensional point cloud data are collected respectively; s2, well lid image data and ground point cloud data are obtained respectively; s3, respectively acquiring first position information and second position information; and S4, if the first position information and the second position information meet the first preset condition, the well lid is judged to be in an open state, and if the first position information and the second position information do not meet the first preset condition, the well lid is judged to be in a closed state. The state of the well lid is judged based on the first position information and the second position information, so that the state of the well lid can be judged to be the open state or the closed state, the state of the well lid is accurately recognized, and the purposes of reminding a driver of the road condition and reducing the driving risk can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of manhole cover recognition, in particular to a method and device for recognizing a road manhole cover and reminding a driver. BACKGROUND

[0002] During driving on urban roads, the state of a manhole cover is crucial for driving safety. If the manhole cover is in an open state, it may cause a vehicle to sink or a collision accident. Traditional manhole cover detection methods mainly rely on manual patrols, which are inefficient and difficult to monitor in real time. In recent years, the development of vehicle-mounted sensor technology has provided new possibilities for real-time monitoring of manhole cover states. However, single-sensor detection methods have limitations, for example, image processing-based recognition methods are susceptible to factors such as light and weather, while laser radar can obtain three-dimensional point cloud data, but the recognition accuracy of manhole covers in complex environments is limited. SUMMARY

[0003] To solve the above problems, the present application provides a method and device for recognizing a road manhole cover and reminding a driver.

[0004] In a first aspect, the present application provides a method for recognizing a road manhole cover and reminding a driver, comprising the following steps: S1: respectively collecting road image data and three-dimensional point cloud data of a first preset area in front of the vehicle; S2: respectively acquiring manhole cover image data based on the road image data and ground point cloud data based on the three-dimensional point cloud data; S3: respectively acquiring first position information corresponding to the manhole cover in the manhole cover image data and second position information corresponding to the pit in the ground point cloud data; S4: judging the state of the manhole cover based on the first position information and the second position information, if the first position information and the second position information satisfy a first preset condition, determining that the state of the manhole cover is open, if the first position information and the second position information do not satisfy the first preset condition, determining that the state of the manhole cover is closed.

[0005] Preferably, after step S4, the method further comprises: if it is determined that the state of the manhole cover is open, issuing a prompt message, and if it is determined that the state of the manhole cover is closed, not issuing a prompt message.

[0006] Preferably, in step S2, acquiring the manhole cover image data based on the road image data specifically comprises: performing a first preset processing on the road image data to obtain processed image data; acquiring a first learning model, and detecting the processed image data through the first learning model, if the processed image data includes a manhole cover feature, regarding the processed image data as the manhole cover image data.

[0007] Preferably, the obtaining the first learning model specifically comprises: constructing a first training model; obtaining first training image samples, the first training image samples comprising a plurality of manhole cover features of different manhole cover states under different environments and / or different illumination conditions; and training the first training model based on the first training image samples and a convolutional neural network to obtain the first learning model.

[0008] Preferably, the first preset processing of the road image data to obtain the processing image data specifically comprises: performing a preprocessing operation on the road image data to obtain first preprocessing data, the preprocessing operation comprising a grayscale operation, a binaryzation operation and a denoising operation; and selecting a first preset region from the first preprocessing data as the processing image data.

[0009] Preferably, the obtaining the ground point cloud data based on the three-dimensional point cloud data in step S2 specifically comprises: performing a second preset processing on the three-dimensional point cloud data to obtain processing point cloud data; obtaining a point cloud separation model based on a RANSAC algorithm, inputting the processing point cloud data into the point cloud separation model, and outputting the ground point cloud data with a preset area from the point cloud separation model.

[0010] Preferably, the obtaining the second position information corresponding to the pit in the ground point cloud data in step S3 specifically comprises: analyzing the ground point cloud data, if the point cloud at a first position in the ground point cloud data is a null value, taking the first position as the pit; obtaining first coordinate information at a peripheral edge position of the pit, obtaining second coordinate information of the pit based on the first coordinate information at the peripheral edge position, and taking the second coordinate information as the second position information corresponding to the pit.

[0011] In a second aspect, an embodiment of the present application provides a device for identifying a manhole cover on a road surface and reminding a driver, comprising a vehicle-mounted camera, a vehicle-mounted laser radar, an image data processing module, a point cloud data processing module, a positioning module, a pit determination module and a manhole cover state acquisition module. The vehicle-mounted camera is configured to collect road image data of a first preset region in front of the vehicle. The vehicle-mounted laser radar is configured to collect three-dimensional point cloud data of the first preset region in front of the vehicle. The image data processing module is configured to obtain manhole cover image data based on the road image data. The point cloud data processing module is configured to obtain ground point cloud data based on the three-dimensional point cloud data. The positioning module is configured to obtain first position information corresponding to a manhole cover in the manhole cover image data. The pit determination module is configured to obtain second position information corresponding to a pit in the ground point cloud data. The well lid state acquisition module is used to judge the well lid state based on the first position information and the second position information, and if the first position information and the second position information satisfy the first preset condition, the well lid state is determined as an open state, and if the first position information and the second position information do not satisfy the first preset condition, the well lid state is determined as a closed state.

[0012] Preferably, the method further comprises a control module, which is used to send a prompt information if the well lid state is determined as the open state, and not send the prompt information if the well lid state is determined as the closed state.

[0013] Preferably, the image data processing module comprises a first processing unit and a first detection unit, and the point cloud data processing module comprises a second processing unit and a first separation unit. The first processing unit is used to perform a first preset processing on the road image data to obtain processed image data. The first detection unit is used to obtain a first learning model, and detect the processed image data through the first learning model, and if the processed image data comprises a well lid feature, the processed image data is taken as well lid image data. The second processing unit is used to perform a second preset processing on the three-dimensional point cloud data to obtain processed point cloud data. The first separation unit is used to obtain a point cloud separation model based on a RANSAC algorithm, input the processed point cloud data into the point cloud separation model, and the point cloud separation model outputs ground point cloud data with a preset area.

[0014] The method has the advantages that the well lid image data is obtained based on the road image data, the first position information corresponding to the well lid in the well lid image data is obtained, the ground point cloud data is obtained based on the three-dimensional point cloud data, the second position information corresponding to the pit in the ground point cloud data is obtained, and the well lid state is judged based on the first position information and the second position information, so that the well lid state can be determined as the open state or the closed state, the state of the well lid is accurately recognized, and the driver can be prompted about the road condition and the driving risk can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0016] Figure 1 A flowchart of a method for recognizing a road well lid and reminding a driver according to an embodiment of the present application is shown in the figure. Figure 2A schematic diagram of the structure of a device for identifying road manhole covers and alerting drivers provided in an embodiment of the present application; Figure 3 A block diagram of the principle of a device for identifying road manhole covers and alerting drivers, provided in an embodiment of the present application; Figure 4 Ground point cloud data corresponding to a road surface with potholes and a road surface without potholes in a method for identifying road manhole covers and alerting drivers provided in an embodiment of the present application; Figure 5 The actual road surface pattern to be detected in the method for identifying a road manhole cover and alerting a driver provided in an embodiment of the present application; Figure 6 In a method for identifying a road manhole cover and reminding the driver provided in an embodiment of the present application Figure 5 The ground point cloud data corresponding to the actual road surface graphics. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0018] In the following introduction, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The following introduction provides multiple embodiments of the present application. Different embodiments can be replaced or combined, so the present application can also be considered to include all possible combinations of the same and / or different embodiments described. Therefore, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present application should also be considered to include embodiments containing one or more of all other possible combinations of A, B, C, and D, even though the embodiment may not be clearly described in the following text.

[0019] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the elements described without departing from the scope of the present application. Various examples may appropriately omit, replace, or add various processes or components. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted, or combined. In addition, features described in some examples may be combined in other examples.

[0020] See also Figure 1 、 3 -6. Figure 1 This is a flow chart of a method for identifying road manhole covers and alerting drivers provided in an embodiment of the present application. Figure 3 This is a block diagram of the principle of a device for identifying road manhole covers and alerting drivers provided in an embodiment of the present application. Figure 4The ground point cloud data corresponding to the actual road surface pattern with pits and the ground point cloud data corresponding to the actual road surface pattern without pits are respectively acquired in a method for identifying a road surface manhole cover and reminding a driver provided in an embodiment of the present application, Figure 5 The actual road surface pattern to be detected is acquired in a method for identifying a road surface manhole cover and reminding a driver provided in an embodiment of the present application, Figure 6 The ground point cloud data corresponding to the actual road surface pattern with pits and the ground point cloud data corresponding to the actual road surface pattern without pits are respectively acquired in a method for identifying a road surface manhole cover and reminding a driver provided in an embodiment of the present application, Figure 5 In an embodiment of the present application, the method comprises the following steps: S1: acquiring road image data and three-dimensional point cloud data of a first preset area in front of a vehicle respectively; S2: acquiring manhole cover image data based on the road image data and acquiring ground point cloud data based on the three-dimensional point cloud data respectively; S3: acquiring first position information corresponding to a manhole cover in the manhole cover image data and acquiring second position information corresponding to pits in the ground point cloud data respectively; S4: judging a manhole cover state based on the first position information and the second position information, and determining that the manhole cover state is an open state if the first position information and the second position information satisfy a first preset condition, or determining that the manhole cover state is a closed state if the first position information and the second position information do not satisfy the first preset condition.

[0021] In an embodiment of the present application, the current vehicle intelligent system only contains a rendering function of a sewer manhole cover and cannot identify the open and close states of the manhole cover. For an open manhole cover, there is a risk of falling or bumping the chassis for the vehicle driving. In the present application, the state of the manhole cover is accurately judged by combining road image data and three-dimensional point cloud data, so as to provide real-time safety prompts for the driver.

[0022] In an embodiment of the present application, road image data of a first preset area in front of a vehicle is acquired by a vehicle-mounted camera, three-dimensional point cloud data of the first preset area in front of the vehicle is acquired by a vehicle-mounted laser radar, the state of the manhole cover is accurately judged by combining the data of the vehicle-mounted camera and the laser radar, and real-time safety prompts are provided for the driver. The detection capabilities of the vehicle-mounted camera and the vehicle-mounted laser radar are combined to intelligently identify the open or closed state of the manhole cover, so as to prompt the driver about the road conditions and reduce the driving risk.

[0023] In an implementation manner, after the step S4, the method further comprises: if it is determined that the manhole cover state is the open state, issuing a prompt information, and if it is determined that the manhole cover state is the closed state, not issuing the prompt information.

[0024] In the embodiment of the present application, the present application sends a prompt information to the driver according to the cover state. If the cover state is a closed state, no prompt information is sent. If the cover state is an open state, a prompt information is sent to the driver to remind the driver to pay attention to the intersection and to suggest detouring or braking.

[0025] In the embodiment of the present application, the step S4 further includes displaying the cover state.

[0026] In the embodiment of the present application, the cover state can be displayed by a display mode, thereby providing an intuitive prompt for the driver.

[0027] In an implementable manner, the step S2 of acquiring the cover image data based on the road image data specifically includes: performing first preset processing on the road image data to acquire processing image data; acquiring a first learning model, and detecting the processing image data by the first learning model. If the processing image data includes a cover feature, the processing image data is taken as the cover image data.

[0028] In the embodiment of the present application, a neural network algorithm is used to identify the road cover in the road image data to acquire the cover image data.

[0029] In an implementable manner, the step of acquiring the first learning model specifically includes: constructing a first training model; acquiring a first training image sample, the first training image sample including a plurality of cover features in different cover states under different environments and / or different illumination conditions; and training the first training model based on the first training image sample and a convolutional neural network to acquire the first learning model.

[0030] In the embodiment of the present application, the first training model can be a deep learning model such as a convolutional neural network (CNN), and common models include Faster R-CNN, YOLO, SSD, etc.

[0031] In an implementable manner, the step of performing first preset processing on the road image data to acquire processing image data specifically includes: performing a preprocessing operation on the road image data to obtain first preprocessing data, the preprocessing operation including a grayscale operation, a binaryzation operation and a denoising operation; and selecting a first preset region from the first preprocessing data as the processing image data.

[0032] In the embodiment of the present application, a convolutional neural network (such as an improved YOLOv5 model) is used to process the image data, and the cover in the image is identified and its position information is judged through feature extraction, pattern recognition and target detection.

[0033] In the embodiments of the present application, the method for obtaining the first position information can be as follows: 1. Data collection: using a vehicle-mounted camera to collect road image data of a first preset area in front of the vehicle; 2. Image preprocessing: preprocessing the collected road image data, including grayscale, binarization, denoising and the like, to improve the image quality and reduce the difficulty of subsequent processing; selecting a region of interest, i.e., a region that may contain a manhole cover, to reduce the amount of calculation and improve the detection efficiency; 3. Model training: collecting a large number of first training image samples labeled with manhole cover features, the first training image samples including a plurality of manhole cover features of different manhole cover states under different environments and / or different illumination conditions; using a deep learning model such as a convolutional neural network (CNN) as a first training model, training the first training model, and common models include Faster R-CNN, YOLO, SSD and the like; in the training process, the first training model is optimized, the network structure, activation function, loss function and the like are adjusted, to improve the accuracy and robustness of the model, thereby obtaining a first learning model; 4. Manhole cover detection: inputting the preprocessed region of interest into the trained first learning model, i.e., inputting the processed image data into the trained first learning model, the first learning model will output the state of the manhole cover in the region, such as whether it is missing or damaged, thereby obtaining manhole cover image data; for manhole covers of different shapes (such as circular, square, etc.), a plurality of sub-models can be trained to improve the accuracy and efficiency of detection; 5. Result processing: combining the first position information of the manhole cover obtained by the positioning module, sending the alarm information and the position information to the background processing module, for subsequent maintenance and management.

[0034] In the embodiments of the present application, obtaining the first position information corresponding to the manhole cover in the manhole cover image data can include: 1. Vehicle sensor fusion positioning: real-time scanning of the front road surface through the dual-camera, millimeter-wave radar and ultrasonic radar (up to 26 sensors in total) mounted on the vehicle, the deep learning algorithm can identify the manhole cover edge profile and damage features within a range of 15 meters, and the suspension system can be adjusted in advance to avoid bumps; the transparent chassis technology uses four-way cameras on the vehicle body to capture road textures (such as manhole cover patterns, lane lines), and through a space-time jigsaw algorithm, historical pictures are dynamically projected onto the display screen at the bottom of the vehicle, realizing a virtual perspective effect (best at a low speed of 5km / h). 2. Special detection equipment: the inspection vehicle is equipped with high-resolution cameras or laser radars to collect road images, and combines a convolutional neural network (CNN) model to detect and segment the manhole cover, and real-time feedback of missing or displacement status.

[0035] In the embodiments of the present application, obtaining the first position information corresponding to the manhole cover in the manhole cover image data can also include: 1, municipal Internet of Things system access: 1, platform data docking: the municipal management platform integrates manhole cover sensor data (inclination angle, water level, displacement, etc.), and authorized vehicles can apply for API interface access to the vehicle-mounted system, and the navigation interface superimposes the real-time position and state of the manhole cover; 2, emergency alarm linkage: when the inclination of the manhole cover is greater than 15° or the displacement is greater than 5 cm, the management platform pushes the avoidance warning and coordinates (Beidou / GPS positioning error ≤ 3 meters) to nearby vehicles; 2, vehicle active detection technology: 1, transparent chassis function: using the camera at the bottom of the vehicle body to scan the road surface, and using image recognition algorithm to mark the manhole cover contour in real time, and projecting to the center control screen (the effect is best when driving at low speed); 2, sensor fusion identification: millimeter wave radar + camera combination detection: radar identifies the metal reflection characteristics of the manhole cover, and visual algorithm matches the texture pattern of the manhole cover, which can display the position of the manhole cover within 10 meters in front of the vehicle on the instrument panel (the vehicle has L2+ level automatic driving hardware).

[0036] In an implementation manner, the ground point cloud data is obtained based on the three-dimensional point cloud data in the step S2, specifically including: performing second preset processing on the three-dimensional point cloud data to obtain processed point cloud data; and inputting the processed point cloud data into a point cloud separation model based on a RANSAC algorithm, so that the ground point cloud data with a preset area is output by the point cloud separation model.

[0037] In the embodiments of the present application, the method for separating the ground point cloud data by using the RANSAC algorithm is as follows: 1, preprocessing: performing depth screening, camera FOV segmentation and other operations on the original point cloud to obtain the preprocessed region of interest (ROI) point cloud, that is, to obtain the processed point cloud data, and the point cloud data can be preprocessed, such as down-sampling, to reduce the data amount and improve the processing efficiency; 2, RANSAC algorithm for separating the ground point cloud: initial parameters such as iteration number K and distance threshold \(D_{m}\) are given; three non-collinear points are randomly selected from the preprocessed point cloud as initial inner points, and a plane model is fitted by using the three points, and the plane model is represented as \(Ax+By+Cz+D=0\); the distance of all points in the point cloud to the plane model is calculated, and the points with a distance less than the set threshold are called inner points, and the remaining points are called outer points; three points are randomly selected from the inner points, and a new plane is calculated, and the above operation is repeated; after a sufficient number of iterations, RANSAC can converge to a model with more inner points, and the model can be considered as the largest plane, that is, the ground point cloud; 3, result processing: the ground point cloud obtained by the RANSAC algorithm is separated from the original point cloud to obtain the separated ground point cloud and non-ground point cloud, and the separated point cloud data can be published to the ROS system for subsequent processing and analysis.

[0038] In one possible implementation, obtaining the second position information corresponding to the pothole in the ground point cloud data in step S3 specifically includes: analyzing the ground point cloud data, and if the point cloud at the first position in the ground point cloud data is a null value, then treating the first position as a pothole; obtaining the first coordinate information at the outer edge position of the pothole, obtaining the second coordinate information of the pothole based on the first coordinate information at the edge position, and using the second coordinate information as the second position information corresponding to the pothole.

[0039] In an embodiment of the present application, if there is a pothole on the ground, the laser radar point cloud is a null value, and if there is no pothole on the road surface, the laser radar can reflect the point cloud normally. If the point cloud at the first position in the ground point cloud data is a null value, the first position is regarded as a pothole.

[0040] In this embodiment of the present application, a vehicle-mounted laser radar acquires three-dimensional point cloud data around the vehicle. A RANSAC algorithm is used to isolate the point cloud data of the road ahead of the vehicle. This ground point cloud data is analyzed, and the location of potholes ahead of the vehicle is determined based on the continuity of the point cloud.

[0041] In an embodiment of the present application, step S4 specifically includes: if the first position information and the second position information are the same, determining that the manhole cover state is open; if the first position information and the second position information are different, determining that the manhole cover state is closed.

[0042] In an embodiment of the present application, the state of the manhole cover (closed or open) is determined by combining the first position information of the manhole cover recognized by the vehicle-mounted camera and the pothole position information determined by the vehicle-mounted laser radar.

[0043] In this application real-time example, see Figure 5 , Figure 5 The figure shows an actual road surface to be tested. A 128-line laser radar was used to test a road with potholes and a flat road. The ground point cloud data obtained is as follows: Figure 6 As shown in the figure, it can be seen from the test results that the lidar point cloud is empty on the ground with potholes; on the road without potholes, the lidar can reflect the point cloud normally.

[0044] In the embodiments of the present application, 1, through image recognition, whether there is a manhole cover in front of the road is obtained, and the position information of the manhole cover is obtained; 2, for the result of the first step, if there is a manhole cover in front, according to the manhole cover position information, through the laser radar point cloud, the pit position information in front of the road can be obtained. Here, the position information obtained by the two sensors needs to be compared, because there may be multiple manhole covers in front. For the front position A perceived by the camera, there is a manhole cover, and the laser radar also perceives that position A has a pit, it is considered that position A has an open manhole cover, which needs to be alarmed; if the front position B perceived by the camera has a manhole cover, and the laser radar perceives that position B has no pit, it is considered that position B has a closed manhole cover, which does not need to be alarmed; similarly, if the camera perceives that there is no manhole cover in front, it directly does not need to be alarmed.

[0045] In the embodiments of the present application, the present application has the following effects: (1) high accuracy: by fusing the data of the vehicle-mounted camera and the vehicle-mounted laser radar, the state of the manhole cover can be more accurately identified, avoiding the limitations of a single sensor; (2) real-time: sensor data can be processed in real time and the state of the manhole cover can be quickly judged, and timely safety prompts can be provided for the driver; (3) environmental adaptability: the convolutional neural network can adapt to different light and weather conditions, and the point cloud data of the laser radar is not affected by light, improving the environmental adaptability of the system; (4) safety: by timely reminding the driver of the state of the manhole cover, traffic accidents caused by the opening of the manhole cover can be effectively avoided, and driving safety can be improved.

[0046] The embodiments of the present application will be described below with reference to the accompanying drawings. Figure 2 A device for identifying a road manhole cover and reminding a driver provided by the embodiments of the present application will be described in detail. It should be noted that the device for identifying a road manhole cover and reminding a driver shown in the accompanying drawings is used to execute the method of the embodiments of the present application. Figure 2 The device for identifying a road manhole cover and reminding a driver shown in the accompanying drawings is used to execute the method of the embodiments of the present application. For ease of description, only parts related to the embodiments of the present application are shown, and specific technical details not disclosed are described with reference to the embodiments shown in the accompanying drawings. Figure 1 The device for identifying a road manhole cover and reminding a driver shown in the accompanying drawings is used to execute the method of the embodiments of the present application. For ease of description, only parts related to the embodiments of the present application are shown, and specific technical details not disclosed are described with reference to the embodiments shown in the accompanying drawings. Figure 1 The device for identifying a road manhole cover and reminding a driver shown in the accompanying drawings is used to execute the method of the embodiments of the present application. For ease of description, only parts related to the embodiments of the present application are shown, and specific technical details not disclosed are described with reference to the embodiments shown in the accompanying drawings.

[0047] Please refer to Figure 2 , Figure 2 is a structural schematic diagram of a device for identifying a road manhole cover and reminding a driver provided by the embodiments of the present application. As shown in Figure 2 , the device comprises a vehicle-mounted camera 201, a vehicle-mounted laser radar 202, an image data processing module 203, a point cloud data processing module 204, a positioning module 205, a pit determination module 206 and a manhole cover state acquisition module 207; The vehicle-mounted camera 201 is used to collect road image data of a first preset area in front of the vehicle; The vehicle-mounted laser radar 202 is configured to collect three-dimensional point cloud data of a first preset area in front of the vehicle. The image data processing module 203 is configured to obtain cover image data based on road image data. The point cloud data processing module 204 is configured to obtain ground point cloud data based on the three-dimensional point cloud data. The positioning module 205 is configured to obtain first position information corresponding to the cover in the cover image data. The pit determination module 206 is configured to obtain second position information corresponding to the pit in the ground point cloud data. The cover state obtaining module 207 is configured to determine the state of the cover based on the first position information and the second position information. If the first position information and the second position information satisfy a first preset condition, it is determined that the state of the cover is an open state. If the first position information and the second position information do not satisfy the first preset condition, it is determined that the state of the cover is a closed state.

[0048] In the embodiments of the present application, the display module configured to display the state of the cover is further included, so as to provide intuitive prompts for the driver.

[0049] In an implementable manner, the control module 208 is further included, configured to issue a prompt information if it is determined that the state of the cover is the open state, and not to issue the prompt information if it is determined that the state of the cover is the closed state.

[0050] In an implementable manner, the image data processing module 203 includes a first processing unit and a first detection unit; and the point cloud data processing module 204 includes a second processing unit and a first separation unit. The first processing unit is configured to perform first preset processing on the road image data to obtain processed image data. The first detection unit is configured to obtain a first learning model, and detect the processed image data through the first learning model. If the processed image data includes cover features, the processed image data is taken as the cover image data. The second processing unit is configured to perform second preset processing on the three-dimensional point cloud data to obtain processed point cloud data. The first separation unit is configured to obtain a point cloud separation model based on a RANSAC algorithm, input the processed point cloud data into the point cloud separation model, and output ground point cloud data with a preset area from the point cloud separation model.

[0051] In the embodiments of the present application, the first learning model obtained in the first detection unit specifically comprises: constructing a first training model; obtaining a first training image sample, the first training image sample comprising a plurality of manhole cover features of different manhole cover states under different environments and / or different illumination conditions; and training the first training model based on the first training image sample and a convolutional neural network to obtain the first learning model.

[0052] In the embodiments of the present application, the first detection unit specifically comprises: performing a first preset processing on the road image data to obtain processed image data, the first preset processing comprising: performing a preprocessing operation on the road image data to obtain first preprocessed data, the preprocessing operation comprising a grayscale operation, a binaryzation operation and a denoising operation; and selecting a first preset region from the first preprocessed data as the processed image data.

[0053] In the embodiments of the present application, the pit determination module 206 further comprises: analyzing the ground point cloud data, if the point cloud at the first position in the ground point cloud data is a null value, then the first position is taken as a pit; obtaining first coordinate information at the edge position of the outer periphery of the pit, obtaining second coordinate information of the pit based on the first coordinate information at the edge position, and taking the second coordinate information as the second position information corresponding to the pit.

[0054] In the embodiments of the present application, the method for obtaining the first position information can be as follows: 1. Data collection: using a vehicle-mounted camera to collect road image data of a first preset area in front of the vehicle; 2. Image preprocessing: preprocessing the collected road image data, including grayscale, binarization, denoising and other operations, to improve the image quality and reduce the difficulty of subsequent processing; selecting a region of interest, i.e. a region that may contain a manhole cover, to reduce the amount of calculation and improve the detection efficiency; 3. Model training: collecting a large number of first training image samples labeled with manhole cover features, the first training image samples including a plurality of manhole cover features of different manhole cover states under different environments and / or different lighting conditions; using a deep learning model such as a convolutional neural network (CNN) as a first training model, training the first training model, and common models include Faster R-CNN, YOLO, SSD, etc.; in the training process, the first training model is optimized, the network structure, activation function, loss function, etc. are adjusted to improve the accuracy and robustness of the model, thereby obtaining a first learning model; 4. Manhole cover detection: inputting the preprocessed region of interest into the trained first learning model, i.e. inputting the processed image data into the trained first learning model, the first learning model will output the state of the manhole cover in the region, such as whether it is missing, whether it is damaged, etc., thereby obtaining manhole cover image data; for manhole covers of different shapes (such as circular, square, etc.), multiple sub-models can be trained to improve the accuracy and efficiency of detection; 5. Result processing: combining the positioning module to obtain the first position information of the manhole cover, sending the alarm information and position information to the background processing module for subsequent maintenance and management.

[0055] In the embodiments of the present application, the method for separating ground point cloud data by RANSAC algorithm is as follows: 1, preprocessing: performing depth screening, camera FOV segmentation and other operations on the original point cloud to obtain the preprocessed region of interest (ROI) point cloud, that is, to obtain the processed point cloud data, the point cloud data can be preprocessed, such as downsampling, to reduce the data amount and improve the processing efficiency; 2, RANSAC algorithm separates ground point cloud: given initial parameters such as iteration number K and distance threshold \(D_{m}\) ; randomly select three non-collinear points from the preprocessed point cloud as initial inliers, and fit a plane model using the three points, the plane model is represented as \(Ax+By+Cz+D=0\) ; calculate the distance of all points in the point cloud to the plane model, the points with a distance less than the set threshold are called inliers, and the remaining points are called outliers; randomly select three points from the inliers again, calculate a new plane, and repeat the above operation; after a sufficient number of iterations, RANSAC can usually converge to a model with more inliers, which can be considered as the largest plane, that is, the ground point cloud; 3, result processing: separate the ground point cloud obtained by the RANSAC algorithm from the original point cloud to obtain the separated ground point cloud and non-ground point cloud, and the separated point cloud data can be published to the ROS system for subsequent processing and analysis.

[0056] In the embodiments of the present application, 1, through image recognition, whether there is a manhole cover in front of the road is obtained, and the position information of the manhole cover is obtained; 2, for the result of the first step, if there is a manhole cover in front, according to the manhole cover position information, the laser radar point cloud can be used to obtain the pit position information in front of the road. Here, the position information obtained by the two sensors needs to be compared, because there may be multiple manhole covers in front. For the front position A perceived by the camera, there is a manhole cover, and the laser radar also perceives that position A has a pit, then it is considered that position A has an open manhole cover and needs to be alarmed; if the front position B perceived by the camera has a manhole cover, and the laser radar perceives that position B has no pit, then it is considered that position B has a closed manhole cover and does not need to be alarmed; similarly, if the camera perceives that there is no manhole cover in front, then it is directly unnecessary to alarm.

[0057] In the embodiments of the present application, the present application has the following effects: (1) high accuracy: by fusing the data of the vehicle-mounted camera and the vehicle-mounted laser radar, the state of the manhole cover can be more accurately identified, avoiding the limitations of a single sensor; (2) real-time: sensor data can be processed in real time and the state of the manhole cover can be quickly judged to provide safety prompts to the driver in a timely manner; (3) environmental adaptability: the convolutional neural network can adapt to different lighting and weather conditions, and the point cloud data of the laser radar is not affected by lighting, improving the environmental adaptability of the system; (4) Safety: By timely reminding the driver of the state of the manhole cover, traffic accidents caused by the opening of the manhole cover can be effectively avoided, and driving safety can be improved.

[0058] Those skilled in the art can clearly understand that the technical solutions of the embodiments of the present application can be implemented by means of software and / or hardware. The "unit", "module", and "part" in the specification refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, and the hardware may, for example, be a field programmable gate array (FPGA), an integrated circuit (IC), and the like.

[0059] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0060] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0061] In the several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division way for actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some services interfaces, devices or units, and can be electrical or other forms.

[0062] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the present embodiment.

[0063] In addition, each of the function units in each of the embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0064] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable memory. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods described in each of the embodiments of the present application. The aforementioned memory includes: a U disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0065] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be instructed by a program to be completed by relevant hardware, and the program can be stored in a computer readable memory, and the memory can include: a flash disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0066] The above is only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will easily think of embodiments of the present disclosure after considering the specification and practicing the disclosure herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for identifying road manhole covers and reminding drivers, characterized in that: The steps include: S1: Collecting road image data and three-dimensional point cloud data of a first preset area in front of the vehicle respectively; S2: Obtain manhole cover image data based on road image data and obtain ground point cloud data based on 3D point cloud data; S3: Obtain first position information corresponding to the manhole cover in the manhole cover image data and second position information corresponding to the pothole in the ground point cloud data; S4: The state of the manhole cover is judged based on the first position information and the second position information. If the first position information and the second position information meet the first preset condition, the state of the manhole cover is judged to be open; if the first position information and the second position information do not meet the first preset condition, the state of the manhole cover is judged to be closed.

2. A method for identifying a road manhole cover and reminding a driver according to claim 1, characterized in that: After step S4, the method further includes: issuing a prompt message if it is determined that the manhole cover is in the open state, and not issuing a prompt message if it is determined that the manhole cover is in the closed state.

3. A method for identifying a road manhole cover and reminding the driver according to claim 1 or 2, characterized in that: Acquiring manhole cover image data based on road image data in step S2 specifically includes: performing a first preset processing on the road image data to obtain processed image data; obtaining a first learning model, and detecting the processed image data through the first learning model. If the processed image data includes manhole cover features, the processed image data is used as manhole cover image data.

4. A method for identifying a road manhole cover and reminding a driver as claimed in claim 3, characterized in that: Obtaining the first learning model specifically includes: constructing a first training model; obtaining a first training image sample, wherein the first training image sample includes multiple manhole cover features of different manhole cover states under different environments and / or different lighting conditions; training the first training model based on the first training image sample and the convolutional neural network to obtain the first learning model.

5. The method for identifying a road manhole cover and reminding a driver as claimed in claim 3, characterized in that: Performing a first preset processing on the road image data to obtain processed image data specifically includes: performing a preprocessing operation on the road image data to obtain first preprocessed data, wherein the preprocessing operation includes a grayscale operation, a binarization operation, and a denoising operation; and selecting a first preset area from the first preprocessed data as processed image data.

6. A method for identifying road manhole covers and reminding drivers according to claim 1 or 2, characterized in that: In step S2, obtaining ground point cloud data based on three-dimensional point cloud data specifically includes: performing a second preset processing on the three-dimensional point cloud data to obtain processed point cloud data; obtaining a point cloud separation model based on the RANSAC algorithm, inputting the processed point cloud data into the point cloud separation model, and the point cloud separation model outputs ground point cloud data with an area of ​​a preset area.

7. A method for identifying road manhole covers and reminding drivers according to claim 1 or 2, characterized in that: Obtaining the second position information corresponding to the pothole in the ground point cloud data in step S3 specifically includes: analyzing the ground point cloud data, and if the point cloud at the first position in the ground point cloud data is a null value, then treating the first position as a pothole; obtaining the first coordinate information at the outer edge position of the pothole, obtaining the second coordinate information of the pothole based on the first coordinate information at the edge position, and using the second coordinate information as the second position information corresponding to the pothole.

8. A device for identifying road manhole covers and alerting drivers, characterized by: It includes an on-board camera, an on-board laser radar, an image data processing module, a point cloud data processing module, a positioning module, a pothole determination module, and a manhole cover status acquisition module; The vehicle-mounted camera is used to collect road image data of a first preset area in front of the vehicle; The vehicle-mounted laser radar is used to collect three-dimensional point cloud data of a first preset area in front of the vehicle; The image data processing module is used to obtain manhole cover image data based on the road image data; The point cloud data processing module is used to obtain ground point cloud data based on the three-dimensional point cloud data; The positioning module is used to obtain first position information corresponding to the manhole cover in the manhole cover image data; The pothole determination module is used to obtain second position information corresponding to the pothole in the ground point cloud data; The manhole cover status acquisition module is used to judge the manhole cover status based on the first position information and the second position information. If the first position information and the second position information meet the first preset condition, the manhole cover status is determined to be open; if the first position information and the second position information do not meet the first preset condition, the manhole cover status is determined to be closed.

9. The device for identifying a road manhole cover and alerting a driver as claimed in claim 8, characterized in that: The device further comprises a control module for issuing a prompt message if it is determined that the state of the manhole cover is open, and not issuing a prompt message if it is determined that the state of the manhole cover is closed.

10. A device for identifying road manhole covers and alerting drivers as claimed in claim 8 or 9, characterized in that: The image data processing module includes a first processing unit and a first detection unit; the point cloud data processing module includes a second processing unit and a first separation unit; The first processing unit is configured to perform a first preset processing on the road image data to obtain processed image data; The first detection unit is configured to obtain a first learning model, detect the processed image data using the first learning model, and if the processed image data includes a manhole cover feature, use the processed image data as manhole cover image data; The second processing unit is configured to perform a second preset processing on the three-dimensional point cloud data to obtain processed point cloud data; The first separation unit is used to obtain a point cloud separation model based on the RANSAC algorithm, input the processed point cloud data into the point cloud separation model, and the point cloud separation model outputs ground point cloud data with a preset area.

Images