Foresight all-in-one machine method based on visual inspection road bumpy road section early warning

Abstract

The invention discloses a foresight all-in-one machine method based on visual detection of road bumpy road section early warning, and relates to the technical field of road traffic. The problems that an existing front-view all-in-one machine product cannot deal with the conditions of road collapse and pits and does not have the active safety capacity of giving out early warning and braking to a driver under the conditions are solved; road collapse and pits in a camera image are detected by using a deep learning algorithm so as to be sent to a software rear end for early warning processing, software post-processing codes consider rules of various scenes, the positions of the collapse and the pits are estimated, and the residual distance and time of a vehicle facing danger are calculated so as to decide early warning and braking, so that the safety of the vehicle is improved. The information is sent to an alarm unit of the vehicle or the vehicle is braked in a speed reduction mode through the control module, and the driving safety is improved.

Description

Technical Field

[0001] The present invention relates to the field of road traffic technology, and in particular to a forward-looking integrated machine method based on visual detection of road pothole section warning. Background Art

[0002] According to statistics, the number of motor vehicles in China has reached 415 million, of which 318 million are cars; the number of motor vehicle drivers exceeds 500 million. At present, the total number of motor vehicles and drivers ranks first in the world. The first characteristic of highway safety accidents is the huge total number of traffic accidents, with more than 4.7 million accidents reported to traffic police each year; the second characteristic is the high mortality rate, with more than 200,000 casualties in road traffic safety accidents each year; the traffic mortality rate is as high as 21% (that is, the proportion of deaths to casualties in accidents), while Japan's traffic accident mortality rate is only 0.54%; the third characteristic is the frequent occurrence of vicious accidents, with multiple one-time multi-person fatalities occurring every year. The main reasons are the large number of vehicles, large number of people, large number of novices, weak safety awareness, insufficient maintenance of road traffic facilities, and traffic accidents caused by natural disasters. There are many traffic accidents caused by road pavement collapse and potholes. On January 9, 2023, a road collapsed at the entrance of Xinzhuang Square on Hongshan Road in Nanjing, Jiangsu, trapping two vehicles; on December 25, 2023, a road collapsed on the G55 Erguang Expressway in Taiyuan direction (K381), causing an accident involving four vehicles, resulting in three deaths and four minor injuries; on May 1, 2024, a landslide on the Mei-Da Expressway in Guangdong caused a total of 20 vehicles to fall, involving 54 people, and 24 deaths. Against the backdrop of rapid urban expansion in my country, urban road collapse accidents are becoming more frequent. According to incomplete public reports, a total of 244 urban road collapse accidents occurred in my country from April 2018 to May 2019, causing 39 injuries and 33 deaths. According to the statistics of the Shenzhen Municipal Ground Defense Office alone, there were 213 ground collapse accidents in the city in 2016, 238 in 2017, and 287 in 2018, showing a high growth trend. This shows that the prevention and control of ground collapse is long-term, complex and arduous, and also shows the urgency of warning and avoiding danger for drivers in the case of ground collapse during driving. However, the active safety of cars on the market can only warn and emergency brake the stationary vehicles and obstacles in front, which is the well-known AEB automatic emergency braking function, and cannot warn and avoid danger for road collapse and potholes.

[0003] Based on the above reasons, in order to provide early warning and avoidance braking for road collapse and potholes during the driving of motor vehicles, the present invention proposes a method for early warning of safe driving on pothole sections based on visual detection. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the shortcomings of the prior art, the present invention solves the problem that the existing front-view integrated machine products are unable to cope with road collapse and potholes, and do not have the active safety capability to issue warnings and brakes to the driver in this situation; the present invention uses a deep learning algorithm to detect road collapse and potholes in the camera image, and then provides them to the software backend for early warning processing. The software post-processing code takes into account the rules of various scenarios, estimates the location of collapse and potholes, and calculates the remaining distance and time that the vehicle faces danger, so as to make decisions on early warning and braking, and sends the information to the vehicle's alarm unit or brakes the vehicle to slow down through the control module, thereby improving driving safety.

[0006] (II) Technical solution

[0007] To achieve the above object, the present invention provides the following technical solution: a forward-looking integrated machine method for early warning of potholes on a road based on visual detection, comprising the following steps:

[0008] S1: Scene construction, artificially creating various types of collapses and potholes on the closed experimental road surface in the physical world;

[0009] S2: Image collection, using a data collection vehicle equipped with a data collection camera to complete image collection in the built scene;

[0010] S3: Neural network model training: neural network model training starts on the AI ​​training server and ends when the number of training times is reached or the training verification performance is optimal.

[0011] S4: Model deployment: Deploy the neural network algorithm to the controller of the front-view integrated machine to detect road collapse and potholes during driving;

[0012] S5: Use of the model, post-processing software completes the decoding and coordinate position calculation of the neural network detection of road collapse and pothole targets, estimates the distance between the current vehicle collapse and pothole targets, and takes corresponding treatment measures.

[0013] Preferably, in step S3, training the neural network model on the AI ​​training server includes the following steps:

[0014] S11: Data enhancement, enhancing the collected image data so that the band collapse and pits can be clearly displayed;

[0015] S12: image annotation, using annotation tools to annotate images with collapse and pits;

[0016] S13: Optimize training, use the processed images to train the model, update the model parameters through the back propagation algorithm, and gradually improve the model performance.

[0017] Preferably, the deep learning AI algorithm used in step S3 is Yolov5.

[0018] Preferably, the hardware of the forward-looking integrated machine in step S4 uses a 5-megapixel / 8-megapixel camera. The detection distance of the camera for different objects may be different. It can recognize traffic signs at 100 meters, bicycles and pedestrians at 140 meters, and cars at 250 meters. The narrow-angle camera FOV: the field of view angle is about 30-40°, and the detection distance is 250 meters. The hardware of the forward-looking integrated machine also has a controller, which integrates a CPU processor, an AI accelerator NPU, and a microcontroller MCU chip.

[0019] Preferably, in step S5, the operation of the post-processing software includes the following steps:

[0020] S21: Using the front-view integrated device to perform a vision-based real-time data shooting operation of a road image ahead of the driving road;

[0021] S22: Decoding the detected target object in the acquired image using a neural network algorithm;

[0022] S23: estimating the distance between the current vehicle and the collapsed and dented target objects, and calculating the time to travel to the collapsed and dented target objects according to the current vehicle speed;

[0023] S24: The post-processing software makes corresponding warning and braking decision plans based on the photographed target and the calculated time, and uploads them to the vehicle computer for execution.

[0024] Preferably, in step S24, the post-processing software also needs to communicate with the vehicle's CAN network to obtain the vehicle's speed, steering wheel angle, braking, and gear information. The post-processing software also needs to issue a warning signal to control the vehicle's deceleration and braking, as well as the steering wheel steering signal.

[0025] (III) Beneficial effects

[0026] Compared with the prior art, the present invention provides a forward-looking integrated machine method for warning of potholes on a road based on visual detection, which has the following beneficial effects: the present invention solves the problem that the existing forward-looking integrated machine products are unable to cope with road collapse and potholes, and do not have the active safety capability to issue warnings and brakes to the driver in this situation; the present invention uses a deep learning algorithm to detect road collapse and potholes in camera images, and then provides them to the software backend for warning processing. The software post-processing code takes into account the rules of various scenarios, estimates the location of collapse and potholes, and calculates the remaining distance and time that the vehicle faces danger, thereby making decisions on warnings and braking, and sends the information to the vehicle's alarm unit or brakes the vehicle to slow down through the control module, thereby improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 This is the algorithm structure diagram of the present invention. DETAILED DESCRIPTION

[0029] In order to better understand the purpose, structure and function of the present invention, and to achieve safe driving, the present invention solves the problem that existing front-view integrated machine products cannot cope with road collapse and potholes, and do not have the active safety capability to issue warnings and brakes to drivers in this situation; the present invention uses a deep learning algorithm to detect road collapse and potholes in camera images, and then gives them to the software backend for early warning processing. The software post-processing code takes into account the rules of various scenarios, estimates the location of collapse and potholes, and calculates the remaining distance and time that the vehicle faces danger, so as to make decisions on early warning and braking, and sends information to the vehicle's alarm unit or brakes the vehicle to reduce speed through the control module, thereby improving driving safety. The front-view integrated machine method of the present invention based on visual detection of road potholes and early warning is described in further detail.

[0030] refer to Figure 1-2 The present invention: a front-view integrated machine method for early warning of pothole sections based on visual detection of roads, comprising the following steps:

[0031] S1: Scene construction, artificially creating various types of collapses and potholes on the closed experimental road surface in the physical world, can achieve a large number of different scene settings, and effectively improve the precision and accuracy of algorithm training;

[0032] S2: Image collection, using a data collection vehicle equipped with a data collection camera to complete image collection in the built scene;

[0033] S3: Neural network model training: neural network model training starts on the AI ​​training server and ends when the number of training times is reached or the training verification performance is optimal.

[0034] The training of the neural network model on the AI ​​training server includes the following steps:

[0035] S11: Data enhancement, enhancing the collected image data so that the band collapse and pits can be clearly displayed;

[0036] S12: image annotation, using annotation tools to annotate images with collapse and pits;

[0037] S13: Optimize training, use the processed images to train the model, update the model parameters through the back propagation algorithm, and gradually improve the model performance.

[0038] Specifically, the deep learning AI algorithm used is Yolov5. Since Yolov5 has good detection capabilities for target objects such as vehicles, collapses, potholes, and ice cream cones, but the AI ​​algorithm is also unexplainable, it is generally necessary to collect image data of road collapses and potholes to complete AI model training and thus detect target objects;

[0039] S4: Model deployment: Deploy the neural network algorithm to the controller of the front-view integrated machine to detect road collapse and potholes during driving;

[0040] Specifically, the hardware of the front-view integrated machine uses a 5-megapixel / 8-megapixel camera. The camera may have different detection distances for different objects. It can identify traffic signs at 100 meters, bicycles and pedestrians at 140 meters, and cars at 250 meters. The narrow-angle camera FOV: the field of view is about 30-40°, and the detection distance is 250 meters. The hardware of the front-view integrated machine also has a controller. This controller integrates a CPU processor, an AI accelerator NPU, and a microcontroller MCU chip. The front-view integrated machine controller uses deep learning technology to detect road collapse and potholes, thereby misjudging the traffic conditions of the road ahead in advance, and promptly issuing sound and image warnings to drivers, and even emergency braking vehicles;

[0041] S5: Use of the model, post-processing software completes the decoding and coordinate position calculation of the road collapse and pothole targets detected by the neural network, estimates the distance between the current vehicle collapse and the pothole target, and makes corresponding treatment measures;

[0042] Specifically, the operation of the post-processing software includes the following steps:

[0043] S21: Using the front-view integrated device to perform a vision-based real-time data shooting operation of a road image ahead of the driving road;

[0044] S22: Decoding the detected target object in the acquired image using a neural network algorithm;

[0045] S23: estimating the distance between the current vehicle and the collapsed and dented target objects, and calculating the time to travel to the collapsed and dented target objects according to the current vehicle speed;

[0046] S24: The post-processing software makes corresponding warning and braking decision plans based on the photographed target and the calculated time, and uploads them to the vehicle computer for execution. The post-processing software also needs to communicate with the vehicle's CAN network to obtain the vehicle's speed, steering wheel angle, braking, and gear information. The post-processing software also needs to issue a warning signal to control the vehicle's deceleration and braking, as well as the steering wheel steering signal.

[0047] Compared with the prior art, the present invention provides a forward-looking integrated machine method for warning of potholes on a road based on visual detection, which has the following beneficial effects: the present invention solves the problem that the existing forward-looking integrated machine products are unable to cope with road collapse and potholes, and do not have the active safety capability to issue warnings and brakes to the driver in this situation; the present invention uses a deep learning algorithm to detect road collapse and potholes in camera images, and then provides them to the software backend for warning processing. The software post-processing code takes into account the rules of various scenarios, estimates the location of collapse and potholes, and calculates the remaining distance and time that the vehicle faces danger, thereby making decisions on warnings and braking, and sends the information to the vehicle's alarm unit or brakes the vehicle to slow down through the control module, thereby improving driving safety.

[0048] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A forward-looking integrated machine method based on visual detection of road pothole warning. It is characterized in that The following steps are involved: S1: Scene construction, artificially creating various types of collapses and potholes on the closed experimental road surface in the physical world; S2: Image collection, using a data collection vehicle equipped with a data collection camera to complete image collection in the built scene; S3: Neural network model training: neural network model training starts on the AI ​​training server and ends when the number of training times is reached or the training verification performance is optimal. S4: Model deployment: Deploy the neural network algorithm to the controller of the front-view integrated machine to detect road collapse and potholes during driving; S5: Use of the model, post-processing software completes the decoding and coordinate position calculation of the neural network detection of road collapse and pothole targets, estimates the distance between the current vehicle collapse and pothole targets, and takes corresponding treatment measures.

2. The front-view integrated machine method for early warning of potholes based on visual detection according to claim 1, It is characterized in that In step S3, training the neural network model on the AI ​​training server includes the following steps: S11: Data enhancement, enhancing the collected image data so that the band collapse and pits can be clearly displayed; S12: image annotation, using annotation tools to annotate images with collapse and pits; S13: Optimize training, use the processed images to train the model, update the model parameters through the back propagation algorithm, and gradually improve the model performance.

3. The front-view integrated machine method for early warning of pothole sections based on visual detection according to claim 2, It is characterized in that The deep learning AI algorithm used in step S3 is Yolov5.

4. The front-view integrated machine method for early warning of pothole sections based on visual detection according to claim 1, It is characterized in that In step S4, the hardware of the forward-looking integrated machine uses a 5-megapixel / 8-megapixel camera. The detection distance of the camera for different objects may be different. It can recognize traffic signs at 100 meters, bicycles and pedestrians at 140 meters, and cars at 250 meters. The narrow-angle camera FOV: the field of view angle is about 30-40°, and the detection distance is 250 meters. The hardware of the forward-looking integrated machine also has a controller, which integrates a CPU processor, an AI accelerator NPU, and a microcontroller MCU chip.

5. The front-view integrated machine method for early warning of pothole sections based on visual detection according to claim 1, Features: In step S5, the operation of the post-processing software includes the following steps: S21: Using the front-view integrated device to perform a vision-based real-time data shooting operation of a road image ahead of the driving road; S22: Decoding the detected target object in the acquired image using a neural network algorithm; S23: estimating the distance between the current vehicle and the collapsed and dented target objects, and calculating the time to travel to the collapsed and dented target objects according to the current vehicle speed; S24: The post-processing software makes corresponding warning and braking decision plans based on the photographed target and the calculated time, and uploads them to the vehicle computer for execution.

6. The front-view integrated machine method for early warning of potholes based on visual detection according to claim 5, It is characterized in that In step S24, the post-processing software also needs to communicate with the vehicle's CAN network to obtain the vehicle's speed, steering wheel angle, braking, and gear information. The post-processing software also needs to issue a warning signal to control the vehicle's deceleration and braking, as well as the steering wheel steering signal.

Images