A universal autonomous driving navigation system based on road surface defect detection

Abstract

The present invention discloses a general automatic driving navigation system based on road surface disease detection. The system hardware refers to an improvement based on the open source Neo suite. The Neo suite consists of four parts: vehicle module, Giraffe module, Panda module and Eon module. The improvement includes combining the functions of the Giraffe module and the Panda module to allow the vehicle CAN port information to be directly exported from the USB port, and adding the heat dissipation function of the Eon module to ensure long-term high-speed operation. The software includes the open source Openpilot software architecture, which consists of eight parts: Carinterface module, grain module, board dd module, Car module, Loggerd module, Controlsd module, Vision module, and Radar module. The present invention combines deep learning-based image processing technology with automatic driving strategy guidance to identify road surface diseases and other road surface objects and select corresponding strategies, laying an important foundation for the automation and unmanned detection of road surface diseases in the future.

Description

Technical Field

[0001] This invention belongs to the field of deep learning image processing and relates to a technology for identifying road conditions and providing driving strategy guidance. It is applicable to intelligent road hazard detection. Background Art

[0002] Pavement defects directly impact pavement service life and performance, severely impacting public transportation safety. Currently, pavement defects are often detected manually, which is time-consuming and labor-intensive. Therefore, timely and essential identification of pavement defects is crucial.

[0003] Due to a lack of technical support, current defect identification is primarily performed through manual on-site inspections or by human-driven road inspection vehicles. Both methods suffer from slow detection and result upload speeds, as well as high human resource requirements. Furthermore, current research on pavement defect image recognition often relies on perpendicular-to-the-ground image sets collected by road inspection vehicles. However, these image sets lack the authenticity and generalizability necessary for autonomous driving, and thus lack practical application value. Furthermore, due to variations in camera and environmental conditions, images can vary significantly, making them potentially unsuitable for batch processing of pavement crack images. Computational speed and efficiency are also relatively low. Summary of the Invention

[0004] This invention aims to identify road surface defects and other road objects and select corresponding strategies by combining deep learning-based image processing technology with autonomous driving strategy guidance. This approach lays an important foundation for automated and unmanned road surface defect detection in the future. The technical solution employed in this invention is a universal autonomous driving navigation system based on road surface defect detection.

[0005] Compared with the popular autonomous driving systems on the market today, the biggest advantages and features of this system are: in addition to traditional road information detection, such as vehicles and pedestrians, it can realize real-time road defects, such as cracks, potholes, and targets caused by special weather conditions, including water and snow, and adjust the corresponding driving strategy. The combination of autonomous driving and road defect recognition can quickly identify and provide feedback on defects, greatly improving the speed of defect detection and identification and reducing human resources.

[0006] The system mainly includes hardware and software.

[0007] The hardware refers to improvements based on the open source Neo kit. The Neo kit mainly consists of four parts: vehicle module, Giraffe module, Panda module and Eon module. The improvements include combining the functions of the Giraffe module and the Panda module to allow the vehicle CAN port information to be exported directly from the USB port, and adding heat dissipation function to the Eon module to ensure long-term high-speed operation.

[0008] The software includes the open-source Openpilot software framework, which consists of eight modules: the CarInterface module, the Grain module, the Board DD module, the Car module, the Logger DD module, the Control DD module, the Vision module, and the Radar module. It utilizes PID control and includes a "Pave Interface" to receive control commands. The software also includes a deep learning network, Faster R-CNN, which, through training, recognizes road surface information and guides vehicle driving strategy adjustments.

[0009] The specific system implementation method is:

[0010] Hardware: The Giraffe module converts information read from the vehicle's CAN port into a standard OBD port. The Panda module converts information obtained from the standard OBD port into a microcomputer or USB port that can only be read by a mobile terminal. The Eon module receives information sent from the Panda's USB port through the Type-C port. The vehicle's control flow is the opposite of the information flow, and control decision information is returned to the vehicle through the Eon module, Panda module, and Giraffe module, thereby achieving automatic vehicle control.

[0011] Software part: The Carinterface module is mainly responsible for docking with the vehicle. The Boardd module realizes the direct conversion between the vehicle CANO (Controller Area Network) information and the data protocol defined by the researchers. The specific conversion standard is specified by Cereal. The Controlsd module is the core component of the controller, which is used to receive data from video and radar. The PID (Proportional-Integral-Derivative) control method is used to realize horizontal and vertical control of the vehicle. Add a "Pave Interface" connected to the Controlsd module to receive control parameters (control commands) according to the actual road or traffic conditions. The Loggerd module is responsible for sending and receiving data. The software system can realize three core functions, including: vehicle information reception, vehicle control, and vehicle driving control.

[0012] The specific implementation steps of the system are as follows:

[0013] Step 1: General automatic navigation system

[0014] The implementation part of the general automatic navigation system is mainly divided into two parts: hardware and software.

[0015] Hardware:

[0016] The present invention adopts the open source Neo suite framework, such as Figure 1 And improve the equipment to meet long-term working requirements.

[0017] The Neo suite framework consists of four main components: the vehicle module, the Giraffe module, the Panda module, and the Eon module. Specifically, the Giraffe module converts information read from the vehicle's CAN port to a standard OBD port; the Panda module converts information obtained from the standard OBD port to a microcomputer or USB port that can only be read by a mobile terminal; and the Eon module receives information sent from the Panda module's USB port via a Type-C port. The vehicle's control flow is the opposite of the information flow, with control decision information returned to the vehicle via the Eon, Panda, and Giraffe modules, enabling autonomous vehicle control.

[0018] First, combine the functions of the Giraffe module and the Panda module to export the information of the vehicle's CAN port directly from the USB port, such as Figure 2 As shown; secondly, the Eon module has been improved and a heat dissipation system has been added to ensure the long-term and efficient operation of autonomous driving, such as Figure 3 shown.

[0019] Software part:

[0020] The present invention is based on the open source Openpilot software framework. The core framework of the software system is as follows Figure 4 shown.

[0021] The software system consists of eight main components: the CarInterface module, the Grain module, the BoardD module, the Car module, the LoggerD module, the ControlSd module, the Vision module, and the Radar module. The CarInterface module is primarily responsible for interfacing with the vehicle. The BoardD module implements direct conversion between vehicle CANO (Controller Area Network) information and a data protocol defined by researchers, with the specific conversion standard specified by Cereal. The ControlSd module is the core component of the controller, responsible for receiving data from video and radar.

[0022] The vehicle uses a PID (Proportional-Integral-Derivative) control method to achieve horizontal and vertical control. To further control the autonomous vehicle's driving strategy, the present invention adds a "PaveInterface" connected to the Controlsd module to receive control parameters (control commands) based on actual road or traffic conditions. This will be demonstrated later in the section on adjusting the autonomous driving strategy based on road conditions.

[0023] In the entire vehicle network, the Loggerd module is responsible for sending and receiving data. The core functions of the software system include three parts: vehicle information reception, vehicle control, and vehicle driving control. Specifically:

[0024] 1. Vehicle information reception, that is, obtaining comprehensive vehicle information.

[0025] In order to realize the automatic driving of the vehicle, it is necessary to obtain the static parameters and dynamic operation information of the vehicle first. Compared with the traditional method of collecting vehicle information through the vehicle's OBD port, the software framework of the present invention directly uses the DBC file to read the vehicle data information through the CAN port, increasing the amount of information obtained from the commonly used 27 to 84. Figure 5 See for more information.

[0026] The present invention utilizes Openpilot, an open-source framework for autonomous driving that can accommodate 60 vehicle types. Furthermore, an information acquisition module based on the Openpilot framework has been developed. This module not only acquires vehicle information through the vehicle's local area network (LAN) system, but also remotely acquires real-time information via the 4G wireless network, improving the real-time and comprehensive nature of vehicle information and operating status.

[0027] 2. Remote control of vehicles

[0028] The present invention designs Figure 6 The vehicle remote control module shown currently uses a 4G network to transmit control signals. The resulting 100ms delay in transmission and reception prevents direct use for real-time vehicle control, but the development of 5G networks will eventually solve this problem.

[0029] 3. Automatic cruising mission of road detection vehicles

[0030] The Visiond module trained by Openpilot is used as an automatic cruise training module in this invention. The main function of this module is to simulate the driver's driving behavior so that the vehicle stays in the lane and drives according to the expected speed. In Openpilot, driving behavior simulation uses a generative adversarial network (GAN) to learn human driving behavior and generate simulated driving behavior based on input signals such as video, throttle, brake, and steering. Figure 7As shown in Figure 1, based on the road conditions and the driving behavior generated by the simulation, the PID control method is used to achieve vehicle detection and automatic driving. When the system does not detect the road or traffic conditions mentioned in step 2, the vehicle will use automatic cruise control.

[0031] Step 2: Road Condition Identification

[0032] This paper uses the candidate region-based Faster R-CNN algorithm to automatically detect road objects. Faster R-CNN is an improved version of Fast R-CNN, using a convolutional neural network (CNN) to directly generate candidate regions. This algorithm reduces the time required for region selection while maintaining detection accuracy.

[0033] The basic structure of Faster R-CNN, such as Figure 8 As shown in Figure 2, it can be divided into four modules according to its functions, namely convolutional neural networks (CONV layers), region proposal networks (RPN), RoI pooling, and classifiers.

[0034] Convolutional Neural Network. A convolutional neural network (CONV layer) is used to extract image features, with the entire image as input and the extracted features as output. Autonomous vehicles process images with high complexity, distinct features of various targets, and high recognition. Therefore, this paper uses the ImageNet pre-trained VGG-16 model as the basic network for classification tasks. The input size of VGG is 1280*720*3. The output of the convolutional layer serves as the output feature map of the Conv layer.

[0035] Region Proposal Network. The extracted convolutional feature maps are processed using a region proposal network. It is used to find regions that may contain a predefined number of objects. It is a convolutional network that inputs the feature map from the Conv layer and outputs region proposals. In the region proposal network, anchors are fixed-size boxes placed on the image using a variety of sizes and scales, serving as reference boxes for the initial prediction of object locations. The output of the binary classification layer is the probability of whether each anchor point is background. The regression layer of the region proposal network outputs the position information of the anchor boxes that match the predicted objects.

[0036] RoI pooling. RoI pooling converts inputs of varying sizes into fixed-length outputs. The inputs are different candidate regions, and interpolation is used to fix the cropped feature maps to a size of 14*14*512. After max pooling, the output is a fixed-size feature map of 7*7*512.

[0037] Classification and regression. This layer outputs the category of the candidate region and its exact location in the image. Its function in Faster R-CNN is similar to that of a fully connected layer in a conventional convolutional neural network. The two output layers contain n+1 and 4n neurons, respectively. One layer provides a score for each candidate object, and the other outputs the location of the predicted boxes for each of the n categories. This network functions identically to the RPN, so sharing weights between the two networks significantly improves computational speed.

[0038] The present invention adopts the stochastic gradient descent method (SGD) and the Momentum optimization algorithm in training. The relevant parameters are set as follows: The present invention specifically identifies 13 types of road feature information, such as Figure 9 As shown in the figure, target detection is mainly divided into three categories: general (typical) unmanned driving target detection, road surface damage detection, and target detection considering climate impacts. General unmanned driving detection targets include pedestrians, bicycles, road markings, large vehicles, and small vehicles; road surface damage detection targets include potholes, cracks, large-area repairs, crack repairs, damaged road markings, and manhole covers; and detection targets considering climate impacts include rain and snow accumulation.

[0039] For typical autonomous vehicle object detection, pedestrians and cyclists are key targets. Compared to regular vehicles, they are more vulnerable on the road, so timely detection and appropriate decision-making are crucial to ensuring safety. Small and large vehicles are also important components of the transportation process, which will guide autonomous vehicles in better route planning.

[0040] When it comes to pavement damage, identifying potholes and cracks that require prompt asphalt repair is key. These impact the safety and comfort of autonomous vehicles. Statistical data on pavement markings and damaged markings can be used to assess the condition of road markings under normal operation. Repairing cracks and large-scale repairs is essential for analyzing the condition of asphalt pavement surfaces. Furthermore, in actual pavement, manhole covers are unstable components of the pavement structure and require careful attention due to differences in rigidity between manhole covers and the asphalt mixture, as well as the stress transmission at the connection between the two.

[0041] For target detection that considers climatic factors, rain and snow accumulation are key considerations in this invention. Rain and snow accumulation on asphalt pavement not only reduce the structural stability of the asphalt pavement, leading to road defects such as potholes, but also reduce the pavement's anti-skid performance, increasing the probability of traffic accidents.

[0042] Step 3: Adjust the autonomous driving strategy based on road conditions

[0043] It is divided into the following three parts:

[0044] 1. Determine the actual effective line of sight distance

[0045] The proposed integrated system for autonomous driving road surface damage detection has two design objectives: fully automatic detection of road surface damage and ensuring the comfort and safety of autonomous vehicles. Therefore, the proposed autonomous driving strategy differs from other widely used strategies that only consider traffic flow and ignore actual road conditions. In this study, the adjustment of driving strategies depends largely on the detection and judgment of various road conditions by video detection sensors. Detection accuracy depends on the effective detection range of the sensors, which is significantly affected by the sensor's placement. Figure 13 The layout diagram of the video detection sensors on the detection vehicle and the corresponding line of sight distance.

[0046] like Figure 12 In (a), the maximum detection range of the video detection sensor is L V , the angle between the sight distance and the horizon is θ V , the sensor layout height is h v Since the visual range line of the video detection sensor intersects the horizon, the effective visual range L d It is not equal to the maximum viewing distance of the video detection sensor, but is as follows:

[0047]

[0048] like Figure 12 As shown in (b), since the visual range line of the video detector sensor does not intersect with the horizon (although the extended part intersects with the horizon), the effective visual range L d It can be calculated from the maximum viewing distance of the video detector:

[0049]

[0050] like Figure 12 As shown in (a), the effective viewing distance L d Depends on the layout height h of the video detection sensor v , Maximum viewing distance L v , the angle θ between the sight line and the horizontal line v The relationship between them is as follows:

[0051]

[0052] According to the experience of video surveillance equipment layout on highways, the actual effective viewing distance is 80% of the maximum viewing distance of the video sensor equipment. d ' is the actual maximum viewing distance:

[0053] L' d =0.8L d (4)

[0054] Considering the obstruction of the sight distance by the vehicle itself and environmental factors, the actual effective sight distance is set to 0.8L d '.

[0055] 2. Changing autonomous driving behavior

[0056] In the road condition recognition part, there are currently 13 road and traffic objects that need to be detected, namely: road markings, damaged road markings, potholes, cracks, crack repairs, large-area repairs, pedestrians, large vehicles, small cars, bicycles, manhole covers, snow, and accumulated water. Based on different autonomous driving behaviors, they can be divided into five types:

[0057] 1) Normal driving

[0058] In this scenario, the vehicle will continue to execute its autonomous driving strategy as normal when detecting road markings, manhole covers, and normal road conditions. These road conditions will not affect driving behavior, and the vehicle will continue to operate under normal conditions, i.e., the vehicle will continue to operate at the programmed speed and path, and the video detection results will not affect driving behavior. In this scenario, there is no need to report an inspection alert to highway authorities.

[0059] 2) Small-scale deceleration

[0060] Second, a minor deceleration. When inspecting damaged road markings, large-scale repairs, crack repairs, and small vehicles, automated inspection vehicles will slow down slightly for a more detailed inspection. Under normal driving conditions, inspection vehicles stay within their lanes, and damaged road markings do not significantly impact driving behavior. Large-scale repairs and crack repairs significantly mitigate road conditions and have little impact on normal driving behavior. Small vehicles are safe to use a minor deceleration. All of these road and traffic conditions do not significantly impact the comfort and safety of automated inspection vehicles under normal circumstances, so no significant changes in driving behavior are required. However, current conditions such as damaged road markings, large-scale repairs, and closed cracks require careful inspection. While complying with local traffic regulations, a slight reduction in speed can also improve the frequency and quality of road inspection images captured by standard cameras. In these cases, detailed documentation of damaged road markings, large-scale repairs, and crack repairs is necessary and can be reported to highway authorities.

[0061] When the video detection sensor detects damaged road markings, large-area repairs, and crack repairs, the vehicle speed is V0. Set the deceleration target to V L1 , the minimum speed limit of the road section is V L , the deceleration distance is L d '. For detecting the speed v of the vehicle from point A t The decreasing process of the velocity v0 to point B is as follows: Figure 13 ,have:

[0062]

[0063] Therefore, the average acceleration a of the deceleration section is obtained:

[0064]

[0065] where v t Satisfy v t =max{V L1 ,V L}.

[0066] 3) Larger range of deceleration

[0067] The third type is a relatively large deceleration, which belongs to the strategy for detecting road cracks, snow and water accumulation. In this case, road cracks may cause slight vibrations of the detection vehicle, thereby affecting driving behavior, but the impact is not serious. However, snow and water accumulation will pose a serious threat to normal driving. The deceleration process in a larger range is similar to the deceleration process in a smaller range. When the detection vehicle decelerates through road cracks, on the one hand, driving safety is improved; on the other hand, ordinary cameras are used to increase the shooting frequency and quality at a certain cost, making it easier to collect more road crack information. In this process, the average acceleration of the deceleration section can also be calculated according to equation (6). Here, v t =max{V L2 ,V L}, where V L2 It is a deceleration within a relatively large range of pre-set speed values.

[0068] 4) Synchronous deceleration and lane changing

[0069] Fourth, synchronized deceleration and lane change. Potholes on the road can cause significant vibrations in the detection vehicle, which can seriously affect the vehicle's driving behavior. In particular, when the vehicle is driving at high speeds, unpredictable horizontal movement and elevation bumps may endanger the safety of the detection vehicle. Therefore, if a pothole is detected, the automatic detection vehicle will synchronize deceleration and lane change to avoid the pothole and improve driving safety. Figure 14 shown.

[0070] When the detection vehicle at position A detects a pothole at position C, it simultaneously turns and decelerates to move to the outer lane to avoid the risk posed by the pothole. The acceleration is calculated as shown in Equation (6). The average steering angle of the vehicle determines the steering amplitude and speed of the vehicle, which is part of the vehicle's lateral driving behavior and is calculated as follows:

[0071]

[0072] Where ω is the average steering angular velocity and θ is the total steering angle of the vehicle. We therefore have:

[0073]

[0074] In this case, detailed information of potholes on the road needs to be quickly reported to the police / road management agency immediately to ensure public safety.

[0075] 5) Manual driving

[0076] When detecting pedestrians, bicycles, and oversized vehicles, the driving strategy is changed to manual driving for the safety of pedestrians and road detection vehicles.

[0077] 3. Algorithm

[0078] The main process of autonomous driving behavior based on intelligent detection of road conditions is as follows: Figure 15 As shown:

[0079] Step 1: Calculate the maximum effective detection range of the video detector based on the sensor layout position and angle on the detection vehicle, and determine the range basis for driving behavior classification;

[0080] Step 2: Based on 13 road and traffic conditions, driving behaviors are categorized into five categories: normal driving, small deceleration, large deceleration, synchronized deceleration and lane change, and manual driving. These control parameters are sent to the pave interface to control the autonomous driving system. During normal driving, the detection vehicle travels at the set speed and along the set path. For small and large decelerations, the detection vehicle remains in the same lane, and the average acceleration of the detection vehicle is calculated based on the difference between pre-set deceleration thresholds to achieve detailed information collection on road surface defects. Synchronized deceleration and lane changes are used to avoid serious defects such as potholes and to collect detailed information on road surface defects. The deceleration process is consistent with Step 4, with steering behavior primarily dependent on initial speed, speed change, and sight distance. For complex traffic conditions such as pedestrians, bicycles, and oversized vehicles, the autonomous driving strategy is switched to manual driving.

[0081] This research on the dynamics of autonomous driving behavior complements the widely used traffic information-based research, not simply considering road conditions. Unlike existing autonomous driving technologies, which only identify traffic signals and road participants, this research allows for a comprehensive study of autonomous driving strategies, combining traffic information and road conditions. Furthermore, this research incorporates 5G technology to enable rapid, real-time data transmission.

[0082] In addition, the recommended deceleration / lane change is intended to illustrate changes in driving strategy, and actual operations must strictly comply with local traffic regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 Hardware framework diagram of the Neo autonomous driving system.

[0084] Figure 2 Functional integration diagram of Giraffe and Panda.

[0085] Figure 3 Increase the Eon of the cooling system.

[0086] Figure 4 Diagram of the core framework of Openpilot software.

[0087] Figure 5 Collect vehicle information. (a) Through the OBD port (b) Through the CAN port

[0088] Figure 6 Schematic diagram of the vehicle information remote collection module.

[0089] Figure 7 Dynamic control of automatic road surface detection.

[0090] Figure 8 Schematic diagram of the basic structure of Faster R-CNN.

[0091] Figure 9 Sample images of labeled objects. (a) Road markings (b) Damaged road markings (c) Large area repairs (d) Potholes (e) Manhole covers (f) Cracks (g) Crack repairs (h) Pedestrians (i) Cyclists (j) Large vehicles (k) Small vehicles (l) Snow accumulation (m) Rain accumulation.

[0092] Figure 10 Vehicle operating parameter information collection test. (a) Door opening and closing test (b) Seat belt insertion and removal test (c) Left turn signal test (d) Throttle signal test (e) Brake signal test (f) Turn signal test (g) Gear shift signal test.

[0093] Figure 11 Schematic diagram of vehicle remote control test.

[0094] Figure 12 Layout diagram of video detection sensors on the detection vehicle and corresponding line of sight distance diagram.

[0095] Figure 13 Schematic diagram of a small-scale deceleration scenario.

[0096] Figure 14 Schematic diagram of the synchronized deceleration and lane change scenario.

[0097] Figure 15 Flowchart of autonomous driving behavior based on intelligent detection of road conditions DETAILED DESCRIPTION

[0098] We conduct relevant tests on the software part of the general automatic navigation system.

[0099] For the vehicle information receiving part, the present invention uses the open source framework of autonomous driving - Openpilot, and further develops and designs an information collection module based on the Openpilot framework. At the same time, the information collection test of vehicle operating parameters such as door opening and closing, seat belt plugging and unplugging, turn signal, brake signal, steering, shift signal, etc. is carried out. The results are as follows Figure 10 As shown, it indicates that this part allows users to quickly and comprehensively obtain vehicle information and operating status.

[0100] For the vehicle remote control portion, we tested the remote control function, sending left and right turn information via a laptop. Figure 11 The testing process for vehicle remote control.

[0101] In the road surface identification part of this invention.

[0102] The real-world example uses the Berkeley DeepDrive Video Dataset (BDDV). It's an open-source autonomous driving dataset shared by the Berkeley DeepDrive research group. Videos are captured from the vehicle's front-facing camera, each approximately 40-50 seconds long. Approximately 10,000 video screenshots were captured, totaling 40,000 images. Images with a high distribution of detection targets were selected as the original dataset.

[0103] A total of 10,020 images were labeled, including all 13 classes of objects shown in Table 1. The original dataset was divided into a training set and a test set in a ratio of 9:1, with 9,019 images in the training set and 901 in the test set, as shown in Table 2.

[0104] Table 1 Calibration summary

[0105]

[0106]

[0107] Table 2 Data partition

[0108]

[0109] During training, the momentum of the Momentum optimization algorithm was set to 0.9. During training, the mini-batch size was set to 256. The learning rate was set to 0.001. After 30,000 epochs (one epoch is equivalent to training once using all the samples in the training set), the learning rate was set to 0.0001. All models in this paper were implemented in TensorFlow.

[0110] After 100,000 training cycles, totaling 18 hours, Faster R-CNN performed well on the test set and could meet basic requirements.

Claims

1. A general-purpose autonomous driving navigation system based on road surface defect detection, characterized by: This includes both hardware and software; The hardware refers to improvements based on the open-source Neo kit, which consists of four parts: the vehicle module, the Giraffe module, the Panda module, and the Eon module. The Giraffe and Panda modules combine their functionality to allow vehicle CAN port information to be exported directly from the USB port, and the Eon module also adds heat dissipation capabilities. The software includes the open-source Openpilot software framework, which consists of eight modules: Carinterface module, grain module, board dd module, Car module, Loggerd module, Controlsd module, Vision module, and Radar module. It uses PID control and adds a "Pave Interface" to receive control commands. The software also includes the Faster R-CNN deep learning network, which is trained to recognize road surface information and guide vehicle driving strategy adjustments. The maximum viewing distance of the video detection sensor is L v , the angle between the sight distance and the horizon is θ v , the layout height of the video detection sensor is h v ; Since the visual range line of the video detection sensor intersects the horizon, the effective visual range L d It is not equal to the maximum viewing distance of the video detection sensor, as follows: Since the visual range line of the video detector sensor does not intersect the horizon, the effective visual range L d Calculated from the maximum viewing distance of the video detection sensor: Effective viewing distance L d Depends on the layout height h of the video detection sensor v , Maximum viewing distance L v , the angle θ between the sight distance and the horizon v The relationship between them is as follows: According to the experience of highway video surveillance equipment layout, the actual maximum viewing distance is 80% of the effective viewing distance of the video detection sensor; assuming L d ′ is the actual maximum viewing distance: L' d =0.8L d (4) Considering the obstruction of the sight distance by the vehicle itself and environmental factors, the actual effective sight distance is set to 0.8L′ d ; Changing autonomous driving behavior: The road condition recognition component detects 13 road and traffic objects: road markings, damaged road markings, potholes, cracks, crack repairs, large-area repairs, pedestrians, large vehicles, small cars, bicycles, manhole covers, snow, and accumulated water. These objects are categorized into five types based on the autonomous driving behavior: 1) Normal driving; In this case, the vehicle will execute the autonomous driving strategy normally when detecting road markings, manhole covers and normal road conditions; 2) Small-scale deceleration; When inspecting damaged road markings, large-scale repairs, crack repairs, and small vehicles, the automated inspection vehicle will slow down for detailed inspections. Damaged road markings, large-scale repairs, and crack closures currently require inspection. While meeting local traffic regulations, the vehicle's speed should be slightly reduced to increase the frequency and quality of road inspection images captured by standard cameras. Documentation is required to record detailed information on damaged road markings, large-scale repairs, and crack repairs. When the video detection sensor detects damaged road markings, large-area repairs, and crack repairs, the vehicle speed is V0; the deceleration target is set to V L1 , the minimum speed limit of the road section is V L ; For the detection vehicle's speed v from point A t The decreasing process of the velocity v0 to point B is: Average acceleration a in the deceleration section: where v t Satisfy v t =max{V L1 ,V L }; 3) Larger range of deceleration; When the detection vehicle decelerates through the road crack, the average acceleration of the deceleration section is calculated according to equation (6); v t =max{V L2 ,V L }, where V L2 It is a deceleration with a relatively large range of pre-set speed values; 4) Synchronous deceleration and lane change; When the detection vehicle at position A detects a pothole at position C, it turns and decelerates synchronously to move to the outer lane to avoid the risk posed by the pothole. The acceleration is calculated as shown in formula (6). The average steering angle of the vehicle determines the steering amplitude and speed of the vehicle, which belongs to the lateral driving behavior of the vehicle and is calculated as follows: Where ω is the average steering angular velocity, θ is the total steering angle of the vehicle; therefore: Detailed information about potholes on the road needs to be quickly reported to the police / immediately to the highway management agency to ensure public safety; 5) Manual driving; When detecting pedestrians, bicycles, and oversized vehicles, the driving strategy is changed to manual driving for the safety of pedestrians and road detection vehicles.

2. The universal autonomous driving navigation system based on road surface defect detection according to claim 1, characterized in that: The Giraffe module converts the information read from the vehicle's CAN port into a standard OBD port; the Panda module converts the information obtained from the standard OBD port into a microcomputer or USB port that can only be read by a mobile terminal; the Eon module receives information sent from the Panda's USB port through the Type-C port; the vehicle's control flow is the opposite of the information flow, and control decision information is returned to the vehicle through the Eon module, Panda module, and Giraffe module, thereby achieving automatic vehicle control.

3. The universal autonomous driving navigation system based on road surface defect detection according to claim 1, characterized in that: The Carinterface module is responsible for interfacing with the vehicle. The Boardd module implements direct conversion between vehicle CANO information and a defined data protocol, using a conversion standard specified by Cereal. The Controlsd module is the core component of the controller, receiving data from video and radar. PID control is used to achieve horizontal and vertical vehicle control. A "Pave Interface" connected to the Controlsd module is added to receive control parameters based on actual road or traffic conditions. The Loggerd module is responsible for sending and receiving data. The software system implements vehicle information reception, vehicle control, and vehicle driving control functions.

4. The universal autonomous driving navigation system based on road surface defect detection according to claim 1, characterized in that: In order to realize automatic driving of the vehicle, it is necessary to obtain the static parameters and dynamic operation information of the vehicle; the vehicle data information is directly read using the DBC file through the CAN port.

5. The universal autonomous driving navigation system based on road surface defect detection according to claim 1, characterized in that: The information collection module based on the openpilot framework obtains vehicle information through the vehicle's local area network system and can also remotely obtain real-time information through the 4G wireless network.

6. The universal autonomous driving navigation system based on road surface defect detection according to claim 1, characterized in that: The Faster R-CNN algorithm based on candidate regions is used to realize automatic detection of road objects; the basic structure of Faster R-CNN can be divided into four modules according to its function, namely convolutional neural network CONV layers, region proposal network RPN, RoI pooling and classifier.

7. The universal autonomous driving navigation system based on road surface defect detection according to claim 6, characterized in that: The features of the image are extracted using the CONV layer of the convolutional neural network CONV layers, where the entire image is used as input and the extracted features are used as output; the ImageNet pre-trained VGG-16 model is used as the basic network for the classification task; the output of the convolution layer is used as the output feature map of the Conv layer.

8. The universal autonomous driving navigation system based on road surface defect detection according to claim 7, characterized in that: The extracted convolutional feature map is processed using the region proposal network (RPN); regions that may contain a predefined number of objects are found; it is a convolutional network that inputs the feature map from the Conv layer and outputs the proposed region; in the region proposal network, the anchor is a fixed-size box placed on the image using a variety of different sizes and scales, which serves as a reference box for the first prediction of the object position; the output of the binary classification layer is the probability of whether each anchor point is background; the regression layer of the region proposal network outputs the position information of the anchor box that matches the predicted object.

9. The universal autonomous driving navigation system based on road surface defect detection according to claim 8, characterized in that: RoI pooling: RoI pooling converts inputs of different sizes into outputs of fixed length. The input is different candidate regions, and the cropped feature map is fixed to a size of 14*14*512 using interpolation. After maximum pooling, a fixed-size feature map of 7*7*512 is output.

10. The universal automatic driving navigation system based on road surface defect detection according to claim 9, characterized in that: The output of the classification and regression layers of the classifier is the category to which the candidate region belongs and the exact location of the candidate region in the image; its role in Faster R-CNN is similar to that of the fully connected layer of a conventional convolutional neural network; the two output layers contain n+1 and 4n neurons.

Images