Vehicle-mounted road defect detection system and method based on binocular vision and deep fusion

Abstract

The invention relates to a vehicle-mounted road defect detection system and method based on binocular vision and deep fusion, and the system comprises a binocular camera module, a main control platform, a positioning module, a communication module and a power supply control module. The system realizes automatic detection and severity quantitative evaluation of various pavement defects such as cracks, pits and ruts, obtains defect positions in combination with a positioning module, and uploads the defect positions to a cloud platform in real time through a communication module, thereby forming a closed-loop urban road defect information perception and management system. According to the method, the quantitative evaluation capability of the identification accuracy and severity of the road defects is remarkably improved, and automatic identification, accurate positioning and real-time cloud synchronous uploading of the road defects are realized; the method has the advantages of flexible deployment, efficient operation, high adaptability and the like, is particularly suitable for urban road intelligent inspection and maintenance management scenes, and has good engineering application prospects and popularization values.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent transportation, vehicle computing and image recognition technology, specifically relating to a vehicle-mounted road defect detection system and method based on binocular vision and deep fusion. Background Technology

[0002] With the acceleration of urbanization, urban roads are facing increasingly severe traffic pressure, leading to various structural defects such as cracks and potholes in the road surface. These defects not only affect driving safety and smoothness, but may also cause accidents, shorten road lifespan, and increase maintenance costs. Efficient and accurate road defect detection has become a key issue for intelligent transportation and urban management.

[0003] Currently, road inspections mainly rely on manual patrols or dedicated inspection vehicles. However, the former is inefficient and prone to errors, while the latter is costly and has limited coverage, making it difficult to meet the high-frequency monitoring needs of urban roads. In recent years, the application of computer vision and deep learning technologies has brought new possibilities for automated inspection, but existing methods still have significant shortcomings: they cannot quantify the severity of defects, lack geographic coordinate positioning, and suffer from poor data real-time performance and collaboration, making it difficult to directly apply the inspection results to actual maintenance work.

[0004] In conclusion, to address these challenges, it is necessary to develop new detection systems that integrate multimodal sensing data and positioning and communication functions to achieve intelligent identification, accurate assessment, and cloud-based collaborative management of road defects, thereby improving the overall efficiency and intelligence level of urban road maintenance. Summary of the Invention

[0005] In view of the current technical shortcomings, this invention provides an on-board road defect detection system and method based on binocular vision and deep fusion. By integrating deep learning models and binocular stereo vision technology, it can automatically detect and quantitatively assess the severity of various road defects such as cracks, potholes, and ruts. The system also uses a positioning module to obtain the location of defects and uploads the information to a cloud platform in real time through a communication module, forming a closed-loop urban road defect information perception and management system.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A vehicle-mounted road defect detection system based on binocular vision and depth fusion is disclosed. The system includes a binocular camera module, a main control platform, a positioning module, a communication module, and a power control module. The binocular camera module includes a left camera and a right camera. The main control platform includes a YOLOv12 vision recognition module, a depth estimation module, and a defect severity assessment module. The binocular camera module, the main control platform, and the communication module are connected sequentially. The positioning module is connected to the communication module. The power control module is connected to the binocular camera module, the main control platform, the positioning module, and the communication module.

[0007] Through the coordinated design of visual recognition, spatial reconstruction, geocalibration and network transmission, the various modules jointly realize an integrated workflow of "automatic perception + severity assessment + real-time positioning + cloud reporting" in vehicle road defect detection.

[0008] Preferably, the main control platform further includes an image preprocessing module and a continued detection module. The YOLOv12 visual recognition module is connected to the image preprocessing module and the continued detection module respectively. The continued detection module is connected to the left eye camera. The image preprocessing module, the depth estimation module, and the defect degree assessment module are connected in sequence. The left eye camera is connected to the YOLOv12 visual recognition module and the image preprocessing module respectively, and the right eye camera is connected to the image preprocessing module.

[0009] Preferably, the positioning module is a GPS positioning module, and the communication module includes an MQTT communication module and a cloud receiving platform. The GPS positioning module and the defect severity assessment module are respectively connected to the cloud receiving platform through the MQTT communication module.

[0010] Preferably, the information recorded by the GPS positioning module includes vehicle information and time, and the vehicle information includes the vehicle's latitude and longitude and speed.

[0011] Preferably, the main control platform is the NVIDIA Jetson platform, and the system is installed at the front of the vehicle.

[0012] This invention also provides a vehicle-mounted road defect detection method based on binocular vision and depth fusion, employing the above system and including the following steps: S1. Vehicle system initialization phase: The system is installed at the front of the vehicle and enters standby mode after power-on; the binocular camera module, YOLOv12 vision recognition module, depth estimation module, positioning module and communication module maintain low power consumption after initialization. The binocular camera module and the main control platform continuously collect real-time image data at a preset frame rate; the system is powered by the vehicle power supply and supports power outage resume function. S2, Defect Image Acquisition and Target Detection Stage: During vehicle movement, the binocular camera module simultaneously acquires left and right eye image data; the left eye image is input into the YOLOv12 model in real time to perform road defect detection, identify the defect category, the bounding box coordinates in the image, and the confidence level; S3. Depth Map Generation and Defect Severity Assessment Stage: The system performs stereo matching based on the left and right eye images, outputs a disparity map and converts it into a depth map; maps the defect bounding boxes output by YOLOv12 to the corresponding regions of the depth map, calculates geometric parameters including the average depth, maximum depth and standard deviation of the defect region; and classifies the severity of defects based on preset thresholds. S4. Positioning and Data Encapsulation Upload Stage: The positioning module continuously records the vehicle's latitude, longitude, speed, and time information; after detecting a defect, the system encapsulates the defect image, detection results, depth parameters, and corresponding geographic coordinates into a standard structured data frame, and uploads it to the cloud receiving platform through the communication module; S5. System Loop and Anomaly Handling Phase: After data upload is completed, the system releases cache resources and enters the next frame detection loop; for abnormal scenarios including image acquisition failure, network interruption, and GPS drift, the system sets up a retry mechanism and module restart strategy; when the vehicle is stationary or there is no image input for a long time, the system switches to sleep mode.

[0013] As a preferred option, step S1 is as follows: After the vehicle starts, the system is powered by the vehicle power supply and starts up. The main control NVIDIA Jetson platform enters the startup process and completes the initialization of each key module in the system in sequence, preparing to enter the detection task process. After the main control NVIDIA Jetson platform completes startup, it loads the embedded operating system and library dependencies, initializes the YOLOv12 visual recognition module and depth estimation module, starts independent image acquisition threads for the left and right channels of the binocular camera module, initializes the camera parameters according to the preset configuration, points the lens towards the road area ahead, and connects the GPS positioning module to the main control platform through the serial port to start the positioning service thread.

[0014] As a preferred option, step S2 specifically involves: S2-1, Image Acquisition: After the system completes initialization, the binocular camera module officially enters the continuous acquisition phase; the left and right camera channels synchronously acquire image frames at a preset frequency; firstly, the main control platform starts the dual-channel image acquisition thread through a multi-threading mechanism, the image acquired by the left camera is denoted as LeftImage(t), and the image acquired by the right camera is denoted as RightImage(t), where t represents the timestamp of the current frame; the system uses a buffer queue to align the dual-channel image data; S2-2, Image Correction and Registration: Based on the pre-calibrated intrinsic and extrinsic parameter matrices of the system, stereo correction is performed on the acquired original images; the correction process includes distortion correction and image geometric alignment operations; the system binds the acquisition timestamp to the left and right images of each frame and performs synchronous annotation using the UTC time obtained by GPS, while assigning a unique frame ID to the image frame.

[0015] As a preferred option, step S3 specifically involves: S3-1, Image preprocessing: The system performs standardization processing on the original image, including normalization, and converts the image format into the Tensor structure required for YOLOv12 model input; The S3-2 and YOLOv12 models run on the Jetson platform using the TensorRT engine for forward inference, outputting predicted bounding boxes for multi-class defect targets; each detection result includes defect category, location box, and confidence score. S3-3. Perform non-maximum suppression on the model output and retain targets with confidence scores greater than a set threshold for subsequent analysis. S3-4. Write each detected defect target into the task queue, waiting to enter the next stage of in-depth statistical processing; record the defect location, type, confidence level, image frame ID and acquisition time information for each task. S3-5, The depth estimation module estimates the actual depth of the current defect based on the principle of binocular stereo vision and classifies the severity level: S3-5-1, Stereo Parallax Calculation Principle: The system performs pixel-level parallax calculation using the SGBM algorithm to obtain a complete parallax map, and then generates a corresponding depth map by calculating the depth of each pixel. S3-5-2, Target Region Depth Statistics: For each detection box output by YOLOv12, extract the Region of Interest (ROI) in the corresponding area of ​​the depth map and calculate the depth standard deviation. ; S3-5-3, Defect Severity Grading: For maximum depth Less than 1cm is considered minor, 1cm ≤ <3cm and <0.5 points indicates moderate severity. ≥3cm or It is categorized as severe.

[0016] Preferably, after completing defect classification and severity estimation, the system will obtain the geographic coordinates of the current location through the GPS positioning module, and upload the detection results in a standardized package. Step S4 is as follows: S4-1, Location Information Acquisition: The GPS positioning module periodically outputs NMEA data, and the main control platform parses the $GPRMC data frame to extract the longitude, latitude, and timestamp corresponding to the defect detection time. S4-2, Data Structure Encapsulation: The system integrates and encapsulates image recognition, depth estimation, defect level, and GPS positioning information to form a unified defect reporting structure; the fields include: defect ID, defect category, location information, bounding box position, average depth, maximum depth, standard deviation, severity level, and timestamp. S4-3, Caching and Preparation for Upload: Structured data is cached in the upload queue in MQTT format, waiting for the communication module to execute the upload process; S4-4, MQTT Communication Upload Phase: After completing defect identification, deep estimation and information encapsulation, the system pushes the detection results to the cloud receiving platform via the MQTT protocol, supporting remote monitoring, data recording and road maintenance scheduling.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention patent realizes automatic detection, severity analysis, accurate positioning and real-time cloud reporting of road defects by deploying on mobile platforms such as buses or cars, avoiding the high cost and low efficiency of traditional manual inspection or special inspection vehicles, and significantly improving the intelligence and coverage of road inspection work; (2) The system of the present invention uses binocular vision and YOLOv12 model to work together, and combines depth map to evaluate the three-dimensional geometric features of defects, thus realizing the organic combination of defect identification and quantitative evaluation; (3) The system of the present invention adopts a modular design, has low power edge inference capability, supports vehicle power supply and intermittent operation mechanism, and is suitable for large-scale mobile deployment needs; the system has a compact structure and clear control logic, which makes it easy to deploy quickly in various urban vehicles such as buses, taxis, and sanitation vehicles. Attached Figure Description

[0018] Figure 1 This is a system structure block diagram of the present invention; Figure 2 This is a schematic diagram of the overall system deployment of the present invention; Figure 3 This is a schematic diagram of the YOLO detection + depth map fusion method of the present invention; Figure 4 This is a schematic diagram of information encapsulation and transmission according to the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be further described in detail below through embodiments. These embodiments are for illustrative purposes only and are not intended to limit the present invention. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Reference Figure 1A vehicle-mounted road defect detection system based on binocular vision and depth fusion is disclosed. The system includes a binocular camera module, a main control platform, a positioning module, a communication module, and a power control module. The binocular camera module includes a left camera and a right camera. The main control platform includes a YOLOv12 vision recognition module, a depth estimation module, and a defect severity assessment module. The binocular camera module, the main control platform, and the communication module are connected sequentially. The positioning module is connected to the communication module. The power control module is connected to the binocular camera module, the main control platform, the positioning module, and the communication module.

[0021] Through the coordinated design of visual recognition, spatial reconstruction, geocalibration and network transmission, the various modules jointly realize an integrated workflow of "automatic perception + severity assessment + real-time positioning + cloud reporting" in vehicle road defect detection.

[0022] The main control platform also includes an image preprocessing module and a continued detection module. The YOLOv12 visual recognition module is connected to the image preprocessing module and the continued detection module respectively. The continued detection module is connected to the left eye camera. The image preprocessing module, the depth estimation module, and the defect degree assessment module are connected in sequence. The left eye camera is connected to the YOLOv12 visual recognition module and the image preprocessing module respectively, and the right eye camera is connected to the image preprocessing module.

[0023] The positioning module is a GPS positioning module, and the communication module includes an MQTT communication module and a cloud receiving platform. The GPS positioning module and the defect severity assessment module are connected to the cloud receiving platform via the MQTT communication module. The information recorded by the GPS positioning module includes vehicle information and time, and the vehicle information includes the vehicle's latitude, longitude, and speed. The main control platform is an NVIDIA Jetson platform.

[0024] The system is installed at the front of the vehicle; specific deployment details are as follows: Figure 2 The system comprises a binocular camera module 1, used to acquire images from both left and right perspectives. Installed at the front of the vehicle, it supports different road field of view widths, providing a foundation for visual input. The main system 2 includes a main control platform, a communication module, and a power control module. The main control platform is responsible for the overall system control logic, including image acquisition scheduling, recognition and depth calculation triggering, data structure encapsulation, and communication flow control. The communication module (MQTT protocol) publishes structured defect information to a cloud server via a 4G network, employing a QoS1 mechanism to ensure that information is transmitted at least once and receives a server response, preventing data loss. The power control module (power supply module) is powered by the vehicle's power interface and can be equipped with a backup lithium battery. Combined with a power management IC, it achieves stable power supply and power-on protection for the equipment. The GPS positioning module 3 is used to obtain the vehicle's current geographical location information, which is then bound to the defect detection results to form traceable spatial information.

[0025] Reference Figure 3 , Figure 4 The specific implementation steps of the above system are as follows: S1. Vehicle System Initialization Phase: The system is installed at the front of the vehicle and enters standby mode after power-on. After the binocular camera module, YOLOv12 vision recognition module, depth estimation module, positioning module, and communication module complete initialization, they maintain low-power operation. The binocular camera module (image acquisition module) and the main control platform continuously acquire real-time image data at a preset frame rate. The system is powered by the vehicle power supply and supports power-off resume function. Step S1 is as follows: After the vehicle starts, the system is powered by the vehicle power supply and starts up. The main control NVIDIA Jetson platform enters the startup process and initializes each key module in the system in sequence to ensure that the software and hardware environment is in a stable state and prepares to enter the detection task process. After the main control NVIDIA Jetson platform completes startup, it loads the embedded operating system and library dependencies, initializes the YOLOv12 vision recognition module and depth estimation module, and starts independent image acquisition threads for the left and right channels of the binocular camera module. The camera parameters are initialized according to the preset configuration, and the lens is facing the road area in front to ensure that a complete front view of the lane is obtained. The GPS positioning module is connected to the main control platform through the serial port and starts the positioning service thread.

[0026] S2, Defect Image Acquisition and Target Detection Stage: During vehicle operation, the binocular camera module simultaneously acquires left and right eye image data; the left eye image is input into the YOLOv12 model in real time to perform road defect detection, identify defect categories (including potholes, cracks, ruts, etc.), bounding box coordinates and confidence scores in the image. Step S2 is as follows: S2-1, Image Acquisition: After the system completes initialization, the binocular camera module officially enters the continuous acquisition phase; the left and right camera channels synchronously acquire image frames at a preset frequency; firstly, the main control platform starts the dual-channel image acquisition thread through a multi-threading mechanism, the image acquired by the left camera is denoted as LeftImage(t), and the image acquired by the right camera is denoted as RightImage(t), where t represents the timestamp of the current frame; the system uses a buffer queue to align the dual-channel image data to avoid image frame errors caused by differences in USB bandwidth; S2-2, Image Correction and Registration: Based on the pre-calibrated intrinsic and extrinsic parameter matrices of the system, stereo correction is performed on the acquired original images. The correction process includes distortion correction and image geometric alignment operations to ensure that the left and right images are at the same scan line position. The system binds the acquisition timestamp to each frame of left and right images and synchronously marks it with the UTC time obtained by GPS. At the same time, a unique frame ID is assigned to the image frame to ensure data traceability in the subsequent depth estimation and target recognition process.

[0027] S3. Depth Map Generation and Defect Severity Assessment Stage: The system performs stereo matching based on the left and right eye images, outputs a disparity map and converts it into a depth map; maps the defect bounding boxes output by YOLOv12 to the corresponding regions of the depth map, calculates geometric parameters including the average depth, maximum depth and standard deviation of the defect region; and classifies the severity of defects based on preset thresholds. Step S3 is as follows: S3-1, Image preprocessing: The system performs standardization processing on the original image, including normalization, and converts the image format into the Tensor structure required for YOLOv12 model input; The S3-2 and YOLOv12 models run on the Jetson platform using the TensorRT engine for efficient forward inference, outputting predicted bounding boxes for multi-class defect targets; each detection result includes defect category, location box, and confidence score. S3-3. Perform non-maximum suppression (NMS) on the model output to eliminate duplicate bounding boxes, retaining targets with confidence scores greater than a set threshold for subsequent analysis; valid bounding boxes will be used for spatial alignment of the depth map, forming a basis for joint analysis of visual and geometric information; S3-4. Write each detected defect target into the task queue, waiting to enter the next stage of in-depth statistical processing; record the defect location, type, confidence level, image frame ID and acquisition time information for each task. S3-5, The depth estimation module estimates the actual depth of the current defect based on the principle of binocular stereo vision and classifies the severity level: S3-5-1, Stereo Parallax Calculation Principle: There is parallax in the left and right views of a binocular camera, meaning the position of the same point in space shifts between the images from the two cameras. Based on the pinhole camera model, the basic formula for stereo depth calculation is: Where Z is the true depth between the target point and the camera (in meters), f is the focal length of the camera (in pixels), B is the baseline distance of the binocular cameras (in meters), and d is the parallax, which represents the lateral pixel offset of the same object point in the left and right images. The system uses the SGBM algorithm to perform pixel-level disparity calculation to obtain a complete disparity map, and then uses the above formula to calculate the depth of each pixel to generate the corresponding depth map. S3-5-2, Target Region Depth Statistics: For each detection box output by YOLOv12, extract the Region of Interest (ROI) in the corresponding area of ​​the depth map and calculate the depth standard deviation. ; in, It is the depth standard deviation. It is the average depth. It represents the depth value of each pixel within the ROI; S3-5-3, Defect Severity Grading: For maximum depth Less than 1cm is considered minor, 1cm ≤ <3cm and <0.5 points indicates moderate severity. ≥3cm or It is categorized as severe.

[0028] S4. Positioning and Data Encapsulation Upload Stage: The positioning module continuously records the vehicle's latitude, longitude, speed, and time information; after detecting a defect, the system encapsulates the defect image, detection results, depth parameters, and corresponding geographic coordinates into a standard structured data frame, and uploads it to the cloud receiving platform via the 4G wireless communication module; the communication protocol supports MQTT and has breakpoint resume and data verification mechanisms to ensure transmission reliability. After completing defect classification and severity estimation, the system will obtain the geographic coordinates of the current location through the GPS positioning module, and upload the standardized and packaged detection results. Step S4 is as follows: S4-1, Location Information Acquisition: The GPS positioning module periodically outputs NMEA data, and the main control platform parses the $GPRMC data frame to extract the longitude, latitude, and timestamp corresponding to the defect detection time. S4-2, Data Structure Encapsulation: The system integrates and encapsulates image recognition, depth estimation, defect level, and GPS positioning information to form a unified defect reporting structure; the fields include: defect ID, defect category, location information, bounding box position, average depth, maximum depth, standard deviation, severity level, and timestamp. S4-3, Caching and Preparation for Upload: Structured data is cached in the upload queue in MQTT format, waiting for the communication module to execute the upload process; S4-4, MQTT Communication Upload Phase: After completing defect identification, deep estimation, and information encapsulation, the system pushes the detection results to the cloud receiving platform via the MQTT protocol, supporting remote monitoring, data recording, and road maintenance scheduling. To ensure transmission reliability, this system adopts the QoS1 transmission level of the MQTT protocol.

[0029] S5. System Loop and Anomaly Handling Phase: After data upload is completed, the system releases cache resources and enters the next frame detection loop. For abnormal scenarios, including image acquisition failure, network interruption, and GPS drift, the system sets up a retry mechanism and module restart strategy to ensure continuous operation. When the vehicle is stationary or there is no image input for a long time (low scene change or parking state), the system switches to sleep mode (automatically enters low frequency operation mode) to reduce the image acquisition rate and calculation frequency and extend the platform's service life.

[0030] In terms of detection efficiency, the system of this invention is based on the NVIDIA Jetson platform and integrates YOLOv12 object detection and binocular depth estimation algorithms. The processing time for a single frame image is controlled within 2 seconds, which is significantly better than traditional manual inspection methods, improving detection efficiency by more than 15 times. Compared with some traditional machine vision systems that use single-modal image processing (with an average detection time of 58 seconds), this system still has a 24-fold speed advantage, which can meet the needs of vehicle platforms for rapid and continuous detection in complex road conditions.

[0031] In terms of recognition accuracy, based on 1000 sets of typical urban road defect images (covering types such as cracks, potholes, and spalling) collected on-site, this system integrates semantic recognition and deep feature analysis, achieving a defect recognition accuracy of 98.3%. This represents a significant improvement over the average accuracy of manual inspection methods and conventional image recognition algorithms (approximately 91-94%). It exhibits extremely low false negative and false positive rates, particularly in the identification of minute defects with crack widths less than 2mm, with an actual false negative rate below 1.3% and a false positive rate below 2.2%.

[0032] In terms of quantifying defect severity, the system performs post-fusion analysis by combining YOLO bounding box results with disparity maps generated by binocular cameras. Combined with depth estimation formulas, it achieves a structured assessment of parameters such as defect depth, width, and area, providing a quantitative basis for road maintenance. In multiple sets of measured samples, the depth error is controlled within ±1.5mm, meeting the needs for refined assessment of road defects in urban scenarios.

[0033] This invention significantly improves the accuracy of road defect identification and the ability to quantitatively assess the severity by fusing YOLO target detection results with binocular depth maps. Combining the highly reliable data communication mechanism of the MQTT protocol and GPS positioning technology, it achieves automatic identification, precise location, and real-time cloud-based synchronous uploading of road defects. This system boasts advantages such as flexible deployment, high efficiency, and strong adaptability, making it particularly suitable for intelligent inspection and maintenance management of urban roads, and possesses promising engineering application prospects and widespread value.

[0034] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A vehicle-mounted road defect detection system based on binocular vision and depth fusion, characterized in that, The system includes a binocular camera module, a main control platform, a positioning module, a communication module, and a power control module. The binocular camera module includes a left camera and a right camera. The main control platform includes a YOLOv12 visual recognition module, a depth estimation module, and a defect severity assessment module. The binocular camera module, the main control platform, and the communication module are connected sequentially. The positioning module is connected to the communication module. The power control module is connected to the binocular camera module, the main control platform, the positioning module, and the communication module.

2. The vehicle-mounted road defect detection system based on binocular vision and depth fusion as described in claim 1, characterized in that, The main control platform also includes an image preprocessing module and a continued detection module. The YOLOv12 visual recognition module is connected to the image preprocessing module and the continued detection module respectively. The continued detection module is connected to the left eye camera. The image preprocessing module, the depth estimation module, and the defect degree assessment module are connected in sequence. The left eye camera is connected to the YOLOv12 visual recognition module and the image preprocessing module respectively, and the right eye camera is connected to the image preprocessing module.

3. The vehicle-mounted road defect detection system based on binocular vision and depth fusion according to claim 2, characterized in that, The positioning module is a GPS positioning module, and the communication module includes an MQTT communication module and a cloud receiving platform. The GPS positioning module and the defect severity assessment module are respectively connected to the cloud receiving platform through the MQTT communication module.

4. The vehicle-mounted road defect detection system based on binocular vision and depth fusion according to claim 3, characterized in that, The information recorded by the GPS positioning module includes vehicle information and time. The vehicle information includes the vehicle's latitude, longitude, and speed.

5. The vehicle-mounted road defect detection system based on binocular vision and depth fusion according to claim 1, characterized in that, The main control platform is the NVIDIA Jetson platform, and the system is installed at the front of the vehicle.

6. A vehicle-mounted road defect detection method based on binocular vision and depth fusion, characterized in that, The system according to any one of claims 1-5 comprises the following steps: S1. Vehicle system initialization phase: The system is installed at the front of the vehicle and enters standby mode after power-on; the binocular camera module, YOLOv12 vision recognition module, depth estimation module, positioning module and communication module maintain low power consumption after initialization. The binocular camera module and the main control platform continuously collect real-time image data at a preset frame rate; the system is powered by the vehicle power supply and supports power outage resume function. S2, Defect Image Acquisition and Target Detection Stage: During vehicle movement, the binocular camera module simultaneously acquires left and right eye image data; the left eye image is input into the YOLOv12 model in real time to perform road defect detection, identify the defect category, the bounding box coordinates in the image, and the confidence level; S3. Depth Map Generation and Defect Severity Assessment Stage: The system performs stereo matching based on the left and right eye images, outputs a disparity map and converts it into a depth map; maps the defect bounding boxes output by YOLOv12 to the corresponding regions of the depth map, calculates geometric parameters including the average depth, maximum depth and standard deviation of the defect region; and classifies the severity of defects based on preset thresholds. S4. Positioning and Data Encapsulation Upload Stage: The positioning module continuously records the vehicle's latitude, longitude, speed, and time information; after detecting a defect, the system encapsulates the defect image, detection results, depth parameters, and corresponding geographic coordinates into a standard structured data frame, and uploads it to the cloud receiving platform through the communication module; S5. System Loop and Anomaly Handling Phase: After data upload is completed, the system releases cache resources and enters the next frame detection loop; for abnormal scenarios including image acquisition failure, network interruption, and GPS drift, the system sets up a retry mechanism and module restart strategy; when the vehicle is stationary or there is no image input for a long time, the system switches to sleep mode.

7. The vehicle-mounted road defect detection method based on binocular vision and depth fusion according to claim 6, characterized in that, Step S1 is as follows: After the vehicle starts, the system is powered by the vehicle power supply and starts up. The main control NVIDIA Jetson platform enters the startup process and initializes each key module in the system in sequence, preparing to enter the detection task process. After the main control NVIDIA Jetson platform completes startup, it loads the embedded operating system and library dependencies, initializes the YOLOv12 visual recognition module and depth estimation module, starts independent image acquisition threads for the left and right channels of the binocular camera module, initializes the camera parameters according to the preset configuration, points the lens towards the road area ahead, and connects the GPS positioning module to the main control platform through the serial port to start the positioning service thread.

8. The vehicle-mounted road defect detection method based on binocular vision and depth fusion according to claim 7, characterized in that, Step S2 is as follows: S2-1, Image Acquisition: After the system completes initialization, the binocular camera module officially enters the continuous acquisition phase; the left and right camera channels synchronously acquire image frames at a preset frequency; firstly, the main control platform starts the dual-channel image acquisition thread through a multi-threading mechanism, the image acquired by the left camera is denoted as LeftImage(t), and the image acquired by the right camera is denoted as RightImage(t), where t represents the timestamp of the current frame; The system uses a cache queue to align dual-channel image data; S2-2, Image Correction and Registration: Based on the pre-calibrated intrinsic and extrinsic parameter matrices of the system, stereo correction is performed on the acquired original images; the correction process includes distortion correction and image geometric alignment operations; the system binds the acquisition timestamp to the left and right images of each frame and performs synchronous annotation using the UTC time obtained by GPS, while assigning a unique frame ID to the image frame.

9. The vehicle-mounted road defect detection method based on binocular vision and depth fusion according to claim 8, characterized in that, Step S3 is as follows: S3-1, Image preprocessing: The system performs standardization processing on the original image, including normalization, and converts the image format into the Tensor structure required for YOLOv12 model input; The S3-2 and YOLOv12 models run on the Jetson platform using the TensorRT engine for forward inference, outputting predicted bounding boxes for multi-class defect targets; each detection result includes defect category, location box, and confidence score. S3-3. Perform non-maximum suppression on the model output and retain targets with confidence scores greater than a set threshold for subsequent analysis. S3-4. Write each detected defect target into the task queue, waiting to enter the next stage of in-depth statistical processing; record the defect location, type, confidence level, image frame ID and acquisition time information for each task. S3-5, The depth estimation module estimates the actual depth of the current defect based on the principle of binocular stereo vision and classifies the severity level: S3-5-1, Stereo Parallax Calculation Principle: The system performs pixel-level parallax calculation using the SGBM algorithm to obtain a complete parallax map, and then generates a corresponding depth map by calculating the depth of each pixel. S3-5-2, Target Region Depth Statistics: For each detection box output by YOLOv12, extract the Region of Interest (ROI) in the corresponding area of ​​the depth map and calculate the depth standard deviation. ; S3-5-3, Defect Severity Grading: For maximum depth Less than 1cm is considered minor, 1cm ≤ <3cm and <0.5 points indicates moderate severity. ≥3cm or It is categorized as severe.

10. The vehicle-mounted road defect detection method based on binocular vision and depth fusion according to claim 8, characterized in that, After completing defect classification and severity estimation, the system will obtain the geographic coordinates of the current location through the GPS positioning module, and upload the standardized and packaged detection results. Step S4 is as follows: S4-1, Location Information Acquisition: The GPS positioning module periodically outputs NMEA data, and the main control platform parses the $GPRMC data frame to extract the longitude, latitude, and timestamp corresponding to the defect detection time. S4-2, Data Structure Encapsulation: The system integrates and encapsulates image recognition, depth estimation, defect level, and GPS positioning information to form a unified defect reporting structure; the fields include: defect ID, defect category, location information, bounding box position, average depth, maximum depth, standard deviation, severity level, and timestamp. S4-3, Caching and Preparation for Upload: Structured data is cached in the upload queue in MQTT format, waiting for the communication module to execute the upload process; S4-4, MQTT Communication Upload Phase: After completing defect identification, deep estimation and information encapsulation, the system pushes the detection results to the cloud receiving platform via the MQTT protocol, supporting remote monitoring, data recording and road maintenance scheduling.

Images