A road inspection method and system based on front and rear view heterogeneous sensor layout

Abstract

The application provides a road inspection method and system based on front and rear view heterogeneous sensor layout, which synchronously collects road data through front view sensors installed at the front end of a vehicle to collect wide area data and rear view sensors installed at the rear end of the vehicle to collect refined data; the front view data is sent to a front view real-time processing pipeline for lightweight target detection and real-time uploading, and the rear view data is sent to a rear view deep analysis pipeline for disease identification and local storage for batch uploading; time and space alignment based on a vehicle pose transformation matrix, Dempster-Shafer evidence fusion and Bayesian inference conflict arbitration are adopted to realize cooperative processing of front and rear view heterogeneous data; real-time early warning information, inspection reports and maintenance suggestions are output. Through the heterogeneous layout of front and rear view sensors and data shunting processing, the application realizes the cooperation of wide area dynamic target rapid detection and refined static disease identification, and compared with single sensor layout, the detection range is expanded and the false detection rate is reduced.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation, road infrastructure inspection and computer vision technology, and in particular to a road inspection method and system based on a front- and rear-view heterogeneous sensor layout. Background Technology

[0002] With the increasing demands for smart city construction and refined road maintenance, road inspection systems are playing an increasingly important role in modern transportation infrastructure management. However, existing road inspection systems have revealed many technical problems in practical applications, making it difficult to simultaneously meet the dual requirements of road inspection for both breadth of coverage and detection accuracy.

[0003] First, existing road inspection systems typically employ a single-sensor layout or a homogeneous multi-sensor layout, making it difficult to simultaneously achieve both wide-area rapid detection and refined defect identification. If a large field-of-view, low-resolution sensor is used, defects such as minute cracks and peeling cannot be effectively identified; if a narrow field-of-view, high-resolution sensor is used, it is difficult to cover road assets and traffic incidents at greater distances.

[0004] Second, the existing inspection systems suffer from homogeneous functionality between front-view and rear-view sensors, failing to differentiate them based on their respective detection characteristics. For instance, the front-view system requires rapid capture of dynamic targets ahead, demanding high real-time performance; the rear-view system performs detailed analysis of the road surface already traversed by the vehicle, requiring high resolution and 3D measurement accuracy. This homogeneous layout leads to wasted sensor resources and hinders the achievement of optimal detection results.

[0005] Third, existing inspection systems generally employ a single data processing flow, failing to differentiate processing based on the real-time nature and importance of different inspection tasks. Real-time-critical early warning tasks and disease identification tasks requiring in-depth analysis compete for limited computing resources on edge computing devices, leading to overall performance degradation.

[0006] Fourth, the existing system lacks a dynamic mechanism for allocating computing power, often resulting in large real-time detection delays or insufficient accuracy in in-depth analysis.

[0007] Fifth, the existing inspection equipment has a complex installation structure, making it difficult to quickly deploy to different vehicle models. Installation usually requires specialized tools and a long time, which limits the system's flexibility and scalability.

[0008] To address the aforementioned issues, there is an urgent need for a road inspection system and method that can simultaneously achieve wide-area dynamic target detection and refined static defect identification, and that is flexible in deployment and has a reasonable allocation of computing power.

[0009] The information disclosed in the above background section is only intended to enhance the understanding of the overall background of the present invention, and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0010] To address the aforementioned shortcomings, the present invention aims to provide a road inspection method and system based on a heterogeneous front- and rear-view sensor layout. By using differentiated configurations of the heterogeneous front- and rear-view sensors and data diversion processing logic, it achieves the synergy of rapid detection of dynamic targets in a wide area and refined analysis of static road surface defects. At the same time, it improves deployment flexibility through a modular quick-release installation structure.

[0011] The technical solution of the present invention is as follows: The first aspect of this invention provides a road inspection method based on a front- and rear-view heterogeneous sensor layout, comprising the following steps: S1: Road data is collected synchronously by a front-view sensor installed at the front of the vehicle and a rear-view sensor installed at the rear of the vehicle. The collected front-view data and rear-view data are preprocessed and a unified timestamp and location label are added. The front-view sensor is used to collect wide-area data of the road in front of the vehicle, and the rear-view sensor is used to collect refined data of the road surface that the vehicle has passed. S2: The pre-processed data is split and processed according to the data source: the forward-looking data is sent to the forward-looking real-time processing pipeline, a lightweight algorithm is used to perform target detection and the detection results are uploaded to the cloud in real time; the rear-looking data is sent to the rear-looking depth analysis pipeline, the pavement defects are analyzed in depth and the analysis results are stored locally and then uploaded in batches. S3: Perform heterogeneous data collaborative processing on the output results of the forward-looking real-time processing pipeline and the rear-looking depth analysis pipeline, including unifying the forward-looking detection results and the rear-looking detection results into the same global coordinate system for spatiotemporal alignment, fusing forward-looking and rear-looking evidence, and arbitrating when the forward-looking and rear-looking detection results conflict, to obtain the road condition assessment result. S4: Output real-time early warning information, inspection reports and maintenance suggestions based on the road condition assessment results.

[0012] Specifically, step S1 involves preprocessing the collected forward-looking and backward-looking data and adding unified timestamps and location tags, which includes: S101: The forward-looking data is preprocessed using Gaussian filtering for noise reduction, Brown-Conrad model distortion correction, and histogram equalization for color enhancement. S102: Preprocessing of rear-view data using high dynamic range processing with multi-exposure fusion and local contrast-limited adaptive histogram equalization to enhance details; S103: Add a unified timestamp and location label to the preprocessed forward-looking and backward-looking data. The location label includes latitude and longitude, altitude and attitude angle.

[0013] Specifically, step S2, which involves splitting the preprocessed data according to its source, includes: S201: Determine the processing priority of each acquisition task based on the following traffic allocation decision model:

[0014] in, For the task Processing priority; For real-time requirements; For the importance of the task; For resource utilization rate; The weighting coefficients are and satisfy the following conditions: ; S202: Dynamically allocate computing resources for the forward-looking real-time processing pipeline and the backward-looking depth analysis pipeline according to the processing priority.

[0015] Specifically, step S2, which involves sending the forward-looking data into the forward-looking real-time processing pipeline, using a lightweight algorithm for target detection, and uploading the detection results to the cloud in real time, includes the following steps executed sequentially: S211: Use the YOLOv5s lightweight convolutional neural network model to perform target detection on the forward-looking data, and identify road assets and traffic events. The road assets include guardrails, signs and streetlights, and the traffic events include accidents, construction and congestion. S212: Predict the motion trajectory of the identified target based on Kalman filtering; S213: Conduct risk assessment based on target type, location, and movement status to obtain early warning information including event type, location, and severity; S214: Upload detection results and early warning information to the cloud in real time via 4G or 5G mobile communication networks.

[0016] Specifically, step S2, which involves sending the rear-view data into the rear-view depth analysis pipeline to perform depth analysis on pavement defects and then storing the analysis results locally before uploading them in batches, includes the following steps executed sequentially: S221: The U-Net semantic segmentation network is used to segment the road surface region of the rear view image, separating the road surface region from non-road surface interference; S222: The ResNet-50 classification network is used to identify defects in the segmented pavement areas. The identified defect types include cracks, potholes and spalling. S223: Calculate the length, width, depth, and area of ​​the lesion based on structured light 3D point cloud data; S224: Calculate the severity score of the disease using the following formula:

[0017] Where S represents the severity score of the disease; L is the crack length; W is the crack width; D is the pit depth; and A is the area of ​​disease. These are weighting coefficients determined based on road grade and type of road damage; S225: Stores raw data and analysis results in local storage media in the form of data packets, and uploads the locally stored data to the data center in batches via Wi-Fi after the inspection is completed.

[0018] Specifically, step S3, which unifies the forward-looking detection results and the backward-looking detection results to the same global coordinate system through spatiotemporal alignment, includes: S301: Based on vehicle pose transformation moment and sensor extrinsic matrix The detection points in the sensor coordinate system are calculated using the following formula. Points transformed to the global coordinate system :

[0019] S302: Linear interpolation is used to compensate for the time difference in detection results acquired at different times of forward and backward sight acquisition.

[0020] Specifically, step S3, which involves fusing forward and backward visual evidence and arbitrating when forward and backward visual detection results conflict, includes: S311: Using the Dempster-Shafer evidence theory, calculate the basic probability distribution of the fused proposition A using the following formula:

[0021] Where: m(A) is the basic probability assignment of the fused proposition A; m1(B) and m2(C) are the basic probability assignments of the foresight and backsight evidence; K is the conflict coefficient; S312: When the conflict coefficient K is greater than a preset threshold, it is determined that there is a conflict between the forward and backward look detection results, and the following Bayesian inference model is used for arbitration based on historical data:

[0022] in Assume H in evidence E 1、 Posterior probability under E2 Let P(H) be the likelihood probability, and let P(H) be the prior probability based on historical data.

[0023] Specifically, step S4, which involves outputting real-time early warning information, inspection reports, and maintenance recommendations based on the road condition assessment results, includes: S401: Push real-time warning information to the driver and command center, the real-time warning information including event type, location, severity and recommended measures; S402: Generate a structured inspection report, which includes a road asset inventory and condition assessment, pavement defect statistics and distribution map, traffic incident records and handling suggestions, and an overall road health score, calculated using the following formula:

[0024] Where H represents the road health score, and W... i S represents the weight of the i-th type of disease. i The severity score is given for the i-th type of disease, where n is the number of disease types; S403: Generate maintenance recommendations based on the following maintenance priority decision model:

[0025] Where Priority represents maintenance priority, Severity represents the severity of damage, Traffic represents the degree of traffic impact, and Cost represents the maintenance cost coefficient. These are the weighting coefficients.

[0026] Specifically, the method supports at least one of the following three working modes: Automatic inspection mode: The vehicle travels along a predetermined route, with the forward-looking sensor providing real-time warnings and the rear-looking sensor performing in-depth analysis. Key inspection mode: Reduce vehicle speed and increase the sampling frequency of rear-view sensors to focus on specific road sections or defects; Emergency Response Mode: The forward-looking sensor focuses on detecting sudden events, while the rear-looking sensor assesses road damage.

[0027] A second aspect of this invention provides a road inspection system based on a heterogeneous front-view and rear-view sensor layout. Applying the aforementioned method, the system includes a front-view sensor module, a rear-view sensor module, an edge computing device, and a communication module. The front-view sensor module is installed at the front of the vehicle and connected to the edge computing device, used to synchronously collect wide-area data of the road ahead of the vehicle and send it to the edge computing device. The rear-view sensor module is installed at the rear of the vehicle and connected to the edge computing device, used to synchronously collect refined data of the road surface already traversed by the vehicle and send it to the edge computing device. The edge computing device is connected to the front-view sensor module, the rear-view sensor module, and the communication module respectively, and is configured with a data offloading processing module and a heterogeneous data collaborative processing module. The data offloading and processing module includes a forward-looking real-time processing pipeline and a rear-looking depth analysis pipeline, used to send data collected by the forward-looking sensor module into the forward-looking real-time processing pipeline for real-time processing, and to send data collected by the rear-looking sensor module into the rear-looking depth analysis pipeline for depth analysis; the heterogeneous data collaborative processing module is used to perform spatiotemporal alignment, evidence fusion, and conflict arbitration on the output results of the forward-looking real-time processing pipeline and the rear-looking depth analysis pipeline, and output road condition assessment results; the communication module is connected to the edge computing device, used to upload the detection results of the forward-looking real-time processing pipeline to the cloud in real time, and to periodically upload the analysis results of the rear-looking depth analysis pipeline to the data center in batches.

[0028] The beneficial effects of this invention are: 1. This invention overcomes the limitations of traditional homogeneous layouts by employing a heterogeneous front- and rear-view sensor arrangement, significantly improving the comprehensiveness and detection accuracy of road inspections. In actual tests on urban main roads, the detection range of this invention is significantly increased compared to a single-sensor layout, improving the identification rate of minor defects and the accuracy of fusion result recognition, while reducing the false detection rate and missed detection rate. Simultaneously, the front- and rear-view sensors serve as redundant backups for each other; in the event of a single sensor failure, the system can degrade its operation while maintaining basic inspection capabilities, thus improving system availability.

[0029] 2. This invention achieves differentiated and precise allocation of edge computing power through data offloading processing logic, resolving the inherent conflict between real-time early warning and deep analysis competing for computing power on edge computing devices. The forward-looking real-time processing pipeline prioritizes low latency, with processing latency controllable within 100ms, meeting the real-time requirements of driver-level early warning; the rear-looking depth analysis pipeline prioritizes high precision, with depth analysis accuracy exceeding 95%, meeting the accuracy requirements of road maintenance decision-making; the offloading decision model achieves a 92% correct allocation rate for data collection tasks.

[0030] 3. A heterogeneous data collaborative algorithm based on Dempster-Shafer evidence theory and Bayesian inference was used to achieve mutual verification and conflict resolution of fore- and post-view detection results, constructing a complete road condition profile. This mechanism not only outputs detection results but also establishes causal relationships between fore-view event areas and corresponding post-view pavement defects, outputting a complete road condition profile with semantic information.

[0031] 4. By flexibly switching between three working modes—automatic inspection, key inspection, and emergency response—the same system can adapt to different task scenarios such as daily inspection, disease review, and disaster emergency response, significantly improving the system's applicability and economic benefits, and extending the service life of roads by reducing the frequency of major repairs. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the hardware configuration of the road inspection system based on a front- and rear-view heterogeneous sensor layout according to the present invention; Figure 2 This is a schematic diagram of the road inspection method based on a front- and rear-view heterogeneous sensor layout according to the present invention; Figure 3 This is a software processing flowchart of an embodiment of the present invention; Figure 4 This is a data splitting and processing logic diagram according to an embodiment of the present invention. Detailed Implementation

[0033] To illustrate the technical content, objectives, and effects of this invention in detail, the following description is provided in conjunction with embodiments and accompanying drawings. In the description of the embodiments, it should be understood that terms indicating orientation or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings and are used only for the convenience of describing this embodiment and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation; therefore, they should not be construed as limitations on this invention.

[0034] like Figure 1 As shown, this embodiment provides a road inspection system based on a heterogeneous front-view and rear-view sensor layout, including a front-view sensor module, a rear-view sensor module, an edge computing device, a communication module, and a modular quick-release installation structure.

[0035] The forward-looking sensor module is mounted above the vehicle's front bumper and connects to an edge computing device to synchronously collect road data in front of the vehicle and send it to the edge computing device. The forward-looking sensor module includes: Wide-angle industrial camera: In this embodiment, a wide-angle industrial camera with a resolution of 2 megapixels, a horizontal field of view of 120°, and a sampling frequency of 30fps is used to capture a panoramic view of the road ahead. In other embodiments, the specific specifications of the camera can be flexibly adjusted according to actual detection needs, but its field of view should be greater than that of the camera in the rear-view sensor module to meet the requirements of wide-area dynamic target detection.

[0036] LiDAR: In this embodiment, an equivalent 128-line LiDAR is used, with a detection range of 200 meters and an angular resolution of 0.1°, to obtain three-dimensional information of obstacles in front.

[0037] GPS / IMU integrated navigation system: used to provide vehicle position, speed and attitude information, and to provide a benchmark for the spatiotemporal alignment of sensor data.

[0038] The rear-view sensor module is mounted above the rear trunk of the vehicle and connects to an edge computing device. It synchronously collects data on the road surface the vehicle has passed and sends it to the edge computing device. The rear-view sensor module includes: High-resolution industrial camera: In this embodiment, a high-resolution industrial camera with a resolution of 5 megapixels, a horizontal field of view of 60°, and a sampling frequency of 15fps is used to capture road surface details. In other embodiments, the specific specifications of the camera can be flexibly adjusted according to actual detection needs, but its resolution should be higher than that of the camera in the forward-looking sensor module to meet the requirements for identifying subtle defects.

[0039] Structured light 3D sensor: In this embodiment, a structured light 3D sensor with a measurement accuracy of ±1mm and a scanning range of 2m×2m is used to acquire the 3D shape of the road surface.

[0040] Supplemental lighting system: It adopts an LED array that can automatically adjust the brightness according to the ambient light to ensure the effectiveness of nighttime inspections.

[0041] It should be noted that, compared to the front-view sensor module, the rear-view sensor module has a significantly higher camera resolution, while its field of view and acquisition frequency are significantly lower. This heterogeneous configuration is the key design feature of this invention: the front-view module prioritizes a large field of view and a high frame rate to meet the need for rapid acquisition of dynamic targets; the rear-view module prioritizes high resolution and high 3D measurement accuracy to meet the need for detailed identification of static road surface defects.

[0042] The specific camera resolution, field of view, acquisition frame rate, number of lidar beams, and other parameters used in the front-view and rear-view sensor modules can be adaptively adjusted according to actual inspection needs, but the heterogeneous layout principle of "front-view field of view is greater than rear-view field of view and rear-view resolution is higher than front-view resolution" should be maintained.

[0043] The edge computing device is installed in the vehicle's trunk and connects to the front-view sensor module, rear-view sensor module, and communication module. The edge computing device is equipped with a data offloading processing module and a heterogeneous data collaborative processing module.

[0044] The data offloading and processing module includes a forward-looking real-time processing pipeline and a rear-looking depth analysis pipeline. It is used to send the data collected by the forward-looking sensor module into the forward-looking real-time processing pipeline for real-time processing, and to send the data collected by the rear-looking sensor module into the rear-looking depth analysis pipeline for depth analysis.

[0045] The heterogeneous data collaborative processing module is used to perform spatiotemporal alignment, evidence fusion, and conflict arbitration on the outputs of the forward-looking real-time processing pipeline and the rear-looking depth analysis pipeline, and output road condition assessment results.

[0046] In this embodiment, the edge computing device uses an NVIDIA Jetson AGX Xavier processor, 32GB of RAM, and a 512GB SSD; the accompanying data acquisition card supports multiple Camera Link interfaces and Ethernet interfaces; and the power management module supports 12V / 24V adaptive input. In other embodiments, the specific model and configuration of the edge computing device can be flexibly replaced according to actual needs.

[0047] The communication module connects to the edge computing device to upload the detection results of the forward-looking real-time processing pipeline to the cloud in real time, and to periodically upload the analysis results of the backward-looking depth analysis pipeline to the data center in batches. In this embodiment, the communication module uses a 4G / 5G dual-mode communication device, supporting real-time data upload and remote control; batch uploads can also be performed using Wi-Fi.

[0048] To enhance the system's deployment flexibility across different vehicle models, the front-view and rear-view sensor modules can be connected to the vehicle body via a modular quick-release mounting structure, thereby improving deployment efficiency.

[0049] During the system deployment phase, the forward-looking and rear-looking sensors need to be calibrated. In this embodiment, the Zhang Zhengyou calibration method is used to calibrate the camera's intrinsic parameters, with a calibration error of less than 0.1 pixels; a joint calibration method is used to calibrate the extrinsic parameters of the LiDAR and camera, with a calibration error of less than 2 cm. The calibration parameters are stored in the configuration file of the edge computing device for use in data acquisition and coordinate transformation during the inspection process. It should be noted that the Zhang Zhengyou calibration method and the LiDAR-camera joint calibration method are mature existing technologies, and those skilled in the art can select appropriate specific implementation methods according to the actual configuration.

[0050] like Figure 2-4As shown, the software processing flow provided in this embodiment includes four steps: data acquisition and preprocessing, data splitting and processing, heterogeneous data collaborative processing, and result output and application. Each step is closely connected through clearly defined data and control flows.

[0051] Step S1: Data Acquisition and Preprocessing Synchronous acquisition by the front and rear view sensors is achieved through hardware trigger signals. In this embodiment, the synchronization error should be less than 1ms.

[0052] In this embodiment, the forward-looking sensor acquires data at a frequency of 30fps, with each frame containing a 1920×1080 RGB image, approximately 3 million points / second of point cloud data, and GPS / IMU data; the rear-looking sensor acquires data at a frequency of 15fps, with each frame containing a 2592×1944 high-resolution image and approximately 500,000 points of 3D point cloud data.

[0053] S101 Forward-looking data preprocessing: The forward-looking data is preprocessed using Gaussian filtering for noise reduction, Brown-Conrad model distortion correction, histogram equalization, and color enhancement. The Gaussian filtering, Brown-Conrad distortion model, and histogram equalization used are all mature technologies in this field and can be flexibly implemented by those skilled in the art according to actual needs.

[0054] S102 Rearview Data Preprocessing: High dynamic range processing is achieved through multi-exposure fusion of the rearview data, and road surface details are enhanced using locally contrast-limited adaptive histogram equalization. Both the multi-exposure fusion method and the locally contrast-limited adaptive histogram equalization method are mature technologies in this field.

[0055] S103 Spatiotemporal Labeling: All data is labeled with a unified timestamp and location label. In this embodiment, the timestamp uses UTC time with an accuracy of 1ms; the location labels include latitude and longitude, altitude, and attitude angle. A spatiotemporal reference is established through the unified timestamp and location label, providing a foundation for subsequent data fusion.

[0056] Through the above data processing, during testing on the test road, the forward-looking camera captured an image containing guardrails and traffic signs. After preprocessing in S101, the image signal-to-noise ratio was improved from 25dB to 35dB, and the distortion correction error was less than 0.5 pixels. The image of road surface cracks captured by the rear-looking camera was processed in S102 high dynamic range, and the visibility of dark details was improved.

[0057] The preprocessed data output in this step is passed to the split processing module in step S2 via shared memory, and the spatiotemporal label provides a benchmark for data fusion in step S3.

[0058] Step S2: Data splitting and processing: like Figure 4As shown, data diversion processing is one of the core innovations of this invention.

[0059] S201 Task Assignment Decision: Based on the source of the forward / backward data and the type of the detected target, tasks are automatically assigned to the corresponding processing pipelines. The task assignment decision is based on task priority and real-time requirements, and the assigned task decision model adopted is as follows:

[0060] Where P(i) is the processing priority of task i; R(i) is the real-time requirement, ranging from 0 to 1, with 1 indicating the highest real-time requirement; A(i) is the importance of the task, ranging from 0 to 1, with 1 indicating the most important task; U(i) is the resource utilization rate, ranging from 0 to 1, with 1 indicating sufficient resources; α, β, and γ are weighting coefficients and satisfy α+β+γ=1.

[0061] In this embodiment, for forward-looking real-time early warning tasks, the weighting coefficients are α=0.6, β=0.3, and γ=0.1, emphasizing real-time priority; for backward-looking depth analysis tasks, the weighting coefficients are α=0.2, β=0.5, and γ=0.3, emphasizing task importance and resource sufficiency. These weighting coefficients can be adjusted by those skilled in the art according to the actual deployment scenario.

[0062] S202 Resource Allocation: Based on the processing priority P(i) calculated in S201, dynamically allocate computing resources for the forward real-time processing pipeline and the backward depth analysis pipeline, giving priority to tasks with high real-time requirements.

[0063] S211 Forward-Looking Object Detection: A lightweight convolutional neural network model, YOLOv5s, is used to detect objects in the forward-looking data, identifying road assets such as guardrails, signs, streetlights, and traffic events such as accidents, construction, and congestion. In other embodiments, YOLOv5s can be replaced with other lightweight deep learning models, such as YOLOv7-tiny, MobileNet-SSD, etc.

[0064] S212 Motion Prediction: Predicts the motion trajectory of the identified target based on Kalman filtering.

[0065] S213 Risk Assessment: Assess the risk level based on the target type, location, and movement status to obtain early warning information including event type, location, and severity.

[0066] S214 Real-time upload: Detection results and early warning information are uploaded to the cloud in real time via 4G or 5G mobile communication networks.

[0067] S221 Road Surface Segmentation: The U-Net semantic segmentation network is used to segment the road surface region of the rear view image, separating the road surface region from non-road surface interference. In other embodiments, U-Net can be replaced by other semantic segmentation networks such as DeepLabV3+.

[0068] S222 Defect Identification: The ResNet-50 classification network is used to identify defects in the segmented pavement areas. The identified defect types include cracks, potholes, and spalling. In other embodiments, ResNet-50 can be replaced with other classification networks such as EfficientNet.

[0069] S223 Parameter Calculation: Calculate the length, width, depth, and area of ​​the lesion based on point cloud data obtained from a structured light 3D sensor.

[0070] S224 Severity Assessment: The severity score of the disease is calculated using the following formula:

[0071] Where S is the severity score of the damage, ranging from 0 to 100; L is the crack length in centimeters; W is the crack width in millimeters; D is the pothole depth in centimeters; A is the area of ​​the damage in square centimeters; w1, w2, w3, and w4 are weighting coefficients, determined according to the road grade and the type of damage.

[0072] For example, in the implementation of this embodiment, a crack was detected by the rear-view sensor. It was segmented by U-Net and identified as a transverse crack by ResNet-50. The crack was calculated to be 35cm long, 2.5mm wide, and 1.2cm deep, with a severity score of 65, which is considered moderately severe.

[0073] S225 Local Storage and Batch Upload: Stores complete data packets containing raw data and analysis results in the local SSD of the edge computing device as data packets; after the daily inspection, the locally stored data is uploaded to the data center in batches via Wi-Fi.

[0074] The detection results output in this step are transmitted to the data fusion module in step S3 via a message queue, while the real-time warning information is simultaneously transmitted to the output module in step S4.

[0075] Step S3: Heterogeneous data collaborative processing: S301 Coordinate System Transformation: Based on GPS / IMU data, the forward and backward look detection results are unified to the same global coordinate system. The spatiotemporal alignment model is as follows:

[0076] Where P_global is a point in the global coordinate system; T_vehicle is the vehicle pose transformation matrix; T_sensor is the sensor extrinsic parameter matrix (obtained during system deployment); and P_sensor is a point in the sensor coordinate system.

[0077] S302 Time Compensation: For detection results at the same spatial location but acquired at different times, linear interpolation is used to compensate for the time difference to ensure spatiotemporal consistency.

[0078] S311 Evidence Fusion: The Dempster-Shafer evidence theory is used to fuse the results of forward and backward look detections.

[0079] Where m(A) is the basic probability assignment of proposition A after fusion; m1(B) and m2(C) are the basic probability assignments of the forward-looking and backward-looking evidence, respectively; and K is the conflict coefficient. The Dempster-Shafer evidence theory can effectively fuse detection evidence from heterogeneous forward-looking and backward-looking sensors, improving the confidence level of the final judgment.

[0080] S312 Conflict Arbitration: When the conflict coefficient K is greater than the preset threshold of 0.7 in this embodiment, it is determined that there is a conflict between the forward and backward look detection results, and arbitration is performed using a Bayesian inference model based on historical data.

[0081] Where P(H|E1,E2) is the posterior probability of hypothesis H under evidence E1 and E2; P(E1|H) and P(E2|H) are the likelihood probabilities; and P(H) is the prior probability based on historical data. The prior probability P(H) is obtained by statistically analyzing historical detection data accumulated by this system during long-term inspections.

[0082] S321 Establishing Correlation: Establishing a correlation between forward-looking detected targets and rear-looking detected targets. The correlation includes the causal correspondence between the event area detected by forward-looking and the corresponding pavement defects detected by rear-looking. For example, "forward-looking detected a construction area → rear-looking detected potholes in that area" reflects the correlation between events perceived by the wide-area forward-looking system and specific defects perceived by the fine rear-looking system.

[0083] S322 Road Condition Profile Output: The road condition profile is output as a component of the road condition profile, forming a complete road condition profile. The complete road condition profile includes not only the targets and defects detected by the forward and rear views, but also the causal correspondences between them, providing richer semantic information for road maintenance decisions.

[0084] For example, in this embodiment, during implementation, the front-view detects a construction area with a confidence level of 0.88, the rear-view detects potholes in the road surface of the same area with a confidence level of 0.91, and the confidence level is increased to 0.95 after Dempster-Shafer fusion, confirming the association relationship through Bayesian inference.

[0085] The fusion results output in this step are passed to the output module of step S4 through the database, and the complete road condition profile is used to generate a comprehensive report.

[0086] Step S4: Result Output and Application: S401 Real-time Warning: Pushes real-time warning information to drivers and the command center. Warning information includes event type, location, severity, and recommended actions. For high-risk events, such as accidents ahead, warnings are pushed immediately for rapid response.

[0087] S402 Inspection Report Generation: A structured inspection report is generated daily. This report includes a road asset inventory and condition assessment, pavement defect statistics and distribution maps, traffic incident records and handling recommendations, and an overall road health score. The overall road health score is calculated using the following formula:

[0088] Where H is the road health score, ranging from 0 to 100, with 100 representing the best; wi is the weight of the i-th type of disease; Si is the severity score of the i-th type of disease; and n is the number of disease types.

[0089] S403 Maintenance Recommendation Generation: Maintenance priorities are generated based on historical data and AI algorithms. The maintenance priority decision model is as follows:

[0090] Where Priority represents maintenance priority; Severity represents the severity of the damage; Traffic represents the degree of traffic impact; Cost represents the maintenance cost coefficient; and α, β, and γ are weighting coefficients. In this embodiment, α, β, and γ can be taken as 0.5, 0.3, and 0.2 respectively, emphasizing that the severity of the damage is the main basis for decision-making.

[0091] For example, in a routine inspection, the system detected 15 defects, generated a road health score of 82 points, and recommended 3 high-priority maintenance points.

[0092] The system of this invention supports the following three working modes, which can be selected and switched through the user interface. The system automatically adjusts the sensor parameters and the resource allocation strategy of the processing pipeline: (1) Automatic inspection mode: The vehicle travels along a predetermined route, and the system operates fully automatically, providing real-time forward warnings and rearward depth analysis. This mode is suitable for daily road inspections and is the system's default operating mode.

[0093] (2) Targeted Inspection Mode: This mode focuses on inspecting specific road sections or defects, reducing vehicle speed to provide a longer data collection window, while increasing the sampling frequency of the rear-view sensors to obtain denser and more detailed road surface data. This mode is suitable for defect re-inspection or construction supervision, and can perform high-precision re-measurement of known problem areas.

[0094] (3) Emergency Response Mode: Upon receiving an emergency task, rapid deployment is carried out. The forward-looking sensor focuses on detecting sudden events, while the rear-looking sensor assesses road damage. This mode is suitable for disaster emergency response, such as rapid assessment of road accessibility after earthquakes and floods.

[0095] The time logic connection principle of this embodiment is as follows: The preprocessed data output from step S1 is passed to the split processing module in step S2 via shared memory. This data includes timestamps, location tags, and preprocessing parameters. The shared memory approach ensures minimal data transfer latency, meeting the low-latency requirements of the forward-looking real-time processing pipeline.

[0096] The detection results output in step S2 are transmitted to the heterogeneous data collaborative processing module in step S3 via a message queue. This message queue method supports asynchronous transmission of forward and backward look detection results, preventing the time-consuming operation of backward look depth analysis from blocking real-time forward look processing.

[0097] The fusion results output from step S3 are passed to the output module of step S4 via a database, and include road condition assessment, correlation, and health score. The database approach facilitates the querying and statistical analysis of historical data.

[0098] As a preferred embodiment, if any step fails, the system records an error log and attempts to retry or downgrade the process to ensure system availability. For example, if the rear-view depth analysis pipeline malfunctions, the system can downgrade to rely solely on the front-view real-time processing pipeline to maintain basic inspection functions.

[0099] As a preferred embodiment, the system dynamically adjusts the computing resources of each step according to the system load, giving priority to ensuring the resource supply of the forward-looking real-time processing pipeline, and avoiding delays in real-time early warning due to insufficient resources.

[0100] This embodiment was tested and verified on an urban main road (5km long, 6 lanes in both directions). The test vehicle was an SUV equipped with this inspection system. The test time was 2 hours each during the morning and evening rush hours on weekdays. The test content included road asset detection, road surface defect identification, and traffic incident early warning.

[0101] Table 1 System performance test results

[0102] Table 2 Economic Benefit Analysis

[0103] In actual testing, the triage decision model correctly allocated 92% of the tasks to the appropriate processing pipeline; the detection confidence improved by an average of 8.3% and a maximum of 15.2% after the fusion of Dempster-Shafer evidence theory; and the road health score results were 89.5% consistent with the expert assessment.

[0104] Although the present invention has been described in detail above with specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A road inspection method based on a front- and rear-view heterogeneous sensor layout, characterized in that, Includes the following steps: S1: Road data is collected synchronously by a front-view sensor installed at the front of the vehicle and a rear-view sensor installed at the rear of the vehicle. The collected front-view and rear-view data are preprocessed and a unified timestamp and location label are added respectively. The front-view sensor is used to collect wide-area data of the road in front of the vehicle, and the rear-view sensor is used to collect refined data of the road surface that the vehicle has passed. S2: The pre-processed data is split and processed according to the data source: the forward-looking data is sent to the forward-looking real-time processing pipeline, a lightweight algorithm is used to perform target detection and the detection results are uploaded to the cloud in real time; the rear-looking data is sent to the rear-looking depth analysis pipeline, the pavement defects are analyzed in depth and the analysis results are stored locally and then uploaded in batches. S3: Perform heterogeneous data collaborative processing on the output results of the forward-looking real-time processing pipeline and the rear-looking depth analysis pipeline, including unifying the forward-looking detection results and the rear-looking detection results into the same global coordinate system for spatiotemporal alignment, fusing forward-looking and rear-looking evidence, and arbitrating when the forward-looking and rear-looking detection results conflict, to obtain the road condition assessment result. S4: Output real-time early warning information, inspection reports and maintenance suggestions based on the road condition assessment results.

2. The road inspection method based on a heterogeneous front- and rear-view sensor layout according to claim 1, characterized in that, Step S1, which involves preprocessing the collected forward-looking and backward-looking data and adding unified timestamps and location tags, specifically includes: S101: The forward-looking data is preprocessed using Gaussian filtering for noise reduction, Brown-Conrad model distortion correction, and histogram equalization for color enhancement. S102: Preprocessing of rear-view data using high dynamic range processing with multi-exposure fusion and local contrast-limited adaptive histogram equalization to enhance details; S103: Add a unified timestamp and location label to the preprocessed forward-looking and backward-looking data. The location label includes latitude and longitude, altitude and attitude angle.

3. The road inspection method based on a front- and rear-view heterogeneous sensor layout according to claim 1, characterized in that, Step S2, which describes the process of splitting the preprocessed data according to its source, specifically includes: S201: Determine the processing priority of each acquisition task based on the following traffic allocation decision model: ; in, For the task Processing priority; For real-time requirements; For the importance of the task; For resource utilization rate; The weighting coefficients are and satisfy the following conditions: ; S202: Dynamically allocate computing resources for the forward-looking real-time processing pipeline and the backward-looking depth analysis pipeline according to the processing priority.

4. The road inspection method based on a heterogeneous front- and rear-view sensor layout according to claim 1, characterized in that, Step S2, which involves sending the forward-looking data into the forward-looking real-time processing pipeline, performing target detection using a lightweight algorithm, and uploading the detection results to the cloud in real time, specifically includes the following steps executed sequentially: S211: Use the YOLOv5s lightweight convolutional neural network model to perform target detection on the forward-looking data, and identify road assets and traffic events. The road assets include guardrails, signs and streetlights, and the traffic events include accidents, construction and congestion. S212: Predict the motion trajectory of the identified target based on Kalman filtering; S213: Conduct risk assessment based on target type, location, and movement status to obtain early warning information including event type, location, and severity; S214: Upload detection results and early warning information to the cloud in real time via 4G or 5G mobile communication networks.

5. The road inspection method based on a front- and rear-view heterogeneous sensor layout according to claim 1, characterized in that, Step S2, which involves sending the rear-view data into the rear-view depth analysis pipeline to perform depth analysis on pavement defects and then storing the analysis results locally before uploading them in batches, specifically includes the following steps executed sequentially: S221: The U-Net semantic segmentation network is used to segment the road surface region of the rear view image, separating the road surface region from non-road surface interference; S222: The ResNet-50 classification network is used to identify defects in the segmented pavement areas. The identified defect types include cracks, potholes and spalling. S223: Calculate the length, width, depth, and area of ​​the lesion based on structured light 3D point cloud data; S224: Calculate the severity score of the disease using the following formula: ; Where S represents the severity score of the disease; L is the crack length; W is the crack width; D is the pit depth; and A is the area of ​​disease. These are weighting coefficients determined based on road grade and type of road damage; S225: Stores raw data and analysis results in local storage media in the form of data packets, and uploads the locally stored data to the data center in batches via Wi-Fi after the inspection is completed.

6. The road inspection method based on a heterogeneous front- and rear-view sensor layout according to claim 1, characterized in that, Step S3, which involves unifying the forward-looking detection results and the backward-looking detection results to the same global coordinate system through spatiotemporal alignment, specifically includes: S301: Based on vehicle pose transformation moment and sensor extrinsic matrix The detection points in the sensor coordinate system are calculated using the following formula. Points transformed to the global coordinate system : ; S302: Linear interpolation is used to compensate for the time difference in detection results acquired at different times of forward and backward sight acquisition.

7. The road inspection method based on a heterogeneous front- and rear-view sensor layout according to claim 1, characterized in that, Step S3, which involves fusing forward and backward visual evidence and arbitrating when forward and backward visual detection results conflict, specifically includes: S311: Using the Dempster-Shafer evidence theory, calculate the basic probability distribution of the fused proposition A using the following formula: ; Where: m(A) is the basic probability assignment of the fused proposition A; m1(B) and m2(C) are the basic probability assignments of the foresight and backsight evidence; K is the conflict coefficient; S312: When the conflict coefficient K is greater than a preset threshold, it is determined that there is a conflict between the forward and backward look detection results, and the following Bayesian inference model is used for arbitration based on historical data: ; in Assume H in evidence E 1、 Posterior probability under E2 Let P(H) be the likelihood probability, and let P(H) be the prior probability based on historical data.

8. The road inspection method based on a heterogeneous front- and rear-view sensor layout according to claim 1, characterized in that, Step S4, which involves outputting real-time early warning information, inspection reports, and maintenance recommendations based on the road condition assessment results, specifically includes: S401: Push real-time warning information to the driver and command center, the real-time warning information including event type, location, severity and recommended measures; S402: Generate a structured inspection report, which includes a road asset inventory and condition assessment, pavement defect statistics and distribution map, traffic incident records and handling suggestions, and an overall road health score, calculated using the following formula: ; Where H represents the road health score, and W... i S represents the weight of the i-th type of disease. i The severity score is given for the i-th type of disease, where n is the number of disease types; S403: Generate maintenance recommendations based on the following maintenance priority decision model: ; Where Priority represents maintenance priority, Severity represents the severity of damage, Traffic represents the degree of traffic impact, and Cost represents the maintenance cost coefficient. These are the weighting coefficients.

9. The road inspection method based on a front- and rear-view heterogeneous sensor layout according to claim 1, characterized in that, The method supports at least one of the following three working modes: Automatic inspection mode: The vehicle travels along a predetermined route, with the forward-looking sensor providing real-time warnings and the rear-looking sensor performing in-depth analysis. Key inspection mode: Reduce vehicle speed and increase the sampling frequency of rear-view sensors to focus on specific road sections or defects; Emergency Response Mode: The forward-looking sensor focuses on detecting sudden events, while the rear-looking sensor assesses road damage.

10. A road inspection system based on a heterogeneous front- and rear-view sensor layout, employing the road inspection method based on a heterogeneous front- and rear-view sensor layout as described in any one of claims 1 to 9, characterized in that, It includes a front-view sensor module, a rear-view sensor module, an edge computing device, and a communication module; The forward-looking sensor module is installed at the front of the vehicle and connected to the edge computing device to synchronously collect wide-area data of the road in front of the vehicle and send it to the edge computing device. The rear-view sensor module is installed at the rear of the vehicle and connected to the edge computing device. It is used to synchronously collect detailed data of the road surface that the vehicle has passed and send it to the edge computing device. The edge computing device is connected to the front-view sensor module, the rear-view sensor module, and the communication module, respectively, and is configured with a data offloading processing module and a heterogeneous data collaborative processing module. The data offloading processing module includes a front-view real-time processing pipeline and a rear-view depth analysis pipeline, used to send the data collected by the front-view sensor module into the front-view real-time processing pipeline for real-time processing, and to send the data collected by the rear-view sensor module into the rear-view depth analysis pipeline for depth analysis. The heterogeneous data collaborative processing module is used to perform spatiotemporal alignment, evidence fusion, and conflict arbitration on the output results of the forward-looking real-time processing pipeline and the rear-looking depth analysis pipeline, and output road condition assessment results. The communication module is connected to the edge computing device and is used to upload the detection results of the forward-looking real-time processing pipeline to the cloud in real time, and to periodically upload the analysis results of the rear-looking depth analysis pipeline to the data center in batches.

Images