Highway pavement quality monitoring method and system based on deep learning
By generating dense depth maps based on two-dimensional image data and sparse depth maps, and combining it with a two-stream network model, the problems of expensive equipment and insufficient data fusion in traditional highway pavement quality monitoring are solved, and efficient and accurate pavement condition monitoring is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Filing Date
- 2026-04-14
- Publication Date
- 2026-07-10
AI Technical Summary
Traditional methods for monitoring the quality of highway pavement rely on single visual images, which lack accurate perception of elevation changes or require high-cost 3D laser scanning equipment. This results in expensive equipment, low acquisition efficiency, and difficulty in routine deployment. Furthermore, the lack of differentiation mechanisms when fusing multi-source data leads to missed detections and false detections.
A method for generating dense depth maps based on two-dimensional image data and sparse depth maps is adopted. The image texture and geometric depth information are processed separately by combining a two-stream network model. Uncertain information is introduced through the depth completion stage to achieve dynamic fusion.
Acquiring high-density road surface geometry information while maintaining economic efficiency improves the comprehensiveness and accuracy of monitoring results and reduces the rate of missed and false detections.
Smart Images
Figure CN122367962A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of highway pavement quality monitoring technology, specifically to a highway pavement quality monitoring method and system based on deep learning. Background Technology
[0002] Highway pavement quality monitoring refers to the regular or real-time monitoring of highway pavement conditions through scientific testing methods and technical means, assessing performance indicators such as pavement smoothness, skid resistance, and cracks to ensure its safety. Monitoring can promptly detect pavement defects and hidden dangers, allowing for early maintenance or repair measures to reduce the risk of traffic accidents and vehicle damage, extend road service life, and ensure smooth and safe road traffic.
[0003] However, traditional methods for monitoring the quality of highway pavement often suffer from technical problems, such as relying on a single visual image or requiring expensive 3D laser scanning equipment. The former lacks accurate perception of changes in pavement elevation, while the latter is expensive, has low acquisition efficiency, and complex data processing, making it difficult to deploy routinely. Traditional methods for monitoring the quality of highway pavement often use only 2D images as a single input, or lack a differentiated fusion mechanism when simply stacking multi-source data. This makes it difficult for the model to accurately distinguish between apparent anomalies and geometric defects in complex pavement scenarios, and it is prone to missed detections and false detections. Summary of the Invention
[0004] To address the aforementioned issues and overcome the shortcomings of existing technologies, this invention provides a deep learning-based method and system for monitoring highway pavement quality. Traditional methods often rely on single visual images or require expensive 3D laser scanning equipment. The former lacks accurate perception of pavement elevation changes, while the latter is expensive, inefficient, and complex to process, making routine deployment difficult. This solution creatively employs a method that generates dense depth maps from 2D image data and sparse depth maps for subsequent processing. This achieves high-density pavement geometric information while maintaining economic efficiency, providing crucial 3D morphological support for precise pavement condition identification. Furthermore, traditional methods often rely solely on 2D images as input or lack differentiated fusion mechanisms when simply stacking multi-source data, making it difficult for models to accurately distinguish between surface anomalies and geometric defects in complex pavement scenarios, leading to missed and false detections. This solution creatively uses a dual-stream network model as the pavement monitoring model. By processing image texture and geometric depth information separately and achieving dynamic fusion, while introducing uncertainty information during the depth completion stage, the comprehensiveness and accuracy of the monitoring results are improved.
[0005] The technical solution adopted by this invention is as follows: The deep learning-based highway pavement quality monitoring method provided by this invention includes the following steps:
[0006] Step S1: Acquisition of raw images;
[0007] Step S2: Data optimization processing;
[0008] Step S3: Depth completion model construction;
[0009] Step S4: Road surface monitoring model construction;
[0010] Step S5: Highway pavement quality monitoring.
[0011] Further, in step S1, the original image acquisition is used to collect the original image data required for highway pavement quality monitoring. Specifically, it is to obtain a pavement monitoring original image dataset through image acquisition. The pavement monitoring original image dataset specifically includes a historical pavement original image dataset and a current pavement original image dataset. Both the historical pavement original image dataset and the current pavement original image dataset contain pavement image data and sparse depth point cloud data. The historical pavement original image dataset also contains historical dense depth data and historical pavement defect annotation data.
[0012] Further, in step S2, the data optimization processing is used to optimize the acquired raw image data, specifically including the following steps:
[0013] Step S21: Data spatiotemporal alignment, used to unify data from different sensors into the image coordinate system for processing. Specifically, it maps sparse three-dimensional laser point clouds onto the corresponding pixel positions on the two-dimensional image plane to obtain a sparse depth map aligned with the RGB image pixels.
[0014] Step S22: Region of Interest Extraction, used to focus computing resources on key road surface areas and eliminate interference. Specifically, it calculates the effective area in the image containing only the road surface based on the geometric parameters of the acquisition device and generates a mask to obtain the road surface area image.
[0015] Step S23: Sequence frame organization, which utilizes the temporal continuity information of the video sequence to enhance the stability of single-frame perception. Specifically, it calculates pixel-level motion vectors between consecutive frames through an optical flow network to obtain a dense optical flow field between adjacent frames.
[0016] Step S24: Data augmentation, used to improve the robustness of the model to different acquisition conditions and noise, specifically by applying synchronous color perturbation, spatial transformation and simulating different proportions of missing laser point clouds to the image and depth data to obtain augmented and standardized data;
[0017] Step S25: Dataset partitioning, used to obtain training data and test data, specifically by partitioning the original historical road surface image dataset into a completion training set and a completion test set;
[0018] The current road surface original image dataset is optimized through the data spatiotemporal alignment, the region of interest extraction, and the sequence frame organization to obtain the current dataset to be processed. The historical road surface original image dataset is optimized through the data spatiotemporal alignment, the region of interest extraction, the sequence frame organization, the data augmentation, and the dataset partitioning to obtain the completed training set and the completed test set.
[0019] Further, in step S3, the depth completion model construction is used to construct a model that generates a dense depth map based on two-dimensional image data and sparse depth map for subsequent processing. Specifically, it constructs an improved codec model as a depth completion model. The improved codec model outputs a dense depth map and a depth uncertainty map by combining planar prior guidance and iterative refinement.
[0020] The construction of the depth completion model specifically includes the following steps:
[0021] Step S31: Iterative initialization, used to construct the initial state for the iterative optimization process, includes the following steps:
[0022] Step S311: Input data concatenation, used to integrate data from different modalities into a unified tensor format, specifically concatenating road surface image data containing texture information and sparse depth map containing geometric information in the channel dimension to obtain a fused feature map as input data;
[0023] Step S312: Initialize iterative parameters, which is used to set the start state and termination conditions of the iterative optimization process. Specifically, the predicted dense depth map and depth uncertainty map are initialized to zero matrices, and the total number of iterations is set.
[0024] Step S313: Mask parameter setting, used to distinguish between valid data areas and invalid background areas when calculating loss. Specifically, it defines a binary valid point mask and a binary road surface mask. For the binary valid point mask, the corresponding element is 1 when the pixel has a LiDAR measurement value, and 0 otherwise. For the binary road surface mask, the corresponding element is 1 when the pixel belongs to the road surface area image, and 0 otherwise.
[0025] Step S32: Planar prior-guided design, used to embed the physical prior knowledge that the road surface is macroscopically approximately planar into the network, suppressing unreasonable undulations in flat areas. The steps include:
[0026] Step S321: Encoder feature extraction, used to progressively extract multi-level, abstract feature representations from the fused input data, specifically by processing the fused feature map of the input through a downsampling convolutional neural network to obtain an enhanced feature map;
[0027] Step S322: Planar parameter prediction, used to decouple the planar model parameters describing the macroscopic geometry of the road surface from the global features. Specifically, after global pooling of the enhanced feature map, a set of parameters is regressed through a fully connected layer, where each set of parameters defines the normal vector and distance of a local plane.
[0028] Step S323: Planar feature generation, which adapts the global planar parameters to the local geometry at different locations in the image. Specifically, it transforms the predicted planar parameters into a planar prior feature map aligned with the feature map space through a deformable convolutional layer based on the image content adaptive sampling offset.
[0029] Step S324: Feature enhancement fusion, which combines the extracted general features with the road surface-specific planar prior features. Specifically, the enhanced feature map output by the encoder is added to the planar prior feature map with learnable weights to obtain the enhanced fused feature map.
[0030] Step S33: Iterative refinement design, used to achieve multi-round, coarse-to-fine optimization, focusing on correcting high uncertainty and mission-critical areas. Steps include:
[0031] Step S331: Spatiotemporal feature fusion, which is used to stabilize the accuracy of single-frame depth prediction by utilizing the strong correlation between consecutive frames. Specifically, during the training phase, the features of adjacent frames are aligned to the current frame coordinate system through optical flow and then fused by weight. The resulting features replace the enhanced fused feature map and are used as the input features of the decoder based on the U-Net structure.
[0032] Step S332: Decoding and output prediction, used to recover a dense depth map and a depth uncertainty map with the same resolution as the input from high-level semantic features. Specifically, the enhanced fusion feature map is used as the input feature of the decoder based on the U-Net structure, and the dense depth map and the depth uncertainty map are output through two independent convolution heads respectively. The dense depth map is composed of the predicted depth value of each pixel, and the depth uncertainty map is composed of the uncertainty in the form of the log-variance of the depth value of each pixel.
[0033] Step S333: Iterative input update, used to build more informative input for the next iteration. Specifically, the dense depth map and depth uncertainty map predicted in the previous round are concatenated with the original road surface image data and sparse depth map in the channel dimension to obtain the model input features for this round.
[0034] Step S334: Iteration termination judgment, used to control the end of the iterative optimization process. Specifically, it checks whether the current iteration number has reached the preset total iteration number. If so, the loop is terminated and the final dense depth map and depth uncertainty map are output as the model output. Otherwise, iterative optimization continues.
[0035] Step S34: Construct and train the model, specifically by integrating the iterative initialization, the planar prior guided design, and the iterative refinement design to construct an improved codec model, train the model based on the completion training set and the completion test set and verify its performance to obtain the improved codec model as a deep completion model;
[0036] The completion training set and the completion test set are used as inputs to the deep completion model, and after processing, the monitoring training set and the monitoring test set are obtained.
[0037] Further, in step S4, the road surface monitoring model is constructed to build a model required for monitoring the quality of highway road surface based on two-dimensional image data and dense depth map. Specifically, a dual-stream network model is constructed as the road surface monitoring model. The dual-stream network model processes the road surface image data and dense depth map through image stream and depth stream respectively, and outputs multi-task prediction results by combining a multi-scale feature fusion mechanism.
[0038] The construction of the road surface monitoring model specifically includes the following steps:
[0039] Step S41: Dual-stream modulation fusion, used to dynamically adjust the contribution weights of image texture information and depth geometric information in feature fusion according to the discrimination requirements of different types of road conditions. The steps include:
[0040] Step S411: Dual-stream feature extraction, used to independently extract effective feature representations from two modalities of data: road surface image and dense depth map. Specifically, two parallel ResNet18-based encoder networks are used to process the road surface image data and the dense depth map respectively, and each outputs multi-scale encoded features in four residual stages.
[0041] Step S412: Depth saliency modulation, used to learn the importance of depth information at each location in the image. Specifically, it involves concatenating the road surface image coding features and depth coding features at the same scale, and then predicting a modulation weight through a small sub-network.
[0042] Step S413: Feature modulation fusion, used to perform feature fusion based on modulation weights. Specifically, the deep coding features are transformed by channels, multiplied element-wise by the modulation weights, and then added to the road image coding features to obtain the modulation fusion features.
[0043] Step S414: Uncertainty-weighted fusion, used to reduce the negative impact of unreliable areas in deep completion on road quality judgment. Specifically, it involves spatially reweighting the modulation fusion features based on the depth uncertainty map to suppress the feature contribution of high uncertainty areas.
[0044] Step S42: Multi-scale deformation sensing, used to simultaneously capture the morphological features of road surface conditions at different scales, includes the following steps:
[0045] Step S421: Deformable convolutional pyramid, used to enhance the adaptability to spatial deformation, specifically by fusing multi-scale uncertainty-weighted features from top to bottom based on deformable convolution;
[0046] Step S422: Multi-scale fusion, used to fuse multi-scale information, specifically by using a 1×1 convolutional layer to process the stitched multi-scale pyramid features, and then upsampling to restore the resolution to be consistent with the model input, to obtain the road surface monitoring feature map;
[0047] Step S43: Multi-task output, used to generate the final pavement quality detection prediction result. Specifically, three model output layers are designed to output pixel-level pavement disease segmentation map, image-level disease classification label and disease severity regression value in parallel based on pavement monitoring feature map.
[0048] Step S44: Construct and train the model. Specifically, the dual-stream network model is constructed by integrating the dual-stream modulation fusion, the multi-scale deformation sensing, and the multi-task output. The model is trained and its performance is verified based on the monitoring training set and the monitoring test set to obtain the dual-stream network model as the road surface monitoring model.
[0049] Further, in step S5, the road surface quality monitoring specifically involves using the current dataset to be processed as input to the depth completion model to complete the depth information, and inputting the output dense depth map and depth uncertainty map into the road surface monitoring model in combination with the current dataset to be processed. The resulting pixel-level road surface defect segmentation map, image-level defect classification label, and defect severity regression value are used as reference results for road surface quality monitoring.
[0050] The highway pavement quality monitoring system based on deep learning provided by this invention includes an original image acquisition module, a data optimization and processing module, a deep completion model construction module, a pavement monitoring model construction module, and a highway pavement quality monitoring module.
[0051] The original image acquisition module is used to acquire raw data, obtain a road surface monitoring original image dataset by acquiring the raw data, and send the road surface monitoring original image dataset to the data optimization processing module;
[0052] The data optimization processing module is used for data optimization processing. Through data optimization processing, it obtains the current dataset to be processed, the completed training set, and the completed test set, and sends the current dataset to be processed to the highway pavement quality monitoring module, and sends the completed training set and the completed test set to the deep completion model construction module.
[0053] The deep completion model construction module is used to construct a deep completion model. By constructing an improved codec model, a deep completion model is obtained. The deep completion model is used to process and complete the training set and the test set respectively to obtain the monitoring training set and the monitoring test set. The deep completion model is then sent to the highway pavement quality monitoring module, and the monitoring training set and the monitoring test set are sent to the pavement monitoring model construction module.
[0054] The road surface monitoring model construction module is used to construct a road surface monitoring model. By constructing a dual-flow network model, a road surface monitoring model is obtained, and the road surface monitoring model is sent to the highway road surface quality monitoring module.
[0055] The highway pavement quality monitoring module is used for highway pavement quality monitoring. It processes the current data by using the deep completion model in combination with the pavement monitoring model and outputs pavement quality monitoring reference results.
[0056] The beneficial effects achieved by the present invention using the above solution are as follows:
[0057] (1) In view of the technical problems of traditional highway pavement quality monitoring methods, which usually rely on a single visual image or require the use of high-cost three-dimensional laser scanning equipment, the former lacks accurate perception of pavement elevation changes, while the latter is expensive, has low acquisition efficiency and complex data processing, making it difficult to deploy routinely, this solution creatively adopts a method of generating dense depth maps based on two-dimensional image data and sparse depth maps for subsequent processing. Under the premise of ensuring economy, high-density pavement geometric information is obtained, thereby providing key three-dimensional morphological support for the fine identification of pavement conditions.
[0058] (2) In view of the technical problems of traditional highway pavement quality monitoring methods, which often use only two-dimensional images as a single input or lack differentiated fusion mechanism when simply stacking multi-source data, making it difficult for the model to accurately distinguish between appearance anomalies and geometric defects in complex pavement scenarios, and easily leading to missed detections and false detections, this solution creatively adopts a dual-stream network model as the pavement monitoring model. By processing image texture and geometric depth information separately and achieving dynamic fusion, and introducing uncertainty information in the depth completion stage, the comprehensiveness and accuracy of monitoring results are improved. Attached Figure Description
[0059] Figure 1A flowchart illustrating the deep learning-based highway pavement quality monitoring method provided by this invention;
[0060] Figure 2 A schematic diagram of the modules of the deep learning-based highway pavement quality monitoring system provided by the present invention;
[0061] Figure 3 A flowchart illustrating the data optimization process in step S2;
[0062] Figure 4 A flowchart illustrating the process of building the deep completion model in step S3;
[0063] Figure 5 This is a flowchart illustrating the process of building the road surface monitoring model in step S4.
[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation
[0065] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0066] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0067] Example 1, see Figure 1 The present invention provides a deep learning-based method for monitoring highway pavement quality, which includes the following steps:
[0068] Step S1: Acquisition of raw images;
[0069] Step S2: Data optimization processing;
[0070] Step S3: Depth completion model construction;
[0071] Step S4: Road surface monitoring model construction;
[0072] Step S5: Highway pavement quality monitoring.
[0073] Example 2, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In step S1, the original image acquisition is used to acquire the original image data required for highway pavement quality monitoring. Specifically, it is to obtain a pavement monitoring original image dataset through image acquisition. The pavement monitoring original image dataset specifically includes a historical pavement original image dataset and a current pavement original image dataset. Both the historical pavement original image dataset and the current pavement original image dataset contain pavement image data and sparse depth point cloud data. The historical pavement original image dataset also contains historical dense depth data and historical pavement defect annotation data.
[0074] The road surface image data specifically refers to a sequence of multiple consecutive two-dimensional road surface images acquired using an RGB industrial camera. The sparse depth point cloud data specifically refers to a sequence of multiple consecutive sparse point cloud data of road surface depth acquired using a low-cost single-line lidar. The historical dense depth data specifically refers to the true value data of road surface dense depth obtained using a ground-based three-dimensional laser scanner on typical road sections.
[0075] Example 3, see Figure 1 , Figure 2 and Figure 3 This embodiment is based on the above embodiment. In step S2, the data optimization processing is used to optimize the acquired raw image data, specifically including the following steps:
[0076] Step S21: Data spatiotemporal alignment, used to unify data from different sensors into the image coordinate system for processing. Specifically, it maps sparse three-dimensional laser point clouds onto the corresponding pixel positions on the two-dimensional image plane to obtain a sparse depth map aligned with the RGB image pixels.
[0077] Step S22: Region of Interest Extraction, used to focus computing resources on key road surface areas and eliminate interference. Specifically, it calculates the effective area in the image containing only the road surface based on the geometric parameters of the acquisition device and generates a mask to obtain the road surface area image.
[0078] Step S23: Sequence frame organization, which utilizes the temporal continuity information of the video sequence to enhance the stability of single-frame perception. Specifically, it calculates pixel-level motion vectors between consecutive frames through an optical flow network to obtain a dense optical flow field between adjacent frames.
[0079] Step S24: Data augmentation, used to improve the robustness of the model to different acquisition conditions and noise, specifically by applying synchronous color perturbation, spatial transformation and simulating different proportions of missing laser point clouds to the image and depth data to obtain augmented and standardized data;
[0080] Step S25: Dataset partitioning, used to obtain training data and test data, specifically by partitioning the original historical road surface image dataset into a completion training set and a completion test set;
[0081] The current road surface original image dataset is optimized through the data spatiotemporal alignment, the region of interest extraction, and the sequence frame organization to obtain the current dataset to be processed. The historical road surface original image dataset is optimized through the data spatiotemporal alignment, the region of interest extraction, the sequence frame organization, the data augmentation, and the dataset partitioning to obtain the completed training set and the completed test set.
[0082] Example 4, see Figure 1 , Figure 2 and Figure 4 This embodiment is based on the above embodiment. In step S3, the depth completion model construction is used to construct a model that generates a dense depth map based on two-dimensional image data and sparse depth map for subsequent processing. Specifically, it constructs an improved codec model as a depth completion model. The improved codec model outputs a dense depth map and a depth uncertainty map by combining planar prior guidance and iterative refinement.
[0083] The construction of the depth completion model specifically includes the following steps:
[0084] Step S31: Iterative initialization, used to construct the initial state for the iterative optimization process, includes the following steps:
[0085] Step S311: Input data concatenation, used to integrate data from different modalities into a unified tensor format, specifically concatenating road surface image data containing texture information and sparse depth map containing geometric information in the channel dimension to obtain a fused feature map as input data;
[0086] Step S312: Initialize iterative parameters, which is used to set the start state and termination conditions of the iterative optimization process. Specifically, the predicted dense depth map and depth uncertainty map are initialized to zero matrices, and the total number of iterations is set.
[0087] Step S313: Mask parameter setting, used to distinguish between valid data areas and invalid background areas when calculating loss. Specifically, it defines a binary valid point mask and a binary road surface mask. For the binary valid point mask, the corresponding element is 1 when the pixel has a LiDAR measurement value, and 0 otherwise. For the binary road surface mask, the corresponding element is 1 when the pixel belongs to the road surface area image, and 0 otherwise.
[0088] Step S32: Planar prior-guided design, used to embed the physical prior knowledge that the road surface is macroscopically approximately planar into the network, suppressing unreasonable undulations in flat areas. The steps include:
[0089] Step S321: Encoder feature extraction, used to progressively extract multi-level, abstract feature representations from the fused input data, specifically by processing the fused feature map of the input through a downsampling convolutional neural network to obtain an enhanced feature map;
[0090] Step S322: Planar parameter prediction, used to decouple the planar model parameters describing the macroscopic geometry of the road surface from the global features. Specifically, after global pooling of the enhanced feature map, a set of parameters is regressed through a fully connected layer. Each set of parameters defines the normal vector and distance of a local plane. The formula used is as follows:
[0091] ;
[0092] In the formula, Pl represents the predicted planar parameter field, and each row is defined as a planar anchor point feature, containing the normal vector and distance of a local plane. This indicates that the planar parameters predict the weights of the second fully connected layer. This indicates that the planar parameters predict the weights of the first fully connected layer. This indicates that the plane parameters predict the bias term of the second fully connected layer. This represents the bias term for the first fully connected layer in planar parameter prediction. Represents the ReLU activation function. This represents the global average pooling function. Represents enhanced feature maps;
[0093] Step S323: Planar feature generation, used to adapt global planar parameters to local geometry at different locations in the image. Specifically, through a deformable convolutional layer, based on an adaptive sampling offset according to image content, the predicted planar parameters are transformed into a planar prior feature map aligned with the feature map space. The formula used is as follows:
[0094] ;
[0095] In the formula, Let Ad represent the predicted deformable convolution offset field, and Let Ad represent the predicted deformable convolution modulation scalar field. This represents the convolutional layer function used to predict deformable convolution parameters. Represents the planar prior feature map at location The value, Let A represent the positional variable of the planar prior feature map, and let B represent the number of planar anchor points and B represent the number of sampling points of the convolution kernel. This represents the deformable convolution weights at the b-th sampling point. Indicates the position The modulation scalar of the a-th planar anchor point This indicates the regular convolution sampling position of the b-th sampling point. Indicates the position The offset of the a-th plane anchor point This represents the feature of the a-th planar anchor point. This represents the fully connected layer function used to convert planar anchor point features into feature modulation factors;
[0096] Step S324: Feature enhancement fusion, which combines the extracted general features with the road surface-specific planar prior features. Specifically, the enhanced feature map output by the encoder is added to the planar prior feature map with learnable weights to obtain the enhanced fused feature map.
[0097] Step S33: Iterative refinement design, used to achieve multi-round, coarse-to-fine optimization, focusing on correcting high uncertainty and mission-critical areas. Steps include:
[0098] Step S331: Spatiotemporal feature fusion, which is used to stabilize the accuracy of single-frame depth prediction by utilizing the strong correlation between consecutive frames. Specifically, during the training phase, the features of adjacent frames are aligned to the current frame coordinate system through optical flow and then fused by weight. The resulting features replace the enhanced fused feature map and are used as the input features of the decoder based on the U-Net structure.
[0099] Step S332: Decoding and output prediction, used to recover a dense depth map and a depth uncertainty map with the same resolution as the input from high-level semantic features. Specifically, the enhanced fusion feature map is used as the input feature of the decoder based on the U-Net structure, and the dense depth map and the depth uncertainty map are output through two independent convolution heads respectively. The dense depth map is composed of the predicted depth value of each pixel, and the depth uncertainty map is composed of the uncertainty in the form of the log-variance of the depth value of each pixel.
[0100] Step S333: Iterative input update, used to build more informative input for the next iteration. Specifically, the dense depth map and depth uncertainty map predicted in the previous round are concatenated with the original road surface image data and sparse depth map in the channel dimension to obtain the model input features for this round.
[0101] Step S334: Iteration termination judgment, used to control the end of the iterative optimization process. Specifically, it checks whether the current iteration number has reached the preset total iteration number. If so, the loop is terminated and the final dense depth map and depth uncertainty map are output as the model output. Otherwise, iterative optimization continues.
[0102] Step S34: Construct and train the model, specifically by integrating the iterative initialization, the planar prior guided design, and the iterative refinement design to construct an improved codec model, train the model based on the completion training set and the completion test set and verify its performance to obtain the improved codec model as a deep completion model;
[0103] The completion training set and the completion test set are used as inputs to the deep completion model, and after processing, the monitoring training set and the monitoring test set are obtained.
[0104] By performing the above operations, this solution addresses the technical problems of traditional highway pavement quality monitoring methods, which typically rely on single visual images or require expensive 3D laser scanning equipment. The former lacks accurate perception of pavement elevation changes, while the latter is expensive, has low acquisition efficiency, and complex data processing, making it difficult to deploy routinely. This solution creatively adopts a method that generates dense depth maps based on 2D image data and sparse depth maps for subsequent processing. This method obtains high-density pavement geometric information while ensuring economy, thus providing crucial 3D morphological support for the precise identification of pavement conditions.
[0105] Example 5, see Figure 1 , Figure 2 and Figure 5 This embodiment is based on the above embodiment. In step S4, the road surface monitoring model is constructed to build a model required for monitoring the quality of highway road surface based on two-dimensional image data and dense depth map. Specifically, a dual-stream network model is constructed as the road surface monitoring model. The dual-stream network model processes the road surface image data and dense depth map through image stream and depth stream respectively, and outputs multi-task prediction results by combining a multi-scale feature fusion mechanism.
[0106] The construction of the road surface monitoring model specifically includes the following steps:
[0107] Step S41: Dual-stream modulation fusion, used to dynamically adjust the contribution weights of image texture information and depth geometric information in feature fusion according to the discrimination requirements of different types of road conditions. The steps include:
[0108] Step S411: Dual-stream feature extraction, used to independently extract effective feature representations from two modalities of data: road surface image and dense depth map. Specifically, two parallel ResNet18-based encoder networks are used to process the road surface image data and the dense depth map respectively, and each outputs multi-scale encoded features in four residual stages.
[0109] Step S412: Depth saliency modulation, used to learn the importance of depth information at each location in the image. Specifically, it involves concatenating the road surface image coding features and depth coding features at the same scale, and then predicting a modulation weight through a small sub-network. The formula used is as follows:
[0110] ;
[0111] In the formula, This represents the modulation weight at the l-th scale. This represents the sigmoid activation function. This represents the weights of the second fully connected layer in the small subnetwork used to obtain the modulation weights. This represents the weights of the first fully connected layer of the small subnetwork used to obtain the modulation weights. This represents the encoded features of the road surface image at the l-th scale. This represents the depth-coded feature at the l-th scale. This represents the bias term used to obtain the modulation weights of the first fully connected layer of the small subnetwork. This represents the bias term used to obtain the modulation weights of the second fully connected layer of the small subnetwork;
[0112] Step S413: Feature modulation fusion, used for feature fusion based on modulation weights. Specifically, the deep encoded features are transformed by channels, multiplied element-wise by the modulation weights, and then added to the road image encoded features to obtain the modulation fused features. The formula used is as follows:
[0113] ;
[0114] In the formula, This represents the modulation fusion feature at the l-th scale. Indicates the channel transformation weights;
[0115] Step S414: Uncertainty-weighted fusion, used to reduce the negative impact of unreliable areas in depth completion on road quality judgment. Specifically, it involves spatially reweighting the modulation fusion features based on the depth uncertainty map to suppress the feature contribution of high uncertainty areas. The formula used is as follows:
[0116] ;
[0117] In the formula, This represents the uncertainty weight at the l-th scale. This represents the uncertainty weight scaling factor. This represents the depth uncertainty map downsampled to the same resolution as the encoded features at the l-th scale. This represents the uncertainty-weighted fusion feature at the l-th scale;
[0118] Step S42: Multi-scale deformation sensing, used to simultaneously capture the morphological features of road surface conditions at different scales, includes the following steps:
[0119] Step S421: Deformable convolutional pyramid, used to enhance adaptability to spatial deformation, specifically, it is based on deformable convolution to fuse multi-scale uncertainty-weighted features in a top-down manner, using the following formula:
[0120] ;
[0121] In the formula, This represents the pyramid features at the fifth scale. This represents the pyramid features at the fourth scale. This represents the pyramid features at the third scale. This represents the pyramid features at the second scale. This represents the uncertainty-weighted fusion feature at the fifth scale. This represents the uncertainty-weighted fusion feature at the fourth scale. This represents the uncertainty-weighted fusion feature at the third scale. This represents the uncertainty-weighted fusion feature of the second scale. This represents a 1×1 convolution function. Represents a deformable convolution function. Indicates the upsampling function;
[0122] Step S422: Multi-scale fusion, used to fuse multi-scale information, specifically by using a 1×1 convolutional layer to process the stitched multi-scale pyramid features, and then upsampling to restore the resolution to be consistent with the model input, to obtain the road surface monitoring feature map;
[0123] Step S43: Multi-task output, used to generate the final pavement quality detection prediction result. Specifically, three model output layers are designed to output pixel-level pavement disease segmentation map, image-level disease classification label and disease severity regression value in parallel based on pavement monitoring feature map.
[0124] Step S44: Construct and train the model. Specifically, the dual-stream network model is constructed by integrating the dual-stream modulation fusion, the multi-scale deformation sensing, and the multi-task output. The model is trained and its performance is verified based on the monitoring training set and the monitoring test set to obtain the dual-stream network model as the road surface monitoring model.
[0125] By performing the above operations, this solution addresses the technical problems of traditional highway pavement quality monitoring methods, which often rely solely on two-dimensional images as a single input or lack differentiated fusion mechanisms when simply stacking multi-source data. This makes it difficult for the model to accurately distinguish between apparent anomalies and geometric defects in complex pavement scenarios, leading to missed detections and false detections. This solution creatively adopts a dual-stream network model as the pavement monitoring model. By processing image texture and geometric depth information separately and achieving dynamic fusion, and by introducing uncertainty information from the depth completion stage, the comprehensiveness and accuracy of the monitoring results are improved.
[0126] Example 6, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In step S5, the road surface quality monitoring specifically involves using the current dataset to be processed as the input to the depth completion model to complete the depth information. The output dense depth map and depth uncertainty map are combined with the current dataset to be processed and input into the road surface monitoring model. The final pixel-level road surface defect segmentation map, image-level defect classification label, and defect severity regression value are used as reference results for road surface quality monitoring.
[0127] Example 7, see Figure 1 and Figure 2 Based on the above embodiments, the deep learning-based highway pavement quality monitoring system provided by the present invention includes an original image acquisition module, a data optimization and processing module, a deep completion model construction module, a pavement monitoring model construction module, and a highway pavement quality monitoring module.
[0128] The original image acquisition module is used to acquire raw data, obtain a road surface monitoring original image dataset by acquiring the raw data, and send the road surface monitoring original image dataset to the data optimization processing module;
[0129] The data optimization processing module is used for data optimization processing. Through data optimization processing, it obtains the current dataset to be processed, the completed training set, and the completed test set, and sends the current dataset to be processed to the highway pavement quality monitoring module, and sends the completed training set and the completed test set to the deep completion model construction module.
[0130] The deep completion model construction module is used to construct a deep completion model. By constructing an improved codec model, a deep completion model is obtained. The deep completion model is used to process and complete the training set and the test set respectively to obtain the monitoring training set and the monitoring test set. The deep completion model is then sent to the highway pavement quality monitoring module, and the monitoring training set and the monitoring test set are sent to the pavement monitoring model construction module.
[0131] The road surface monitoring model construction module is used to construct a road surface monitoring model. By constructing a dual-flow network model, a road surface monitoring model is obtained, and the road surface monitoring model is sent to the highway road surface quality monitoring module.
[0132] The highway pavement quality monitoring module is used for highway pavement quality monitoring. It processes the current data by using the deep completion model in combination with the pavement monitoring model and outputs pavement quality monitoring reference results.
[0133] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0134] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.
[0135] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.
Claims
1. A deep learning-based method for monitoring highway pavement quality, characterized by: The method includes the following steps: S1: Raw image acquisition. Through image acquisition, a raw image dataset for road surface monitoring is obtained. Specifically, the raw image dataset for road surface monitoring includes a historical raw image dataset and a current raw image dataset for road surface. S2: Data optimization processing, which optimizes the collected raw data to obtain the current dataset to be processed, the training set to be completed, and the test set to be completed; S3: Depth Completion Model Construction, used to construct a model that generates a dense depth map based on two-dimensional image data and sparse depth map for subsequent processing. Specifically, it constructs an improved codec model as the depth completion model. The improved codec model outputs a dense depth map and a depth uncertainty map by combining planar prior guidance and iterative refinement. The completion training set and the completion test set are used as inputs to the depth completion model. After processing, a monitoring training set and a monitoring test set are obtained. S4: Road surface monitoring model construction, used to construct the model required for highway road surface quality monitoring based on two-dimensional image data and dense depth map. Specifically, a dual-stream network model is constructed as the road surface monitoring model. The dual-stream network model processes the road surface image data and dense depth map through image stream and depth stream respectively, and outputs multi-task prediction results by combining a multi-scale feature fusion mechanism. S5: Highway pavement quality monitoring, specifically, the current dataset to be processed is used as the input of the depth completion model to complete the depth information, and the output dense depth map and depth uncertainty map are combined with the current dataset to be processed and input into the pavement monitoring model. The final pixel-level pavement disease segmentation map, image-level disease classification label and disease severity regression value are used as the reference results for pavement quality monitoring.
2. The deep learning-based highway pavement quality monitoring method according to claim 1, characterized in that: The construction of the depth completion model specifically includes the following steps: Step S31: Iterative initialization, used to construct the initial state for the iterative optimization process, includes the following steps: Step S311: Input data concatenation, used to integrate data from different modalities into a unified tensor format, specifically concatenating road surface image data containing texture information and sparse depth map containing geometric information in the channel dimension to obtain a fused feature map as input data; Step S312: Initialize iterative parameters, which is used to set the start state and termination conditions of the iterative optimization process. Specifically, the predicted dense depth map and depth uncertainty map are initialized to zero matrices, and the total number of iterations is set. Step S313: Mask parameter setting, used to distinguish between valid data areas and invalid background areas when calculating loss. Specifically, it defines a binary valid point mask and a binary road surface mask. For the binary valid point mask, the corresponding element is 1 when the pixel has a LiDAR measurement value, and 0 otherwise. For the binary road surface mask, the corresponding element is 1 when the pixel belongs to the road surface area image, and 0 otherwise. Step S32: Planar prior-guided design, used to embed the physical prior knowledge that the road surface is macroscopically approximately planar into the network, suppressing unreasonable undulations in flat areas. The steps include: Step S321: Encoder feature extraction, used to progressively extract multi-level, abstract feature representations from the fused input data, specifically by processing the fused feature map of the input through a downsampling convolutional neural network to obtain an enhanced feature map; Step S322: Planar parameter prediction, used to decouple the planar model parameters describing the macroscopic geometry of the road surface from the global features. Specifically, after global pooling of the enhanced feature map, a set of parameters is regressed through a fully connected layer, where each set of parameters defines the normal vector and distance of a local plane. Step S323: Planar feature generation, which adapts the global planar parameters to the local geometry at different locations in the image. Specifically, it transforms the predicted planar parameters into a planar prior feature map aligned with the feature map space through a deformable convolutional layer based on the image content adaptive sampling offset. Step S324: Feature enhancement fusion, which combines the extracted general features with the road surface-specific planar prior features. Specifically, the enhanced feature map output by the encoder is added to the planar prior feature map with learnable weights to obtain the enhanced fused feature map. Step S33: Iterative refinement design, used to achieve multi-round, coarse-to-fine optimization, focusing on correcting high uncertainty and mission-critical areas. Steps include: Step S331: Spatiotemporal feature fusion, which is used to stabilize the accuracy of single-frame depth prediction by utilizing the strong correlation between consecutive frames. Specifically, during the training phase, the features of adjacent frames are aligned to the current frame coordinate system through optical flow and then fused by weight. The resulting features replace the enhanced fused feature map and are used as the input features of the decoder based on the U-Net structure. Step S332: Decoding and output prediction, used to recover a dense depth map and a depth uncertainty map with the same resolution as the input from high-level semantic features. Specifically, the enhanced fusion feature map is used as the input feature of the decoder based on the U-Net structure, and the dense depth map and the depth uncertainty map are output through two independent convolution heads respectively. The dense depth map is composed of the predicted depth value of each pixel, and the depth uncertainty map is composed of the uncertainty in the form of the log-variance of the depth value of each pixel. Step S333: Iterative input update, used to build more informative input for the next iteration. Specifically, the dense depth map and depth uncertainty map predicted in the previous round are concatenated with the original road surface image data and sparse depth map in the channel dimension to obtain the model input features for this round. Step S334: Iteration termination judgment, used to control the end of the iterative optimization process. Specifically, it checks whether the current iteration number has reached the preset total iteration number. If so, the loop is terminated and the final dense depth map and depth uncertainty map are output as the model output. Otherwise, iterative optimization continues. Step S34: Construct and train the model, specifically by integrating the iterative initialization, the planar prior guided design, and the iterative refinement design to construct an improved codec model, train the model based on the completion training set and the completion test set and verify its performance to obtain the improved codec model as a deep completion model; The completion training set and the completion test set are used as inputs to the deep completion model, and after processing, the monitoring training set and the monitoring test set are obtained.
3. The deep learning-based highway pavement quality monitoring method according to claim 1, characterized in that: The construction of the road surface monitoring model specifically includes the following steps: Step S41: Dual-stream modulation fusion, used to dynamically adjust the contribution weights of image texture information and depth geometric information in feature fusion according to the discrimination requirements of different types of road conditions. The steps include: Step S411: Dual-stream feature extraction, used to independently extract effective feature representations from two modalities of data: road surface image and dense depth map. Specifically, two parallel ResNet18-based encoder networks are used to process the road surface image data and the dense depth map respectively, and each outputs multi-scale encoded features in four residual stages. Step S412: Depth saliency modulation, used to learn the importance of depth information at each location in the image. Specifically, it involves concatenating the road surface image coding features and depth coding features at the same scale, and then predicting a modulation weight through a small sub-network. Step S413: Feature modulation fusion, used to perform feature fusion based on modulation weights. Specifically, the deep coding features are transformed by channels, multiplied element-wise by the modulation weights, and then added to the road image coding features to obtain the modulation fusion features. Step S414: Uncertainty-weighted fusion, used to reduce the negative impact of unreliable areas in deep completion on road quality judgment. Specifically, it involves spatially reweighting the modulation fusion features based on the depth uncertainty map to suppress the feature contribution of high uncertainty areas. Step S42: Multi-scale deformation sensing, used to simultaneously capture the morphological features of road surface conditions at different scales, includes the following steps: Step S421: Deformable convolutional pyramid, used to enhance the adaptability to spatial deformation, specifically by fusing multi-scale uncertainty-weighted features from top to bottom based on deformable convolution; Step S422: Multi-scale fusion, used to fuse multi-scale information, specifically by using a 1×1 convolutional layer to process the stitched multi-scale pyramid features, and then upsampling to restore the resolution to be consistent with the model input, to obtain the road surface monitoring feature map; Step S43: Multi-task output, used to generate the final pavement quality detection prediction result. Specifically, three model output layers are designed to output pixel-level pavement disease segmentation map, image-level disease classification label and disease severity regression value in parallel based on pavement monitoring feature map. Step S44: Construct and train the model. Specifically, the dual-stream network model is constructed by integrating the dual-stream modulation fusion, the multi-scale deformation sensing, and the multi-task output. The model is trained and its performance is verified based on the monitoring training set and the monitoring test set to obtain the dual-stream network model as the road surface monitoring model.
4. The deep learning-based highway pavement quality monitoring method according to claim 1, characterized in that: Both the historical road surface original image dataset and the current road surface original image dataset contain road surface image data and sparse depth point cloud data. The historical road surface original image dataset also contains historical dense depth data and historical road surface defect annotation data.
5. The deep learning-based highway pavement quality monitoring method according to claim 1, characterized in that: The data optimization process specifically includes the following steps: Step S21: Data spatiotemporal alignment, used to unify data from different sensors into the image coordinate system for processing. Specifically, it maps sparse three-dimensional laser point clouds onto the corresponding pixel positions on the two-dimensional image plane to obtain a sparse depth map aligned with the RGB image pixels. Step S22: Region of Interest Extraction, used to focus computing resources on key road surface areas and eliminate interference. Specifically, it calculates the effective area in the image containing only the road surface based on the geometric parameters of the acquisition device and generates a mask to obtain the road surface area image. Step S23: Sequence frame organization, which utilizes the temporal continuity information of the video sequence to enhance the stability of single-frame perception. Specifically, it calculates pixel-level motion vectors between consecutive frames through an optical flow network to obtain a dense optical flow field between adjacent frames. Step S24: Data augmentation, used to improve the robustness of the model to different acquisition conditions and noise, specifically by applying synchronous color perturbation, spatial transformation and simulating different proportions of missing laser point clouds to the image and depth data to obtain augmented and standardized data; Step S25: Dataset partitioning, used to obtain training data and test data, specifically by partitioning the original historical road surface image dataset into a completion training set and a completion test set; The current road surface original image dataset is optimized through the data spatiotemporal alignment, the region of interest extraction, and the sequence frame organization to obtain the current dataset to be processed. The historical road surface original image dataset is optimized through the data spatiotemporal alignment, the region of interest extraction, the sequence frame organization, the data augmentation, and the dataset partitioning to obtain the completed training set and the completed test set.
6. A deep learning-based highway pavement quality monitoring system, used to implement the deep learning-based highway pavement quality monitoring method as described in any one of claims 1-5, characterized in that: It includes a raw image acquisition module, a data optimization and processing module, a deep completion model construction module, a road surface monitoring model construction module, and a highway road surface quality monitoring module.
7. The deep learning-based highway pavement quality monitoring system according to claim 6, characterized in that: The original image acquisition module is used to acquire raw data, obtain a road surface monitoring original image dataset by acquiring the raw data, and send the road surface monitoring original image dataset to the data optimization processing module; The data optimization processing module is used for data optimization processing. Through data optimization processing, it obtains the current dataset to be processed, the completed training set, and the completed test set, and sends the current dataset to be processed to the highway pavement quality monitoring module, and sends the completed training set and the completed test set to the deep completion model construction module. The deep completion model construction module is used to construct a deep completion model. By constructing an improved codec model, a deep completion model is obtained. The deep completion model is used to process and complete the training set and the test set respectively to obtain the monitoring training set and the monitoring test set. The deep completion model is then sent to the highway pavement quality monitoring module, and the monitoring training set and the monitoring test set are sent to the pavement monitoring model construction module. The road surface monitoring model construction module is used to construct a road surface monitoring model. By constructing a dual-flow network model, a road surface monitoring model is obtained, and the road surface monitoring model is sent to the highway road surface quality monitoring module. The highway pavement quality monitoring module is used for highway pavement quality monitoring. It processes the current data by using the deep completion model in combination with the pavement monitoring model and outputs pavement quality monitoring reference results.