A road maintenance decision-making method and system combining visual measurement and disease identification

Abstract

The application relates to a road maintenance decision-making method and system combining visual measurement and disease identification, and belongs to the technical field of road engineering detection. The method comprises the following steps: establishing a perspective transformation matrix, extracting a first profile and surface texture features of diseases in road surface depth data, extracting a second profile based on an orthographic image, projecting the second profile to a road surface depth data view plane, performing geometric comparison with the first profile, detecting abnormal mutation points, determining deviation categories, generating a low confidence label, taking the second profile as a boundary constraint, constructing a Poisson equation based on intact road surface point cloud depth gradient field, performing implicit curved surface reconstruction on disease area distortion point cloud, obtaining three-dimensional measurement parameters, determining disease grades, constructing urgency scores, determining maintenance priorities, and outputting a maintenance decision-making report. The application realizes cross verification and fusion repair of vehicle-mounted depth data and unmanned aerial vehicle orthographic images, and can be applied to the measurement and maintenance decision-making field of road diseases.

Description

Technical Field

[0001] This invention belongs to the field of road engineering inspection technology, specifically relating to a road maintenance decision-making method and system that integrates visual measurement and defect identification. Background Technology

[0002] Road surface distress detection and maintenance decision-making are core aspects of routine highway maintenance management. The task is to accurately identify, three-dimensionally measure, and assess the severity of road surface defects such as cracks, potholes, and subsidence, and then formulate targeted maintenance measures accordingly. Currently, road surface distress detection methods based on vehicle-mounted mobile measurement systems are widely used. Common technical solutions include: using vehicle-mounted cameras to acquire road surface images and detecting distressed areas using image recognition technology; using 3D laser scanning or binocular stereo vision to acquire 3D point clouds of the road surface and analyze the geometric properties of the defects; and combining GPS or inertial navigation systems to spatially locate the defects. In recent years, some solutions have attempted to combine UAV aerial imagery with vehicle-mounted data, using a top-down perspective to supplement blind spots from the ground perspective.

[0003] However, existing methods for pavement distress detection and maintenance decision-making still have the following shortcomings: Ground-based vehicle-mounted vision systems are susceptible to perspective distortion in depth data acquired by binocular stereo cameras due to vehicle posture and road slope variations. Furthermore, dynamic occlusion by other vehicles or roadside trees during vehicle movement can lead to localized loss or distortion of depth data, severely impacting the accuracy of 3D measurement of the damage. Existing methods lack proactive verification mechanisms for depth data quality and cannot identify whether distortion or occlusion exists in the depth data within the damage area.

[0004] In existing methods, vehicle-mounted ground data and UAV overhead data are mostly collected and processed independently, lacking effective cross-validation and fusion. The accurate contour information of orthophotos cannot be used to verify and correct the measurement deviation of vehicle-mounted depth data, resulting in the failure to fully leverage the advantages of the two data sources.

[0005] For diseased areas that have been obscured or distorted, existing methods usually discard the data of that area directly or fill it with simple interpolation, lacking precise repair methods based on geometric constraints, resulting in inaccurate three-dimensional measurement parameters of the disease.

[0006] Most existing maintenance decision-making methods classify maintenance based solely on the area of ​​the disease or a single depth index, without combining the geometric severity of the disease with the density of texture damage. Furthermore, they lack a quantitative ranking mechanism for maintenance priorities that takes into account the impact of traffic flow, resulting in a low degree of matching between the decision-making results and actual maintenance needs. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this invention provides a road maintenance decision-making method that integrates visual measurement and defect identification. The objective of this invention can be achieved through the following technical solutions: A road maintenance decision-making method integrating visual measurement and defect identification includes the following steps: The system acquires left and right eye images and synchronous pose data from the vehicle-mounted binocular stereo camera, generates road surface depth data, and simultaneously acquires top-view orthophotos from the UAV. Based on the pose data, the system performs spatiotemporal alignment between the two and establishes a perspective transformation matrix from the top-view orthophoto to the road surface depth data view plane. The first contour and surface texture features of the defects are extracted based on pavement depth data. The second contour is extracted based on the top-view orthophoto. The second contour is back-projected onto the pavement depth data view plane using a perspective transformation matrix to obtain the theoretical projection contour. The Hausdorff distance and the angle distribution of the edge normal vectors between the theoretical projection contour and the first contour are calculated. First-order difference and sliding variance statistics are performed along the contour arc length to detect abnormal abrupt changes exceeding the threshold. If the abrupt changes are uniformly distributed along the entire contour, they are judged as systematic perspective deviations. If they are concentrated in local arc segments, they are judged as local distortions caused by dynamic occlusion. Based on this, low-confidence labels are generated. In response to low confidence labels, using the second contour as the boundary constraint, a Poisson equation is constructed using the depth gradient field of the surrounding intact road surface point cloud. Implicit surface reconstruction is performed on the distorted point cloud of the diseased area to obtain corrected three-dimensional measurement parameters including the maximum depth, average depth and volume of the disease. By integrating and correcting the three-dimensional measurement parameters and surface texture features, the level of damage is determined, and an urgency score is constructed based on the level of damage, the volume of damage, and the traffic flow of the road section. The maintenance priority is determined by the maximum score, and a maintenance decision report is output.

[0008] Specifically, the process of establishing the perspective transformation matrix includes: Based on the pose data, the acquisition time of the top-view orthophoto data and the acquisition time of the road surface depth data are aligned on the time axis; Based on the vehicle position and attitude angle in the pose data, the spatial plane equation corresponding to the road surface depth data view plane is solved, and the coordinates of the top-view orthophoto image are mapped to the road surface depth data view plane to establish a perspective transformation matrix.

[0009] Specifically, the extraction process of the first contour includes: The road surface depth data is input into the semantic segmentation network, and the probability value of belonging to the disease category is output pixel by pixel. Pixels with probability values ​​exceeding a preset threshold are marked as disease pixels, and the region formed by disease pixels is extracted as the disease pixel region. Connectivity analysis is performed on the diseased pixel regions, and interconnected pixel regions are marked as the same disease target. The set of boundary pixels of the disease target is extracted as the first contour.

[0010] Specifically, the extraction process of the surface texture features includes: Image patches of the diseased area are extracted from the road surface depth data. After grayscale processing of the image patches, a grayscale co-occurrence matrix is ​​constructed, and energy, contrast and correlation texture features are extracted. Local binary mode encoding is performed on the image patch, and the frequency of occurrence of each encoding mode is counted to form a texture histogram feature; the texture histogram feature is concatenated with the gray-level co-occurrence matrix feature to form the surface texture feature.

[0011] Specifically, the extraction process of the second contour includes: The top-view orthophoto data is preprocessed by grayscale conversion and Gaussian filtering, and edge detection is performed on the preprocessed top-view orthophoto data to obtain a binarized edge map; Morphological closure operations are performed on the binarized edge map to connect the broken edges. The region enclosed by the closed edges is extracted as a disease candidate region, and the set of boundary pixels of the disease candidate region is extracted as the second contour.

[0012] Specifically, the detection process of the abnormal mutation points includes: Calculate the Hausdorff distance and the angle distribution of edge normals between the theoretical projected profile and the first profile, and establish a correspondence between the Hausdorff distance and the arc length position of the theoretical projected profile. The distance change rate sequence is obtained by performing a first-order difference operation on the Hausdorf distance along the arc length direction, and the angle dispersion sequence is obtained by performing sliding window variance statistics on the angle between the edge normal vectors along the arc length direction. Abnormal abrupt changes exceeding their respective preset thresholds are detected in the distance change rate sequence and the angle dispersion sequence, respectively.

[0013] Specifically, the process for determining the deviation category includes: The location distribution of abnormal mutation points on the theoretical projected contour arc length is obtained. If the abnormal mutation points are evenly distributed along the entire contour arc length, they are determined to be systematic deviations caused by perspective projection. If the abnormal mutation points are concentrated in local arc segments of the contour, they are determined to be local distortions caused by dynamic occlusion. The low confidence label contains label category information corresponding to the deviation category.

[0014] Specifically, the reconstruction process of the Poisson equation includes: The inner region of the second contour is taken as the reconstruction target region. The intact road surface point cloud within a preset range around the second contour is extracted from the road surface depth data. The depth value is fitted to the intact road surface point cloud to obtain a continuous depth field. The gradient of the continuous depth field is calculated and the divergence is taken as the right-hand side of the Poisson equation. The depth value at the second contour boundary is used as the Dirichlet boundary condition. The Poisson equation is solved to obtain the depth value of each pixel position in the reconstruction target area. The original distorted depth value is replaced with the depth value to complete the implicit surface reconstruction.

[0015] Specifically, the corrected three-dimensional measurement parameters include: The vertical distance between the depth value of each pixel in the reconstructed target area and the fitting reference plane of the intact road surface point cloud is taken as the disease depth. The maximum value of the disease depth is taken as the maximum disease depth. The arithmetic mean of the disease depth in the reconstructed target area is taken as the average disease depth. The projected area of ​​the spatial distribution range of the disease depth on the road surface is taken as the disease projected area. The product of the disease projected area and the average disease depth is taken as the disease volume.

[0016] Specifically, the process for determining the severity of the disease includes: A geometric feature vector is constructed using three-dimensional measurement parameters, and an apparent feature vector is constructed using surface texture features. The geometric feature vector and the apparent feature vector are normalized according to preset weights and then concatenated in a feature layer to form a multi-dimensional disease feature vector. The multidimensional disease feature vector is input into a pre-trained classifier. The classifier outputs the posterior probability of the disease belonging to each preset level. Each preset level is classified according to the geometric severity of the disease and the density of texture damage. The level with the highest posterior probability is output as the judgment result.

[0017] Specifically, the process of determining the maintenance priority includes: The geometric severity score is obtained by weighting the grade coefficient corresponding to the disease level and the normalized value of the disease volume, and the influence range score is obtained by weighting the geometric severity score and the influence range score. All diseases are ranked from high to low according to the maintenance urgency score, and the ranking order is used as the maintenance priority.

[0018] The beneficial effects of this invention are as follows: This invention achieves proactive verification of the quality of vehicle-mounted road surface depth data by performing spatiotemporal alignment and perspective transformation with UAV top-down orthophoto data. It uses the second contour extracted from the orthophoto as an independent reference benchmark and performs cross-validation with the first contour extracted from the vehicle-mounted depth data based on Hausdorff distance and the angle between edge normal vectors. This enables the identification of whether there is perspective distortion or dynamic occlusion in the defect area, filling the gap in the lack of depth data quality verification mechanism in existing methods.

[0019] This invention detects abrupt changes in distance by performing first-order difference analysis on the Hausdorff distance along the contour arc length, and simultaneously detects abrupt changes in angle by performing sliding variance analysis on the angle between edge normal vectors. The distribution characteristics of abrupt changes are distinguished into systematic deviations that are uniformly distributed along the entire contour and local distortions that are concentrated in local arc segments. This achieves a fine classification of deviation categories and provides a basis for subsequent differentiated repair strategies for different types of distortion.

[0020] This invention targets low-confidence areas, using the second contour provided by UAV orthophotos as boundary constraints. It constructs a Poisson equation using the depth gradient field of the point cloud of the surrounding intact road surface, performs implicit surface reconstruction on the distorted point cloud of the damaged area, and obtains a smooth depth field that satisfies the boundary constraints by solving the Laplace equation. This effectively restores the true three-dimensional shape of the damaged area and improves the accuracy of the depth and volume measurement of the damaged area.

[0021] This invention integrates modified three-dimensional measurement parameters (maximum depth, average depth, projected area, and volume) with the texture features of the diseased area (gray-level co-occurrence matrix features and local binary pattern features) in a multi-dimensional manner. By jointly determining the disease level through a pre-trained classifier, it achieves a comprehensive evaluation of geometric severity and surface damage. At the same time, it constructs a maintenance urgency score based on disease level, disease volume, and road traffic flow, realizing a quantitative ranking of maintenance priorities. This makes maintenance decisions more scientific and reasonable, and highly matched with actual maintenance management needs. Attached Figure Description

[0022] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0023] Figure 1 This is an overall flowchart of a road maintenance decision-making method that integrates visual measurement and defect identification according to the present invention; Figure 2 This is a diagram illustrating the cross-validation and bias classification in this invention. Figure 3 This is a flowchart of the implicit surface reconstruction process based on the Poisson equation in this invention. Detailed Implementation

[0024] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0025] Please see Figures 1-3 , Figure 1The invention demonstrates the overall process of the method, starting from the acquisition of vehicle-mounted binocular images and UAV orthophotos, through spatiotemporal alignment, perspective transformation, cross-validation, deviation determination, Poisson reconstruction, and finally the output of the maintenance decision report, embodying the technical line of "acquisition-verification-repair-decision".

[0026] Figure 2 The process of cross-validation and deviation category determination is demonstrated. The first and second contours are extracted from vehicle depth data and UAV orthophotos, respectively. Geometric comparison is performed by back projection and Hausdorff distance calculation. Abnormal abrupt change points are detected along the contour arc length. Based on the distribution characteristics of the abrupt change points, they are classified into two categories: perspective systematic deviation and dynamic occlusion local distortion.

[0027] Figure 3 The process of implicit surface reconstruction using the Poisson equation is demonstrated. Using the second contour as the boundary constraint, the depth field of the point cloud of the surrounding intact road surface is extracted, the Poisson equation is constructed and solved, and the distorted depth value of the damaged area is replaced with the solution to correct the three-dimensional measurement parameters.

[0028] A road maintenance decision-making method integrating visual measurement and defect identification includes the following steps: The system acquires left and right eye images and synchronous pose data from the vehicle-mounted binocular stereo camera, generates road surface depth data, and simultaneously acquires top-view orthophotos from the UAV. Based on the pose data, the system performs spatiotemporal alignment between the two and establishes a perspective transformation matrix from the top-view orthophoto to the road surface depth data view plane. The first contour and surface texture features of the defects are extracted based on pavement depth data. The second contour is extracted based on the top-view orthophoto. The second contour is back-projected onto the pavement depth data view plane using a perspective transformation matrix to obtain the theoretical projection contour. The Hausdorff distance and the angle distribution of the edge normal vectors between the theoretical projection contour and the first contour are calculated. First-order difference and sliding variance statistics are performed along the contour arc length to detect abnormal abrupt changes exceeding the threshold. If the abrupt changes are uniformly distributed along the entire contour, they are judged as systematic perspective deviations. If they are concentrated in local arc segments, they are judged as local distortions caused by dynamic occlusion. Based on this, low-confidence labels are generated. In response to low confidence labels, using the second contour as the boundary constraint, a Poisson equation is constructed using the depth gradient field of the surrounding intact road surface point cloud. Implicit surface reconstruction is performed on the distorted point cloud of the diseased area to obtain corrected three-dimensional measurement parameters including the maximum depth, average depth and volume of the disease. By integrating and correcting the three-dimensional measurement parameters and surface texture features, the level of damage is determined, and an urgency score is constructed based on the level of damage, the volume of damage, and the traffic flow of the road section. The maintenance priority is determined by the maximum score, and a maintenance decision report is output.

[0029] The process of establishing the perspective transformation matrix specifically includes: Using the pose data output by the vehicle-mounted inertial navigation positioning system as the time reference, the acquisition time of the top-view orthophoto data and the acquisition time of the road surface depth data are paired frame by frame on the time axis. For each frame of data after pairing, the position and orientation of the road surface depth data view plane in three-dimensional space are determined according to the vehicle position coordinates, heading angle, pitch angle and roll angle in the pose data, and then the spatial plane equation corresponding to the view plane is solved. The pixel coordinates in the top-view orthophoto coordinate system are mapped to the road surface depth data view plane through the spatial plane equation, and a perspective transformation matrix between the two coordinate systems is established.

[0030] The extraction process of the first contour specifically includes: Road surface depth data is input into a pre-trained semantic segmentation network. The semantic segmentation network adopts an encoder-decoder structure. The encoder extracts multi-scale depth features of the road surface depth data through multi-layer convolution and downsampling. The decoder restores spatial resolution through multi-layer upsampling and feature fusion. Finally, it outputs a probability map of the same size as the input. The value of each pixel position in the probability map represents the probability that the pixel belongs to the disease category. The probability map is binarized with a preset probability threshold. Pixels with probability values ​​exceeding the threshold are marked as disease pixels, and pixels with probability values ​​not exceeding the threshold are marked as non-disease pixels. The continuous region formed by all disease pixels is extracted as the disease pixel region. Eight-neighbor connected component analysis is performed on the disease pixel region. Disease pixels that are connected to each other are grouped into the same connected region. The same connected region is marked as the same disease target. The set of pixels representing the outer boundary of the disease target is extracted as the first contour.

[0031] The process of extracting surface texture features specifically includes: A rectangular image block centered on the diseased area is extracted from the road surface depth data. This image block is then converted to grayscale, transforming the color image into a grayscale image. On the grayscale image, all pixel pairs are traversed with a preset pixel spacing and angle, and the frequency of different grayscale level combinations is counted to construct a grayscale co-occurrence matrix. Based on this matrix, energy, contrast, and correlation values ​​are calculated as the first texture feature. The energy value reflects the uniformity of grayscale distribution, the contrast value reflects the clarity of the texture and the depth of the grooves, and the correlation value reflects the linear dependence of grayscale in the row or column direction. Simultaneously, circular local binary mode encoding is performed on the grayscale image. Neighboring pixels are binarized using the grayscale value of the center pixel as a threshold. The binarization results are then summed bit-weighted to obtain the local binary mode encoding value of the center pixel. After traversing all pixels, the frequency of each encoding value is counted to construct a texture histogram feature as the second texture feature. The first and second texture features are concatenated to form the surface texture feature.

[0032] The extraction process of the second contour specifically includes: The top-view orthophoto data is converted to grayscale by performing grayscale processing. A Gaussian kernel is used to smooth and filter the grayscale image to suppress noise, resulting in a filtered image. Canny edge detection is then performed on the filtered image, sequentially performing gradient magnitude and direction calculations, non-maximum suppression, and double-threshold hysteresis tracking to extract edge pixels and output a binarized edge map. Morphological closing operations are performed on the binarized edge map. A pre-defined structuring element is used to first dilate the edge map to fill holes at edge breaks, and then an erosion operation is performed on the dilated result to restore the original edge size, resulting in a closed edge map. The region enclosed by the closed edges is extracted from the closed edge map as a disease candidate region. This candidate region is then filtered by area, eliminating noise regions with areas smaller than a pre-defined area threshold. The set of boundary pixels of the remaining disease candidate regions is extracted as the second contour.

[0033] The detection process for abnormal mutation points specifically includes: The second contour is mapped to the road surface depth data view plane using the perspective transformation matrix to obtain the theoretical projection contour. The theoretical projection contour and the first contour are compared point by point in the same coordinate system. The Euclidean distance between each contour point on the theoretical projection contour and its nearest point on the first contour is calculated, and the maximum value of all Euclidean distances is taken as the one-way Hausdorff distance between the theoretical projection contour and the first contour. At the same time, the Euclidean distance between each contour point on the first contour and its nearest point on the theoretical projection contour is calculated, and the maximum value is taken. The larger of the two one-way Hausdorff distances is taken as the two-way Hausdorff distance output. The edge normal vector at each contour point on the theoretical projection contour and the edge normal vector at the corresponding nearest point on the first contour are calculated. The angle between vectors is used to obtain the edge normal vector angle distribution; a correspondence is established between the bidirectional Hausdorff distance and the arc length position of the theoretical projected contour, that is, a distance distribution curve along the arc length is constructed with the arc length parameter as the abscissa and the bidirectional Hausdorff distance at each arc length position as the ordinate; a first-order difference operation is performed on the distance distribution curve along the arc length to obtain a distance change rate sequence, and points in the distance change rate sequence whose absolute value exceeds a first preset threshold are identified as distance abnormal change points; at the same time, a sliding window variance statistics are performed on the edge normal vector angle distribution along the arc length direction to obtain an angle dispersion sequence, and points in the angle dispersion sequence that exceed a second preset threshold are identified as angle abnormal change points; the union of the distance abnormal change points and the angle abnormal change points is taken as the abnormal change point.

[0034] The process for determining the deviation category specifically includes: Obtain the set of position coordinates of all abnormal mutation points on the arc length of the theoretical projection contour, and statistically analyze the distribution density of abnormal mutation points with the arc length position as the horizontal axis; calculate the distribution variance of all abnormal mutation points within the entire arc length range. If the distribution variance is less than a preset uniform distribution variance threshold, it is determined that the abnormal mutation points exhibit a uniform distribution characteristic along the entire contour arc length, indicating that the deviation of the theoretical projection contour from the first contour is basically consistent in each arc segment. This situation is determined to be a systematic deviation caused by perspective projection. If the distribution variance is not less than the uniform distribution variance threshold, the abnormal mutation points are clustered according to the arc length position. If the abnormal mutation points are concentrated in one or more isolated local arc segments while there are no abnormal mutation points in other arc segments, it is determined that the abnormal mutation points exhibit a local clustering distribution characteristic, indicating that the theoretical projection contour deviates from the first contour only in specific local segments. This situation is determined to be a local distortion caused by dynamic occlusion; generate a low confidence label, which contains a deviation category field, and the deviation category field is labeled as systematic deviation or local distortion.

[0035] The reconstruction process of the Poisson equation specifically includes: The pixel range covered by the inner region of the second contour on the road surface depth data view plane is determined as the reconstruction target region. All road surface point clouds within a preset distance range extending outward from the second contour are extracted from the road surface depth data. Pixels outside the damaged area in all road surface point clouds are considered intact road surface point clouds. Scatter interpolation is performed on the projection points of the intact road surface point clouds on the view plane to fit a continuous depth field function. This depth field function is defined in a two-dimensional coordinate domain on the road surface depth data view plane and approximates the corresponding depth value at each intact road surface point cloud location. The gradient of the depth field function within the reconstruction target region is calculated. The vector field is then used to calculate the divergence of the gradient vector field, and this divergence is used as the right-hand side of the Poisson equation. The depth values ​​of each pixel position at the second contour boundary are used as Dirichlet boundary conditions to construct a Poisson equation defined on the reconstruction target region. The Poisson equation is discretized into a system of linear equations using the finite difference method. The unknowns of the system of linear equations are the depth values ​​of each pixel position within the reconstruction target region. The system of linear equations is solved iteratively to obtain the depth values ​​of each pixel position within the reconstruction target region. The obtained depth values ​​are used to replace the original distorted depth values ​​of the corresponding pixel positions in the road surface depth data to complete the implicit surface reconstruction.

[0036] The revised 3D measurement parameters specifically include: The spatial plane obtained by plane fitting the depth values ​​of the intact road surface point cloud is used as the reference plane, which represents the original elevation of the road surface before the damage occurs. The vertical distance from the depth value of each pixel in the reconstruction target area to the reference plane is calculated as the damage depth at each pixel position. The maximum value among all damage depths is taken as the maximum damage depth, and the arithmetic mean of all damage depths in the reconstruction target area is taken as the average damage depth. The reconstruction target area is projected along the horizontal and vertical coordinate axes of the road surface depth data, respectively. The product of the horizontal span and the vertical span of the projected coverage area is taken as the damage projection area. The product of the damage projection area and the average damage depth is taken as the damage volume.

[0037] The process of determining the severity of a disease includes: The maximum depth, average depth, projected area, and volume of the disease in the corrected three-dimensional measurement parameters are arranged in order to form a geometric feature vector, and the surface texture features are used to form an apparent feature vector. Each dimension of the geometric and apparent feature vectors is normalized to map the values ​​of each dimension to a unified numerical range to eliminate the influence between different units. The normalized geometric and apparent feature vectors are concatenated along their dimensional directions to form a multidimensional disease feature vector. This multidimensional disease feature vector is input into a pre-trained classifier, which is implemented using a support vector machine or deep neural network and pre-trained with a large number of sample feature vectors labeled with disease levels. These sample feature vectors contain geometric and texture features of various diseases. After receiving the multidimensional disease feature vector, the classifier outputs the posterior probability of the disease belonging to four preset levels: mild, moderate, severe, and extremely severe, based on the pre-learned feature-to-level mapping relationship. The four posterior probabilities are compared, and the level with the highest posterior probability is output as the disease level determination result.

[0038] The process of determining maintenance priorities specifically includes: The geometric severity score is calculated by weighting the grade coefficient corresponding to the disease level with the normalized value of the disease volume. The grade coefficient increases sequentially from low to high according to the disease level, with a larger grade coefficient for higher disease levels. The normalized value of the traffic flow of the road segment is used as the impact range score. The traffic flow of the road segment is provided by road sensors or historical statistical data and is used to reflect the degree of impact of the disease on passing vehicles. The geometric severity score and the impact range score are weighted and summed according to preset weights to obtain the maintenance urgency score of each disease. The preset weights are used to balance the relative importance of the severity of the disease itself and the external impact range in the urgency evaluation. All diseases in the same inspection road segment are sorted in descending order of maintenance urgency score, and the maintenance priority of each disease is determined by the sorting result. The disease with the highest maintenance urgency score is given the highest maintenance priority.

[0039] In this embodiment, the patrol vehicle is equipped with a binocular stereo camera and an inertial navigation positioning system. The baseline distance of the binocular stereo camera is denoted as B, the camera focal length as f, and the principal point coordinates as (u0, v0). The UAV inspection system is equipped with a high-resolution orthophoto camera, which performs overhead photography of the same road segment according to a preset route to acquire overhead orthophoto image data.

[0040] I. Multi-source data acquisition and spatiotemporal alignment During the patrol vehicle's operation, a binocular stereo camera continuously acquires left and right-eye images of the road at a fixed frame rate. Stereo matching is performed on the acquired left and right-eye images: using the pixel with coordinates (u, v) in the left-eye image as the target pixel, a matching pixel is searched along the same horizontal scan line in the right-eye image. The disparity value d(u, v) is calculated. Based on the triangulation principle Z(u, v) = B·f / d(u, v), the disparity value is mapped pixel-by-pixel to a depth value Z(u, v). After traversing the entire image, a depth map is generated. The depth map and the left-eye color image are then stitched together according to pixel position to form road surface depth data. Each pixel in this road surface depth data contains four channels: red channel value R, green channel value G, blue channel value B, and depth value Z.

[0041] The UAV inspection system flies along a preset route and takes vertically downward pictures with its high-resolution orthophoto camera to obtain orthophoto data from above. After orthorectification, the pixel coordinates of the orthophoto data directly correspond to the ground plane coordinates and are recorded as pixel coordinates (x_ortho, y_ortho) in the orthophoto coordinate system. The corresponding grayscale value is recorded as I_ortho(x_ortho, y_ortho).

[0042] Using the pose data output by the inertial navigation system as the time reference, the acquisition times of the top-view orthophoto data and the road surface depth data are paired frame by frame on the time axis. For each paired frame, the position and orientation of the road surface depth data view plane in three-dimensional space are determined based on the vehicle position coordinates P_veh=(X_v, Y_v, Z_v), heading angle α, pitch angle β, and roll angle γ in the pose data. The normal vector n=(n_x, n_y, n_z) of the road surface depth data view plane is determined by trigonometric functions of the heading angle, pitch angle, and roll angle. n_x = sin(α)·cos(β) n_y = cos(α)·cos(β) n_z=sin(β) Any point P=(X,Y,Z) on the view plane satisfies the equation of the space plane: n_x·(X-X_v)+n_y·(Y-Y_v)+n_z·(Z-Z_v)=0 The pixel coordinates (x_ortho, y_ortho) in the top-view orthophoto coordinate system are mapped to the road surface depth data view plane through perspective transformation, and a perspective transformation matrix H is established. This matrix is ​​a 3×3 matrix: [u, v, 1]ᵀ=H·[x_ortho, y_ortho, 1]ᵀ Where (u, v) are the corresponding pixel coordinates on the view plane of the road surface depth data.

[0043] II. Extraction of First Contour and Surface Texture Features Road surface depth data is input into a pre-trained semantic segmentation network. This network takes the four-channel information (R, G, B, Z) of each pixel in the road surface depth data as input, passes through an encoder-decoder structure for forward propagation, and outputs a probability map of the same size as the input. The value p(u, v) of each pixel in the probability map represents the probability that the pixel belongs to the defect category. The probability map is binarized using a preset probability threshold p_th. Pixels with p(u, v) > p_th are marked as defect pixels, and pixels with p(u, v) ≤ p_th are marked as non-defect pixels. A continuous region composed of all defect pixels is extracted as the defect pixel region. Eight-neighbor connected component analysis is performed on the defect pixel region, grouping interconnected defect pixels into the same connected region, which is then marked as the same defect target. The first contour of the k-th defect target is denoted as C1_k = {(u_i, v_i) | i = 1, 2, ..., n1_k}, where n1_k is the total number of pixels on the contour.

[0044] A rectangular image block centered on the centroid of the k-th defect area is extracted from the road surface depth data. This image block is then converted to grayscale, transforming the color image into a grayscale image. All pixel pairs in the grayscale image are traversed using a preset pixel spacing δ and orientation angle θ. The frequency of different grayscale level combinations is counted, constructing a grayscale co-occurrence matrix M_GLCM. The matrix element M_GLCM(a, b) represents the number of times grayscale levels a and b occur simultaneously at a relative position of (δ, θ). The first texture feature is calculated based on the grayscale co-occurrence matrix. Energy value E = Σ_aΣ_bM_GLCM(a, b) 2 Contrast value C = Σ_aΣ_b|ab| 2 ·M_GLCM(a, b) Correlation value Corr=Σ_aΣ_b(a-μ_x)(b-μ_y)·M_GLCM(a, b) / (σ_x·σ_y) Where μ_x and μ_y are the mean values ​​of the gray-level co-occurrence matrix in the row and column directions, respectively, and σ_x and σ_y are the standard deviations of the gray-level co-occurrence matrix in the row and column directions, respectively. Simultaneously, circular local binary mode encoding is performed on the gray-level image. Using the gray-level value g_c of the center pixel as a threshold, the gray-level values ​​g_p of each pixel within a circular neighborhood of radius R_LBP and number of neighborhood points P_LBP are binarized. The binarization result is s(g_p-g_c), which takes the value 1 when g_p-g_c≥0, and 0 otherwise. The binarization results are then summed bit-weightedly to obtain the local binary mode encoded value of the center pixel. After traversing all pixels, the frequency of each encoded value is counted to construct a texture histogram feature H_LBP. This feature is then concatenated with the gray-level co-occurrence matrix features E, C, and Corr to form the surface texture feature.

[0045] III. Second Contour Extraction and Cross-Validation The top-view orthophoto data is converted to grayscale, transforming the color orthophoto into a grayscale image. A Gaussian kernel is used to smooth the grayscale image; the standard deviation of the Gaussian kernel is denoted as σ_g, and the filtered image is denoted as I_smooth. Canny edge detection is performed on the filtered image, sequentially performing gradient magnitude and direction calculations, non-maximum suppression, and double-threshold hysteresis tracking to extract edge pixels and output a binarized edge map E_edge. Morphological closing operations are performed on the binarized edge map, first dilating it with the structuring element K. E_dilate=E_edge⊕K Then perform an etching operation on the expansion results: E_close=E_dilate⊖K Obtain the closed edge map E_close. Extract the region enclosed by the closed edges from the closed edge map as the disease candidate region. Perform area screening on the disease candidate region, and remove noise regions with an area smaller than the preset area threshold A_th. Extract the set of boundary pixels of the retained disease candidate regions as the second contour, denoted as C2={(x_j, y_j)|j=1,2,...,n2}, where n2 is the total number of pixels on the contour.

[0046] The second profile C2 is mapped to the road surface depth data view plane using the perspective transformation matrix H, resulting in the theoretical projected profile C_proj. [u_j', v_j', 1]ᵀ=H·[x_j, y_j, 1]ᵀ C_proj={(u_j', v_j')|j=1, 2,..., n2} The theoretical projected contour C_proj and the first contour C1_k are compared point-by-point in the same coordinate system. The Euclidean distance between each contour point (u_j', v_j') on the theoretical projected contour and its nearest point on the first contour is calculated, and the maximum value of all Euclidean distances is taken as the first one-way Hausdorff distance h(C_proj, C1_k). The Euclidean distance between each contour point (u_i, v_i) on the first contour and its nearest point on the theoretical projected contour is calculated, and the maximum value is taken to obtain the second one-way Hausdorff distance h(C1_k, C_proj). The larger of the two-way Hausdorff distance H(C_proj, C1_k) is taken. H(C_proj, C1_k)=max{h(C_proj, C1_k), h(C1_k, C_proj)} Simultaneously, the angle θ_j between the edge normal vector n_proj(u_j', v_j') at each contour point on the theoretical projection contour and the edge normal vector n1(u_i, v_i) at the corresponding nearest point on the first contour is calculated, resulting in the edge normal vector angle distribution {θ_j|j=1,2,...,n2}.

[0047] A correspondence is established between the bidirectional Hausdorff distance and the arc length position of the theoretical projected profile. The arc length parameter *s* is plotted as the abscissa, and the bidirectional Hausdorff distance at each arc length position is plotted as the ordinate, forming the distance distribution curve H(s) along the arc length. A first-order difference operation is performed on the distance distribution curve along the arc length to obtain the distance change rate sequence: ΔH(s_i)=H(s_{i+1})-H(s_i) Points in the distance change rate sequence whose absolute value exceeds a first preset threshold τ_dist are identified as distance anomalous abrupt change points. Simultaneously, a sliding window variance analysis is performed on the edge normal vector angle distribution along the arc length direction, with the window length denoted as W, yielding an angle dispersion sequence Var_θ(s_i). Points in this sequence that exceed a second preset threshold τ_angle are identified as angle anomalous abrupt change points. The union of the distance and angle anomalous abrupt change points is taken as the anomalous change point set M.

[0048] IV. Bias Category Determination and Low Confidence Label Generation Obtain the set of position coordinates {s_m|m∈M} of all anomalous mutation points on the theoretical projected profile arc length, and statistically analyze the distribution density of these anomalous mutation points with the arc length position as the horizontal axis. Calculate the variance of the distribution of all anomalous mutation points over the entire arc length range: σ_M 2 =(1 / |M|)·Σ_{m∈M}(s_m-μ_s) 2 Where μ_s is the average arc length of all anomalous mutation points. If the distribution variance σ_M 2 Less than the preset uniform distribution variance threshold σ_uniform 2 If abnormal mutation points exhibit a uniform distribution along the entire arc length of the contour, this is considered a systematic deviation caused by perspective projection. If the distribution variance is not less than the uniform distribution variance threshold, the abnormal mutation points are clustered according to their arc length positions. Abnormal mutation points with an arc length position spacing less than the preset clustering distance threshold d_cluster are grouped into the same cluster. If abnormal mutation points are concentrated in one or more isolated local arc segments while other arc segments do not contain abnormal mutation points, this is considered a local distortion caused by dynamic occlusion. A low-confidence label L_low is generated, containing a deviation category field, indicating either systematic deviation or local distortion.

[0049] V. Poisson Equation Reconstruction and 3D Measurement Parameter Correction The pixel range covered by the inner region of the second contour C2 on the road surface depth data view plane is defined as the reconstruction target region Ω. All road surface point clouds within the outward extension distance R_extend of the second contour are extracted from the road surface depth data, and the pixels outside the defect area are designated as intact road surface point clouds P_good. Scatter interpolation is performed on the projection points of the intact road surface point clouds on the view plane to fit a continuous depth field function Z_good(u, v). This function is defined in the two-dimensional coordinate domain {(u, v)} of the road surface depth data view plane and approximates the corresponding depth value at each intact road surface point cloud location.

[0050] Calculate the gradient vector field of the depth field function within the reconstructed target region. Then, calculate the divergence of the gradient vector field. The divergence value is used as the right-hand side term f(u, v) of the Poisson equation. The depth values ​​at each pixel location on the second contour boundary are used as the Dirichlet boundary condition Z(u, v)|_{∂Ω}=Z_boundary(u, v), and a Poisson equation defined on the reconstructed target region is constructed: f(u, v) = f(u, v), (u, v) ∈ Ω Z(u, v)=Z_boundary(u, v), (u, v)∈∂Ω in For the Laplace operator.

[0051] The Poisson equation is discretized into a system of linear equations using the finite difference method. The reconstructed target region Ω is divided into a finite difference grid according to the pixel grid, and the Laplacian operator is discretized at each grid point (i, j) as follows: , j)≈[Z(i+1,j)+Z(i-1,j)+Z(i,j+1)+Z(i,j-1)-4Z(i,j)] / h 2 Where h is the grid step size. The discretized equations are rearranged into a linear system of equations A·Z=b, where the unknowns are the depth values ​​Z(i,j) of each pixel location within the reconstructed target region. This linear system of equations is solved using an iterative method (such as the Gauss-Seidel iteration). , , , , ,j-1)-h 2 ·f(i,j)] / 4 After iterative convergence, the depth values ​​Z_recon(i,j) of each pixel position in the target reconstruction area are obtained. The obtained depth values ​​are used to replace the original distorted depth values ​​of the corresponding pixel positions in the road surface depth data to complete the implicit surface reconstruction.

[0052] The spatial plane obtained by plane fitting the depth values ​​of the intact road surface point cloud is used as the reference plane Z_base(u, v), which represents the original elevation of the road surface before the defects occurred. The vertical distance from the depth value of each pixel in the reconstructed target area to the reference plane is calculated as the defect depth. Depth(u,v)=Z_recon(u,v)-Z_base(u,v) The maximum depth of all disease depths is taken as the maximum depth of disease, Depth_max = max_{(u, v)∈Ω}Depth(u, v), and the arithmetic mean of all disease depths within the reconstructed target area is taken as the average depth of disease. Depth_avg=(1 / |Ω|)·Σ_{(u, v)∈Ω}Depth(u, v) The reconstructed target area is projected along both the horizontal and vertical coordinate axes of the road surface depth data. The projected area Area_proj = W_proj × L_proj is calculated as the product of the horizontal span W_proj and the vertical span L_proj. The projected area volume Volume = Area_proj × Depth_avg is calculated as the product of the projected area and the average depth of the disease. The maximum depth, average depth, projected area, and volume of the disease constitute the corrected three-dimensional measurement parameters.

[0053] VI. Disease Severity Determination The geometric feature vector is constructed by sequentially arranging the maximum depth, average depth, projected area, and volume of the disease from the corrected three-dimensional measurement parameters: F_geo=[Depth_max, Depth_avg, Area_proj, Volume] The apparent feature vector F_app is constructed using the energy value, contrast value, correlation value, and local binary pattern histogram of surface texture features. Each dimension of both the geometric and apparent feature vectors is normalized, mapping the values ​​of each dimension to the interval between 0 and 1. F'_geo=(F_geo-F_geo_min) / (F_geo_max-F_geo_min) The normalized geometric feature vector and the normalized apparent feature vector are concatenated along the dimensional direction to form a multidimensional disease feature vector F_total=[F'_geo, F'_app].

[0054] A multidimensional disease feature vector is input into a pre-trained classifier. This classifier is implemented using a support vector machine (SVM) and is pre-trained with feature vectors from a large number of labeled disease levels. During training, a kernel function maps the input features to a high-dimensional space, and an optimal hyperplane is found in this high-dimensional space to separate samples of different levels. After receiving the multidimensional disease feature vector F_total, the classifier outputs the posterior probabilities P_class=[P_1, P_2, P_3, P_4] for each of the four preset levels: mild, moderate, severe, and extremely severe, based on the pre-learned feature-to-level mapping. P_1 corresponds to the probability of the mild level, and P_4 corresponds to the probability of the extremely severe level. The four posterior probabilities are compared, and the level with the highest posterior probability is output as the disease level determination.

[0055] VII. Determining Maintenance Priorities and Outputting Decision Reports The geometric severity score, Score_geo = λ_grade + w_vol·Volume_norm, is calculated by weighting the grade coefficient λ_grade corresponding to the disease level with the normalized value of the disease volume. The grade coefficient λ_grade increases sequentially from low to high according to the disease level, with the grade coefficients for minor, moderate, severe, and extremely severe levels denoted as λ1, λ2, λ3, and λ4, respectively, where λ1 < λ2 < λ3 < λ4. The normalized value of the road segment traffic flow, Traffic_norm, is used as the impact range score. The road segment traffic flow is provided by road sensors and is used to reflect the degree of impact of the disease on passing vehicles. The geometric severity score and the impact range score are weighted and summed according to preset weights w_geo and w_traffic to obtain the maintenance urgency score for the k-th disease. Urgency_k=w_geo·Score_geo_k+w_traffic·Traffic_norm_k All defects within the same patrol section are sorted according to their maintenance urgency scores from highest to lowest. The maintenance priority of each defect is determined by the sorting results, with the defect having the highest maintenance urgency score receiving the highest maintenance priority.

[0056] The disease number, disease level, corrected three-dimensional measurement parameters, maintenance urgency score, and maintenance priority are output as a maintenance decision report in a unified format.

[0057] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A road maintenance decision-making method integrating visual measurement and defect identification, characterized in that, Includes the following steps: Based on pose data, the road surface depth data and the top-view orthophoto data are spatiotemporally aligned, and a perspective transformation matrix is ​​established. Extract the first contour and surface texture features of pavement defects from pavement depth data, and extract the second contour based on top-view orthophoto data; use the perspective transformation matrix to back-project the second contour onto the pavement depth data view plane, perform geometric comparison with the first contour, detect abnormal abrupt points, determine the deviation category, and generate low confidence labels. In response to low confidence labels, using the second contour as the boundary constraint, a Poisson equation is constructed based on the depth gradient field of the pre-set intact road surface point cloud to perform implicit surface reconstruction on the distorted point cloud of the diseased area, and the corrected three-dimensional measurement parameters are obtained. By integrating three-dimensional measurement parameters and surface texture features to determine the level of damage, an urgency score is constructed based on the level of damage, the volume of damage, and the traffic flow of the road section to determine the maintenance priority and output a maintenance decision report.

2. The method according to claim 1, characterized in that, The process of establishing the perspective transformation matrix specifically includes: Based on the pose data, the acquisition time of the top-view orthophoto data and the acquisition time of the road surface depth data are aligned on the time axis; Based on the vehicle position and attitude angle in the pose data, the spatial plane equation corresponding to the road surface depth data view plane is solved, and the coordinates of the top-view orthophoto image are mapped to the road surface depth data view plane to establish a perspective transformation matrix.

3. The method according to claim 1, characterized in that, The extraction process of the first contour specifically includes: The road surface depth data is input into the semantic segmentation network, and the probability value of belonging to the disease category is output pixel by pixel. Pixels with probability values ​​exceeding a preset threshold are marked as disease pixels, and the region formed by disease pixels is extracted as the disease pixel region. Connectivity analysis is performed on the diseased pixel regions, and interconnected pixel regions are marked as the same disease target. The set of boundary pixels of the disease target is extracted as the first contour.

4. The method according to claim 1, characterized in that, The extraction process of the surface texture features specifically includes: Image patches of the diseased area are extracted from the road surface depth data. After grayscale processing of the image patches, a grayscale co-occurrence matrix is ​​constructed, and energy, contrast and correlation texture features are extracted. Local binary mode encoding is performed on the image patch, and the frequency of occurrence of each encoding mode is counted to form a texture histogram feature; the texture histogram feature is concatenated with the gray-level co-occurrence matrix feature to form the surface texture feature.

5. The method according to claim 1, characterized in that, The extraction process of the second contour specifically includes: The top-view orthophoto data is preprocessed by grayscale conversion and Gaussian filtering, and edge detection is performed on the preprocessed top-view orthophoto data to obtain a binarized edge map; Morphological closure operations are performed on the binarized edge map to connect the broken edges. The region enclosed by the closed edges is extracted as a disease candidate region, and the set of boundary pixels of the disease candidate region is extracted as the second contour.

6. The method according to claim 1, characterized in that, The detection process for the abnormal mutation points specifically includes: Calculate the Hausdorff distance and the angle distribution of edge normals between the theoretical projected profile and the first profile, and establish a correspondence between the Hausdorff distance and the arc length position of the theoretical projected profile. The distance change rate sequence is obtained by performing a first-order difference operation on the Hausdorf distance along the arc length direction, and the angle dispersion sequence is obtained by performing sliding window variance statistics on the angle between the edge normal vectors along the arc length direction. Abnormal abrupt changes exceeding their respective preset thresholds are detected in the distance change rate sequence and the angle dispersion sequence, respectively.

7. The method according to claim 1, characterized in that, The process for determining the deviation category specifically includes: The location distribution of abnormal mutation points on the theoretical projected contour arc length is obtained. If the abnormal mutation points are evenly distributed along the entire contour arc length, they are determined to be systematic deviations caused by perspective projection. If the abnormal mutation points are concentrated in local arc segments of the contour, they are determined to be local distortions caused by dynamic occlusion. The low confidence label contains label category information corresponding to the deviation category.

8. The method according to claim 1, characterized in that, The reconstruction process of the Poisson equation specifically includes: The inner region of the second contour is taken as the reconstruction target region. The intact road surface point cloud within a preset range around the second contour is extracted from the road surface depth data. The depth value is fitted to the intact road surface point cloud to obtain a continuous depth field. The gradient of the continuous depth field is calculated and the divergence is taken as the right-hand side of the Poisson equation. The depth value at the second contour boundary is used as the Dirichlet boundary condition. The Poisson equation is solved to obtain the depth value of each pixel position in the reconstruction target area. The original distorted depth value is replaced with the depth value to complete the implicit surface reconstruction.

9. The method according to claim 1, characterized in that, The corrected three-dimensional measurement parameters specifically include: The vertical distance between the depth value of each pixel in the reconstructed target area and the fitting reference plane of the intact road surface point cloud is taken as the disease depth. The maximum value of the disease depth is taken as the maximum disease depth. The arithmetic mean of the disease depth in the reconstructed target area is taken as the average disease depth. The projected area of ​​the spatial distribution range of the disease depth on the road surface is taken as the disease projected area. The product of the disease projected area and the average disease depth is taken as the disease volume.

10. The method according to claim 1, characterized in that, The process for determining the severity of the disease specifically includes: A geometric feature vector is constructed using three-dimensional measurement parameters, and an apparent feature vector is constructed using surface texture features. The geometric feature vector and the apparent feature vector are normalized according to preset weights and then concatenated in a feature layer to form a multi-dimensional disease feature vector. The multidimensional disease feature vector is input into a pre-trained classifier. The classifier outputs the posterior probability of the disease belonging to each preset level. Each preset level is classified according to the geometric severity of the disease and the density of texture damage. The level with the highest posterior probability is output as the judgment result.

11. The method according to claim 1, characterized in that, The process of determining the maintenance priority specifically includes: The geometric severity score is obtained by weighting the grade coefficient corresponding to the disease level and the normalized value of the disease volume, and the influence range score is obtained by weighting the geometric severity score and the influence range score. All diseases are ranked from high to low according to the maintenance urgency score, and the ranking order is used as the maintenance priority.

12. A road maintenance decision system integrating visual measurement and defect identification, comprising a processor for performing the method as described in any one of claims 1 to 11.

Images