A road crack dynamic detection method and system based on visual sensing

Abstract

This invention discloses a method and system for dynamic detection of road cracks based on visual sensing, belonging to the interdisciplinary field of computer vision and intelligent transportation systems. It includes: simultaneously acquiring continuous video frames of the road surface and strictly time-aligned six-DOF vehicle pose data; calculating the relative transformation matrix of adjacent frames based on the pose data and performing spatial alignment to construct a spatiotemporal feature map containing temporal statistical information; utilizing multi-view observations formed by vehicle movement, adaptively selecting the optimal multi-frame images and reconstructing the crack depth based on the principle of motion parallax; performing segmented planar constraint optimization on the reconstructed initial 3D point cloud to generate an optimized 3D crack model and outputting length, width, and depth parameters. This method significantly reduces the false alarm rate, achieves high-precision 3D quantitative assessment of cracks, and provides reliable data support for road maintenance decisions.

Description

Technical Field

[0001] This application belongs to the interdisciplinary field of computer vision and intelligent transportation systems, specifically relating to a method and system for dynamic detection of road cracks based on visual sensing. Background Technology

[0002] With the continuous advancement of intelligent transportation and road maintenance technologies, visual sensing-based road defect detection has become a key means to ensure the safe operation of infrastructure. Traditional detection methods mainly rely on vehicle-mounted cameras to collect static road surface images and combine them with deep learning models to identify and classify crack targets in a single frame. While this type of method has a certain degree of effectiveness in controlled environments, it faces multiple challenges in real-world dynamic road inspection scenarios, especially in dealing with image quality degradation caused by complex lighting, random occlusion, and vehicle movement.

[0003] The crack detection strategy based on single-frame images has inherent limitations. First, common road surface obstructions such as fallen leaves, tire tracks, water stains, and tree shadows are highly similar to real cracks in texture, color, and shape, making them easily misidentified as damaged areas, resulting in a high false alarm rate. Second, single-frame images lack spatial depth information, making it impossible to quantitatively assess the actual damage level of cracks (such as depth, volume, or three-dimensional morphology), which restricts the scientific nature of subsequent maintenance decisions. In addition, image blurring caused by vehicle vibration or high-speed shooting is common during vehicle movement, and existing methods, lacking cross-frame redundant information, struggle to effectively suppress noise and restore clear features.

[0004] Therefore, there is an urgent need for a dynamic road crack detection method that can integrate spatiotemporal context information, has anti-interference capabilities, and supports three-dimensional parameter output. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for dynamic detection of road cracks based on visual sensing, which can effectively solve the problems in the background art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a dynamic detection method for road cracks based on visual sensing, comprising the following specific steps:

[0007] Simultaneously acquire continuous video frames of the road surface and vehicle six-DOF pose data that are strictly time-aligned with each frame;

[0008] Based on the vehicle's six-degree-of-freedom pose data, the relative pose transformation matrix of adjacent frame images is calculated, and the adjacent frame images are spatially aligned based on motion vectors to construct a spatiotemporal feature map containing time dimension statistical information.

[0009] Suspected crack regions are extracted from the spatiotemporal feature map. Through continuous tracking for more than a preset number of frames and morphological change constraints, the temporal consistency of the extracted suspected crack regions is determined to filter out non-persistent interference and obtain real crack candidate regions.

[0010] By utilizing multi-view observations generated by vehicle movement, the optimal multi-frame observation images are adaptively selected from continuously tracked real crack candidate regions, and the crack depth is reconstructed based on the principle of motion parallax to obtain the initial three-dimensional point cloud.

[0011] The initial 3D point cloud is segmented and plane-constrained for optimization to generate an optimized 3D crack model, and outputs 3D detection data including length, width and depth.

[0012] Furthermore, continuous video frames of the road surface and vehicle six-DOF pose data strictly time-aligned with each frame are acquired simultaneously, including:

[0013] The vehicle-mounted camera acquires a sequence of road surface images at a first frequency, and the inertial measurement unit acquires vehicle motion state data at a second frequency higher than the first frequency.

[0014] Based on the IEEE 1588 precision time protocol, a nanosecond-level clock synchronization is established between the camera device and the main control processor of the inertial measurement unit. A unified timestamp tag with an accuracy better than 1 microsecond is attached to each frame of image and each set of motion state data to achieve hardware-level time synchronization of image and motion data.

[0015] The synchronized motion state data is fused and calculated to output the vehicle's six-degree-of-freedom pose in the world coordinate system at a frequency that matches the image acquisition frequency.

[0016] Furthermore, spatial alignment of adjacent frame images based on motion vectors is performed, including:

[0017] Based on the relative pose transformation matrix and camera intrinsic parameters, the current frame image is transformed by back projection mapping to obtain the initial registered image registered to the coordinate system of the previous frame.

[0018] By introducing optical flow constraints, the dense optical flow field between adjacent frames is solved by minimizing the energy function under the assumption of constant brightness. The dense optical flow field is then used to perform residual optimization on the initial registered image, resulting in an optimized registered image with sub-pixel accuracy, thus controlling the image alignment error to within 0.3 pixels.

[0019] Furthermore, a spatiotemporal feature map containing statistical information along the time dimension is constructed, including:

[0020] A sliding window mechanism is used to stack optimized and registered images from multiple consecutive frames within a window;

[0021] For each spatial pixel location, calculate its mean grayscale value and temporal variance on the sliding window time axis;

[0022] The grayscale mean and temporal variance are treated as independent channels, which together with the original grayscale channel of the current frame form a three-channel spatiotemporal feature map to explicitly encode the stability features of pixels in the temporal dimension.

[0023] Furthermore, through continuous tracking of more than a preset number of frames and morphological change constraints, the temporal consistency of the extracted suspected crack regions is determined, including:

[0024] An independent Kalman filter is initialized for each extracted suspected crack region to track its state between consecutive frames;

[0025] The time consistency judgment criteria are set as follows: the suspected crack area must exist continuously in the image sequence for no less than a preset frame number threshold, and the area change rate and center displacement between adjacent frames must be less than the corresponding preset change threshold.

[0026] The suspected crack region is determined as a real crack candidate region and retained only when the consecutive existence frame number threshold, area change rate threshold, and center displacement threshold are met simultaneously; otherwise, it is discarded as a transient interference.

[0027] Furthermore, the optimal multi-frame observation images are adaptively selected from continuously tracked real crack candidate regions, including:

[0028] Based on the vehicle's six-DOF pose sequence, calculate the baseline length of the same crack region between different frames;

[0029] Evaluate the sharpness of the corresponding frame image;

[0030] Select a set of continuous and uniformly distributed frames from the tracking results whose baseline length is within the preset optimal disparity range and whose image sharpness is higher than the preset threshold, and use them as observation frames for depth reconstruction.

[0031] Furthermore, the initial 3D point cloud is subjected to piecewise planar constraint optimization to generate an optimized 3D crack model, including:

[0032] The crack skeleton is extracted based on the binary mask of the crack region, and the crack region is divided into multiple overlapping local windows along the skeleton.

[0033] Within each local window, a random sampling consensus algorithm is used to perform plane fitting on the initial 3D point cloud within the window to obtain the local plane equation and the fitting residual;

[0034] For each pixel within the crack region, the desired depth value is calculated based on the planar equation of one or more local windows to which it belongs, and then weighted and fused with the initial depth value based on the disparity map to obtain the optimized depth field.

[0035] Furthermore, the output includes 3D detection data containing length, width, and depth, including:

[0036] The three-dimensional point set of the crack skeleton is extracted, and the three-dimensional geodesic distance from the starting point to the ending point of the skeleton is calculated using Dijkstra's shortest path algorithm, which is taken as the actual length of the crack.

[0037] For a point on the skeleton, perform sub-pixel edge detection along a direction perpendicular to its local principal direction, determine the three-dimensional coordinates of the two sides of the boundary, calculate the Euclidean distance, and obtain the width of the point.

[0038] The reference plane is fitted with the non-crack pixels in the crack mask outer region. The vertical distance from each pixel in the crack region to the reference plane is calculated to obtain the depth value. The approximate volume of the crack is obtained based on the depth field integral.

[0039] Furthermore, the method also includes: extracting the length, average width, maximum depth and aspect ratio of the output crack as feature vectors, inputting them into a pre-trained decision tree classifier, and semantically classifying the crack as a transverse crack, a longitudinal crack, a crazing crack or a block crack.

[0040] On the other hand, the road crack dynamic detection system based on vision sensing disclosed in this application includes:

[0041] The data acquisition and synchronization module is used to synchronously acquire continuous video frames of the road surface and vehicle six-degree-of-freedom pose data that is strictly time-aligned with each frame of the image.

[0042] The image alignment and spatiotemporal feature construction module is used to calculate the relative pose transformation matrix of adjacent frame images based on the vehicle's six-degree-of-freedom pose data, spatially align adjacent frame images, and construct a spatiotemporal feature map containing time dimension statistical information.

[0043] The temporal consistency tracking module is used to extract suspected crack regions on the spatiotemporal feature map and to determine temporal consistency through continuous tracking for more than a preset number of frames and morphological change constraints, filtering out non-persistent interference and obtaining real crack candidate regions.

[0044] The 3D depth reconstruction module is used to utilize the multi-view observation formed by vehicle movement, adaptively select the optimal multi-frame observation images from the continuously tracked real crack candidate regions, reconstruct the crack depth based on the principle of motion parallax, and perform segmented planar constraint optimization on the generated initial 3D point cloud to obtain the optimized 3D crack model.

[0045] The parameter calculation and output module is used to calculate and output three-dimensional detection data containing length, width and depth based on the three-dimensional crack model.

[0046] In summary, this application includes at least one of the following beneficial technical effects:

[0047] 1. This application effectively distinguishes between real cracks and transient interference such as fallen leaves, tire tracks, and water stains by introducing a time consistency tracking mechanism. Since real cracks exhibit stable spatial location and morphological characteristics across multiple consecutive frames, while interference items typically appear only in a single frame or a few frames, this invention reduces the false alarm rate to a preset low level by setting a threshold for sustained existence for more than a preset number of frames and morphological change constraints, resulting in a reduction of more than a predetermined amount compared to traditional single-frame detection methods.

[0048] 2. This application utilizes the multi-view observation conditions formed by vehicle movement and combines the principle of motion parallax to reconstruct crack depth information, overcoming the inherent limitation of monocular vision's lack of depth perception. Through an adaptive frame selection strategy, it ensures that the optimal parallax baseline can be obtained at different vehicle speeds; by introducing a stereo matching algorithm with prior crack shape, it improves the matching robustness of weakly textured crack regions; through piecewise planar constraint optimization, it effectively suppresses outlier noise, making the depth field continuous, smooth, and physically consistent; through refined three-dimensional parameter calculation, it not only obtains accurate length, width, and depth but also estimates crack volume and quantifies the uniformity of depth distribution.

[0049] 3. This application achieves high-precision image spatial alignment by synchronously fusing vehicle motion state data and image sequences, effectively compensating for image blurring and misalignment caused by vehicle vibration and bumps. Even under high-speed driving or adverse road conditions, the system can maintain stable feature extraction and matching performance, ensuring the reliability and repeatability of detection results. Attached Figure Description

[0050] Figure 1 This is an overall schematic diagram of a road crack dynamic detection method based on visual sensing.

[0051] Figure 2 This is a schematic diagram illustrating the principle of time-consistency tracking and motion parallax depth reconstruction.

[0052] Figure 3 It is a logical diagram that synchronously acquires road surface video frames and vehicle motion state data and constructs a spatiotemporal feature map;

[0053] Figure 4 It is a multi-level interactive relationship diagram of multi-frame image spatial alignment, continuous tracking of suspected crack areas, and three-dimensional parameter output. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0055] The road crack dynamic detection method based on vision sensing in this application is implemented in the following steps:

[0056] First, regarding step S1: synchronously acquire continuous video frames of the road surface and vehicle motion status data. In order to achieve high-precision dynamic detection of road cracks, it is first necessary to synchronously acquire high frame rate road surface images and vehicle motion status data, which specifically includes the following sub-steps.

[0057] Step S101: The vehicle-mounted camera acquires a sequence of road surface images. The vehicle-mounted high frame rate camera continuously captures the road surface image sequence at a capture frequency of no less than 200 Hz. This camera is equipped with a global shutter CMOS image sensor with an effective pixel resolution of 2048×1536 and a dynamic range of no less than 70 dB. The exposure time can be adjusted from 10 microseconds to 50 milliseconds. An automatic exposure control algorithm adjusts the exposure parameters in real time according to the ambient light intensity, ensuring clear images without motion blur can be obtained under strong direct sunlight or low-light conditions at night.

[0058] This automatic exposure control algorithm is based on image grayscale histogram statistics to calculate the average brightness value of the current image and compare it with a preset target brightness threshold. A proportional-integral-derivative (PID) controller is used to perform closed-loop adjustment of the exposure time, stabilizing the image brightness within the target range to adapt to changes in lighting conditions. The camera device integrates an active infrared illumination module, composed of an 850nm wavelength infrared LED array, with a maximum output power of 10 watts and an illumination angle of 90 degrees.

[0059] When the ambient illuminance is below 5 lux, the supplementary lighting module is automatically activated to provide uniform and shadow-free near-infrared illumination, maintaining the image signal-to-noise ratio above 30 dB.

[0060] Step S102: The inertial measurement unit (IMU) collects vehicle motion state data. The IMU records the vehicle's motion state data in three-dimensional space in real time at a sampling frequency of not less than 500 Hz. The IMU includes a three-axis accelerometer and a three-axis gyroscope. The accelerometer has a range of ±16g and a zero-bias instability of less than 50 micrograms per square Hz; the gyroscope has a zero-bias instability of less than 0.5 degrees per hour and an angular random walk coefficient of less than 0.15 degrees per square hour.

[0061] The raw acceleration and angular velocity data are tightly coupled and fused with the wheel speed signal provided by the wheel speed encoder via an extended Kalman filter. The wheel speed encoder is mounted on the drive shaft and has a resolution of at least 1000 pulses per revolution; the vehicle's forward speed can be calculated by counting the pulses. The extended Kalman filter is designed as follows:

[0062] The state vector is defined as:

[0063]

[0064] in This represents the vehicle's three-dimensional position in the world coordinate system, in meters. This refers to three-dimensional velocity, measured in meters per second. A unit quaternion representing attitude; This is zero bias for the gyroscope, measured in radians per second. This is for accelerometer zero bias, measured in meters per square second.

[0065] The state transition model predicts based on measurements from the inertial measurement unit. Let the sampling interval be... The gyroscope measurement value The accelerometer measurement value True angular velocity and acceleration The relationship with the measured value is as follows:

[0066]

[0067] in and It is zero-mean Gaussian white noise, and its covariance matrix is ​​given in the device datasheet.

[0068] The attitude quaternion update uses discretized kinematic equations:

[0069]

[0070] in To represent quaternion multiplication, For angular velocity The pure quaternions formed This is a quaternion exponential mapping. The position and velocity updates are as follows:

[0071]

[0072] acceleration The gravity vector needs to be subtracted. And convert it to the world coordinate system.

[0073] The observation model utilizes vehicle speed information provided by a wheel speed encoder. The vehicle's forward speed, calculated from the encoder pulse frequency, is denoted as... Its observation equation is:

[0074]

[0075] in , , The roll angle, pitch angle, and yaw angle are obtained from quaternions. The variance of the observed noise is set according to the encoder resolution. Furthermore, vehicle incompleteness constraints can be used to assume zero lateral and vertical velocities as virtual observations, further enhancing observability.

[0076] The extended Kalman filter operates recursively according to the standard prediction-update steps: first, it uses the state transition model and IMU measurements to make predictions, obtaining the state prior estimate and covariance matrix; then, when wheel speed observations are available, it calculates the Kalman gain and fuses the observations to update the state posterior estimate.

[0077] Through this tightly coupled fusion method, the vehicle's six-degree-of-freedom pose in the world coordinate system is finally output at a frequency of 200 Hz, including three translational components and three Euler angles.

[0078] The above steps S101 and S102 acquire image and motion data at rates of 200 Hz and 500 Hz or higher, respectively, providing high-density raw information for achieving spatiotemporal alignment.

[0079] Step S103: Time synchronization of image sequence and motion state data. The image sequence and motion state data are precisely aligned through a hardware-level time synchronization mechanism. Specifically, the IEEE 1588 Precision Time Protocol (PTP) is used to establish nanosecond-level clock synchronization between the camera device and the main control processor of the inertial measurement unit. All data frames are appended with a uniform timestamp tag, with a timestamp accuracy better than 1 microsecond.

[0080] This mechanism ensures that the image frame captured at any given time corresponds strictly one-to-one with its corresponding vehicle pose data on the time axis, with a time alignment error not exceeding one-tenth of the single-frame image acquisition cycle.

[0081] Through the above step S1, the system obtains a strictly aligned high frame rate road surface image sequence and high-precision vehicle pose data, providing a reliable data foundation for subsequent image registration, crack tracking and 3D reconstruction, ensuring that subsequent processing can be carried out in a unified spatiotemporal coordinate system, and effectively suppressing the adverse effects of motion blur and attitude changes.

[0082] Step S2: Perform spatial alignment of adjacent frame images based on motion vectors to construct a spatiotemporal feature map.

[0083] After obtaining synchronized high-frame-rate road surface image sequences and vehicle pose data, precise spatial alignment of consecutive image frames is required to eliminate image offsets caused by vehicle motion and extract stable features from the temporal dimension. This involves the following sub-steps.

[0084] Step S201: Calculate the relative pose transformation matrix between adjacent frames. Based on the six-DOF vehicle pose in the world coordinate system output in step S1, it first needs to be converted into the camera pose. The installation relationship of the vehicle-mounted camera device relative to the vehicle carrier coordinate system is obtained through offline hand-eye calibration to obtain the extrinsic parameter matrix. , representing the rigid body transformation from the camera coordinate system to the vehicle coordinate system. Then, the camera pose in the world coordinate system at frame k is:

[0085]

[0086] in This is the vehicle pose in the k-th frame output from step S1. Similarly, the camera pose in the (k-1)-th frame can be obtained. .

[0087] The relative transformation matrix from the k-th frame camera to the (k-1)-th frame camera is:

[0088]

[0089] This matrix is ​​composed of rotation matrices. Translation vector Composition. To maintain geometric consistency, Lie algebraic exponential mapping can be used, but directly using rotation matrices and translation vectors for subsequent calculations can also meet the accuracy requirements.

[0090] Step S202: Subpixel-level image registration based on back projection, using a back projection mapping strategy, to register the current frame... Each pixel in Switch to the previous frame The image coordinate system. First, the camera intrinsic parameter matrix needs to be known. This matrix was obtained through offline calibration:

[0091]

[0092] in Focal length in pixels. These are the coordinates of the principal point of the image.

[0093] Pixels through Inverse transformation to normalized camera coordinate system Then, using the relative transformation matrix Inverse transformation to the previous frame camera coordinate system Finally passed Projecting back onto the image plane yields the corresponding pixel coordinates:

[0094]

[0095] Since the mapping results are usually non-integer coordinates, bilinear interpolation is required. The grayscale value at that location. Let... Four integer pixels around it The grayscale values ​​are respectively ,in The interpolated grayscale value is:

[0096]

[0097] in , The above transformation is performed on all pixels to obtain the registered image. .

[0098] Step S203: Residual optimization with optical flow constraints. To further improve alignment accuracy, an optical flow constraint term is introduced to perform residual optimization on the registration results of step S202. Define the optical flow field. Given the pixel displacement vectors between adjacent frames, minimize the energy function under the assumption of constant brightness:

[0099]

[0100] in For the image region, the first term is a data term that forces brightness to remain constant, and the second term is a smoothing term that constrains the local smoothing of the optical flow field. These are the smoothing weighting coefficients.

[0101] The Horn-Schunck algorithm is used iteratively to solve for this energy function. Let... where are the partial derivatives of image brightness with respect to spatial and temporal directions, respectively, and can be obtained through difference approximation. The iterative update formula for the optical flow field is:

[0102]

[0103] in The average optical flow of the current pixel's neighborhood (usually eight neighborhoods) is indicated by the superscript. This indicates the number of iterations. The initial value of the iteration is set to 0, and it is repeated until the change is less than a preset threshold (e.g., ...). (or reach the maximum number of iterations (e.g., 100). After obtaining the dense optical flow field, it is used to correct the initial registration result: for each pixel in the current frame, its corresponding correction position in the previous frame is... The optimized registration image was obtained by using bilinear interpolation again. This ensures that the final image alignment error is controlled within 0.3 pixels.

[0104] Step S204: Constructing a spatiotemporal feature map using a sliding window. After multi-frame registration, a spatiotemporal feature map is constructed using a sliding window mechanism. The window length is set to 8 frames. Whenever a new frame enters the system, the window slides forward one frame, discarding the oldest frame and incorporating the newest frame. All aligned images within the window are processed. For each spatial location along the time axis Statistical analysis was performed on the pixel value sequence to calculate the time mean:

[0105]

[0106] And time variance:

[0107]

[0108] Will and They are treated as two independent channels, and compared with the original grayscale channel (e.g., taking the latest frame). The grayscale values ​​together constitute the three-channel spatiotemporal feature map. ,in and These represent the image height and width, respectively. This spatiotemporal feature map not only preserves static texture information but also explicitly encodes stability features in the temporal dimension: the mean reflects the average brightness of the pixel at that location over time, while the variance characterizes the degree of brightness fluctuation, thus providing an enhanced basis for subsequent crack region extraction.

[0109] In summary, through step S2, the system achieves image spatial alignment based on motion vectors and constructs a feature map containing spatiotemporal statistical information, providing high-quality input for subsequent temporal consistency tracking and crack identification.

[0110] For step S3: Continuous tracking of suspected crack areas using a time consistency operator filters out non-persistent transient noise. After obtaining the spatiotemporal feature map, candidate crack regions need to be extracted. Through continuous tracking over multiple frames, transient interference such as fallen leaves and water stains are removed, retaining only genuine cracks with temporal consistency. This step specifically includes the following sub-steps.

[0111] Step S301: Initial suspected crack region extraction based on convolutional neural network. First, the spatiotemporal feature map generated in step S2... The method utilizes a pre-trained convolutional neural network crack detector to extract initial suspected crack regions. This detector employs an encoder-decoder structure, with the encoder using a lightweight MobileNetV3 as its backbone and the decoder using a feature pyramid to progressively restore spatial resolution, ultimately outputting a crack probability map of the same size as the input.

[0112] To ensure detector performance, a training set containing various road surface types, lighting conditions, and crack morphologies needs to be constructed. Images of interfering elements such as fallen leaves and water stains are added and manually annotated at the pixel level, with crack areas marked as 1 and background as 0. The training images are uniformly scaled to 256×256 pixels and data augmentation techniques such as random cropping, flipping, brightness adjustment, and noise addition are used to obtain a dataset of no less than 50,000 images.

[0113] The loss function employs a combination of cross-entropy loss and Dice loss. Cross-entropy loss measures pixel-level classification error, while Dice loss measures the overlap between predicted and ground truth regions; each has a weight of 0.5. Training uses the Adam optimizer with an initial learning rate of 0.001, decaying by 0.1 every 10 epochs for a total of 50 epochs. After training, the network is applied to spatiotemporal feature maps, outputting a probability map. Then, through 0.5 threshold segmentation and connected component analysis, an initial set of suspected crack regions is obtained. Each region is characterized by its bounding box, binary mask, and confidence score.

[0114] Step S302: Initialize the Kalman filter for each suspected crack region, for each region Initialize independent Kalman filters for continuous frame tracking. The state vector is six-dimensional:

[0115]

[0116] in The coordinates of the region center For area, Aspect ratio, The velocity of the center of motion.

[0117] The state transition adopts a uniform speed model with a frame interval. seconds, the transition matrix is:

[0118]

[0119] The observation vector is extracted directly from the detection results. Observation matrix The first four dimensions are mapped to the unit.

[0120] Observation noise covariance Dynamically adjusted based on image signal-to-noise ratio (SNR): when SNR ≥ 30 dB, use diag([1,1,10,0.01]); otherwise, use diag([4,4,40,0.04]). Process noise covariance. Adaptive based on vehicle vibration intensity: if acceleration standard deviation If the initial covariance is positive, use diag([0.1,0.1,5,0.005,0.05,0.05]); otherwise, use diag([0.5,0.5,25,0.025,0.25,0.25]). Let it be diag([10,10,100,0.1,1,1]).

[0121] Step S303: Kalman filter prediction and update loop. With each new frame, standard Kalman prediction-update is performed on the surviving regions. First, prediction is made based on the state of the previous frame:

[0122]

[0123] Then, the detection result of the current frame is associated and matched with the prediction result (e.g., using IoU matching). If a new observation is matched... Then calculate and update the Kalman gain:

[0124]

[0125] If no observation is matched, the prediction state is retained but not updated.

[0126] Step S304: Temporal consistency determination and false region removal. A dual threshold determination is applied to each continuous tracking region: it must exist continuously for no less than 5 frames, and the area change rate between adjacent frames must be... Central displacement Pixels. If both conditions are met, the crack is considered a real crack candidate and retained; otherwise, it is immediately discarded.

[0127] Through step S3 above, the system achieves crack tracking based on time consistency, effectively filtering out instantaneous interference and providing reliable candidates for subsequent 3D reconstruction.

[0128] Step S4: Reconstruct the crack depth using the motion parallax between multiple frames, and output 3D detection data including length, width, and depth.

[0129] After obtaining the candidate crack regions verified by time consistency, it is necessary to further recover the three-dimensional geometric information of the cracks. This step is based on multi-view observations formed by vehicle movement, and uses the principle of motion parallax to reconstruct the crack depth and calculate its three-dimensional parameters. Specifically, it includes the following sub-steps.

[0130] Step S401: Adaptively select the optimal multi-frame observation images and calculate geometric constraints. First, based on the vehicle pose sequence output in step S1, adaptively select a set of observation frames from the continuous tracking results of the same crack region according to the vehicle's movement speed and image sharpness.

[0131] Specifically, the baseline length (by the magnitude of the translation vector t) and the image blur index (e.g., Laplacian variance) between adjacent frames are calculated. When the baseline length is within a preset optimal disparity range (e.g., 0.1m to 0.5m) and the image blur is below a threshold (e.g., Laplacian variance > 100), the frame pair is included in the candidate set.

[0132] Five consecutive and evenly distributed frames are selected from the candidate set as the final observation frames. These frames correspond to different camera poses T1, T2, ..., T5 ∈ SE (3).

[0133] If the candidate set has fewer than 5 frames, the baseline length threshold is appropriately relaxed or adjacent frames are added. This adaptive selection strategy ensures that sufficient disparity baselines are obtained at different vehicle speeds, avoiding the decrease in depth accuracy due to excessively short baselines or the difficulty in image matching caused by excessively long baselines, thus laying the foundation for subsequent high-precision 3D reconstruction. For each pair of adjacent frames i and i+1, the rotation matrix R_{i→i+1} and the translation vector t_{i→i+1} are extracted based on the relative transformation matrix obtained in step S201, and then the fundamental matrix is ​​calculated:

[0134]

[0135] in Represents the translation vector antisymmetric matrix, This is the camera intrinsic parameter matrix (already calibrated in step S202). express The inverse transpose of . The fundamental matrix encodes the epipolar geometric relationship between the two views, providing a basis for subsequent epipolar line correction.

[0136] Step S402: Epipolar line correction and semi-global stereo matching. To simplify stereo matching to a one-dimensional horizontal search, epipolar line correction needs to be performed on the image pairs. Based on the fundamental matrix... Calculate the correction transformation matrix of the left and right images. and The original image is remapped to coplanar and row-aligned corrected image pairs, ensuring that corresponding points lie on the same scan line. This correction process is a standard technique in the field and can be implemented using the Bouguet algorithm or epipolar correction methods based on the fundamental matrix.

[0137] On the corrected image pairs, the improved semi-global matching (SGM) algorithm is used to solve for the optimal disparity d pixel by pixel. The cost function is defined as:

[0138]

[0139] Where p=(u,v) are the pixel coordinates of the left image, and d is the disparity to be determined; and These are the grayscale values ​​of the corresponding pixels in the left and right images, respectively. Let be the angle between the gradient direction in the neighborhood of point p and the horizontal direction; The disparity prior value can be predicted from the disparity map or local plane of the previous frame to enhance the temporal consistency of matching; weighting coefficients , , .

[0140] When dynamically programming the aggregation cost, an adaptive path weight based on the crack skeleton orientation is introduced: for extracted suspected crack regions, their skeleton orientation is calculated, and in the eight paths of cost aggregation, the aggregation weight along the skeleton tangent direction is increased (e.g., multiplied by 1.2) to enhance crack continuity; for non-crack regions, isotropic aggregation is used. This improvement makes the matching algorithm more adaptable to the slender structure of cracks, effectively reducing false matches caused by texture repetition or weak textures.

[0141] The cost function combines grayscale consistency, local texture direction, disparity prior, and crack shape prior, making the matching more robust to weak texture crack regions. Dynamic programming is used to aggregate costs in eight directions, and a winner-takes-all strategy is used to select the disparity corresponding to the minimum cumulative cost, resulting in a disparity map with a disparity resolution of up to 0.1 pixels.

[0142] Step S403: Triangulation is used to calculate the 3D coordinates and obtain the disparity map, which is then combined with the camera intrinsic parameters. With external parameters, the three-dimensional spatial coordinates of each pixel in the crack region are calculated using triangulation. For a pair of points with the same name, the coordinates in the left figure are... The right figure (number) The corresponding coordinates in the frame are The formula for calculating three-dimensional coordinates is:

[0143]

[0144] in and For from the first Frame to the The relative rotation matrix and translation vector of the frame; Represents the cross product of vectors. Represents the dot product of vectors; This is the inverse of the intrinsic parameter matrix. The formula is based on vector geometry, obtaining vectors perpendicular to the two line-of-sight directions through the cross product, and then using the dot product with the translation vector to obtain depth information. This is equivalent to solving for the intersection of the two line-of-sight directions.

[0145] Perform the above calculations on all pixels within the crack area to obtain the initial 3D point cloud.

[0146] Step S404: Segmented Planar Constraint Optimization of the Depth Field. To further improve depth accuracy, segmented planar constraint optimization is performed based on the crack mask and skeleton. This includes the following sub-steps:

[0147] Step S404a: Crack Segmentation and Local Window Division. Based on the crack binary mask obtained in step S301, the crack skeleton is extracted and discretized into a sequence of key points. Along the skeleton, the crack region is divided into multiple overlapping local windows with a fixed step size (e.g., 10 pixels). Each window covers a continuous segment of the skeleton and its neighborhood (e.g., a window size of 30×30 pixels). The design of the overlapping windows ensures a smooth transition between adjacent segments.

[0148] Step S404b: Feature Point Selection and Plane Fitting. Within each local window, initial 3D points of all pixels (obtained in step S403) are selected as candidate points. The RANSAC algorithm is used to iteratively fit the plane: three points are randomly selected to calculate the plane equation, and the distance from all points within the window to the plane is counted. Points with a distance less than a threshold (e.g., 1 mm) are selected as inliers. This process is repeated multiple times (e.g., 100 times), and the plane model with the most inliers is selected. Then, the plane is refitted using the least squares method with the inliers to obtain the local plane equation AX+BY+CZ+D=0 and the fitting residual (the root mean square distance from the inlier to the plane). If the fitting residual of a certain window is greater than a preset threshold (e.g., 2 mm), the window size is reduced or a quadratic surface fitting is used, and RANSAC is re-executed. This step effectively eliminates interference from deep pits or noise points inside cracks on the plane fitting.

[0149] Step S404c: Planar Blending and Depth Optimization. For each pixel, determine one or more local windows to which it belongs (due to window overlap). Calculate the desired depth value for that pixel based on the planar equations of each window. Simultaneously, the initial depth is obtained from the disparity map transformation in step S403. And calculate its confidence weight. (For example, related to the inverse of the matching cost). Furthermore, geometric weights are calculated based on the distance from the pixel to the crack skeleton. (The closer the distance, the greater the weight). Depth value after fusion. The weighted average is obtained as follows:

[0150]

[0151] in The weights for the corresponding windows can be determined based on the fitting residuals and the distance from the pixel to the center of the window (the smaller the fitting residuals and the closer to the center, the greater the weight). After calculation for all pixels, the optimized depth field Z(x,y) is obtained. This depth field retains the original parallax observations and achieves smoothing and noise reduction through piecewise plane constraints, significantly improving the accuracy of depth measurement, especially the depth continuity at the crack edges.

[0152] Step S405: Refined calculation of 3D crack geometry parameters. Based on the optimized depth field and the 2D crack mask obtained in step S301, a refined algorithm is used to calculate the 3D geometry parameters of the crack.

[0153] Length L: Extract the crack skeleton (single pixel width), map the skeleton points to 3D space to obtain a 3D point set { The geodesic distance from the start point to the end point of the skeleton is calculated using Dijkstra's shortest path algorithm, which involves sequentially accumulating the three-dimensional Euclidean distances between adjacent skeleton points: For branched cracks, the length of each branch is calculated separately, and the total length and the length of the main branch are recorded. This method reflects the actual three-dimensional extension of the crack better than simple pixel accumulation.

[0154] Width W: For each point on the skeleton First, the local principal direction of the point is calculated (by performing principal component analysis (PCA) on neighboring skeleton points to obtain the eigenvector corresponding to the largest eigenvalue). Then, sub-pixel edge detection is performed along a direction perpendicular to the principal direction and within the tangent plane. Specifically, sampling is performed along this direction in the depth field, and the positions of the two side boundaries are determined using depth gradient changes or grayscale gradient changes (sub-pixel accuracy can be achieved by combining B-spline interpolation). The 3D coordinates of the two side boundaries are obtained, and the Euclidean distance between them is calculated as the width of the point. The average width is obtained by averaging the widths of all skeleton points. Record the maximum width at the same time. And location. For network cracks such as crazing, the width can be calculated separately within each grid cell.

[0155] Depth D: Define the reference plane: Within a region of non-crack pixels extending a certain distance (e.g., 5cm) beyond the crack mask, the equation of the reference plane is obtained by fitting the plane using RANSAC. The perpendicular distance from each 3D pixel within the crack region to the reference plane is the depth of that point. The statistical value of all point depths, such as the 5th percentile, is taken as the bottom depth of the crack. (To exclude the influence of accidental protrusions), or take the average value as the average depth. Simultaneously, the standard deviation of the depth distribution is calculated to assess depth uniformity.

[0156] Volume V: Based on the depth field within the crack region, the approximate volume of the crack is obtained by multiplying the area of ​​each pixel's corresponding infinitesimal element by the depth and integrating. Where Δx and Δy are the actual physical dimensions corresponding to each pixel (obtained through camera calibration). Volume parameters can intuitively reflect the degree of crack damage and provide a basis for estimating the amount of maintenance materials needed.

[0157] All parameters are output in millimeters or cubic millimeters, along with confidence information (such as standard deviation) to facilitate subsequent disease severity assessment and maintenance decisions.

[0158] Step S406: Semantic classification of cracks. To facilitate maintenance decisions, semantic classification is performed on the output 3D crack data. The feature vector of each crack is extracted: length. Average width Maximum depth and aspect ratio These features are then input into a pre-trained decision tree classifier for classification.

[0159] The decision tree classifier is constructed using the CART algorithm with a depth of 8 layers, and the splitting criterion is minimizing Gini impurity. The training data construction process is as follows: A large number of labeled crack types are collected, including transverse cracks, longitudinal cracks, alligator cracks, and block cracks, with no fewer than 1000 samples per class. The feature vector of each crack is calculated in step S405, and its class label is manually assigned. The sample set is divided into a training set and a validation set in a 7:3 ratio. The CART decision tree is trained using the training set, and parameters are adjusted through cross-validation to ensure a classification accuracy of over 95%.

[0160] Based on the classification results, the cracks are divided into:

[0161] Horizontal cracks: ;

[0162] Longitudinal cracks: ;

[0163] Cracks: And the number of branches The number of branches can be obtained through skeleton analysis;

[0164] Blocky cracks: Furthermore, it is a closed polygon, which can be determined through contour analysis;

[0165] Through the above step S4, the system realizes crack depth reconstruction and three-dimensional parameter output based on motion parallax, providing accurate quantitative data for road maintenance.

[0166] In summary, this application addresses the problem of misjudgment of interfering objects in single-frame detection through temporal consistency tracking, overcomes the limitation of monocular vision's lack of depth perception by utilizing motion parallax, and compensates for image motion blur by leveraging vehicle motion status to ensure detection stability in dynamic inspection scenarios. The entire solution provides accurate and reliable 3D defect data for road maintenance, demonstrating significant technological advancements and practical value.

[0167] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0168] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for dynamic detection of road cracks based on visual sensing, characterized in that, Includes the following steps: Simultaneously acquire continuous video frames of the road surface and vehicle six-DOF pose data that are strictly time-aligned with each frame; Based on the vehicle's six-degree-of-freedom pose data, the relative pose transformation matrix of adjacent frame images is calculated, and the adjacent frame images are spatially aligned based on motion vectors to construct a spatiotemporal feature map containing time dimension statistical information. Suspected crack regions are extracted from the spatiotemporal feature map. Through continuous tracking for more than a preset number of frames and morphological change constraints, the temporal consistency of the extracted suspected crack regions is determined to filter out non-persistent interference and obtain real crack candidate regions. By utilizing multi-view observations generated by vehicle movement, the optimal multi-frame observation images are adaptively selected from continuously tracked real crack candidate regions, and the crack depth is reconstructed based on the principle of motion parallax to obtain the initial three-dimensional point cloud. The initial 3D point cloud is segmented and plane-constrained for optimization to generate an optimized 3D crack model, and outputs 3D detection data including length, width and depth. 2.The road crack dynamic detection method based on visual sensing according to claim 1, characterized in that, Simultaneously acquire continuous video frames of the road surface and vehicle six-DOF pose data that is strictly time-aligned with each frame, including: The vehicle-mounted camera acquires a sequence of road surface images at a first frequency, and the inertial measurement unit acquires vehicle motion state data at a second frequency higher than the first frequency. Based on the IEEE 1588 precision time protocol, a nanosecond-level clock synchronization is established between the camera device and the main control processor of the inertial measurement unit. A unified timestamp tag with an accuracy better than 1 microsecond is attached to each frame of image and each set of motion state data to achieve hardware-level time synchronization of image and motion data. The synchronized motion state data is fused and calculated to output the vehicle's six-degree-of-freedom pose in the world coordinate system at a frequency that matches the image acquisition frequency. 3.The road crack dynamic detection method based on visual sensing according to claim 1, characterized in that, Spatial alignment of adjacent frames based on motion vectors includes: Based on the relative pose transformation matrix and camera intrinsic parameters, the current frame image is transformed by back projection mapping to obtain the initial registered image registered to the coordinate system of the previous frame. By introducing optical flow constraints, the dense optical flow field between adjacent frames is solved by minimizing the energy function under the assumption of constant brightness. The dense optical flow field is then used to perform residual optimization on the initial registered image, resulting in an optimized registered image with sub-pixel accuracy, thus controlling the image alignment error to within 0.3 pixels. 4.The method of claim 3, wherein, Construct a spatiotemporal feature map that includes statistical information in the time dimension, including: A sliding window mechanism is used to stack optimized and registered images from multiple consecutive frames within a window; For each spatial pixel location, calculate its mean grayscale value and temporal variance on the sliding window time axis; The grayscale mean and temporal variance are treated as independent channels, which together with the original grayscale channel of the current frame form a three-channel spatiotemporal feature map to explicitly encode the stability features of pixels in the temporal dimension. 5.The road crack dynamic detection method based on visual sensing according to claim 1, wherein, By continuously tracking for more than a preset number of frames and constraining morphological changes, the temporal consistency of the extracted suspected crack regions is determined, including: An independent Kalman filter is initialized for each extracted suspected crack region to track its state between consecutive frames; The time consistency judgment criteria are set as follows: the suspected crack area must exist continuously in the image sequence for no less than a preset frame number threshold, and the area change rate and center displacement between adjacent frames must be less than the corresponding preset change threshold. The suspected crack region is determined as a real crack candidate region and retained only when the consecutive existence frame number threshold, area change rate threshold, and center displacement threshold are met simultaneously; otherwise, it is discarded as a transient interference. 6.The method of claim 1, wherein, The optimal multi-frame observation images are adaptively selected from continuously tracked real crack candidate regions, including: Based on the vehicle's six-DOF pose sequence, calculate the baseline length of the same crack region between different frames; Evaluate the sharpness of the corresponding frame image; Select a set of continuous and uniformly distributed frames from the tracking results whose baseline length is within the preset optimal disparity range and whose image sharpness is higher than the preset threshold, and use them as observation frames for depth reconstruction.

7. The method for dynamic detection of road cracks based on vision sensing according to claim 1, characterized in that, The initial 3D point cloud is segmented and planar constrained to generate an optimized 3D crack model, including: The crack skeleton is extracted based on the binary mask of the crack region, and the crack region is divided into multiple overlapping local windows along the skeleton. Within each local window, a random sampling consensus algorithm is used to perform plane fitting on the initial 3D point cloud within the window to obtain the local plane equation and the fitting residual; For each pixel within the crack region, the desired depth value is calculated based on the planar equation of one or more local windows to which it belongs, and then weighted and fused with the initial depth value based on the disparity map to obtain the optimized depth field.

8. The method for dynamic detection of road cracks based on vision sensing according to claim 7, characterized in that, The output includes 3D detection data containing length, width, and depth, including: The three-dimensional point set of the crack skeleton is extracted, and the three-dimensional geodesic distance from the starting point to the ending point of the skeleton is calculated using Dijkstra's shortest path algorithm, which is taken as the actual length of the crack. For a point on the skeleton, perform sub-pixel edge detection along a direction perpendicular to its local principal direction, determine the three-dimensional coordinates of the two sides of the boundary, calculate the Euclidean distance, and obtain the width of the point. The reference plane is fitted with the non-crack pixels in the crack mask outer region. The vertical distance from each pixel in the crack region to the reference plane is calculated to obtain the depth value. The approximate volume of the crack is obtained based on the depth field integral.

9. The method for dynamic detection of road cracks based on vision sensing according to claim 1, characterized in that, The method also includes: extracting the length, average width, maximum depth, and aspect ratio of the output crack as feature vectors, inputting them into a pre-trained decision tree classifier, and semantically classifying the crack as a transverse crack, a longitudinal crack, a crazing crack, or a block crack.

10. A road crack dynamic detection system based on vision sensing, used to implement the method described in any one of claims 1-9, characterized in that, include: The data acquisition and synchronization module is used to synchronously acquire continuous video frames of the road surface and vehicle six-degree-of-freedom pose data that is strictly time-aligned with each frame of the image. The image alignment and spatiotemporal feature construction module is used to calculate the relative pose transformation matrix of adjacent frame images based on the vehicle's six-degree-of-freedom pose data, spatially align adjacent frame images, and construct a spatiotemporal feature map containing time dimension statistical information. The temporal consistency tracking module is used to extract suspected crack regions on the spatiotemporal feature map and to determine temporal consistency through continuous tracking for more than a preset number of frames and morphological change constraints, filtering out non-persistent interference and obtaining real crack candidate regions. The 3D depth reconstruction module is used to utilize the multi-view observation formed by vehicle movement, adaptively select the optimal multi-frame observation images from the continuously tracked real crack candidate regions, reconstruct the crack depth based on the principle of motion parallax, and perform segmented planar constraint optimization on the generated initial 3D point cloud to obtain the optimized 3D crack model. The parameter calculation and output module is used to calculate and output three-dimensional detection data containing length, width and depth based on the three-dimensional crack model.

Images