Real-time crack detection and trajectory tracking system and method based on video inter-frame relationship

Abstract

The invention relates to the technical field of road maintenance automatic detection, in particular to a real-time crack detection and trajectory tracking system and method based on a video inter-frame relationship, and the system comprises a video collection module, a preprocessing module, an MFCO detection module, an inter-frame feature matching module, a trajectory tracking module, a dynamic parameter adjustment module and an output module. The method comprises the steps of vehicle-mounted video stream collection, single-frame preprocessing, single-frame crack detection, inter-frame feature matching, crack track generation, dynamic parameter adjustment and result output, and real-time detection, redundancy elimination and continuous track generation of pavement cracks in a vehicle-mounted dynamic scene are achieved. The maintenance priority is correlated through crack length-depth quantification, the problems of redundancy detection, crack fracture misjudgment, poor multi-material adaptability and the like of an existing static crack detection technology in a vehicle-mounted scene are solved, the method can be directly applied to road maintenance vehicle-mounted inspection, and the detection efficiency and the decision supporting capacity are improved.

Description

Technical Field

[0001] This invention relates to the field of automated highway maintenance detection technology, specifically a real-time crack detection and trajectory tracking system method based on the inter-frame relationship of decorative elements. Background Technology

[0002] Road surface cracks are a key indicator reflecting the safety and durability of road structures, and rapid and accurate crack detection is a core aspect of highway maintenance and management. Existing road surface crack detection technologies are mainly divided into two categories: traditional manual inspection and static image processing technology. Traditional manual inspection is inefficient and highly subjective, making it difficult to meet the needs of long-distance highway inspections. Static image processing technologies (such as Otsu threshold segmentation, Canny edge detection, and single-frame deep learning segmentation models) have achieved a certain degree of automation, but they are only designed for single-frame static images and cannot adapt to vehicle-mounted mobile scenarios. Vehicle-mounted video streams suffer from problems such as redundant crack detection in consecutive frames (the same crack is repeatedly identified in multiple frames, leading to wasted computing power), false crack breakage due to motion blur (single-frame blurring makes cracks appear discontinuous, and the model cannot correlate inter-frame information), and interference from road material (blurred crack edges in asphalt pavements, high textural noise in concrete pavements, and color interference from colored anti-skid pavements). The detection accuracy and real-time performance are insufficient to meet engineering requirements.

[0003] Existing vehicle-mounted dynamic detection technologies have many limitations. Currently, most publicly available vehicle-mounted video stream detection technologies focus on traffic sign recognition and vehicle target tracking, without optimizing for the linear / mesh features of road surface cracks or their adaptability to multiple materials. Some crack detection-related patents (such as application number 202310787133.7) only involve multi-scale feature extraction from static images and do not introduce inter-frame correlation mechanisms, thus failing to address redundancy and fragmentation issues in dynamic scenarios. Furthermore, existing technologies do not establish a correlation between "road surface material - vehicle speed" and detection parameters. Single-parameter models show a significant decrease in detection accuracy at different vehicle speeds (such as 30km / h and 60km / h) and on different road surface materials. Moreover, they lack the linkage between quantitative parameters such as crack length and depth and maintenance decisions, and can only output segmentation results, failing to directly support the formulation of maintenance plans. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time crack detection and trajectory tracking system and method based on video frame relationships, which combines video frame relationship optimization, dynamic parameter adjustment and trajectory tracking to meet the road surface crack detection requirements of "real-time, high precision, multi-material adaptation and quantitative decision-making" in vehicle dynamic scenarios.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A real-time crack detection and trajectory tracking system based on inter-frame relationships in video, comprising: Video acquisition module: used to acquire road video streams in vehicle-mounted mobile scenarios, including an industrial camera (resolution ≥1920×1080, frame rate ≥25fps), a laser ranging unit (to obtain the real-time distance L between the camera and the road surface, unit: mm), and a positioning unit (to record the road surface position for each frame). Preprocessing module: The preprocessing module is connected to the video acquisition module and is used for single-frame image preprocessing, including dynamic blur suppression, image normalization and noise filtering; Multi-scale Feature Fusion Cascaded Optimization Detection Network Module (MFCO Detection Module): This module communicates with the preprocessing module and is used for crack segmentation and detection, outputting the number of pixels in the binarized crack image (denoted as the total number of crack pixels after removing isolated noise <3 pixels). The multi-scale feature fusion cascaded optimization detection network module includes a multi-scale dilated convolutional feature extraction submodule, an improved decoding submodule, and a hierarchical feature fusion optimization submodule. The dilation rate in the multi-scale dilated convolutional feature extraction submodule is dynamically adjusted, and the hierarchical feature fusion optimization submodule is used to enhance the utilization of low-level features and improve the coherence of crack segmentation. Inter-frame feature matching module: The inter-frame feature matching module is communicatively connected to the video acquisition module and the multi-scale feature fusion cascaded optimization detection network module, respectively, and is used for the correlation of cracks in adjacent frames; Trajectory Tracking Module: The trajectory tracking module communicates with the inter-frame feature matching module and is used to generate continuous crack trajectories (including spatial coordinate sequences and actual lengths). The trajectory tracking module includes a breakpoint recovery submodule. When the confidence level of a single frame is <0.6, the breakpoint recovery submodule predicts the position of the current frame by fitting a quadratic polynomial to the crack center coordinates of the previous 3 frames, and completes the trajectory. Dynamic parameter adjustment module: The dynamic parameter adjustment module communicates with the video acquisition module, the multi-scale feature fusion cascaded optimization detection network module, and the inter-frame feature matching module, adjusting parameters according to vehicle speed. (km / h) and the void rate and overlap threshold of the multi-scale void convolution feature extraction submodule for road surface material adjustment; Output module: The output module communicates with the trajectory tracking module and the positioning unit to associate the crack trajectory with the positioning information and generate a structured inspection report, including crack location, length, direction, inspection time and maintenance priority.

[0007] The output module supports 4G / 5G wireless communication and can transmit structured inspection reports to a remote highway maintenance management platform in real time. The report is in JSON format and includes the following fields: latitude, longitude, crack_length (crack length, mm), crack_direction (crack direction), confidence (inspection confidence level), and maintenance_level (maintenance priority: emergency / routine / observation).

[0008] Preferably, the laser ranging unit in the video acquisition module is used to obtain the real-time distance L between the camera and the road surface; Industrial cameras include an automatic exposure adjustment unit that adjusts exposure based on changes in light intensity. , The current frame's illumination intensity. When the light intensity of the previous frame is used, the exposure parameters are automatically adjusted to avoid distortion of crack features.

[0009] Preferably, the dynamic parameter adjustment module pre-stores a three-dimensional mapping table of vehicle speed, road surface material, and void ratio; The dynamic parameter adjustment module includes a road surface material recognition submodule, which is a lightweight CNN classifier (parameter count < 1M). The input features are the "grayscale mean + LBP texture features" of the preprocessed image, with a classification accuracy of ≥ 92%. The module also adjusts the edge smoothing weights of the hierarchical feature fusion optimization submodule in the multi-scale feature fusion cascade optimization detection network module according to the material, with the weight for concrete pavement being 15%-20% higher than that for asphalt pavement.

[0010] A real-time crack detection and trajectory tracking method includes the following steps: S1. Video Stream Acquisition: The vehicle-mounted video acquisition module acquires road video streams and simultaneously obtains the real-time distance between the camera and the road surface. Vehicle speed and the position information of each frame; S2. Single-frame preprocessing: The single-frame image obtained in step S1 is subjected to dynamic blur suppression, image normalization and noise filtering through the preprocessing module. S3. Single-frame crack detection: Input the preprocessed image obtained in step S2 into the multi-scale feature fusion cascaded optimization detection network module, and output the number of pixels in the crack binarized image. And confidence level, and calculate crack density. ; S4. Inter-frame feature matching: The inter-frame feature matching module calculates the pixel displacement vector of the gap between adjacent frames based on the improved Lucas-Kanade optical flow method. And calculate the overlap. If the overlap If the value exceeds the preset threshold, it is determined to be the same crack, and redundant results are discarded. S5. Crack Trajectory Generation: Using the trajectory tracking module, a spatial coordinate sequence is generated based on the correlation results of the same crack in consecutive frames, and the actual length is calculated. The direction is determined by fitting a direction vector through a coordinate sequence; when the single-frame confidence is low, the current frame position is predicted by the breakpoint recovery submodule to complete the crack trajectory. S6. Dynamic Parameter Adjustment: The road surface material is identified through the road surface material identification submodule in the dynamic parameter adjustment module, combined with the vehicle speed obtained in step S1. The void rate of the multi-scale void convolution feature extraction submodule in the multi-scale feature fusion cascade optimization detection network module is adjusted according to the three-dimensional mapping table of vehicle speed-road surface material-void rate; additional color channel normalization is performed on colored anti-skid road surfaces to eliminate color interference. S7. Result Output: Based on the actual length obtained in step S5, the output module outputs the result. and depth calculation results Determine maintenance priorities; integrate crack trajectory, location information, and maintenance priorities to generate a structured inspection report.

[0011] Preferably, the specific process of single-frame preprocessing in step S2 includes: Dynamic blur suppression quantifies the degree of blur by using the mean of the L2 norm of the inter-frame optical flow vector. :when When a 3×3 deblurring convolution kernel is selected, when Select 5x5 cores, when Select 7×7 cores for pixel resolution; Image normalization unifies the size of a single frame image to 480×320 pixels, and records the total number of pixels in the image as . ; Noise filtering employs adaptive Gaussian filtering, with the filter kernel variance in the crack region. Noisy areas such as oil stains / shadows .

[0012] Preferably, the specific process of single-frame crack detection in step S3 includes: Define crack density for: (1) in, The number of pixels in the binarized image of the crack. This represents the total number of pixels in the image.

[0013] Preferably, the specific process of inter-frame feature matching in step S4 includes: Crack pixel displacement vector calculated based on improved Lucas-Kanade optical flow method And define the pixel overlap of the inter-frame gap. (The same crack) is identified as: (2) in, This represents the number of pixels intersecting the cracks between adjacent frames. Let the number of unions be denoted by 'Number'. The initial preset threshold is the minimum overlap value that achieves "trajectory accuracy ≥ 95%" in a dataset test with 1000+ frames. Subsequent thresholds are determined based on crack density. Make fine adjustments: when When the suture is high-density, the threshold increases by 5%-10%; when When the density is low, the threshold decreases by 5%-10%.

[0014] Preferably, the specific process of generating the crack trajectory in step S5 includes: The actual length of the crack Calculated using camera parameters: (3) in, The crack pixel length (total crack skeleton length, unit: pixels). The physical size of the camera pixel (mm / pixel, inherent parameter). The camera focal length (mm, an inherent parameter); When the confidence level of a single frame is less than 0.6, the crack center coordinates of the first three frames are used. Fitting a quadratic polynomial Solve the simultaneous equations and find the polynomial coefficients: (4) in, For frame number, The vertical coordinate of the crack center is... These are the polynomial coefficients; After obtaining the polynomial coefficients, substitute them into the current frame number. Calculate predicted coordinates Complete the breakpoints.

[0015] Preferably, the specific process of adjusting the dynamic parameters in step S6 includes: In the three-dimensional mapping table of vehicle speed-road surface material-void ratio: when the vehicle speed At that time, the void ratio was 3-5, and the vehicle speed was... At that time, the void ratio was 1-2.

[0016] Preferably, the specific process of outputting the result in step S7 includes: Deep inference results Using a linear regression model: (5) in, The average gray value of the crack region is calculated from the binarized image of the crack. The regression slope, The regression intercept was obtained through training with 500+ sets of "grayscale mean - measured borehole depth" samples, with an error ≤0.5mm; When the actual length And the deep calculation results The maintenance priority is "urgent"; when and At that time, the maintenance priority is "routine"; when and At that time, the maintenance priority is "observation".

[0017] The beneficial effects of this invention are:

[0018] This invention breaks through the detection accuracy bottleneck in vehicle-mounted dynamic scenes. By using a multi-scale dilated convolution feature extraction submodule (dynamic dilation rate) and an inter-frame feature matching mechanism, it solves the core pain points of existing static crack detection technologies in vehicle-mounted moving scenes, effectively improving the detection score and reducing the false detection rate. In dynamic scenes with vehicle speeds of 30-60 km / h, the crack detection F1 score is ≥88%, which is 5%-8% higher than that of traditional static models. The false detection rate (identifying road noise as cracks) is reduced by more than 15%, and the missed detection rate (small / fuzzy cracks) is reduced by more than 10%.

[0019] This invention adapts to multiple materials such as asphalt, concrete, and colored anti-slip pavement through a dynamic parameter adjustment module, as well as extreme interferences such as rainstorm reflection, shadows, and oil stains, thereby enhancing robustness in multiple scenarios. The detection accuracy fluctuation is ≤3% under different scenarios, avoiding the problem of insufficient scenario adaptability of single parameter models.

[0020] This invention optimizes the MFCO detection module to a lightweight design, achieving a single-frame inference time of ≤0.02s. It supports real-time processing of vehicle-mounted video streams at 30fps, enabling daily highway inspections of over 500 kilometers. This represents a 5-10 times improvement in efficiency compared to manual foot inspections, significantly shortening the inspection cycle for highway maintenance. It achieves vehicle-level real-time detection, meeting the high-efficiency requirements of vehicle-mounted inspections. Furthermore, it reduces the cost of manual intervention, automating the entire process from "video acquisition → crack detection → trajectory tracking → maintenance suggestion generation" without requiring manual setting of detection thresholds or data filtering, reducing manual data processing workload by over 80%.

[0021] This invention can output structured quantization results by optimizing sub-modules and trajectory tracking mechanisms through hierarchical feature fusion. It can not only output binary images of cracks, but also generate quantization parameters such as the actual length of cracks (error ≤ 5%), direction, and depth estimation (error ≤ 0.5 mm), thus avoiding the problem of insufficient information in traditional detection that only outputs segmentation images.

[0022] This invention correlates maintenance priorities (urgent / routine / observation) with quantitative parameters. The generated inspection report can be directly connected to the highway maintenance management platform, reducing intermediate steps from inspection to maintenance plan formulation, directly supporting maintenance decisions, and providing integrated support for building "inspection-quantification-decision-making", thereby improving the accuracy of maintenance resource allocation by more than 20%.

[0023] The MFCO detection module used in this invention has ≤10M parameters, supports INT8 quantization and TensorRT acceleration, is compatible with automotive embedded devices, and can be deployed on ordinary automotive embedded devices (such as NVIDIA Jetson AGX series and Snapdragon automotive-grade processors). It does not rely on high-performance servers, which greatly reduces the threshold for system implementation and reduces hardware deployment costs by more than 60%. Furthermore, this invention reduces over-maintenance or delayed maintenance of roads through accurate detection and maintenance decisions, extends the service life of highways by 3-5 years, and reduces the average annual maintenance cost by 15%-20%, achieving long-term cost savings.

[0024] The modules of this invention are highly scalable. The multi-scale dilated convolution submodule and the improved decoding submodule are all modularly designed. In the future, the accuracy of detecting small cracks can be further improved by accessing LiDAR data and upgrading the super-resolution submodule, etc., to adapt to more complex road detection scenarios (such as bridges and tunnel pavements), and it has a wider range of applications. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the internal structure of the MFCO detection module in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the three-dimensional mapping of "vehicle speed-road surface material-void ratio" in Embodiment 1 of the present invention; Figure 4 This is a flowchart illustrating the architecture of Embodiment 2 of the present invention; Figure 5 This is a visual comparison chart of the segmentation results of the same type of method in Embodiment 2 of the present invention. Detailed Implementation

[0026] Example 1 The following are specific embodiments of the combination that further explain the present invention, such as Figure 1As shown, this embodiment is a real-time crack detection and trajectory tracking system based on the relationship between video frames, including: Video acquisition module: used to acquire road video streams in vehicle-mounted mobile scenarios, including an industrial camera (resolution ≥1920×1080, frame rate ≥25fps), a laser ranging unit (to obtain the real-time distance L between the camera and the road surface, unit: mm) and a positioning unit (to record the road surface position for each frame, which can be a GPS positioning unit).

[0027] The video acquisition module includes a laser ranging unit to obtain the real-time distance L between the camera and the road surface; the industrial camera includes an automatic exposure adjustment unit that adjusts the exposure based on changes in light intensity. , The current frame's illumination intensity. When the light intensity of the previous frame is used, the exposure parameters are automatically adjusted to avoid distortion of crack features.

[0028] Preprocessing module: The preprocessing module communicates with the video acquisition module and is used for single-frame image preprocessing, including dynamic blur suppression, image normalization and noise filtering.

[0029] Multi-scale Feature Fusion Cascaded Optimization Detection Network Module (MFCO Detection Module): This module communicates with the preprocessing module and is used for crack segmentation and detection, outputting the number of pixels in the binarized crack image (denoted as the total number of crack pixels after removing isolated noise <3 pixels). ) and crack pixel confidence; such as Figure 2 As shown, the multi-scale feature fusion cascaded optimization detection network module includes a multi-scale dilated convolution feature extraction submodule, an improved decoding submodule, and a hierarchical feature fusion optimization submodule. The dilation rate in the multi-scale dilated convolution feature extraction submodule is dynamically adjusted, and the hierarchical feature fusion optimization submodule is used to enhance the utilization of low-level features and improve the coherence of crack segmentation.

[0030] Inter-frame feature matching module: The inter-frame feature matching module is connected to the video acquisition module and the multi-scale feature fusion cascaded optimization detection network module respectively, and is used for the correlation of cracks in adjacent frames.

[0031] Trajectory Tracking Module: The trajectory tracking module communicates with the inter-frame feature matching module and is used to generate continuous crack trajectories (including spatial coordinate sequences and actual lengths). The trajectory tracking module includes a breakpoint recovery submodule. When the confidence level of a single frame is less than 0.6, the breakpoint recovery submodule predicts the position of the current frame by fitting a quadratic polynomial to the crack center coordinates of the previous 3 frames, thus completing the trajectory.

[0032] Dynamic parameter adjustment module: The dynamic parameter adjustment module communicates with the video acquisition module, the multi-scale feature fusion cascaded optimization detection network module, and the inter-frame feature matching module, adjusting parameters according to vehicle speed. (km / h) and the void ratio and overlap threshold of the multi-scale void convolution feature extraction submodule for road surface material adjustment; such as Figure 3 As shown, the dynamic parameter adjustment module has a pre-stored three-dimensional mapping table of "vehicle speed-road surface material-void ratio".

[0033] The dynamic parameter adjustment module includes a road surface material recognition submodule, which is a lightweight CNN classifier (parameter count < 1M). The input features are the "grayscale mean + LBP texture features" of the preprocessed image, with a classification accuracy of ≥ 92%. The module also adjusts the edge smoothing weights of the hierarchical feature fusion optimization submodule in the multi-scale feature fusion cascade optimization detection network module according to the material, with the weight for concrete pavement being 15%-20% higher than that for asphalt pavement.

[0034] Output module: The output module communicates with the trajectory tracking module and the positioning unit to associate the crack trajectory with the positioning information and generate a structured inspection report, including crack location, length, direction, inspection time and maintenance priority.

[0035] The output module supports 4G / 5G wireless communication and can transmit structured inspection reports to a remote highway maintenance management platform in real time. The report is in JSON format and includes the following fields: latitude, longitude, crack_length (crack length, mm), crack_direction (crack direction), confidence (inspection confidence level), and maintenance_level (maintenance priority: emergency / routine / observation).

[0036] Example 2 This embodiment describes a method for real-time crack detection and trajectory tracking based on the real-time crack detection and trajectory tracking system based on video frame relationships proposed in Embodiment 1. Figure 2 As shown, it includes the following steps:

[0037] S1. Video Stream Acquisition: The vehicle-mounted video acquisition module acquires road video streams and simultaneously obtains the real-time distance between the camera and the road surface. Vehicle speed And the position information for each frame.

[0038] S2. Single-frame preprocessing: The single-frame image obtained in step S1 is subjected to dynamic blur suppression, image normalization and noise filtering through the preprocessing module.

[0039] Dynamic blur suppression quantifies the degree of blur by using the mean of the L2 norm of the inter-frame optical flow vector. :when When a 3×3 deblurring convolution kernel is selected, when Select 5x5 cores, when When processing pixels, a 7×7 core is selected; image normalization unifies the size of a single frame image to 480×320 pixels, and records the total number of pixels in the image as... Noise filtering employs adaptive Gaussian filtering, with the filter kernel variance in the crack region... Noisy areas such as oil stains / shadows .

[0040] S3. Single-frame crack detection: Input the preprocessed image obtained in step S2 into the multi-scale feature fusion cascaded optimization detection network module, and output the number of pixels in the crack binarized image. And confidence level, and calculate crack density. .

[0041] Define crack density for: (1) in, The number of pixels in the binarized image of the crack. This represents the total number of pixels in the image.

[0042] S4. Inter-frame feature matching: The inter-frame feature matching module calculates the pixel displacement vector of the gap between adjacent frames based on the improved Lucas-Kanade optical flow method. And calculate the overlap. If the overlap If the result exceeds the preset threshold, it is determined to be the same crack, and redundant results are discarded.

[0043] Define inter-frame gap pixel overlap (The same crack) is identified as: (2) in, This represents the number of pixels intersecting the cracks between adjacent frames. Let be the number of the union.

[0044] The initial preset threshold is the minimum overlap value that achieves "trajectory accuracy ≥ 95%" in a dataset test with 1000+ frames. Subsequent thresholds are determined based on crack density. Make fine adjustments: when When the suture is high-density, the threshold increases by 5%-10%; when When the density is low, the threshold decreases by 5%-10%.

[0045] S5. Crack Trajectory Generation: Using the trajectory tracking module, a spatial coordinate sequence is generated based on the correlation results of the same crack in consecutive frames, and the actual length is calculated. The direction is determined by fitting the direction vector through the coordinate sequence; when the single-frame confidence is low, the current frame position is predicted by the breakpoint recovery submodule to complete the crack trajectory.

[0046] The actual length of the crack Calculated using camera parameters: (3) in, The crack pixel length (total crack skeleton length, unit: pixels). The physical size of the camera pixel (mm / pixel, inherent parameter). The focal length of the camera is (mm, an inherent parameter).

[0047] When the confidence level of a single frame is less than 0.6, the crack center coordinates of the first three frames are used. Fitting a quadratic polynomial Solve the simultaneous equations and find the coefficients of the polynomial: (4) in, For frame number, The vertical coordinate of the crack center is... These are the polynomial coefficients; after obtaining the polynomial coefficients, substitute them into the current frame number. Calculate predicted coordinates Complete the breakpoints.

[0048] The crack direction is calculated using the crack center coordinate direction vector across consecutive frames: Let the crack center coordinates of two consecutive frames be... (mm, converted from location information and image coordinates), then the directional angle. The formula for calculating the angle with due north is: (6) in, It is a four-quadrant arctangent function with an angle range of 0°-360°.

[0049] S6. Dynamic Parameter Adjustment: The road surface material is identified through the road surface material identification submodule in the dynamic parameter adjustment module, combined with the vehicle speed obtained in step S1. The void rate of the multi-scale void convolution feature extraction submodule in the multi-scale feature fusion cascade optimization detection network module is adjusted according to the three-dimensional mapping table of vehicle speed-road surface material-void rate; additional color channel normalization is performed on colored anti-skid road surfaces to eliminate color interference.

[0050] In the three-dimensional mapping table of vehicle speed-road surface material-voidity: when the vehicle speed... At that time, the void ratio was 3-5, and the vehicle speed was... At that time, the void ratio was 1-2.

[0051] S7. Result Output: Based on the actual length obtained in step S5, the output module outputs the result. and depth calculation results Determine maintenance priorities; integrate crack trajectory, location information, and maintenance priorities to generate a structured inspection report.

[0052] Deep inference results Using a linear regression model: (5) in, The average gray value of the crack region is calculated from the binarized image of the crack. The regression slope, The regression intercept was obtained by training with 500+ sets of "grayscale mean - measured borehole depth" samples, with an error ≤0.5mm.

[0053] Average gray value of crack area The calculation formula is: (7) in, This represents the set of crack pixels in the binary image of the crack. For pixels grayscale value, This represents the number of pixels in the binarized image of the crack.

[0054] When the actual length And the deep calculation results The maintenance priority is "urgent"; when and At that time, the maintenance priority is "routine"; when and At that time, the maintenance priority is "observation".

[0055] To verify the effectiveness of the MFCO algorithm, this embodiment conducts a comparative experiment with representative methods in the field (such as U-Net and DeepLabv3+), and analyzes the experimental results based on segmentation performance. The segmentation performance of different methods on the test set is shown below. Figure 5As shown: The first row of images comes from the CFD dataset, where the crack patterns are relatively simple; the second and third rows of images come from the Crack500 dataset, where the cracks are mostly small and continuous. As can be seen from the figure, for the first row of images, U-Net suffers from segmentation discontinuity, while DeepLabv3+ exhibits over-segmentation, with its predicted width being much larger than the labeled value. The MFCO model proposed in this paper, however, performs better overall. For the second and third rows of images, the MFCO algorithm's segmentation results are basically consistent with those of the aforementioned representative models.

[0056] In this embodiment, a public crack dataset CFD+Crack500+autonomous vehicle video dataset containing 1317 images with multiple materials and interference scenes was used for testing. The training / validation / test sets were divided in an 8:1:1 ratio. The test hardware environment was an NVIDIA Jetson AGX automotive embedded platform with JetPack 5.1 system version and TensorRT 8.6 acceleration. The test results showed that the method mentioned in this embodiment achieved an F1 score of ≥88% and an inference time of ≤0.02s (the computation time for crack detection in a single frame).

[0057] In summary, the MFCO algorithm proposed in this embodiment outperforms other algorithms in terms of the completeness, segmentation accuracy, and edge smoothness of crack segmentation tasks. In particular, it has significant advantages in the complete preservation and accurate localization of crack regions, and the smoothness of the segmentation edges is more in line with the needs of practical application scenarios.

[0058] The above description is merely a further explanation of the present invention in conjunction with specific embodiments. All descriptions made do not imply any limitation on the scope of protection of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time crack detection and trajectory tracking system based on inter-frame relationship of video frames, characterized in that: Comprise: Video acquisition module: for collecting the video stream of the road surface in the vehicle-mounted mobile scene, including industrial camera, laser ranging unit and positioning unit; Preprocessing module: the preprocessing module is in communication connection with the video acquisition module, and is used for single frame image preprocessing, including dynamic blur suppression, image normalization and noise filtering; Multi-scale feature fusion cascade optimization detection network module: the multi-scale feature fusion cascade optimization detection network module is in communication connection with the preprocessing module, and is used for crack segmentation detection, and outputs crack binary image and crack pixel confidence; the multi-scale feature fusion cascade optimization detection network module comprises a multi-scale hollow convolution feature extraction submodule, an improved decoding submodule and a hierarchical feature fusion optimization submodule; the hollow rate in the multi-scale hollow convolution feature extraction submodule is dynamically adjusted, and the hierarchical feature fusion optimization submodule is used for strengthening the use of low-level features to improve the continuity of crack segmentation; Inter-frame feature matching module: the inter-frame feature matching module is in communication connection with the video acquisition module and the multi-scale feature fusion cascade optimization detection network module respectively, and is used for adjacent frame crack association; Trajectory tracking module: the trajectory tracking module is in communication connection with the inter-frame feature matching module, and is used for generating crack continuous trajectory; the trajectory tracking module comprises a breakpoint recovery submodule, and when single frame confidence is less than 0.6, the breakpoint recovery submodule predicts the current frame position by fitting a quadratic polynomial according to the center coordinates of the previous three frames, and completes the trajectory; The dynamic parameter adjustment module is in communication connection with the video acquisition module, the multi-scale feature fusion cascade optimization detection network module and the inter-frame feature matching module, and adjusts the dynamic parameters according to the vehicle-mounted speed and the road surface material adjusts the multi-scale hollow convolution feature extraction submodule's hollow rate and the overlap threshold. Output module: the output module is in communication connection with the trajectory tracking module and the positioning unit, and is used for associating crack trajectory and positioning information, generating a structured detection report, including crack position, length, trend, detection time and maintenance priority.

2. The real-time crack detection and trajectory tracking system based on inter-frame relationship of video according to claim 1, wherein: The laser ranging unit in the video acquisition module is used to obtain the real-time distance L between the camera and the road surface; The industrial camera includes an automatic exposure adjustment unit that adjusts the exposure based on changes in light intensity. At the same time, the exposure parameters are automatically adjusted to avoid distortion of crack features.

3. The real-time crack detection and trajectory tracking system based on video inter-frame relationship according to claim 2, characterized in that: The dynamic parameter adjustment module pre-stores a three-dimensional mapping table of vehicle speed-road surface material-hollow rate; The dynamic parameter adjustment module comprises a road surface material identification submodule, the road surface material identification submodule is a lightweight CNN classifier, the input features are the gray mean value and LBP texture features of the preprocessed image, and the edge smoothing weight of the hierarchical feature fusion optimization submodule in the multi-scale feature fusion cascade optimization detection network module is adjusted according to the material.

4. The real-time crack detection and trajectory tracking method of the system according to claim 3, characterized in that: The method comprises the following steps: S1, video stream collection: collecting a road video stream through the vehicle-mounted video collection module, and synchronously acquiring real-time distance of a camera and a road , vehicle-mounted speed and position information of each frame; S2, single frame preprocessing: the single frame image obtained in step S1 is subjected to dynamic blur suppression, image normalization and noise filtering treatment through the preprocessing module; S3, single-frame crack detection: input the pre-processed image obtained in step S2 into the multi-scale feature fusion cascade optimization detection network module, output the pixel number of the crack binary image and the confidence, and calculate the crack density ; S4, inter-frame feature matching: through the inter-frame feature matching module, the improved Lucas-Kanade optical flow method is used to calculate the displacement vector of the crack pixels between adjacent frames , and the degree of overlap is calculated ; if the degree of overlap is greater than a preset threshold, it is determined that the same crack is obtained, and the redundant result is removed. S5, crack trajectory generation: through the trajectory tracking module, based on the same crack association results of continuous frames to generate a sequence of spatial coordinates, calculate the actual length , and determine the strike by fitting a direction vector with the coordinate sequence; in low single-frame confidence, predict the current frame position through the breakpoint recovery submodule to complete the crack trajectory; S6, dynamic parameter adjustment: the road surface material is identified through the road surface material identification submodule in the dynamic parameter adjustment module, the vehicle speed obtained in step S1 is combined , the three-dimensional mapping table of vehicle speed-road surface material-hole rate is used to adjust the hole rate of the multi-scale hole convolution feature extraction submodule in the multi-scale feature fusion cascade optimization detection network module; color channel normalization is additionally performed on the color anti-skid road surface to eliminate color interference; S7, result output: through the output module, based on the actual length and depth calculation result determining the maintenance priority; integrating the crack trajectory, location information and maintenance priority to generate a structured detection report.

5. The real-time crack detection and trajectory tracking method of claim 4, wherein: The specific process of the step S2 single frame preprocessing comprises: The dynamic blur suppression quantifies the blur degree by inter-frame optical flow vector L2 norm mean : when a 3x3 deblur kernel is selected, when a 5x5 kernel is selected, and when a pixel is selected, a 7x7 kernel is selected; The image normalization unifies the size of a single frame image to 480x320 pixels, and records the total number of image pixels as ; The noise filtering employs an adaptive Gaussian filter, with a crack region filter kernel variance , oil stain / shadow, etc. noise region .

6. The real-time crack detection and trajectory tracking method of claim 5, wherein: The specific process of the step S3 single frame crack detection comprises: Definition of fracture density is: ，（1） wherein, is the number of pixels of the crack binary image, is the total number of pixels of the image.

7. The real-time crack detection and trajectory tracking method of claim 6, wherein: The specific process of the step S4 inter-frame feature matching comprises: Based on the improved Lucas-Kanade optical flow method to calculate the crack pixel displacement vector And define the inter-frame crack pixel overlap degree Is: ，（2） wherein, is the number of intersection of adjacent frame crack pixels, is the number of union; The preset threshold initial value is the minimum overlap degree when the trajectory accuracy is greater than or equal to 95% in the data set test, and the subsequent threshold is adjusted according to the crack density Fine tuning: when is high-density crack, the threshold is increased by 5%-10%; when is low-density crack, the threshold is reduced by 5%-10%.

8. The real-time crack detection and trajectory tracking method of claim 7, wherein: The specific process of the step S5 crack trajectory generation comprises: Actual length of the crack By camera parameter conversion: ，（3） wherein, is the length of the crack pixel, is the physical size of the camera pixel, is the camera focal length; When single frame confidence < 0.6, take the first 3 frame crack center coordinates Fitting a quadratic polynomial Solve for the polynomial coefficients: ，（4） wherein is a frame number, is a crack center ordinate, is a polynomial coefficient; After the polynomial coefficients are found, substitute the current frame number Calculate the predicted coordinates , complete the breakpoint.

9. The real-time crack detection and trajectory tracking method of claim 8, wherein: The specific process of the step S6 dynamic parameter adjustment comprises: In the three-dimensional mapping table of vehicle speed-road material-voidity: when the vehicle speed... At that time, the void ratio was 3-5, and the vehicle speed was... At that time, the void ratio was 1-2.

10. The real-time crack detection and trajectory tracking method of claim 9, wherein: The specific process of the step S7 result output comprises: the depth estimation result using a linear regression model: ，（5） wherein, is the mean gray value of the crack region, calculated from the crack binary image; is the regression slope, is the regression intercept, obtained by training the sample of mean gray value-drilling measured depth; when the actual length and the depth estimation result is "urgent"; when and the maintenance priority is "routine"; when and the maintenance priority is "watch".

Images