Road guardrail damage and deformation detection method and system based on multi-frame fusion

Abstract

The application discloses a road guardrail damage and deformation detection method and system based on multi-frame fusion. The method comprises the following steps: acquiring a road environment video; analyzing the road environment video, locking and tracking key feature points on the guardrail, calculating the motion trajectory of the key feature points in the three-dimensional space, and forming a sparse space feature flow field; fitting a virtual ideal trajectory, and comparing the sparse space feature flow field to identify a geometric abnormal area; detecting the physical deformation of the guardrail based on the geometric abnormal area, and analyzing the color abnormality and high-frequency texture fracture condition to obtain the deformation depth and the rust area; comparing the deformation depth and the rust area with a safety threshold value, and generating a warning work order when at least one of the deformation depth and the rust area exceeds the safety threshold value. Through the method of the application, high-precision and full-coverage detection of the physical deformation and surface damage of the guardrail can be realized without increasing too much cost.

Description

Technical Field

[0001] This invention relates to computer vision, and more specifically to a method and system for detecting damage and deformation of road guardrails based on multi-frame fusion. Background Technology

[0002] In highway and urban expressway systems, road guardrails are crucial infrastructure for ensuring traffic safety, and their integrity directly impacts traffic safety. However, due to factors such as vehicle collisions, natural aging and corrosion, and ground settlement, guardrails inevitably deform and become damaged. Current inspection and maintenance methods mainly include manual visual inspection and automated inspection using LiDAR scanning vehicles. Manual inspection is inefficient and poses safety hazards; while LiDAR solutions, although precise, are limited by their high cost.

[0003] In addition, traditional AI detection techniques based on single-frame images also face challenges: on the one hand, they struggle to distinguish between normal bending in a design and denting deformation caused by an impact, as both appear as curves in two-dimensional images; on the other hand, dynamic obstructions such as roadside vegetation or moving vehicles can interfere with detection results, leading to missed detections or false positives. These limitations highlight the inadequacies of existing technologies in terms of accuracy and reliability, especially under complex environmental conditions.

[0004] Therefore, it is necessary to design a new method that can achieve high-precision, full-coverage detection of physical deformation and surface damage of guardrails without increasing costs too much, thereby effectively improving the efficiency and safety of road guardrail inspection work. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for detecting road guardrail damage and deformation based on multi-frame fusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a road guardrail damage and deformation detection method based on multi-frame fusion, comprising:

[0007] Acquire road environment video;

[0008] The road environment video is analyzed to identify and track key feature points on the guardrail, calculate the motion trajectory of the key feature points in three-dimensional space, and form a sparse spatial feature flow field.

[0009] Fit a virtual ideal trajectory and compare it with the sparse spatial feature flow field to identify geometrically anomalous regions; wherein, the virtual ideal trajectory refers to the ideal motion path of key feature points under no-damage conditions;

[0010] The physical deformation of the guardrail is detected based on the geometric anomaly area, and the color anomaly and high-frequency texture fracture are analyzed to obtain the deformation depth and corrosion area.

[0011] The deformation depth and corrosion area are compared with a safety threshold. When at least one of the deformation depth and corrosion area exceeds the safety threshold, an early warning work order is generated.

[0012] The further technical solution is as follows: the fitting of the virtual ideal trajectory and the comparison with the sparse spatial characteristic flow field to identify geometrically anomalous regions include:

[0013] Based on road design specifications, the parametric geometric manifold equations that the guardrail surface follows within the sliding window are determined, and the parameters are iteratively optimized using a robust regression algorithm to approximate the mathematical surface of the sparse spatial characteristic flow field. The weights are adjusted according to the residuals between the data points and the model to obtain the virtual ideal trajectory.

[0014] By comparing the virtual ideal trajectory with the sparse spatial characteristic flow field, the normal distance is calculated;

[0015] Geometrically abnormal regions are identified and marked based on the relationship between the normal distance and a preset deviation threshold.

[0016] The further technical solution is as follows: The fitting of the virtual ideal trajectory and the comparison with the sparse spatial feature flow field to identify geometrically abnormal regions; wherein, the virtual ideal trajectory refers to the ideal motion path of key feature points under no-damage conditions, and further includes:

[0017] When the geometrically abnormal region is occluded, perspective transformation and texture restoration are performed on the occluded region to update the geometrically abnormal region.

[0018] A further technical solution is as follows: when the geometrically abnormal region is occluded, perspective transformation and texture restoration are performed on the occluded region to update the geometrically abnormal region, including:

[0019] When the geometrically abnormal region is occluded, the occluder is identified and a corresponding occlusion region mask is generated to obtain the current occluded region;

[0020] Search and locate historical video frames containing the same scene but not occluded as reference frames in the time dimension;

[0021] Compare the common feature points in the current frame and the reference frame to calculate the homography matrix;

[0022] The calculated homography matrix is ​​applied to the clear texture in the reference frame corresponding to the current occluded region. After perspective transformation, the texture is filled into the corresponding position in the current frame to update the geometrically anomalous region.

[0023] The further technical solution is as follows: the step of searching and locating historical video frames containing the same scene but not occluded as reference frames in the time dimension includes:

[0024] By utilizing the information redundancy in the time dimension, a backtracking search is performed in the road environment video to find unobstructed historical video frames of the same location at different time points, so as to obtain reference frames.

[0025] The further technical solution is as follows: The physical deformation of the guardrail is detected based on the geometric anomaly area, and color anomalies and high-frequency texture fractures are analyzed to obtain the deformation depth and corrosion area, including:

[0026] For the geometrically anomalous region, the normal difference between the sparse spatial characteristic flow field and the virtual ideal trajectory is calculated to detect and quantify the structural deformation caused by physical impact, and to obtain the deformation depth.

[0027] The analysis of color anomalies and high-frequency texture fractures in the geometrically abnormal areas identifies surface corrosion and cracks, and determines the corrosion area in conjunction with set standards.

[0028] The further technical solution is as follows: Analyzing the road environment video, locking and tracking key feature points on the guardrail, calculating the motion trajectory of the key feature points in three-dimensional space, and forming a sparse spatial feature flow field includes:

[0029] The video of the road environment is filtered to include areas outside the guardrail, and key feature points on the guardrail surface are identified.

[0030] Track the position changes of the key feature points in consecutive frames and calculate the pixel displacement vector of the key feature points;

[0031] Calculate the physical displacement of the camera between two frames;

[0032] Combining the principles of multi-view geometric triangulation, pixel displacement vector, and physical displacement, the three-dimensional coordinates of key feature points are calculated, and the motion trajectory of key feature points in three-dimensional space is constructed to obtain the trajectory vector of key feature points.

[0033] The trajectory vectors of all key feature points are unified into the same spatial coordinate system to form a sparse spatial feature flow field that describes the physical properties of the guardrail surface.

[0034] The further technical solution is as follows: the calculation of the physical displacement of the camera between two frames includes:

[0035] Acquire camera pose change data to obtain physical displacement; wherein the pose change data is calculated by a vehicle kinematics model and / or obtained by solving the features of consecutive frames of images through visual odometry.

[0036] The further technical solution is as follows: the early warning work order includes GPS coordinates, damage type, quantitative data and on-site screenshot, wherein the quantitative data includes deformation depth and corrosion area.

[0037] This invention also provides a road guardrail damage and deformation detection system based on multi-frame fusion, comprising:

[0038] The acquisition unit is used to acquire video of the road environment.

[0039] The feature flow field calculation unit is used to analyze the road environment video, lock and track key feature points on the guardrail, calculate the motion trajectory of the key feature points in three-dimensional space, and form a sparse spatial feature flow field.

[0040] A fitting unit is used to fit a virtual ideal trajectory and compare it with the sparse spatial feature flow field to identify geometrically abnormal regions; wherein, the virtual ideal trajectory refers to the ideal motion path of key feature points under no-damage conditions;

[0041] The dual-flow damage determination unit is used to detect the physical deformation of the guardrail based on the geometric anomaly area, and to analyze the color anomaly and high-frequency texture fracture to obtain the deformation depth and corrosion area.

[0042] The generation unit is used to compare the deformation depth and corrosion area with a safety threshold, and generate an early warning work order when at least one of the deformation depth and corrosion area exceeds the safety threshold.

[0043] The advantages of this invention compared to existing technologies are as follows: This invention acquires road environment video and analyzes and locks key feature points of the guardrail, calculates its three-dimensional spatial motion trajectory to form a sparse spatial feature flow field, and then compares it with a virtual ideal trajectory under no-damage conditions to identify geometrically abnormal areas. It also combines color anomaly and high-frequency texture fracture analysis to quantify the degree of physical deformation and corrosion. Finally, it compares the deformation depth and corrosion area with safety thresholds to generate early warning work orders. This method can achieve high-precision, full-coverage detection of physical deformation and surface damage of guardrails using low-cost video surveillance equipment, effectively improving the efficiency and safety of road guardrail inspection work. It also reduces reliance on expensive sensors and the cost of manual inspection, providing an economical and efficient solution for ensuring the safety of road facilities.

[0044] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating the method for detecting road guardrail damage and deformation based on multi-frame fusion provided in an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a sparse spatial characteristic flow field provided in an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a virtual ideal trajectory provided in an embodiment of the present invention;

[0049] Figure 4 A schematic diagram comparing the virtual ideal trajectory and the characteristic flow field of sparse space provided for embodiments of the present invention;

[0050] Figure 5 This is a schematic diagram of the repair of the obscured area provided in an embodiment of the present invention;

[0051] Figure 6 A schematic diagram showing the deformation depth and corrosion area provided in an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of an early warning work order provided in an embodiment of the present invention;

[0053] Figure 8 This is a schematic block diagram of a road guardrail damage and deformation detection system based on multi-frame fusion provided in an embodiment of the present invention.

[0054] Figure 9 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0057] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0058] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0059] Please see Figure 1 , Figure 1 This is a flowchart illustrating a road guardrail damage and deformation detection method based on multi-frame fusion, as provided in an embodiment of the present invention. This method is applied to a server. By utilizing multi-frame video analysis and 3D spatial feature tracking technology, high-precision, full-coverage detection of physical deformation and surface damage of road guardrails is achieved without significantly increasing costs. The method first analyzes road environment video to lock and track key feature points of the guardrail, forming a sparse spatial feature flow field, and fitting a virtual ideal trajectory to identify geometrically abnormal areas. Next, it further analyzes color anomalies and high-frequency texture fractures to obtain deformation depth and corrosion area. Finally, when the detected deformation or corrosion exceeds a safety threshold, an early warning work order containing GPS coordinates, damage type, and quantitative data is automatically generated. This technology not only improves inspection efficiency and reduces the need for manual inspection but also enhances detection accuracy and safety, ensuring timely detection and handling of potential safety hazards.

[0060] Figure 1 This is a flowchart illustrating the road guardrail damage and deformation detection method based on multi-frame fusion provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S150.

[0061] S110, Obtain road environment video.

[0062] In this embodiment, road environment video refers to continuous image data of the road and its surrounding environment recorded by cameras installed on inspection vehicles or fixed monitoring points. This video data contains detailed visual information about the guardrail and its surrounding environment, which forms the basis for subsequent feature extraction, analysis, and damage detection.

[0063] The specific process of acquiring road environment videos includes:

[0064] Equipment Preparation and Setup: First, ensure that the camera equipment used for collecting video data is correctly installed and tested. This typically involves a professional camera mounted on the roof of an inspection vehicle or a fixed surveillance camera along the roadside. To ensure video quality, parameters such as the camera's angle and focal length also need to be adjusted to clearly capture the condition of the guardrails along the target road section.

[0065] Data Acquisition: When the inspection vehicle travels along the predetermined route or the monitoring system begins operation, the camera automatically records video of the road environment. This video is continuously recorded at a certain frame rate (e.g., 30fps) to ensure that any detailed changes that may occur on the road can be captured, including but not limited to changes in the condition of guardrails, the growth of roadside vegetation, and traffic flow.

[0066] Synchronous Data Collection: In addition to video data, the system also collects other relevant information synchronously, such as the vehicle's motion status (speed, acceleration, etc.) and orientation information (via CAN bus data). This is crucial for subsequent calculation of the 3D coordinates of key feature points. Furthermore, in some cases, it may also be necessary to synchronously collect timestamps, GPS coordinates, and other data to accurately mark the location information of each video frame.

[0067] Preprocessing stage: The acquired raw video data typically undergoes a series of preprocessing steps, such as noise reduction and color correction, to improve the accuracy of subsequent analysis. Simultaneously, to focus on the guardrail area, semantic segmentation techniques may be used to filter out non-guardrail regions in the video, retaining only the parts of interest (i.e., the guardrail area), thus preparing for the subsequent identification and extraction of key feature points.

[0068] Through the above steps, the system can effectively acquire high-quality road environment video data as a basis for further analysis and processing, ultimately achieving accurate detection of physical deformation and surface damage of road guardrails. This process not only improves the efficiency of the detection work but also greatly enhances the reliability and practicality of the detection results.

[0069] like Figure 2As shown, the current frame image is first acquired (set as time T). To eliminate interference from the road surface and background, the algorithm uses a semantic segmentation mask to retain only the guardrail area. Within this area, the FAST corner detection algorithm is used to identify points with significant texture features on the guardrail surface. Assuming the inspection vehicle is traveling on a highway, the system locks onto a specific guardrail bolt (Point A) in the image as the tracking target. Figure 2 In the diagram, the tail (starting point) of each arrow represents a locked feature point.

[0070] S120. Analyze the road environment video, lock and track key feature points on the guardrail, calculate the motion trajectory of the key feature points in three-dimensional space, and form a sparse spatial feature flow field.

[0071] In this embodiment, as Figure 2 As shown, sparse spatial characteristic flow field refers to the process of analyzing road environment videos, identifying and tracking key feature points on the guardrail, and calculating the motion trajectories of these key feature points in three-dimensional space. These trajectory vectors are aggregated to form a sparse spatial characteristic flow field that describes the physical properties of the guardrail surface. This flow field can reflect the geometric orientation of the guardrail surface and any potential physical deformations or damage.

[0072] In one embodiment, step S120 described above may include steps S121 to S125.

[0073] S121. Filter out non-guardrail areas from the road environment video and identify key feature points on the guardrail surface.

[0074] In this embodiment, key feature points refer to those points with significant texture features, typically used for subsequent tracking and analysis. Examples include bolts, seams, or other parts with unique textures on the guardrail surface. Semantic segmentation techniques can remove background interference, focusing on the guardrail area, and then the FAST corner detection algorithm is used to identify these key feature points.

[0075] S122. Track the position changes of the key feature points in consecutive frames and calculate the pixel displacement vector of the key feature points.

[0076] In this embodiment, the pixel displacement vector represents the positional change of key feature points from one frame to the next, in pixels. This step uses pyramid optical flow to track the new position of each key feature point and calculate their movement distance and direction (i.e., pixel displacement vector) on the image plane.

[0077] S123. Calculate the physical displacement of the camera between two frames.

[0078] In this embodiment, the pose change data of the camera is acquired; the pose change data is calculated by a vehicle kinematics model (such as CAN bus data), and / or obtained by solving the features of consecutive frame images through visual odometry, so as to obtain the physical displacement of the camera between two frames.

[0079] S124. Combining the principles of multi-view geometric triangulation, pixel displacement vector, and physical displacement, calculate the three-dimensional coordinates of key feature points, construct the motion trajectory of key feature points in three-dimensional space, and obtain the trajectory vector of key feature points.

[0080] In this embodiment, the trajectory vector of the key feature point is calculated using the multi-view geometric triangulation principle based on the data obtained in the above steps. It includes not only the displacement information of the feature point on the two-dimensional image plane, but also its true depth value Z relative to the camera and its three-dimensional coordinates (X, Y, Z), thus forming a vector representing the "movement" of the feature point in three-dimensional space. This trajectory vector actually reflects the physical orientation of the guardrail surface and any possible deformations.

[0081] S125. Unify the trajectory vectors of all key feature points into the same spatial coordinate system to form a sparse spatial feature flow field that describes the physical properties of the guardrail surface.

[0082] In this step, the trajectory vectors of all key feature points are mapped to a unified spatial coordinate system, thus forming a holistic sparse spatial feature flow field. This flow field effectively depicts the physical properties of the guardrail surface, including but not limited to its natural curvature and any potential physical deformation or damage.

[0083] In this embodiment, when the video enters the next frame (time T+1), the pyramid optical flow method is used to search for the new position of the "guardrail bolt" in frame T+1. The pixel displacement vector u=(u, v) of this point in the image plane is calculated. For example, assuming the vehicle is traveling at a speed of 60km / h and the frame rate is 30fps, the "guardrail bolt" shifts 15 pixels to the left in the image during the two-frame interval. At this time, the following is obtained: Figure 2 The two-dimensional projection direction of the middle arrow is unknown, but the actual physical distance it represents is still unknown.

[0084] The system synchronously reads vehicle CAN bus data to obtain real-time vehicle speed V and steering wheel angle δ. Based on the Ackermann steering geometry model, it calculates the physical pose change of the camera within a two-frame interval Δt.

[0085] Calculation formula: Camera physical displacement .

[0086] Based on the case study (the values ​​presented are only one example, not the only standard): The vehicle speed is read as V = 60 km / h (≈16.7 m / s), and the steering wheel angle δ = 0° (straight ahead). Then, within Δt = 0.0033 s, the physical displacement of the camera... This 0.055 meters is the solution. Figure 2 The key measure of the true length of the middle arrow.

[0087] By combining the obtained pixel displacement (15 pixels) and camera physical displacement (0.055 meters), the depth value Z and three-dimensional coordinates (X, Y, Z) of the feature point relative to the camera are calculated using the principle of multi-view geometric triangulation.

[0088] Figure 2 The arrows in the image are actually a visualization of the motion trajectory vectors of these feature points in three-dimensional space. The length of the arrow quantifies the motion parallax of the feature points in visual space (i.e., pixel displacement modulus), and the arrangement of the arrows represents the physical geometric orientation of the guardrail surface.

[0089] These continuous trajectory vectors are mapped onto a unified spatial coordinate system to form a set of sparse spatial characteristic flow fields, which characterize the physical orientation of the guardrail surface.

[0090] Through the four steps described above, the system can effectively locate and track key feature points on the guardrail and accurately calculate the trajectory of these points in three-dimensional space. This process not only eliminates background interference but also accurately quantifies the geometric characteristics of the guardrail surface and any possible deformation or damage. This method greatly improves detection accuracy, reduces false alarm rates, and significantly increases patrol efficiency and coverage. Simultaneously, it provides solid data support for subsequent refined and intelligent greening management.

[0091] In summary, the goal of step S120 is to accurately locate and track key feature points on the guardrail through in-depth analysis of road environment video, calculate their three-dimensional spatial motion trajectories, and ultimately construct a sparse spatial feature flow field that accurately reflects the guardrail's condition. This process is crucial for achieving high-precision detection of road guardrail damage and deformation.

[0092] S130. Fit a virtual ideal trajectory and compare it with the sparse spatial feature flow field to identify geometrically abnormal regions; wherein, the virtual ideal trajectory refers to the ideal motion path of key feature points under no-damage conditions.

[0093] In this embodiment, the geometrically abnormal region refers to the region where there is a significant deviation between the actual observed surface features of the guardrail and the ideal trajectory fitted based on road design specifications (such as the normal distance exceeding a preset threshold), indicating that there may be physical deformation or damage.

[0094] In one embodiment, step S130 described above may include steps S131 to S133.

[0095] S131. Based on road design specifications, determine the parametric geometric manifold equation that the guardrail surface follows within the sliding window, and iteratively optimize the parameters through a robust regression algorithm to approximate the mathematical surface of the sparse spatial characteristic flow field. Adjust the weights according to the residual between the data points and the model to obtain the virtual ideal trajectory.

[0096] Specifically, parametric geometric models include, but are not limited to, cubic polynomials, B-spline curves, or piecewise smooth functions.

[0097] In this embodiment, based on road design specifications, the parametric geometric manifold equations that the guardrail surface follows within the sliding window are determined. .

[0098] like Figure 3 As shown, a robust regression algorithm is used to iteratively optimize the parameters, approximating the mathematical surface of the sparse spatial characteristic flow field generated in the previous step. During this process, the weights are dynamically adjusted based on the residual between the data points and the model. Specifically, data points that conform to the model's trend are assigned high weights, while data points that deviate from the model are assigned low or very low weights, minimizing their impact on the final model.

[0099] S132. Compare the virtual ideal trajectory with the sparse spatial characteristic flow field and calculate the normal distance.

[0100] like Figure 4 As shown, the virtual ideal trajectory obtained in the previous step is compared with the actual sparse spatial characteristic flow field, and the normal distance between the two is calculated. This process helps to quantify the deviation between the actually observed trajectory and the expected ideal trajectory.

[0101] S133. Identify and mark geometrically abnormal regions based on the relationship between the normal distance and the preset deviation threshold.

[0102] Based on the calculated normal distance, it is compared with a preset deviation threshold. If the normal distance of a certain region is significantly greater than the set threshold (e.g., ...), the error is considered. If the area is marked as a geometric anomaly, it is likely that there is physical deformation or damage.

[0103] In this embodiment, a local sliding window manifold constraint mechanism is introduced during the fitting process of the virtual ideal trajectory. This is an advanced technique for identifying and reconstructing geometrically anomalous regions on the guardrail surface. The specific process is as follows:

[0104] The local sliding window manifold constraint first defines the concept of a sliding window that moves along the monitored guardrail surface. Within each sliding window, the system assumes that a normal guardrail surface should follow a specific geometric manifold equation. This means that, in the absence of physical deformation or damage, the surface features of the guardrail should be accurately described by some mathematical model.

[0105] To find the optimal mathematical surface that best represents these spatial flow field characteristics, the system employs a robust regression algorithm. The advantage of this method lies in its high resistance to outliers (i.e., data points deviating from the normal range). First, based on road design specifications, the system establishes an ideal cubic polynomial model for the horizontal trajectory of the guardrail, namely… .here, , , , These are parameters to be determined, which determine the specific shape of the curve.

[0106] Next, a robust regression algorithm is used to iteratively optimize the model. During this process, the algorithm automatically assigns weights to data points based on the residual magnitude. Data points that conform to the model's trend (referred to as Class A points) are given higher weights, while concave points that deviate from the model (Class B points) are given extremely low weights, minimizing their impact on the final curve. In this way, even with a certain degree of noise or outliers, the system can fit an ideal trajectory that ignores the damage—the so-called virtual ideal trajectory.

[0107] The final step is to compare the calculated virtual ideal trajectory with the actually observed trajectory on the guardrail surface (the solid red line). This is done by calculating the normal distance between the two and setting a deviation threshold (e.g., in this case...). This allows us to identify regions where deviations are significant and orientations are anomalous. These regions are considered geometric anomaly zones, i.e., candidate areas where physical deformation or damage may have occurred.

[0108] The entire process not only effectively separates normal and abnormal sections, but also provides a precise method to quantify and locate potential problems on the guardrail surface, which is crucial for timely maintenance and safety assessments.

[0109] In addition, after step S130 above, the following steps are also included:

[0110] like Figure 5 As shown, when the geometrically abnormal region is occluded, perspective transformation and texture restoration are performed on the occluded region to update the geometrically abnormal region.

[0111] Specifically, firstly, when the geometrically abnormal area is obstructed, the obstruction is identified and a corresponding obstruction area mask is generated to obtain the current obstruction area; in this embodiment, the current obstruction area refers to a specific image area within the detected geometrically abnormal area of ​​the road guardrail that cannot be directly observed or analyzed due to obstructed vision caused by roadside vegetation or passing vehicles.

[0112] Next, historical video frames containing the same scene but not occluded are searched and located in the time dimension as reference frames. Specifically, utilizing the information redundancy in the time dimension, a backtracking search is performed in the road environment video to find unoccluded historical video frames of the same location at different time points, thus obtaining the reference frames. In this embodiment, the reference frame is an image frame selected from historical video frames that is at the same location as the current frame but at a different time, and whose corresponding occluded area is not occluded by any object, providing a clear view for recovering the texture of the occluded area.

[0113] Then, the homography matrix is ​​calculated by comparing the common feature points in the current frame and the reference frame. The homography matrix is ​​a mathematical transformation matrix used to describe the perspective relationship between two image planes. It is calculated by matching feature points and can accurately map the clear texture in the reference frame to the occluded area of ​​the current frame, thereby achieving visual elimination of occlusion and complete restoration of the guardrail surface.

[0114] Finally, the calculated homography matrix is ​​applied to perform a perspective transformation on the clear texture in the reference frame corresponding to the current occluded region, and then filled into the corresponding position in the current frame to update the geometrically anomalous region.

[0115] In this embodiment, once the system determines that a certain area is a geometrically anomalous region, if there are occlusions in that area (such as roadside vegetation or passing vehicles), these occlusions must first be identified. This is typically accomplished using semantic segmentation techniques, which involve analyzing the current frame image, distinguishing between occluders and other non-occluded parts, and generating a binary mask for the occluded region. This mask identifies which pixels belong to the occluders and which do not.

[0116] Leveraging the temporal redundancy of road environment video, the system searches forward or backward along the timeline to find unobstructed historical video frames at the same location as reference frames. This process relies on the temporal continuity and spatial consistency of the video stream. Ideally, the system selects the clearest frame closest to the current moment as the reference to minimize additional changes caused by time intervals (such as lighting conditions, weather changes, etc.).

[0117] Once a suitable reference frame is found, the system compares the common feature points between the current frame and the reference frame. By matching these feature points, a transformation relationship between the two frames—the homography matrix—can be calculated. The homography matrix describes how one image is transformed to align with another image, and is particularly important for perspective transformations.

[0118] The final step is to apply the previously calculated homography matrix to perform a perspective transformation on the sharp texture in the reference frame corresponding to the currently occluded area. Perspective transformation corrects image distortion caused by different viewpoints, ensuring that the texture in the reference frame is accurately mapped to the occluded area in the current frame. After the transformation, this sharp texture information is filled back into the corresponding positions in the current frame, thus "erasing" the influence of the occlusion and restoring the true texture of the guardrail surface.

[0119] In summary, when roadside vegetation or passing vehicles cause occlusion, redundant information in the temporal dimension is utilized to address this issue. The process first identifies and generates a mask for the occluded area in the current frame caused by occlusions (such as bushes or cars), thus pinpointing the image portion requiring repair. Next, the system searches the video stream before and after the current frame (e.g., tk or t+k frames) to find the optimal historical reference frame—a clear image at the same physical location that is not occluded, such as the image from 0.5 seconds prior before the vehicle reached the occluded area. Once such a reference frame is found, the system calculates a homography matrix by matching common feature points within the unoccluded area, establishing a precise perspective mapping between the historical and current frames. Finally, based on this homography matrix, the clear texture from the historical frame is accurately projected and filled into the occluded area of ​​the current frame after perspective transformation, achieving a visual "erasure" of these occlusions while restoring the complete texture of the guardrail surface. In this way, even with occlusion, key information in the surveillance video is not lost, maintaining high-quality visual monitoring.

[0120] This approach not only helps to more accurately assess the physical state of the guardrail, but also improves the robustness and reliability of the overall system, especially in dynamic environments. This method effectively combines spatiotemporal information, enhancing the understanding and handling of complex scenarios.

[0121] S140. Detect the physical deformation of the guardrail based on the geometric anomaly area, and analyze the color anomaly and high-frequency texture breakage to obtain the deformation depth and corrosion area.

[0122] In this embodiment, as Figure 6 As shown, the deformation depth refers to the degree of structural deformation caused by physical impact, which is determined by comparing the normal difference between the observed spatial characteristic flow field and the ideal manifold trajectory.

[0123] The rust area refers to the specific area covered by rust or cracks on the surface.

[0124] In one embodiment, step S140 described above may include steps S141 to S142.

[0125] S141. For the geometrically abnormal region, calculate the normal difference between the sparse spatial characteristic flow field and the virtual ideal trajectory to detect and quantify the structural deformation caused by physical impact, and obtain the deformation depth.

[0126] In this embodiment, for each point marked as a geometric anomaly, the system calculates its vertical distance relative to the ideal manifold surface (i.e., the fitted virtual ideal trajectory). This vertical distance is the deformation depth of that point.

[0127] Once the deformation depth at each anomaly point is obtained, the severity of deformation within the entire geometric anomaly region can be assessed based on this data. If the deformation depth at a certain location exceeds a preset safety threshold (e.g., 3 cm), it is determined that deformation was caused by a physical impact at that location.

[0128] S142. Analyze the color anomalies and high-frequency texture fractures in the geometrically abnormal areas, identify surface corrosion and cracks, and determine the corrosion area in conjunction with the set standards.

[0129] In order to accurately measure the rust area, in step S142:

[0130] A convolutional neural network is used to perform detailed analysis of color information and high-frequency textures in the image. For unoccluded frames, the original image is used directly; while for reconstructed images after spatiotemporal restoration, the focus is on detailed analysis within the restored area. By detecting color changes (e.g., reddish-brown representing rust) and surface texture discontinuities, the location and distribution of rust and cracks can be effectively identified. Once the specific locations of rust and cracks are identified, the total rust area is calculated based on their projected area on the image (considering the actual size ratio). If this area exceeds a preset safety threshold (e.g., 2 square centimeters), the area is considered to have a serious corrosion problem and requires maintenance.

[0131] Suppose a section of road guardrail is diagnosed as potentially having physical deformation and surface damage. This is achieved by performing steps S141 and S142 above:

[0132] In S141, the maximum deformation depth of a certain geometric anomaly area reached 5 cm, far exceeding the safety threshold, indicating that this area may have been subjected to a large impact force.

[0133] Further analysis in S142 revealed a rust area of ​​approximately 3 square centimeters in the same region, which also exceeded the safety threshold, indicating that not only was the structure damaged, but the surface protective layer had also failed.

[0134] Ultimately, the system will combine these two indicators to generate an early warning work order, which includes detailed GPS coordinates, damage type (physical deformation and / or corrosion), specific quantitative data (deformation depth, corrosion area), and high-definition screenshots of the site, so as to carry out targeted maintenance and repair work in the future.

[0135] S150. Compare the deformation depth and corrosion area with a safety threshold. When at least one of the deformation depth and corrosion area exceeds the safety threshold, generate an early warning work order.

[0136] In this embodiment, as Figure 7 As shown, the early warning work order includes GPS coordinates, damage type, quantitative data, and on-site screenshots. The quantitative data includes deformation depth and corrosion area.

[0137] In this embodiment, step S150 involves comparing the detected guardrail deformation depth and corrosion area with preset safety thresholds, and generating an early warning work order based on the comparison results. This process ensures effective monitoring of the road guardrail condition, enabling timely detection and handling of potential safety hazards.

[0138] Specifically, based on the actual deformation depth data calculated in the previous steps, the system compares it with a set safety threshold (e.g., 3cm). If the actual deformation depth exceeds this threshold, it indicates that the guardrail may have undergone structural deformation due to a physical impact.

[0139] Similarly, for identified areas of surface rust and cracks, their total area is calculated and compared with a set standard (e.g., 2 square centimeters). If the area exceeds this threshold, it indicates that this section of the railing has a serious corrosion problem.

[0140] When at least one of the deformation depth or corrosion area exceeds the corresponding safety threshold, the system will automatically generate an early warning work order. This means that even if only one indicator exceeds the standard, the early warning mechanism will be triggered, ensuring that all potential risks are promptly monitored and addressed.

[0141] Once it is determined that an alert work order needs to be generated, the work order will contain the following key information:

[0142] GPS coordinates: Accurately record the location of the guardrail where an anomaly occurs, making it easier for subsequent maintenance personnel to quickly locate the problem area.

[0143] Damage type: Clearly indicate whether it is physical deformation, corrosion, or both, to provide specific guidance for repair.

[0144] Quantitative data:

[0145] The specific value of the deformation depth helps assess the extent of damage;

[0146] The specific value of the rusted area is used to determine the severity of the corrosion.

[0147] On-site screenshots: accompanied by clear, high-definition images, visually demonstrating the extent of the damage and assisting technicians in understanding the actual situation and making more accurate repair plans.

[0148] This design not only effectively monitors the condition of road guardrails but also rapidly responds to any changes exceeding safe limits, ensuring the safety and reliability of public facilities. Furthermore, this multi-frame fusion-based method, combined with spatiotemporal manifold reconstruction and spatiotemporal tube texture restoration techniques, not only improves detection accuracy but also overcomes the misjudgment and missed detection problems inherent in traditional methods, achieving high-precision, full-coverage detection of guardrail damage.

[0149] This embodiment aims to address two major problems in existing road guardrail visual inspection technologies: firstly, the tendency to misjudge inherent road geometric curvature as impact deformation; and secondly, missed detections due to obstruction by roadside vegetation or vehicles. To overcome these problems, the method in this embodiment not only works effectively under low-cost video surveillance conditions but also utilizes spatiotemporal manifold reconstruction and spatiotemporal tube texture restoration technologies to eliminate geometric interference and overcome dynamic occlusion, thereby achieving high-precision, full-coverage detection of physical deformation and surface damage of guardrails.

[0150] In this embodiment, the temporal dimension information in the video data is processed to create a spatiotemporal tube structure. The spatiotemporal tube structure is an intermediate data structure used to generate sparse spatial feature flow fields. This structure can help the system accurately track the position and state of guardrail feature points when facing dynamic changes (such as vehicle movement and vegetation swaying).

[0151] The module reconstructs the ideal shape of the guardrail using a mathematical model, a so-called "virtual ideal manifold." This module can identify and eliminate errors caused by normal geometric variations in actual observation, thus focusing on detecting real physical deformation or damage.

[0152] By analyzing the data from the two outputs mentioned above, this module can accurately detect any physical deformation or surface damage on the guardrail and quantify it.

[0153] By employing spatiotemporal manifold reconstruction technology, the method in this embodiment can effectively identify and ignore normal geometric curvatures, thereby reducing the possibility of false positives. Multi-frame fusion technology allows the system to utilize information from historical frames to fill in occluded portions in the current frame, significantly improving the comprehensiveness and accuracy of detection. The method in this embodiment can not only be used for detecting road guardrails, but its core technology can also be applied to object monitoring in other scenarios, such as monitoring the status of green belts, thus promoting intelligent urban management.

[0154] Highly efficient and accurate detection tasks can be accomplished using low-cost video surveillance equipment, reducing the need for expensive sensors and lowering maintenance and manual inspection costs.

[0155] In summary, the method presented in this embodiment not only addresses the main shortcomings of traditional visual inspection technologies for road guardrails but also brings several added benefits. By integrating advanced image processing algorithms and intelligent data analysis methods, the method in this embodiment makes the management and maintenance of road facilities more efficient, economical, and accurate. This innovative methodology provides strong support for the safety and security of future urban infrastructure.

[0156] The aforementioned multi-frame fusion-based method for detecting road guardrail damage and deformation acquires road environment video, analyzes and identifies key feature points of the guardrail, calculates its three-dimensional spatial motion trajectory to form a sparse spatial feature flow field, and then compares it with a virtual ideal trajectory under no-damage conditions to identify geometrically abnormal areas. It also combines color anomaly and high-frequency texture fracture analysis to quantify the degree of physical deformation and corrosion. Finally, it compares the deformation depth and corrosion area with a safety threshold to generate an early warning work order. This method can achieve high-precision, full-coverage detection of guardrail physical deformation and surface damage using low-cost video surveillance equipment, effectively improving the efficiency and safety of road guardrail inspection work. It also reduces reliance on expensive sensors and the cost of manual inspection, providing an economical and efficient solution for ensuring the safety of road facilities.

[0157] Figure 8 This is a schematic block diagram of a road guardrail damage and deformation detection system 300 based on multi-frame fusion provided in an embodiment of the present invention. Figure 8 As shown, corresponding to the above-described road guardrail damage and deformation detection method based on multi-frame fusion, the present invention also provides a road guardrail damage and deformation detection system 300 based on multi-frame fusion. This road guardrail damage and deformation detection system 300 based on multi-frame fusion includes a unit for executing the above-described road guardrail damage and deformation detection method based on multi-frame fusion, and the system can be configured in a server. Specifically, please refer to... Figure 8The road guardrail damage and deformation detection system 300 based on multi-frame fusion includes an acquisition unit 301, a feature flow field calculation unit 302, a fitting unit 303, a dual-flow damage determination unit 304, and a generation unit 305.

[0158] The acquisition unit 301 is used to acquire road environment video; the feature flow field calculation unit 302 is used to analyze the road environment video, lock and track key feature points on the guardrail, calculate the motion trajectory of the key feature points in three-dimensional space, and form a sparse spatial feature flow field; the fitting unit 303 is used to fit a virtual ideal trajectory and compare it with the sparse spatial feature flow field to identify geometrically abnormal areas; wherein, the virtual ideal trajectory refers to the ideal motion path of the key feature points under no-damage conditions; the dual-flow damage determination unit 304 is used to detect the physical deformation of the guardrail based on the geometrically abnormal areas, and analyze color anomalies and high-frequency texture fractures to obtain deformation depth and corrosion area; the generation unit 305 is used to compare the deformation depth and corrosion area with a safety threshold, and generate an early warning work order when at least one of the deformation depth and corrosion area exceeds the safety threshold.

[0159] In one embodiment, the fitting unit 303 includes:

[0160] The virtual ideal trajectory fitting subunit is used to determine the parametric geometric manifold equation followed by the guardrail surface within the sliding window based on road design specifications, and to iteratively optimize the parameters through a robust regression algorithm to approximate the mathematical surface of the sparse spatial characteristic flow field. The weights are adjusted according to the residual between the data points and the model to obtain the virtual ideal trajectory. The calculation subunit is used to compare the virtual ideal trajectory with the sparse spatial characteristic flow field and calculate the normal distance. The identification subunit is used to identify and mark geometrically abnormal regions based on the relationship between the normal distance and a preset deviation threshold.

[0161] In one embodiment, the road guardrail damage and deformation detection system 300 based on multi-frame fusion further includes:

[0162] The repair unit is used to perform perspective transformation and texture repair on the occluded area when the geometrically abnormal area is occluded, so as to update the geometrically abnormal area.

[0163] In one embodiment, the repair unit includes:

[0164] An occlusion recognition subunit is used to identify occlusion objects and generate corresponding occlusion area masks when occlusion occurs in the geometrically anomalous region, so as to obtain the current occluded region; a reference frame determination subunit is used to search and locate historical video frames containing the same scene but not occluded as reference frames in the time dimension; a comparison subunit is used to compare common feature points in the current frame and the reference frame and calculate the homography matrix; a transformation filling subunit is used to apply the calculated homography matrix, perform perspective transformation on the clear texture in the reference frame corresponding to the current occluded region, and fill it into the corresponding position in the current frame to update the geometrically anomalous region.

[0165] In one embodiment, the reference frame determination subunit is used to utilize the information redundancy in the time dimension to perform a backtracking search in the road environment video to find unobstructed historical video frames of the same location at different time points, so as to obtain a reference frame.

[0166] In one embodiment, the dual-flow damage determination unit 304 includes:

[0167] The deformation depth determination subunit is used to calculate the normal difference between the sparse spatial characteristic flow field and the virtual ideal trajectory in the geometrically abnormal region, so as to detect and quantify the structural deformation caused by physical impact and obtain the deformation depth; the area determination subunit is used to analyze the color anomaly and high-frequency texture fracture in the geometrically abnormal region, identify the surface corrosion and cracks, and determine the corrosion area in combination with the set standard.

[0168] In one embodiment, the characteristic flow field calculation unit 302 includes:

[0169] The system includes a feature point recognition subunit for filtering out non-guardrail areas from the road environment video and identifying key feature points on the guardrail surface; a tracking subunit for tracking the position changes of the key feature points in consecutive frames and calculating the pixel displacement vectors of the key feature points; a displacement calculation subunit for calculating the physical displacement of the camera between two frames; a trajectory vector construction subunit for combining the principles of multi-view geometric triangulation, pixel displacement vectors, and the physical displacement to calculate the three-dimensional coordinates of the key feature points and construct the motion trajectory of the key feature points in three-dimensional space to obtain the trajectory vectors of the key feature points; and a unification subunit for unifying the trajectory vectors of all key feature points into the same spatial coordinate system to form a sparse spatial feature flow field describing the physical properties of the guardrail surface.

[0170] In one embodiment, a displacement calculation subunit is used to acquire camera pose change data to obtain physical displacement; wherein the pose change data is calculated by a vehicle kinematics model and / or obtained by solving the features of consecutive frame images through visual odometry.

[0171] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the road guardrail damage and deformation detection system 300 based on multi-frame fusion and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0172] The aforementioned road guardrail damage and deformation detection system 300 based on multi-frame fusion can be implemented as a computer program, which can, for example... Figure 9 It runs on the computer device shown.

[0173] Please see Figure 9 , Figure 9 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0174] See Figure 9 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0175] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a road guardrail damage and deformation detection method based on multi-frame fusion.

[0176] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0177] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a road guardrail damage and deformation detection method based on multi-frame fusion.

[0178] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0179] The processor 502 is used to run the computer program 5032 stored in the memory to implement all the steps of the road guardrail damage and deformation detection method based on multi-frame fusion.

[0180] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0181] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0182] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all the steps of the road guardrail damage and deformation detection method based on multi-frame fusion.

[0183] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0184] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0185] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0186] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0187] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0188] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting damage and deformation of road guardrails based on multi-frame fusion, characterized in that, include: Acquire road environment video; The road environment video is analyzed to identify and track key feature points on the guardrail, calculate the motion trajectory of the key feature points in three-dimensional space, and form a sparse spatial feature flow field. Fit a virtual ideal trajectory and compare it with the sparse spatial feature flow field to identify geometrically anomalous regions; wherein, the virtual ideal trajectory refers to the ideal motion path of key feature points under no-damage conditions; The physical deformation of the guardrail is detected based on the geometric anomaly area, and the color anomaly and high-frequency texture fracture are analyzed to obtain the deformation depth and corrosion area. The deformation depth and corrosion area are compared with a safety threshold. When at least one of the deformation depth and corrosion area exceeds the safety threshold, an early warning work order is generated.

2. The method for detecting road guardrail damage and deformation based on multi-frame fusion according to claim 1, characterized in that, The fitting of the virtual ideal trajectory and the comparison with the sparse spatial characteristic flow field to identify geometrically anomalous regions include: Based on road design specifications, the parametric geometric manifold equations that the guardrail surface follows within the sliding window are determined, and the parameters are iteratively optimized using a robust regression algorithm to approximate the mathematical surface of the sparse spatial characteristic flow field. The weights are adjusted according to the residual between the data points and the model to obtain a virtual ideal trajectory. The data points are the key feature points that have been transformed by the sparse spatial characteristic flow field and iteratively optimized by the robust regression algorithm. By comparing the virtual ideal trajectory with the sparse spatial characteristic flow field, the normal distance is calculated; Geometrically abnormal regions are identified and marked based on the relationship between the normal distance and a preset deviation threshold.

3. The method for detecting road guardrail damage and deformation based on multi-frame fusion according to claim 1, characterized in that, The fitting of the virtual ideal trajectory and comparison with the sparse spatial feature flow field to identify geometrically anomalous regions; wherein, the virtual ideal trajectory refers to the ideal motion path of key feature points under damage-free conditions, and further includes: When the geometrically abnormal region is occluded, perspective transformation and texture restoration are performed on the occluded region to update the geometrically abnormal region.

4. The method for detecting road guardrail damage and deformation based on multi-frame fusion according to claim 1, characterized in that, When the geometrically anomalous region is occluded, perspective transformation and texture restoration are performed on the occluded region to update the geometrically anomalous region, including: When the geometrically abnormal region is occluded, the occluder is identified and a corresponding occlusion region mask is generated to obtain the current occluded region; Search and locate historical video frames containing the same scene but not occluded as reference frames in the time dimension; Compare the common feature points in the current frame and the reference frame to calculate the homography matrix; The calculated homography matrix is ​​applied to the clear texture in the reference frame corresponding to the current occluded region. After perspective transformation, the texture is filled into the corresponding position in the current frame to update the geometrically anomalous region.

5. The method for detecting road guardrail damage and deformation based on multi-frame fusion according to claim 4, characterized in that, The step of searching for and locating historical video frames containing the same scene but not occluded as reference frames in the time dimension includes: By utilizing the information redundancy in the time dimension, a backtracking search is performed in the road environment video to find unobstructed historical video frames of the same location at different time points, so as to obtain reference frames.

6. The method for detecting road guardrail damage and deformation based on multi-frame fusion according to claim 1, characterized in that, The method of detecting physical deformation of the guardrail based on the geometric anomaly area, and analyzing color anomalies and high-frequency texture fractures to obtain deformation depth and corrosion area includes: For the geometrically anomalous region, the normal difference between the sparse spatial characteristic flow field and the virtual ideal trajectory is calculated to detect and quantify the structural deformation caused by physical impact, and to obtain the deformation depth. The analysis of color anomalies and high-frequency texture fractures in the geometrically abnormal areas identifies surface corrosion and cracks, and determines the corrosion area in conjunction with set standards.

7. The method for detecting road guardrail damage and deformation based on multi-frame fusion according to claim 1, characterized in that, The analysis of the road environment video, the identification and tracking of key feature points on the guardrail, and the calculation of the motion trajectory of the key feature points in three-dimensional space to form a sparse spatial feature flow field include: The video of the road environment is filtered to include areas outside the guardrail, and key feature points on the guardrail surface are identified. Track the position changes of the key feature points in consecutive frames and calculate the pixel displacement vector of the key feature points; Calculate the physical displacement of the camera between two frames; Combining the principles of multi-view geometric triangulation, pixel displacement vector, and physical displacement, the three-dimensional coordinates of key feature points are calculated, and the motion trajectory of key feature points in three-dimensional space is constructed to obtain the trajectory vector of key feature points. The trajectory vectors of all key feature points are unified into the same spatial coordinate system to form a sparse spatial feature flow field that describes the physical properties of the guardrail surface.

8. The method for detecting road guardrail damage and deformation based on multi-frame fusion according to claim 7, characterized in that, The calculation of the camera's physical displacement between two frames includes: Acquire camera pose change data to obtain physical displacement; wherein the pose change data is calculated by a vehicle kinematics model and / or obtained by solving the features of consecutive frames of images through visual odometry.

9. The method for detecting road guardrail damage and deformation based on multi-frame fusion according to claim 1, characterized in that, The early warning work order includes GPS coordinates, damage type, quantitative data, and on-site screenshots. The quantitative data includes deformation depth and corrosion area.

10. A road guardrail damage and deformation detection system based on multi-frame fusion, characterized in that, include: The acquisition unit is used to acquire video of the road environment. The feature flow field calculation unit is used to analyze the road environment video, lock and track key feature points on the guardrail, calculate the motion trajectory of the key feature points in three-dimensional space, and form a sparse spatial feature flow field. A fitting unit is used to fit a virtual ideal trajectory and compare it with the sparse spatial feature flow field to identify geometrically abnormal regions; wherein, the virtual ideal trajectory refers to the ideal motion path of key feature points under no-damage conditions; The dual-flow damage determination unit is used to detect the physical deformation of the guardrail based on the geometric anomaly area, and to analyze the color anomaly and high-frequency texture fracture to obtain the deformation depth and corrosion area. The generation unit is used to compare the deformation depth and corrosion area with a safety threshold, and generate an early warning work order when at least one of the deformation depth and corrosion area exceeds the safety threshold.

Images