Guardrail bolt missing detection and positioning method and system based on visual transformation and reasoning

Abstract

This invention discloses a method and system for detecting and locating missing guardrail bolts based on visual transformation and reasoning, belonging to the field of guardrail detection technology. The invention calculates the object-side coverage and overlap regions of an image based on camera parameters, object distance, and trigger interval, and then crops and stitches the images to obtain a stitched guardrail image. The stitched image is input into a bolt detection network, which adaptively injects global and multi-scale semantic features extracted by a self-supervised pre-trained visual transformer into a first-stage target detection sub-network through a feature fusion node to detect bolt targets and suspected missing locations. Contextual reasoning is performed based on the bolt design spacing and arrangement relationship to confirm the missing bolts. The road mileage and spatial coordinates of the missing location are calculated using pose data, and severity classification and warning are performed. This invention achieves efficient deduplication, high-precision small target detection, and accurate localization, significantly improving the automation level and reliability of guardrail bolt missing detection.

Description

Technical Field

[0001] This invention relates to the field of guardrail inspection technology, and in particular to a method and system for detecting and locating missing guardrail bolts based on visual transformation and reasoning. Background Technology

[0002] Corrugated beam steel guardrails are critical protective facilities for highway traffic safety, and their structural integrity directly affects the protection of occupants and vehicles in the event of an accident. Guardrails typically consist of posts, corrugated beams, and connecting bolts, with the bolts playing a crucial role in connection and force transmission. Missing or loose bolts significantly weaken the integrity and impact resistance of the guardrail, potentially leading to its disintegration or failure in the event of a vehicle collision, causing serious secondary accidents. Therefore, regularly, efficiently, and accurately inspecting the condition of guardrail bolts is of great importance for ensuring road safety and fulfilling maintenance management responsibilities.

[0003] Currently, the inspection of guardrail bolt condition mainly relies on the following two methods:

[0004] Manual inspection method: Maintenance personnel inspect along the road by vehicle or on foot, visually observing and recording the status of missing bolts. This method has significant drawbacks: (a) Low efficiency: The inspection speed is slow and it is difficult to cover a large-scale road network; (b) Poor accuracy and consistency: The detection results are highly dependent on the experience, attention and physical condition of the personnel, resulting in a high rate of missed detections and false detections, and the data is difficult to quantify; (c) High safety risks: Personnel working next to or in the lanes of high-speed traffic face significant threats to their personal safety; (d) High cost: It consumes a lot of manpower, and the data recording, sorting and analysis are cumbersome.

[0005] Automated visual inspection technology: In recent years, with the development of computer vision technology, methods based on vehicle-mounted cameras to acquire images and perform algorithm analysis have emerged. However, in practical engineering applications, there is a lack of effective solutions for detecting missing guardrail bolts, specifically manifested in the following ways: (a) images continuously acquired by vehicle-mounted cameras have a large number of overlapping areas, resulting in the same bolt being detected and counted multiple times; (b) bolts are small in size and occupy a small proportion in the image, leading to low detection accuracy of traditional detection algorithms; (c) detection robustness is insufficient due to factors such as changes in lighting, weather conditions, and occlusion; (d) it is difficult to achieve precise positioning of missing bolts, failing to meet the needs of maintenance and repair; and (e) existing technologies typically require complex hardware systems and large amounts of computing resources, making them difficult to promote and apply in practical engineering.

[0006] In summary, there is an urgent need for a technical solution that can overcome the above-mentioned shortcomings, achieve efficient deduplication and splicing, high-precision small target detection, strong environmental adaptability, accurate positioning output, and make full use of structural priors for automated detection and positioning of missing guardrail bolts, so as to improve the level of intelligence and work efficiency of highway facility maintenance. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for detecting and locating missing guardrail bolts based on visual transformation and reasoning. It combines geometric parameter deduplication and splicing, fusion of self-supervised large visual model and single-stage target detection network, fusion of multi-view results, and a systematic solution for multi-sensor positioning and severity assessment to achieve automatic detection, accurate positioning and graded early warning of missing guardrail bolts, thereby improving the automation and intelligence level of road traffic safety facility inspection.

[0008] To achieve the above objectives, this application proposes a method for detecting and locating missing guardrail bolts based on visual transformation and reasoning, comprising the following steps:

[0009] Step S1: Control the inspection vehicle to travel along the guardrail, and use at least one camera on the inspection vehicle facing the guardrail to collect a sequence of guardrail images at a preset fixed-distance trigger interval, and simultaneously record the sensor pose information of the inspection vehicle when each frame of image is collected.

[0010] Step S2: Based on the camera's intrinsic parameters, pixel size, object distance from the camera to the guardrail, and the fixed-distance trigger interval, calculate the object coverage area of ​​each frame of guardrail image in the guardrail extension direction and the object overlap area between adjacent frame guardrail images; based on the object overlap area, crop the adjacent frame guardrail images and stitch them together along the guardrail extension direction to obtain a stitched guardrail image.

[0011] Step S3: Input the spliced ​​guardrail image into the bolt detection network for detection to obtain detection results containing bolt targets and suspected missing locations; wherein, the bolt detection network includes a self-supervised pre-trained visual transformer feature extraction sub-network and a single-stage detection backbone network, and a feature fusion module is set in the input end of the single-stage detection backbone network and at least one backbone intermediate layer;

[0012] Step S4: Based on the designed spacing of the guardrail bolts along the extension direction of the guardrail and the preset positional relationship on the guardrail cross section, perform context-aware reasoning on the detection results, identify and confirm the location of the missing bolts, and obtain the bolt missing detection results;

[0013] Step S5: Based on the image coordinates of each missing bolt in the spliced ​​guardrail image from the bolt missing detection results, and the corresponding sensor pose information, calculate the road mileage and spatial coordinates of each missing bolt; and classify the severity according to the number and distribution of missing bolts to generate a detection report.

[0014] As a further solution, step S1 is based on the odometer and / or wheel speed sensor to achieve fixed-distance triggering. When the inspection vehicle travels a preset interval distance, the camera is triggered to acquire a frame of guardrail image, and the acquisition time of the frame of guardrail image is bound to the sensor pose information output by the multi-sensor fusion module.

[0015] As a further solution, step S2 is performed through the following specific steps:

[0016] Step S21: Calculate the horizontal field of view of the camera based on its focal length, pixel size, and number of horizontal pixels;

[0017] Step S22: Calculate the object coverage width W of a single frame guardrail image in the guardrail extension direction based on the horizontal field of view and the object distance;

[0018] Step S23: Using the fixed-distance trigger interval L as the spacing between the acquisition positions of adjacent frame images, calculate the object-side overlap length L_overlap=max(0,WL) of the guardrail images in adjacent frames;

[0019] Step S24: Based on the object overlap length, the main overlapping parts of two adjacent guardrail images are cropped, and the images are spliced ​​together after retaining a preset length of overlap tolerance zone.

[0020] As a further solution, in step S24, within the overlapping fault tolerance zone, local matching is performed on two adjacent guardrail images based on a preset bolt template, and the position of the next guardrail image in the splicing direction is finely adjusted according to the matched bolt correspondence before splicing.

[0021] As a further solution, in step S3, the visual transformer feature extraction subnetwork is a visual transformer model pre-trained on a large-scale unlabeled image dataset using a self-supervised learning method; the single-stage detection backbone network is a single-stage object detection network based on a convolutional neural network.

[0022] The feature fusion module is configured to perform a two-point feature injection operation, including:

[0023] At the input end of the single-stage detection backbone network, the global semantic features extracted by the visual transformer feature extraction subnetwork are injected.

[0024] In the middle layer of the backbone of the single-stage detection backbone network, multi-scale semantic features extracted by the visual transformer feature extraction subnetwork are injected.

[0025] The injected semantic features are fused with the features of the single-stage detection backbone network itself through a learnable adaptive weighting mechanism.

[0026] The detection head outputs the bolt detection frame, category, confidence level, and suspected missing location.

[0027] As a further solution, step S4 is performed through the following specific steps:

[0028] Step S41: Based on the guardrail design parameters, obtain the nominal spacing d of the bolts along the extension direction of the guardrail;

[0029] Step S42: Sort the bolt targets in the detection results according to the extension direction of the guardrail to form a bolt sequence;

[0030] Step S43: Traverse the bolt sequence. If the distance between two adjacent bolt targets is greater than (1+α)·d, where α is a preset tolerance coefficient, then mark the position between the two adjacent bolt targets as a missing candidate position.

[0031] Step S44: Based on the preset vertical arrangement of bolts on the guardrail cross section, verify the missing candidate positions and confirm the final missing bolt positions.

[0032] As a further solution, in step S1, the camera includes at least two of the following: a horizontal-view camera, a top-view camera, and a bottom-view camera, to acquire guardrail image sequences from different perspectives.

[0033] In step S2, geometric deduplication and stitching are performed on the guardrail image sequences from each viewpoint to obtain stitched guardrail images from multiple viewpoints.

[0034] In step S3, the spliced ​​guardrail images from each viewpoint are input into the bolt detection network for detection to obtain the detection results for each viewpoint.

[0035] Step S4 further includes multi-view result fusion: mapping the detection results of each view to a unified guardrail coordinate system. When multiple views detect bolts at the same object position, the confidence of the bolt detection result is increased; when multiple views do not detect bolts at the same expected bolt position, the position is confirmed as a missing bolt position.

[0036] As a further solution, after step S4, the method also includes a result-level deduplication step:

[0037] The bolt detection results in adjacent multi-frame images or spliced ​​segments are mapped onto a one-dimensional coordinate system of the guardrail based on road mileage;

[0038] The clustering radius is set according to the nominal bolt spacing and the system positioning accuracy. Multiple bolt detection results that are less than the clustering radius on the one-dimensional coordinate system are clustered and merged into a single detection result corresponding to the same physical bolt to eliminate duplicate counting.

[0039] As a further solution, the method also includes an online model update step:

[0040] During system operation, samples with detection confidence levels in the preset middle range, inconsistent detection results at different times, or those that have been manually reviewed and corrected are collected to build a difficult sample library;

[0041] The bolt detection network is incrementally trained periodically using the problematic sample library to generate an updated detection model.

[0042] Based on the statistical distribution of the confidence level of bolt targets in the current detection task, the confidence threshold used to determine whether a bolt exists or is missing is dynamically adjusted.

[0043] On the other hand, the present invention also provides a guardrail bolt missing detection and positioning system based on visual transformation and reasoning, comprising:

[0044] Image acquisition unit: controls the inspection vehicle to travel along the guardrail, and acquires a sequence of guardrail images at a preset fixed-distance trigger interval through at least one camera set on the inspection vehicle facing the guardrail, and synchronously records the sensor pose information of the inspection vehicle when each frame of image is acquired.

[0045] Geometric deduplication and stitching unit: Based on the camera's intrinsic parameters, pixel size, object distance from the camera to the guardrail, and the fixed-distance trigger interval, calculate the object coverage area of ​​each frame of guardrail image in the guardrail extension direction and the object overlap area between adjacent frame guardrail images; based on the object overlap area, crop the adjacent frame guardrail images and stitch them along the guardrail extension direction to obtain the stitched guardrail image.

[0046] Bolt detection unit: The spliced ​​guardrail image is input into the bolt detection network for detection to obtain detection results containing bolt targets and suspected missing locations; wherein, the bolt detection network includes a self-supervised pre-trained visual transformer feature extraction sub-network and a single-stage detection backbone network, and a feature fusion module is set in the input end of the single-stage detection backbone network and at least one intermediate layer of the backbone;

[0047] Contextual reasoning unit: Based on the designed spacing of the guardrail bolts along the extension direction of the guardrail and the preset positional relationship on the guardrail cross section, the detection results are subjected to contextual reasoning to identify and confirm the location of the missing bolts, and the bolt missing detection results are obtained.

[0048] Location and evaluation unit: Based on the image coordinates of each missing bolt in the spliced ​​guardrail image in the bolt missing detection results, and the corresponding sensor pose information, calculate the road mileage and spatial coordinates of each missing bolt; and classify the severity according to the number and distribution of missing bolts, and generate a detection report.

[0049] Compared with related technologies, the present invention provides a method and system for detecting and locating missing guardrail bolts based on visual transformation and reasoning, which has the following advantages:

[0050] 1. High detection efficiency and accuracy: Image deduplication and stitching based on camera geometric parameters avoids complex pixel-level matching, resulting in fast processing speed and eliminating image redundancy at the source, laying the foundation for accurate subsequent statistics. Combining the powerful semantic understanding capabilities of the self-supervised visual transformer feature extraction sub-network with the efficient detection capabilities of the single-stage detection network, deep fusion is achieved through dual-point feature injection, significantly improving the detection accuracy and recall rate of small-sized bolts and their missing states.

[0051] 2. Strong environmental adaptability and robustness: The system introduces a multi-view detection result fusion mechanism, utilizing the complementarity of information from different perspectives to effectively address issues such as occlusion and reflection under single-view conditions. Simultaneously, contextual reasoning based on the spacing and arrangement rules of guardrail bolts allows for the verification and correction of detection results, effectively reducing false positives and false negatives under complex lighting and soiled backgrounds, significantly enhancing the overall robustness of the system.

[0052] 3. Accurate positioning and good engineering practicality: By fusing image detection results with information from multiple sensors such as GPS, IMU, and odometer, the system achieves precise mapping of missing bolts from image coordinates to road mileage and spatial coordinates, directly guiding maintenance and repair. The entire solution can be implemented based on conventional vehicle-mounted hardware and edge computing equipment, eliminating the need for expensive high-precision stereo vision systems, thus keeping costs under control and facilitating integration and promotion within existing maintenance systems.

[0053] 4. Continuous optimization capability: The statistical accuracy is further guaranteed by the result-level deduplication mechanism, and an active learning and threshold adaptation strategy based on a difficult sample library is designed, which enables the system to continuously learn new scenarios in long-term operation, dynamically optimize model performance and decision threshold, and maintain the advanced and stable detection capability. Attached Figure Description

[0054] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0056] Figure 1 A schematic diagram illustrating the steps of a method for detecting and locating missing guardrail bolts based on visual transformation and reasoning provided by the present invention;

[0057] Figure 2 This is a schematic diagram of the guardrail bolt missing detection device provided by the present invention;

[0058] Figure 3 A schematic diagram of the image deduplication and stitching method based on geometric parameters provided by the present invention;

[0059] Figure 4 This is a schematic diagram of the bolt detection network structure provided by the present invention;

[0060] Figure 5 This is a schematic diagram of the missing bolt location and severity assessment process of the present invention;

[0061] Figure 6 A schematic diagram of a guardrail bolt missing detection and positioning system based on visual transformation and reasoning provided by the present invention;

[0062] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0064] Please see Figure 1 This embodiment provides a method for detecting and locating missing guardrail bolts based on visual transformation and reasoning, including the following steps:

[0065] Step S1: Control the inspection vehicle to travel along the guardrail, and use at least one camera on the inspection vehicle facing the guardrail to collect a sequence of guardrail images at a preset fixed-distance trigger interval, and simultaneously record the sensor pose information of the inspection vehicle when each frame of image is collected.

[0066] Step S2: Based on the camera's intrinsic parameters, pixel size, object distance from the camera to the guardrail, and the fixed-distance trigger interval, calculate the object coverage area of ​​each frame of guardrail image in the guardrail extension direction and the object overlap area between adjacent frame guardrail images; based on the object overlap area, crop the adjacent frame guardrail images and stitch them together along the guardrail extension direction to obtain a stitched guardrail image.

[0067] Step S3: Input the spliced ​​guardrail image into the bolt detection network for detection to obtain detection results containing bolt targets and suspected missing locations; wherein, the bolt detection network includes a self-supervised pre-trained visual transformer feature extraction sub-network and a single-stage detection backbone network, and a feature fusion module is set in the input end of the single-stage detection backbone network and at least one backbone intermediate layer;

[0068] Step S4: Based on the designed spacing of the guardrail bolts along the extension direction of the guardrail and the preset positional relationship on the guardrail cross section, perform context-aware reasoning on the detection results, identify and confirm the location of the missing bolts, and obtain the bolt missing detection results;

[0069] Step S5: Based on the image coordinates of each missing bolt in the spliced ​​guardrail image from the bolt missing detection results, and the corresponding sensor pose information, calculate the road mileage and spatial coordinates of each missing bolt; and classify the severity according to the number and distribution of missing bolts to generate a detection report.

[0070] It should be noted that this invention fully utilizes the geometric priors and multi-sensor information collected by the vehicle to achieve rapid deduplication and stitching of guardrail images without the need for complex feature matching. By combining a self-supervised visual large model with a single-stage detection network, the detection accuracy of small bolts and missing states is improved, and the precise location and severity assessment of missing bolts are achieved, making it highly applicable to engineering projects.

[0071] In a specific embodiment, such as Figure 2 The guardrail bolt missing detection device shown is installed on the inspection vehicle 101. The inspection vehicle 101 can be a highway maintenance vehicle, an inspection vehicle, or other vehicles with onboard power supply. When it travels along the road, a multi-view camera system 102, an adaptive lighting system 103, and an intelligent anti-shake gimbal 104 are deployed on the side closest to the guardrail. A multi-sensor fusion module 105 and an edge computing unit 106 are installed inside the vehicle, and data is exchanged with the management platform 107 through wired or wireless networks.

[0072] The main components of the guardrail bolt missing detection device include:

[0073] Multi-view camera system 102: includes a horizontal view camera, a top view camera, and a bottom view camera; recommended configuration specifications: resolution of 2048×1536 pixels; frame rate of 30fps; lens focal length of 12mm; pixel size of 3.45μm.

[0074] Adaptive supplemental lighting system 103: includes white LED, infrared LED, and ultraviolet LED, and monitors lighting conditions in real time through an ambient light sensor; recommended configuration parameters: automatic adjustment range of supplemental lighting intensity: 0-1000 lmo; response time: <100 ms.

[0075] Intelligent Anti-shake Gimbal 104: Employs dual-axis gyroscope stabilization; Recommended specifications: Anti-shake angle: ±15°; Control accuracy: 0.1°; Response frequency: 50Hz.

[0076] Multi-sensor fusion module 105: Employs Kalman filter data fusion; Recommended configuration specifications: GPS module: positioning accuracy 1-5 meters; IMU module: attitude accuracy 0.1°; Odometer: resolution 1000 pulses / revolution;

[0077] Edge computing unit 106: It adopts a GPU acceleration module and carries the DINO-YOLO detection network to perform model optimization quantization and pruning.

[0078] Management Platform 107: It has data storage, report generation, early warning system and GIS visualization system.

[0079] The workflow of the guardrail bolt missing detection device includes:

[0080] 1. Image acquisition stage: The inspection vehicle travels at a speed of 60-100 km / h. The odometer triggers the camera to acquire one frame of image every 2 meters. Multiple sensors synchronously record pose data, and the supplementary lighting system automatically adjusts according to the ambient light.

[0081] 2. Data processing stage: Geometric deduplication and stitching processing speed >30fps, bolt detection accuracy >95%, multi-view fusion time <100ms, positioning evaluation accuracy <0.5 meters;

[0082] 3. Results Output Stage: Generate a detection report including location, quantity, and severity; visualize the results using a GIS map; include early warning information for severe deficiencies; and provide maintenance recommendations including suggested remedial measures.

[0083] The guardrail bolt missing detection device is based on the odometer and / or wheel speed sensor to achieve fixed-distance triggering. When the inspection vehicle travels a preset interval distance, the camera is triggered to collect a frame of guardrail image, and the collection time of the frame of guardrail image is bound with the sensor pose information output by the multi-sensor fusion module.

[0084] Furthermore, such as Figure 3 As shown, the image deduplication and stitching method based on geometric parameters provided in this embodiment specifically includes the following steps:

[0085] Step S21: Calculate the horizontal field of view of the camera based on its focal length, pixel size, and number of horizontal pixels;

[0086] Specifically, the camera intrinsic parameters to be read include: focal length f=12mm, pixel size s=3.45μm, where f is the focal length of the camera lens and s is the pixel size; the object distance D is obtained: the distance from the camera to the guardrail, obtained through laser ranging or a preset value; the trigger interval L=2 meters (which can be adjusted according to actual needs), where L is the object-side interval length triggered by the camera (fixed-distance trigger interval).

[0087] Calculate the horizontal field of view (FOV):

[0088]

[0089] Where FOV is the horizontal field of view of the camera, f is the focal length of the camera lens, s is the pixel size, and N is the number of pixels in the horizontal direction. For a 2048×1536 camera, N=2048.

[0090] Step S22: Calculate the object coverage width W of a single frame guardrail image in the guardrail extension direction based on the horizontal field of view and the object distance;

[0091] Specifically, calculate the object coverage width W:

[0092]

[0093] In the formula: W is the object coverage width along the guardrail direction, D is the object distance from the camera to the guardrail, and FOV is the horizontal field of view of the camera; when D=3 meters, W≈2.08 meters.

[0094] Step S23: Using the fixed-distance trigger interval L as the spacing between the acquisition positions of adjacent frame images, calculate the object-side overlap length L_overlap=max(0,WL) of the guardrail images in adjacent frames;

[0095] Specifically, Frame1 covers the range [0, W], which is the length of W covered along the guardrail direction starting from the current fixed-distance trigger position; Frame2 covers the range [L, L+W], which is the length of W covered along the guardrail direction starting from the next fixed-distance trigger position L, where L is the fixed-distance trigger interval and W is the object coverage width.

[0096] Therefore, the overlap length of the two images in the direction of the guardrail is approximately W-L. There is an overlap when W is greater than L, otherwise the overlap length is 0. Calculated by L_overlap=max(0,WL), when W=2.08 meters and L=2 meters, the overlap length is approximately 0.08 meters.

[0097] Step S24: Based on the object overlap length, the main overlapping parts of two adjacent guardrail images are cropped, and the images are spliced ​​together after retaining a preset length of overlap tolerance zone.

[0098] Specifically, the Frame1 reserved area mainly retains the non-overlapping section near the starting point, and only retains a preset length in the overlapping area with Frame2 to ensure a smooth transition of splicing.

[0099] Frame2 Reserved Area: Mainly retains the non-overlapping section near the end point, and only retains a preset length in the overlapping area with Frame1, and is spatially connected with the reserved area of ​​Frame1.

[0100] Direct splicing avoids pixel-level matching, and its splicing error is less than 1 pixel.

[0101] Furthermore, in step S24, within the overlapping fault tolerance zone, local matching is performed on two adjacent guardrail images based on a preset bolt template, and the position of the next guardrail image in the splicing direction is finely adjusted according to the matched bolt correspondence, before splicing.

[0102] Specifically, retain a 10-20cm tolerance overlap area and perform quick alignment of bolt templates (e.g., circular and hexagonal templates).

[0103] For local matching fine-tuning (e.g., fine-tuning 1-2 pixels for 2-3 bolt positions), a matching success rate of >98% can be achieved.

[0104] The image deduplication and stitching method based on the above geometric parameters has a time complexity of O(n), resulting in low computational complexity; it can process a single frame image in less than 10ms. It boasts fast stitching speed, an error of less than 1 pixel, high stitching accuracy, and is unaffected by lighting or weather conditions, exhibiting strong robustness.

[0105] Furthermore, in step S3, the visual transformer feature extraction sub-network is a visual transformer model pre-trained on a large-scale unlabeled image dataset using a self-supervised learning method; the single-stage detection backbone network is a single-stage object detection network based on a convolutional neural network.

[0106] The feature fusion module is configured to perform a two-point feature injection operation, including:

[0107] At the input end of the single-stage detection backbone network, the global semantic features extracted by the visual transformer feature extraction subnetwork are injected.

[0108] In the middle layer of the backbone of the single-stage detection backbone network, multi-scale semantic features extracted by the visual transformer feature extraction subnetwork are injected.

[0109] The injected semantic features are fused with the features of the single-stage detection backbone network itself through a learnable adaptive weighting mechanism.

[0110] The detection head outputs the bolt detection frame, category, confidence level, and suspected missing location.

[0111] like Figure 4 As shown, the bolt detection network in this embodiment adopts a deep fusion architecture, aiming to achieve high-precision and high-efficiency identification of small-sized bolts and bolts in missing states. The network mainly consists of four parts:

[0112] 1. Self-supervised visual transformer feature extraction sub-network: This part, for example, adopts a DINOv3-based architecture and is pre-trained on large-scale unlabeled datasets such as ImageNet-22k through self-supervised learning to learn general semantic representations of road scenes. It takes a 224×224 pixel input and outputs 768-dimensional multi-scale feature maps (e.g., 14×14, 7×7, 3×3), combining fine-grained details with global receptive field information, providing a strong semantic foundation for small object detection.

[0113] 2. Single-stage detection backbone network: This part is based on an improved YOLO architecture, using CSPDarknet as the backbone network to extract multi-scale features, and PANet for bottom-up path aggregation to enhance feature fusion. The network design supports multi-scale detection such as 80×80, 40×40, and 20×20, and includes enhancements for small targets (bolts) to improve localization sensitivity.

[0114] 3. Feature Fusion Module (Dual-Point Feature Injection Mechanism): This is the core innovation of the network, aiming to deeply integrate semantic understanding and object detection capabilities. It employs a dual-point feature injection strategy: First, the global semantic features extracted by the visual transformer are injected into the input of the detection network, providing high-level semantic guidance for the detection task; second, in the middle layer of the detection network's backbone, the multi-scale semantic features of the visual transformer are adaptively fused with the corresponding scale feature maps of the detection network. This module contains learnable parameters that can dynamically optimize the fusion ratio of the two types of features, achieving complementary advantages.

[0115] 4. Decoupled Detection Head: The final detection task is completed by the decoupled detection head, which separates the target classification and bounding box regression tasks to avoid mutual interference. The classification branch uses VarifocalLoss to focus on hard examples, while the regression branch uses CIoULoss to optimize localization accuracy.

[0116] Training and Performance: The network was trained using a learning rate of 0.001, a batch size of 32, and the AdamW optimizer. After optimization, the model achieved a real-time processing speed of 30fps on a GPU, with a bolt detection rate exceeding 98%, a missing state recognition rate exceeding 95%, and an overall detection accuracy (mAP@0.5) exceeding 96%, effectively balancing accuracy and speed to meet the needs of engineering applications.

[0117] Furthermore, step S4 is performed through the following specific steps:

[0118] Step S41: Based on the guardrail design parameters, obtain the nominal spacing d of the bolts along the extension direction of the guardrail;

[0119] Step S42: Sort the bolt targets in the detection results according to the extension direction of the guardrail to form a bolt sequence;

[0120] Step S43: Traverse the bolt sequence. If the distance between two adjacent bolt targets is greater than (1+α)·d, where α is a preset tolerance coefficient, then mark the position between the two adjacent bolt targets as a missing candidate position.

[0121] Step S44: Based on the preset vertical arrangement of bolts on the guardrail cross section, verify the missing candidate positions and confirm the final missing bolt positions.

[0122] In step S3, the spliced ​​guardrail images from each viewpoint are input into the bolt detection network for detection to obtain the detection results for each viewpoint.

[0123] Step S4 further includes multi-view result fusion: mapping the detection results of each view to a unified guardrail coordinate system. When multiple views detect bolts at the same object position, the confidence of the bolt detection result is increased; when multiple views do not detect bolts at the same expected bolt position, the position is confirmed as a missing bolt position.

[0124] Specifically, this embodiment provides a multi-view detection result fusion and context-aware reasoning method to improve the accuracy and stability of the detection results;

[0125] I. Fusion of Multi-view Detection Results

[0126] To improve the robustness of detection, the system adopts a strategy of multi-view camera collaboration and result fusion.

[0127] Viewpoint configuration: Deploy three fixed-view cameras on the inspection vehicle to collectively cover the bolt area of ​​the guardrail.

[0128] Horizontal view camera: Its optical axis is roughly perpendicular to the facade of the guardrail, and it is used to obtain a front view of the guardrail.

[0129] Top-view camera: Its optical axis is tilted downwards by about 30°, used to observe the upper surface of the guardrail beam and the upper connection area, reducing the obstruction of the beam to the lower bolts.

[0130] The upward-looking camera, with its optical axis tilted upwards at approximately 30°, is used to observe the lower surface of the guardrail beam and the lower connecting area. Three cameras are installed at a spacing of 15-20 cm in the direction of vehicle travel to obtain images with parallax.

[0131] Coordinate unification and mapping: After each camera independently completes image acquisition and stitching, the bolt detection results (image pixel coordinates) in the stitched images from each viewpoint are mapped to a unified object-side two-dimensional coordinate system (which can be simplified to distance and height coordinates along the guardrail) using pre-calibrated camera extrinsic parameters. This coordinate transformation process is completed through matrix operations, with a transformation accuracy better than 0.5 cm and a single-frame processing time of less than 5 milliseconds.

[0132] Results fusion strategy: Fusion of detection results from different perspectives under a unified coordinate system.

[0133] Position matching: Set a position matching threshold of 5 cm. If the centers of two or more detection boxes from different viewpoints are within this threshold range, they are determined to correspond to the same physical bolt.

[0134] Confidence Fusion and Decision: Successfully matched detection results are integrated based on their confidence levels. The specific strategy is as follows: if detected from only one viewpoint, its original confidence level is retained but multiplied by a coefficient of 0.7; if detected from two viewpoints simultaneously, the higher confidence level is used and multiplied by a coefficient of 0.9; if detected from all three viewpoints, the highest confidence level is used (multiplied by a coefficient of 1.0). Finally, the fused confidence level is compared with a preset threshold to determine whether a bolt exists at that location. This fusion enhances the system's adaptability to complex occlusion and lighting changes, improving overall detection accuracy by 5-10%.

[0135] II. Context-Aware Reasoning

[0136] Based on the test results, we further utilize structured prior knowledge of the guardrail bolt arrangement to perform logical reasoning in order to verify the results and identify the actual defects.

[0137] Structural prior import: The system pre-sets or acquires the design parameters of the guardrail in real time, including: the nominal spacing of bolts along the extension direction of the guardrail (e.g., fixed value), the vertical arrangement of bolts on the guardrail cross section (e.g., upper row, lower row), and the standard arrangement density (e.g., 4-6 bolts distributed every 2 meters).

[0138] Detection sequence analysis and anomaly identification: The detection results of a single row of bolts are sorted according to the direction along the guardrail. The actual spacing between adjacent bolts in the sorted sequence is calculated and compared with the nominal spacing. When the spacing between adjacent bolts deviates from the nominal spacing by more than 20%, this position is marked as an abnormal spacing interval. This interval may be a signal of missed detection (spacing too large) or false detection (spacing too small).

[0139] Missing Validation and False Detection Removal: Reasoning based on the above analysis:

[0140] Missing bolt identification: Within an abnormal spacing range, if a bolt should be present based on the nominal spacing, but no valid detection results are given from any viewpoint, then this location is identified as a missing bolt location. Furthermore, consecutive missing locations are clustered to identify continuous missing regions.

[0141] False detection rejection: If the detected bolts are located in a way that seriously violates the vertical arrangement relationship (e.g., appearing between two known rows without design basis), or if their presence causes the local spacing to be much smaller than the nominal spacing and cannot be explained by the double bolt design, they are judged as false detections that do not conform to the structural rules and are rejected from the final results.

[0142] By implementing multi-view fusion and context-aware reasoning in this embodiment, the system can add spatial consistency and logical rationality checks on top of pixel-level detection. Practical applications show that this method can reduce the false negative rate of bolt detection by 3-5%, the false positive rate by 5-8%, and improve the overall stability of detection results by 10-15%.

[0143] Furthermore, after step S4, the method further includes a result-level deduplication step:

[0144] The bolt detection results in adjacent multi-frame images or spliced ​​segments are mapped onto a one-dimensional coordinate system of the guardrail based on road mileage;

[0145] The clustering radius is set according to the nominal bolt spacing and the system positioning accuracy. Multiple bolt detection results that are less than the clustering radius on the one-dimensional coordinate system are clustered and merged into a single detection result corresponding to the same physical bolt to eliminate duplicate counting.

[0146] Specifically, this embodiment further sets up a result-level deduplication and statistical correction module on the basis of the image deduplication and stitching method based on geometric parameters. This module is used to eliminate duplicate counts of the same bolt in adjacent frames or stitching boundary areas, thereby improving the accuracy of missing statistics and severity assessment.

[0147] The processing procedure in this embodiment is as follows:

[0148] (1) Map to the guardrail coordinate system

[0149] Bolt detection results from adjacent multi-frame stitched guardrail images are mapped to a unified guardrail coordinate system. A one-dimensional coordinate axis along the guardrail direction is constructed, using the road center marker or guardrail starting mileage as a reference. Based on camera extrinsic parameters and vehicle positioning information, the image coordinates of each bolt detection box are converted into projected coordinates along the guardrail direction.

[0150] (2) Result-level clustering merging based on nominal distance

[0151] Based on the nominal bolt spacing in the guardrail design and the system positioning accuracy, a cluster radius is set. For multiple bolt detection frames whose distance in the guardrail direction is less than this cluster radius, they are treated as the same physical bolt and spatially clustered. The cluster center is determined using a confidence-weighted method and is used as the final retained detection result, thereby eliminating duplicate counting of the same bolt in multiple frames of images.

[0152] (3) Adjustment of vehicle speed adaptive parameters

[0153] Based on the real-time speed of the inspection vehicle and the fixed-distance trigger interval, the length of the object overlap area and the cluster radius are adaptively adjusted: the cluster radius is appropriately increased when the vehicle speed is high and appropriately decreased when the vehicle speed is low, so as to ensure that the projection offset of the same bolt in adjacent frames does not exceed the cluster radius under different vehicle speed conditions, thereby suppressing duplicate counting while avoiding missed merging.

[0154] (4) Statistical correction and effect analysis

[0155] In a 50km test section of a highway from K100 to K150, without enabling the result-level deduplication module, the bolt duplication count rate was about 20% because the same bolt was repeatedly detected in multiple frames of images. The statistical result of missing bolts was about 18% higher than that of manual review. After enabling the result-level deduplication and statistical correction module of this embodiment, the bolt duplication count rate was reduced to less than 2%, the difference between the statistical result of missing bolts and the result of manual review was controlled within ±5 bolts, and the overall detection mAP was improved to about 96.8%, which significantly improved the reliability of missing bolt statistics and severity assessment. The test results are shown in Table 1.

[0156] Table 1 Comparison of performance of the result-level deduplication module before and after.

[0157]

[0158] Specifically, such as Figure 5 As shown, after obtaining the bolt missing detection results, this embodiment uses precise geometric mapping and multi-sensor fusion technology to physically locate the missing position, and classifies the severity of the guardrail safety status according to a scientific evaluation model, providing a direct basis for maintenance decisions.

[0159] I. Method for Precisely Locating Missing Bolts

[0160] To achieve accurate conversion from images to on-site maintenance work orders, the system executes the following positioning process:

[0161] Coordinate mapping chain:

[0162] Image coordinates to object coordinates: Based on the camera calibration intrinsic parameters (focal length, principal point, etc.) and the known vertical object distance from the camera to the guardrail, the pixel coordinates of the missing bolt in the stitched image are solved by inverse perspective projection transformation to obtain its two-dimensional object coordinates (distance, height) on the guardrail plane perpendicular to the driving direction.

[0163] Object coordinates to road coordinates: Combining the high-precision vehicle pose data (including position, heading, and attitude) recorded synchronously at the time of image acquisition, the above object coordinates are transformed into three-dimensional spatial coordinates in a geocentric reference coordinate system (such as WGS-84). Through this mapping chain, the final positioning accuracy can be better than 0.5 meters.

[0164] Multi-sensor fusion localization: Vehicle pose data comes from a multi-sensor fusion module, which uses Kalman filtering as its core algorithm and fuses information from the following sensors:

[0165] GPS satellite positioning: provides absolute geographical location with an accuracy of approximately 1-5 meters.

[0166] Inertial Measurement Unit (IMU): Provides high-frequency vehicle three-axis acceleration and angular velocity, which are integrated to obtain attitude and position changes with an attitude accuracy of 0.1°, and is used to compensate for GPS signal loss or jumps.

[0167] High-precision odometer: measures vehicle travel distance with an accuracy of 0.1%, providing continuous and reliable incremental location information.

[0168] Through fusion, the system outputs a stable, continuous vehicle pose sequence with higher accuracy than any single sensor, laying the foundation for the spatiotemporal correlation of image coordinates.

[0169] Road information association:

[0170] Mileage station calculation: The obtained spatial coordinates are matched with the road design alignment and the benchmark mileage station to calculate the accurate road mileage station where the missing bolt is located, with an accuracy of within 1 meter.

[0171] Lane position recognition: Based on the position of the coordinates relative to the road centerline, it automatically determines whether the missing bolt is located on the guardrail of the left or right lane.

[0172] II. Severity Assessment Method for Guardrail Safety Condition

[0173] Based on precise positioning, the system performs quantitative assessment and classification of the guardrail condition:

[0174] Evaluation index calculation: The system uses a fixed length (e.g., 100 meters) as an evaluation unit, and automatically collects and calculates three core indicators:

[0175] Missing Quantity: The total number of missing bolts identified within this unit.

[0176] Continuous missing length: The maximum number of consecutive missing bolts that appear in this unit.

[0177] Missing bolt rate: The percentage of the total number of missing bolts out of the total number of theoretically designed bolts in the unit (calculated based on the design spacing and length).

[0178] Severity grading criteria: Based on preset quantitative thresholds, the safety status of the guardrails in each assessment unit is divided into three levels:

[0179] Minor missing bolts: The missing bolts are isolated and the overall missing rate is <5%. This type of problem can be included in the routine maintenance plan.

[0180] Moderate loss: Two consecutive bolts are missing, or the overall loss rate is between 5% and 15%. This indicates that the connection strength has been significantly weakened and repairs should be arranged in the short term.

[0181] Severe loss: Three or more bolts are missing consecutively, or the overall loss rate is >15%. This constitutes a serious safety hazard and requires the initiation of an emergency repair procedure.

[0182] Automated assessment process: The system automatically completes the assessment according to the following steps: data statistics → indicator calculation → level determination → report generation. The final output inspection report will clearly list the location (including station number, coordinates, lane), quantity, and severity level of each missing point, and can further generate a visualized road section risk map.

[0183] This embodiment transforms discrete visual inspection results into management information with spatial location and severity quantification that can directly guide maintenance actions, achieving the following core values:

[0184] Provide quantitative basis for maintenance decisions: shift from "experience-based judgment" to "data-driven" approach, making maintenance priority ranking more scientific.

[0185] Optimize the allocation of maintenance resources: Based on the severity level, manpower and materials can be allocated reasonably, and high-risk road sections should be prioritized for handling.

[0186] Improve maintenance efficiency and effectiveness: Precise positioning reduces on-site search time, and clear standards enhance the targetedness and effectiveness of maintenance.

[0187] Reduce life-cycle maintenance costs: By implementing preventative maintenance and precise repairs, minor issues can be prevented from escalating into major accidents, thereby reducing long-term maintenance costs and the social costs associated with accidents.

[0188] Furthermore, the method also includes an online model update step:

[0189] During system operation, samples with detection confidence levels in the preset middle range, inconsistent detection results at different times, or those that have been manually reviewed and corrected are collected to build a difficult sample library;

[0190] The bolt detection network is incrementally trained periodically using the problematic sample library to generate an updated detection model.

[0191] Based on the statistical distribution of the confidence level of bolt targets in the current detection task, the confidence threshold used to determine whether a bolt exists or is missing is dynamically adjusted.

[0192] Specifically, this embodiment, based on the bolt detection network structure, further provides a training method and online update strategy for the bolt detection network to ensure the detection accuracy and adaptability of the system during long-term operation.

[0193] I. Training Dataset Construction Process

[0194] Building high-quality, diverse training datasets is the cornerstone of model performance. The specific process is as follows:

[0195] Diverse Data Collection: Organize multiple rounds of on-site inspections, covering roads of different grades, including highways, urban expressways, and ordinary national and provincial roads. Data collection activities must encompass various weather and lighting conditions, such as sunny, cloudy, rainy, foggy, and nighttime, and include major guardrail types such as double-wave beams, triple-wave beams, combined guardrails, and bridge railings, to fully ensure the diversity and representativeness of the data samples.

[0196] Rigorous data cleaning and filtering: Raw images undergo a combination of automated and manual cleaning to remove severely blurry, overexposed / underexposed, or camera-obstructed images. Simultaneously, data packets with incorrect timestamps or missing pose information due to equipment malfunctions are repaired or filtered. Furthermore, algorithms are used to deduplicate batch data, preventing an excessively high proportion of images from the same road segment under similar conditions and ensuring a balanced data distribution.

[0197] High-precision manual annotation: Using a professional image annotation platform, fine-grained annotations are performed on geometrically stitched guardrail images. Annotation objects include categories such as "intact bolts" (marked with bounding boxes), "missing bolt locations" (marked with dots or boxes at where bolts should be present), and "severely damaged / obscured suspected missing bolts." The annotation process employs a mechanism of independent annotation by two people, cross-checking, and expert random review to ensure the accuracy and consistency of the annotation results to the greatest extent possible.

[0198] Targeted data augmentation and hard example mining: Given the small size of bolt targets, training samples are augmented with random cropping, multi-scale scaling, brightness / contrast perturbations, and simulated fog / rain weather to improve model scale invariance and illumination robustness. After initial model training, missed and false positive cases on the validation set are analyzed. These "hard example" samples are proactively identified, enhanced, and added to subsequent training loops to specifically improve the model's discrimination ability in complex scenes.

[0199] II. Model Training and Optimization Process

[0200] Based on the above dataset, the bolt detection network was systematically trained:

[0201] Two-stage training strategy:

[0202] Phase 1: Self-supervised pre-training: First, the visual transformer feature extraction sub-network is pre-trained in a self-supervised manner on a large-scale unlabeled general road scene image dataset to enable it to obtain a general and powerful semantic feature extraction capability for road environment elements.

[0203] Phase Two: Supervised Transfer Learning and End-to-End Fine-Tuning: The pre-trained visual transformer weights are fixed or used as initialization and loaded into the complete bolt detection network. Subsequently, the single-stage detection backbone network is trained in a supervised manner on a labeled bolt-specific dataset, and the feature fusion module connecting the two is jointly optimized simultaneously to achieve deep adaptation between the detection task and powerful semantic features.

[0204] Loss Function and Training Techniques: During training, the classification branch employs a loss function such as VarifocalLoss, which focuses on difficult samples to better distinguish foreground from difficult background; the regression branch uses a consistency loss such as CIoULoss, which considers overlap area, center point distance, and aspect ratio. The training process employs strategies such as learning rate warm-up and cosine annealing to ensure stable convergence.

[0205] Lightweight Model Deployment: To meet the real-time requirements of in-vehicle edge computing units, the model is compressed after training convergence. While maintaining controllable accuracy loss, techniques such as network pruning, weight quantization, and knowledge distillation are employed to lightweight the model. Furthermore, the network depth and width are adjusted based on hardware characteristics to ensure the optimized model achieves real-time processing speeds exceeding 30fps on in-vehicle GPUs.

[0206] III. Online Continuous Learning and Update Mechanism

[0207] To enable the system to adapt to new scenarios and problems that arise during long-term operation, an online update strategy should be established:

[0208] Dynamic Construction of the Difficult Sample Library: During routine system inspections, the following three types of samples are automatically collected and stored in the "Difficult Sample Library": (a) Samples with a detection confidence level in the medium range (e.g., 0.3-0.7); (b) Samples with inconsistent detection results at the same location in different vehicle inspections; (c) Samples that have been manually reviewed and corrected by the background maintenance management personnel. These samples represent the "fuzzy areas" or errors in the model's current decision-making.

[0209] Incremental Training and Version Iteration: Technical personnel regularly review and fine-tagged the samples in the difficult sample library. Once a certain number have been accumulated, these samples are used as incremental datasets. While retaining the original training data, small-batch incremental training (fine-tuning) is performed on the model in the production environment. The new model version generated after fine-tuning is rigorously evaluated on independent test sets and difficult sample sets. Only when the new version significantly outperforms the live version in both overall accuracy (mAP) and stability (such as the recognition rate of difficult examples) is it securely deployed to the vehicle edge computing unit.

[0210] Secure Rollback Guarantee: All model update operations adhere to the principles of "canary release" and "secure rollback." During updates, a complete backup of the previous stable version is retained. If unforeseen performance degradation or anomalies occur in real-vehicle testing of the new version, a quick and automatic rollback to the old version can be performed, ensuring that the continuity and reliability of inspection operations are not affected.

[0211] Through the above-mentioned systematic training and dynamic update strategy, the bolt detection network of the present invention can not only have high accuracy in the early stage of deployment, but also continuously absorb new knowledge and adapt to new environments in long-term operation, so as to realize the self-evolution and long-term maintenance of detection performance, fundamentally guaranteeing the practical value and life cycle of the automated inspection system.

[0212] Please see Figure 6 The present invention also provides a guardrail bolt missing detection and positioning system based on visual transformation and reasoning, comprising:

[0213] Image acquisition unit: controls the inspection vehicle to travel along the guardrail, and acquires a sequence of guardrail images at a preset fixed-distance trigger interval through at least one camera set on the inspection vehicle facing the guardrail, and synchronously records the sensor pose information of the inspection vehicle when each frame of image is acquired.

[0214] Geometric deduplication and stitching unit: Based on the camera's intrinsic parameters, pixel size, object distance from the camera to the guardrail, and the fixed-distance trigger interval, calculate the object coverage area of ​​each frame of guardrail image in the guardrail extension direction and the object overlap area between adjacent frame guardrail images; based on the object overlap area, crop the adjacent frame guardrail images and stitch them along the guardrail extension direction to obtain the stitched guardrail image.

[0215] Bolt detection unit: The spliced ​​guardrail image is input into the bolt detection network for detection to obtain detection results containing bolt targets and suspected missing locations; wherein, the bolt detection network includes a self-supervised pre-trained visual transformer feature extraction sub-network and a single-stage detection backbone network, and a feature fusion module is set in the input end of the single-stage detection backbone network and at least one intermediate layer of the backbone;

[0216] Contextual reasoning unit: Based on the designed spacing of the guardrail bolts along the extension direction of the guardrail and the preset positional relationship on the guardrail cross section, the detection results are subjected to contextual reasoning to identify and confirm the location of the missing bolts, and the bolt missing detection results are obtained.

[0217] Location and evaluation unit: Based on the image coordinates of each missing bolt in the spliced ​​guardrail image in the bolt missing detection results, and the corresponding sensor pose information, calculate the road mileage and spatial coordinates of each missing bolt; and classify the severity according to the number and distribution of missing bolts, and generate a detection report.

[0218] It should be noted that, in order to verify the improvement effect of the technical solution of the present invention compared with the comparative scheme and the traditional manual inspection method, a comparative test of detection performance and a comparative analysis of inspection efficiency and cost were carried out on a typical highway test section. The results are shown in Tables 2 and 3.

[0219] (1) Comparison of detection performance: The results of comparing different detection schemes in terms of detection accuracy, false negative rate, false positive rate and duplicate count control are shown in Table 2.

[0220] Table 2 Comparison of Bolt Testing Performance

[0221]

[0222] (2) Comparison of inspection efficiency and cost: The differences between the traditional manual inspection method and the system of the present invention in terms of unit manpower inspection efficiency, annual inspection workload and inspection cost are shown in Table 3.

[0223] Table 3 Comparison of Inspection Efficiency and Cost (Taking 500km of guardrail inspected twice a year as an example)

[0224]

[0225] In addition, this embodiment is also integrated into the highway maintenance business system to realize a complete business closed loop from automatic detection to maintenance decision-making and verification, thereby improving the intelligence and precision of maintenance management.

[0226] I. System Integration Architecture and Data Flow

[0227] To achieve efficient conversion of detection data into maintenance actions, the system is designed with a three-layer integrated architecture:

[0228] Data Interface Layer: The management platform of this invention acts as the data producer, connecting with existing business systems such as the highway asset management system and road property management platform through standardized interfaces such as RESTful APIs or message queues. The pushed data service package structurally includes a list of missing bolts, where each record is associated with key attributes such as spatial coordinates, road mileage marker, route code, lane information, and severity level, ensuring that downstream systems can directly utilize it.

[0229] Business rule configuration layer: In the business system that receives data, maintenance managers can pre-configure automated maintenance strategies that match different severity levels according to management needs. For example:

[0230] Mild missing parts: Automatically categorized and included in subsequent daily inspection plans or the next regular maintenance cycle.

[0231] Moderate deficiency: Triggers an alert, and the system suggests generating a task within a maintenance cycle to schedule centralized repairs.

[0232] Severe Missing Cases: Automatically generate high-priority emergency work orders and push them to mobile devices, requiring immediate action. This layer supports setting differentiated handling thresholds and response time limits based on the importance, traffic volume, or management affiliation of different road segments, enabling flexible and precise strategy management.

[0233] Work orders and closed-loop management:

[0234] Automatic work order generation: Based on the received test results and preset rules, the business system automatically creates electronic work orders containing precise location, quantity of missing items, standard repair procedures and safety precautions, and assigns them to the corresponding maintenance unit or work team.

[0235] Maintenance process feedback and work order closure: After arriving on site, maintenance personnel can confirm the work order, upload photos and records of the maintenance process via mobile terminal. After the maintenance is completed, a completion report is submitted via the terminal, realizing a digital closed loop for the work order.

[0236] Effect verification and feedback: In the next round of inspections, the system can automatically compare the test results before and after the repair at this location to verify the repair effect, and feed back indicators such as "recurrence rate" and "timeliness of treatment" to the system for evaluating maintenance quality and updating the disease model.

[0237] II. Application of Multi-Source Data Collaboration and Comprehensive Analysis

[0238] The detection results of this system can be linked with other road detection systems to build a comprehensive safety evaluation system:

[0239] Linking with road safety facility inspections: Data on missing guardrail bolts is integrated and analyzed with the automated inspection results of other traffic safety facilities such as signs, markings, and delineators on the same road segment to construct a unified "road safety facility risk map" at the road segment or road network level. This map can provide management departments with an integrated, multi-dimensional safety status assessment and early warning, supporting more macro-level safety decision-making.

[0240] Linking with Bridge and Tunnel Structural Health Monitoring: For bridge railings, the bolt connection status is related to the safety of the main structure. This system can push bolt loss data of bridge railings to the bridge health monitoring system or structural health monitoring platform via an interface. Combined with structural strain, displacement, and other response data for cross-analysis, it can provide data support for assessing the impact of local connection failures on the overall structure and assist in determining deeper safety risks.

[0241] Integrating with maintenance performance evaluation mechanisms: Based on the objective and traceable closed-loop data of detection and maintenance provided by the system, management departments can establish quantifiable performance evaluation indicators, such as problem detection rate, timely maintenance response rate, maintenance qualification rate, and defect recurrence rate. Linking system data with the performance of maintenance units can effectively promote the standardization and quality improvement of daily inspections and maintenance operations.

[0242] To illustrate the engineering applicability and implementation effect of the technical solution of the present invention, this embodiment provides a practical application case on a section of a highway.

[0243] I. Application Scenarios and System Configuration

[0244] Application Scenario: A practical test was conducted on a section of a highway from K100 to K150. This section is 50 kilometers long, and the guardrails along the route are standard corrugated beam steel guardrails. It was calculated that the total number of guardrail connection bolts on this section is approximately 12,000.

[0245] System Configuration: The detection system described in this invention is mounted on a standard highway maintenance vehicle. During inspection, the vehicle travels at a normal speed of 60-100 km / h. The image acquisition system is set to a fixed-distance trigger mode, triggering once every 2 meters of forward movement. A three-view camera system consisting of horizontal, overhead, and under-view cameras simultaneously acquires images. All image processing and detection tasks are completed in real time on the edge computing unit inside the vehicle.

[0246] II. Test Results and Performance Analysis

[0247] In a complete inspection job, the system obtained the following results:

[0248] Work efficiency: The inspection of guardrails on both sides of a 50-kilometer road section was completed in less than 1 hour, demonstrating extremely high inspection efficiency.

[0249] Detection statistics: The system identified and located a total of 11,856 bolts, and found 423 missing bolt locations.

[0250] Detection accuracy indicators: Based on the results of post-detection manual review, the overall accuracy of the system in this detection reached 96.8%. Among them, the missed detection rate (undetected objects that should have been detected) was 3.2%, and the false detection rate (misdetecting background or other objects as bolts) was 2.1%.

[0251] Severity distribution: The system automatically classified the 423 missing values ​​according to the standards described in the above embodiment.

[0252] Mild deletions (single isolated deletions): 287, accounting for 67.8% of the total deletions.

[0253] Moderate deletion (two consecutive deletions): 98 instances, accounting for 23.2%.

[0254] Severe deletions (≥3 consecutive deletions): 38 instances, accounting for 9.0%.

[0255] III. Comprehensive Application Effect Evaluation

[0256] Compared with traditional manual inspection methods, this application case fully demonstrates the significant advantages of the present invention:

[0257] Revolutionary improvement in inspection efficiency: Traditional manual inspection of this section of road usually takes 1-2 working days (about 8-16 hours), while this system can complete it in 1 hour, improving efficiency by more than 10 times.

[0258] Significantly improved detection accuracy: Traditional manual inspections are limited by fatigue and viewing angle, resulting in a missed detection rate that can often exceed 15%. This system stably controls the missed detection rate at 3.2%, significantly improving the defect detection capability.

[0259] The safety of the operation has been fundamentally improved: "Inspection while the vehicle is in motion" has been realized, and maintenance personnel do not need to get out of the vehicle or approach the lane during the inspection process, which completely eliminates the traffic safety risks faced by manual inspection.

[0260] Data management is fully digitalized: all detection results (location, images, severity) are automatically generated into structured data reports and electronic map annotations, realizing the digital archiving, querying and analysis of detection information, and providing a data foundation for scientific decision-making.

[0261] Significant economic benefits: Taking into account the costs of manpower, vehicles, and time, the use of this system for routine inspections is expected to reduce the cost of special inspection and maintenance of guardrail bolts on this section of road by more than 30%, resulting in significant economic benefits.

[0262] This application case demonstrates that the technical solution of this invention not only has advanced technical indicators, but also achieves efficient, accurate, safe and economical detection of missing guardrail bolts in real engineering scenarios, and has practical value for large-scale promotion and application.

[0263] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for detecting and locating missing guardrail bolts based on visual transformation and reasoning, characterized in that, Includes the following steps: Step S1: Control the inspection vehicle to travel along the guardrail, and use at least one camera on the inspection vehicle facing the guardrail to collect a sequence of guardrail images at a preset fixed-distance trigger interval, and simultaneously record the sensor pose information of the inspection vehicle when each frame of image is collected. Step S2: Based on the camera's intrinsic parameters, pixel size, object distance from the camera to the guardrail, and the fixed-distance trigger interval, calculate the object coverage area of ​​each frame of guardrail image in the guardrail extension direction and the object overlap area between adjacent frames of guardrail images. Based on the object overlap interval, adjacent frame guardrail images are cropped and spliced ​​along the guardrail extension direction to obtain a spliced ​​guardrail image. Step S3: Input the spliced ​​guardrail image into the bolt detection network for detection to obtain detection results containing bolt targets and suspected missing locations; wherein, the bolt detection network includes a self-supervised pre-trained visual transformer feature extraction sub-network and a single-stage detection backbone network, and a feature fusion module is set in the input end of the single-stage detection backbone network and at least one backbone intermediate layer; Step S4: Based on the designed spacing of the guardrail bolts along the extension direction of the guardrail and the preset positional relationship on the guardrail cross section, perform context-aware reasoning on the detection results, identify and confirm the location of the missing bolts, and obtain the bolt missing detection results; Step S5: Based on the image coordinates of each missing bolt in the spliced ​​guardrail image from the bolt missing detection results, and the corresponding sensor pose information, calculate the road mileage and spatial coordinates of each missing bolt; and classify the severity according to the number and distribution of missing bolts to generate a detection report; In step S3, the visual transformer feature extraction sub-network is a visual transformer model pre-trained on a large-scale unlabeled image dataset using a self-supervised learning method; the single-stage detection backbone network is a single-stage target detection network based on a convolutional neural network. The feature fusion module is configured to perform a two-point feature injection operation, including: At the input end of the single-stage detection backbone network, the global semantic features extracted by the visual transformer feature extraction subnetwork are injected. In the middle layer of the backbone of the single-stage detection backbone network, multi-scale semantic features extracted by the visual transformer feature extraction subnetwork are injected. The injected semantic features are fused with the features of the single-stage detection backbone network itself through a learnable adaptive weighting mechanism. The detection head outputs the bolt's detection frame, category, confidence level, and suspected missing location; In step S1, the camera includes at least two of the following: a horizontal-view camera, a top-view camera, and a bottom-view camera, which respectively acquire guardrail image sequences from different perspectives. In step S2, geometric deduplication and stitching are performed on the guardrail image sequences from each viewpoint to obtain stitched guardrail images from multiple viewpoints. In step S3, the spliced ​​guardrail images from each viewpoint are input into the bolt detection network for detection to obtain the detection results for each viewpoint. Step S4 further includes multi-view result fusion: mapping the detection results of each view to a unified guardrail coordinate system. When multiple views detect bolts at the same object position, the confidence of the bolt detection result is increased; when multiple views do not detect bolts at the same expected bolt position, the position is confirmed as a missing bolt position.

2. The method for detecting and locating missing guardrail bolts based on visual transformation and reasoning according to claim 1, characterized in that, Step S1 is based on the odometer and / or wheel speed sensor to achieve fixed-distance triggering. When the inspection vehicle travels a preset interval distance, the camera is triggered to acquire a frame of guardrail image, and the acquisition time of the frame of guardrail image is bound to the sensor pose information output by the multi-sensor fusion module.

3. The method for detecting and locating missing guardrail bolts based on visual transformation and reasoning according to claim 1, characterized in that, Step S2 is performed through the following specific steps: Step S21: Calculate the horizontal field of view of the camera based on its focal length, pixel size, and number of horizontal pixels; Step S22: Calculate the object coverage width W of a single frame guardrail image in the guardrail extension direction based on the horizontal field of view and the object distance; Step S23: Using the fixed-distance trigger interval L as the spacing between the acquisition positions of adjacent frame images, calculate the object-side overlap length L_overlap=max(0,WL) of the guardrail images in adjacent frames; Step S24: Based on the object overlap length, the main overlapping parts of two adjacent guardrail images are cropped, and the images are spliced ​​together after retaining a preset length of overlap tolerance zone.

4. The method for detecting and locating missing guardrail bolts based on visual transformation and reasoning according to claim 3, characterized in that, In step S24, within the overlapping fault tolerance zone, local matching is performed on two adjacent guardrail images based on a preset bolt template. The position of the next guardrail image in the splicing direction is fine-tuned according to the matched bolt correspondence, and then splicing is performed.

5. The method for detecting and locating missing guardrail bolts based on visual transformation and reasoning according to claim 1, characterized in that, Step S4 is performed through the following specific steps: Step S41: Based on the guardrail design parameters, obtain the nominal spacing d of the bolts along the extension direction of the guardrail; Step S42: Sort the bolt targets in the detection results according to the extension direction of the guardrail to form a bolt sequence; Step S43: Traverse the bolt sequence. If the distance between two adjacent bolt targets is greater than (1+α)·d, where α is a preset tolerance coefficient, then mark the position between the two adjacent bolt targets as a missing candidate position. Step S44: Based on the preset vertical arrangement of bolts on the guardrail cross section, verify the missing candidate positions and confirm the final missing bolt positions.

6. A method for detecting and locating missing guardrail bolts based on visual transformation and reasoning according to any one of claims 1 to 5, characterized in that, Following step S4, the method further includes a result-level deduplication step: The bolt detection results in adjacent multi-frame images or spliced ​​segments are mapped onto a one-dimensional coordinate system of the guardrail based on road mileage; The clustering radius is set according to the nominal bolt spacing and the system positioning accuracy. Multiple bolt detection results that are less than the clustering radius on the one-dimensional coordinate system are clustered and merged into a single detection result corresponding to the same physical bolt to eliminate duplicate counting.

7. A method for detecting and locating missing guardrail bolts based on visual transformation and reasoning according to any one of claims 1 to 5, characterized in that, The method also includes an online model update step: During system operation, samples with detection confidence levels in the preset middle range, inconsistent detection results at different times, or those that have been manually reviewed and corrected are collected to build a difficult sample library; The bolt detection network is incrementally trained periodically using the problematic sample library to generate an updated detection model. Based on the statistical distribution of the confidence level of bolt targets in the current detection task, the confidence threshold used to determine whether a bolt exists or is missing is dynamically adjusted.

8. A guardrail bolt missing detection and positioning system based on visual transformation and reasoning, characterized in that, include: Image acquisition unit: controls the inspection vehicle to travel along the guardrail, and acquires a sequence of guardrail images at a preset fixed-distance trigger interval through at least one camera set on the inspection vehicle facing the guardrail, and synchronously records the sensor pose information of the inspection vehicle when each frame of image is acquired. Geometric deduplication stitching unit: Based on the camera's intrinsic parameters, pixel size, object distance from the camera to the guardrail, and the fixed-distance trigger interval, calculate the object coverage area of ​​each frame of guardrail image in the guardrail extension direction and the object overlap area between adjacent frames of guardrail images. Based on the object overlap interval, adjacent frame guardrail images are cropped and spliced ​​along the guardrail extension direction to obtain a spliced ​​guardrail image. Bolt detection unit: The spliced ​​guardrail image is input into the bolt detection network for detection to obtain detection results containing bolt targets and suspected missing locations; wherein, the bolt detection network includes a self-supervised pre-trained visual transformer feature extraction sub-network and a single-stage detection backbone network, and a feature fusion module is set in the input end of the single-stage detection backbone network and at least one intermediate layer of the backbone; Contextual reasoning unit: Based on the designed spacing of the guardrail bolts along the extension direction of the guardrail and the preset positional relationship on the guardrail cross section, the detection results are subjected to contextual reasoning to identify and confirm the location of the missing bolts, and the bolt missing detection results are obtained. Location and evaluation unit: Based on the image coordinates of each missing bolt in the spliced ​​guardrail image in the bolt missing detection results, and the corresponding sensor pose information, calculate the road mileage and spatial coordinates of each missing bolt; and classify the severity according to the number and distribution of missing bolts, and generate a detection report; In the bolt detection unit, the visual transformer feature extraction subnetwork is a visual transformer model pre-trained on a large-scale unlabeled image dataset using a self-supervised learning method; the single-stage detection backbone network is a single-stage target detection network based on a convolutional neural network. The feature fusion module is configured to perform a two-point feature injection operation, including: At the input end of the single-stage detection backbone network, the global semantic features extracted by the visual transformer feature extraction subnetwork are injected. In the middle layer of the backbone of the single-stage detection backbone network, multi-scale semantic features extracted by the visual transformer feature extraction subnetwork are injected. The injected semantic features are fused with the features of the single-stage detection backbone network itself through a learnable adaptive weighting mechanism. The detection head outputs the bolt's detection frame, category, confidence level, and suspected missing location; In the image acquisition unit, the camera includes at least two of the following: a horizontal-view camera, a top-view camera, and a bottom-view camera, which respectively acquire guardrail image sequences from different perspectives. In the geometric deduplication and stitching unit, geometric deduplication and stitching are performed on the guardrail image sequences from each viewpoint to obtain stitched guardrail images from multiple viewpoints. In the bolt detection unit, the spliced ​​guardrail images from each viewpoint are input into the bolt detection network for detection, and the detection results from each viewpoint are obtained. The context reasoning unit also includes multi-view result fusion: mapping the detection results of each view to a unified guardrail coordinate system. When multiple views detect bolts at the same object location, the confidence of the bolt detection result is increased; when multiple views do not detect bolts at the same expected bolt location, the location is confirmed as a missing bolt location.

Images