Road guardrail bolt detection method and system based on grading detection

Abstract

The invention discloses a road guardrail bolt detection method and system based on hierarchical detection. The method comprises the following steps: acquiring two road guardrail images acquired at the same position of a road guardrail by an upper camera and a lower camera; respectively inputting into a trained first detection model to identify and cut out a bolt group, and obtaining two to-be-detected bolt images; a trained second detection model is utilized to perform bolt type and state identification on each bolt in the to-be-detected bolt image, a first identification result set and a second identification result set are obtained, and the identification result set comprises a bolt identification result corresponding to each bolt and a bolt detection frame coordinate; and de-weighting the bolt identification images in the first identification result set and the second identification result set by using the coordinates of the bolt detection frame to obtain a bolt identification result of the road guardrail at the current position. The two detection models are used for detecting and de-weighting guardrail bolt images shot by the upper camera and the lower camera in sequence, so that the accuracy of a detection result is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection based on deep learning, and specifically to a highway guardrail bolt detection method and system based on hierarchical detection. Background Art

[0002] Among the existing modes of transportation, the number of traffic scenes on highways is increasing day by day, and due to the extension of highways and the increase in vehicle speeds, higher requirements are placed on the stable and safe operation of roads. To ensure the safety and efficiency of highways, timely inspection and maintenance of guardrail bolts are crucial. Guardrail bolts are safety protection facilities at key parts of highways. During long-term use, the bolts may become loose, fall off, rust, or break. These problems may be caused by a variety of factors, such as vehicle vibration, environmental corrosion, and material fatigue. The existence of these defects may lead to road safety hazards, thereby affecting traffic smoothness and driving safety. Therefore, it is necessary to regularly inspect guardrail bolts and take measures such as maintenance or replacement in advance to maintain the safety and stable operation of the highway system.

[0003] In the existing detection algorithm for guardrail bolts, the position and category of existing targets in the image are detected by inputting image features. If the guardrail is a corrugated guardrail, the bolts fixed on it will also be distributed in a curve due to the curved cross-section of the corrugated guardrail. As a result, the bolts photographed from the front by the high-speed camera are not in the same horizontal plane, and some bolts are severely obscured and missing in the image, which has a significant impact on the actual accuracy of highway guardrail bolt detection. Summary of the invention

[0004] The purpose of the present invention is to provide a highway guardrail bolt detection method and system based on hierarchical detection, which adopts two upper and lower high-speed cameras to shoot the guardrail bolt images at the same position from different angles, and uses two detection models to detect the guardrail bolt images taken by the upper and lower cameras in turn, and then obtains the final recognition result after de-duplication and merging the detection results, thereby solving the problem that the bolts are not in the same plane and ensuring the accuracy of the detection results.

[0005] To achieve the above objectives, this application adopts the following solutions: On the one hand, the present invention provides a highway guardrail bolt detection method based on hierarchical detection, which specifically includes the following steps: S1, respectively acquiring a first highway guardrail image and a second highway guardrail image captured by an upper camera and a lower camera at the same position of the highway guardrail; S2, inputting the first highway guardrail image and the second highway guardrail image into the trained first detection model respectively, identifying and cutting out the bolt group from the first highway guardrail image and the second highway guardrail image respectively, to obtain a first bolt image to be detected and a second bolt image to be detected; S3, using the trained second detection model, respectively identifying the bolt type and state of each bolt in the first bolt image to be detected and the second bolt image to be detected, to obtain a first recognition result set and a second recognition result set, wherein the recognition result set includes a bolt recognition result and a bolt detection frame coordinate corresponding to each bolt; S4. After deduplication of the bolt recognition images in the first recognition result set and the second recognition result set using the bolt detection frame coordinates, the bolt recognition result of the highway guardrail at the current position is obtained.

[0006] In some specific implementation schemes, the first detection model adopts the YOLOv8 model, and the training process of the first detection model is: S21, obtaining a number of highway guardrail bolt images captured by the upper camera and the lower camera to construct a first detection model image data set, classifying and annotating the highway guardrail bolt images in the first detection model image data set according to the bolt distribution, and then sending them to the yolov8 network to detect the bolt distribution; S22. Cut out a bolt group from the highway guardrail bolt image according to the bolt distribution, classify and label each bolt in the bolt group, and store them in a corresponding classified image dataset as data for training the second detection model. The classified image dataset includes: connecting bolts, splicing bolts, contour marks, and bolt-off image datasets.

[0007] In some specific implementation schemes, the second detection model adopts a YOLOv8-bolt model, which includes an input layer, a backbone network, a neck structure, and a head output layer, wherein the neck structure adopts an adaptive spatial fusion network of an asymptotic feature pyramid, and introduces an aggregation-distribution mechanism in the adaptive spatial fusion network.

[0008] In some specific implementation schemes, the fusion process of the adaptive spatial fusion network is: S31, extract features of different layers from the backbone network to obtain a multi-scale feature map set {C3, C4, C5} and its corresponding feature layer numbers are {n, 2n, 4n} respectively; S32. For each multi-scale feature map in the multi-scale feature map set, up-sample the multi-scale feature map by 1×1 convolution and bilinear interpolation, down-sample the multi-scale feature map by two rounds of 2×2 convolution with a stride of 2, down-sample the multi-scale feature map by four rounds of 4×4 convolution with a stride of 4, and down-sample the multi-scale feature map by eight rounds of 8×8 convolution with a stride of 8, and finally obtain the feature map set to be fused {C'3, C'4, C'5}; S33, the feature map set to be fused {C'3, C'4, C'5} is adaptively spatially fused through multi-level features under different spatial weights to obtain the corresponding synthetic feature set {B3, B4, B5}.

[0009] In some specific embodiments, the aggregation in the aggregation-distribution mechanism includes: High-level feature alignment module High-FAM, It is used to reduce the feature layer size of each synthetic feature of the synthetic feature set {B3, B4, B5} to the same size and then export it through the channel , ; A high-level information fusion module, including several stacked transformer fusion modules, is used to combine the information derived from the high-level feature alignment module. F f , ; The split module is used to split F f Divided along the channel dimension into F M4 and F M5 , .

[0010] In some embodiments, the high-level feature alignment module High-FAM It includes two pooling layers avgpool1 and avgpool2. avgpool1 reduces the feature layer size of the synthetic feature B4 to the same as B5, and avgpool2 reduces the feature layer size of the synthetic feature B3 to the same as B5.

[0011] In some specific implementations, each transformer fusion module includes a multi-head attention block and a feed-forward network. hf The output after inputting into the multi-head attention block is the same as that from F hf After residual calculation, it is input into the feedforward network; The feedforward network consists of two 1x1 convolutional layers. A deep convolutional layer is introduced between the two 1x1 convolutional layers. The expansion factor of the feedforward network is set to 2. All GELU activations are replaced with ReLU activations. The input and output of the feedforward network are calculated again to obtain F. f .

[0012] In some specific embodiments, the distribution in the aggregation-distribution mechanism includes: The attention operation is used to fuse information, and each synthesized feature is input as the local feature of the current layer, which is expressed as F local ={B4, B5}, and the output of the split module is used as the global feature input of the current layer, expressed as F global ={F M4 , F M5}; right F global Each feature in is calculated using two different convolutions, and we get and , by F local Input into the convolution calculation module and get ; By , and Perform attention fusion calculation and according to F global The size of , using average pooling or bilinear interpolation to scale and , and finally get the fused features F out At the end of each attention fusion calculation, RepBlock is used to further F out The information is extracted and fused, and finally the output of the neck structure is obtained.

[0013] In some specific implementation schemes, the specific process of deduplication in step S4 is: According to the first recognition result set, the bolt detection frame coordinates corresponding to the m bolts are marked as , according to the second recognition result set, the bolt detection frame coordinates corresponding to the n bolts are marked as ,in, λ is the detection frame; Respectively F up and F down The maximum values ​​of the horizontal and vertical coordinates in the coordinates of each detection frame constitute the relative coordinates and , then F up and F down The coordinates of the remaining detection boxes in are mapped to the same coordinate system by subtracting the corresponding relative coordinates, and we get , ; Then F 上 and F 下 The IoU calculation is performed on all the bolt detection frame coordinates in the image to determine whether the corresponding bolts are the same bolt. If they are not the same bolt, the current bolt detection frame coordinates are added to the F 上 The calculation method is as follows: in, is the preset overlap factor. In a second aspect, the present application provides a highway guardrail bolt detection system based on hierarchical detection, comprising: A camera image acquisition module, comprising an upper camera and a lower camera installed on the same vehicle and photographing the same position of a highway guardrail from different directions; A data acquisition module, used to respectively acquire a first highway guardrail image and a second highway guardrail image captured by an upper camera and a lower camera at the same position of the highway guardrail; A first detection module is used to input the first highway guardrail image and the second highway guardrail image into the trained first detection model, respectively, identify and cut out the bolt group from the first highway guardrail image and the second highway guardrail image, and obtain a first bolt image to be detected and a second bolt image to be detected; A second detection module is used to use the trained second detection model to identify the type and state of each bolt in the first bolt image to be detected and the second bolt image to be detected, respectively, to obtain a first recognition result set and a second recognition result set, wherein the recognition result set includes a bolt recognition result and a bolt detection frame coordinate corresponding to each bolt; The deduplication module is used to deduplicate the bolt recognition images in the first recognition result set and the second recognition result set by using the bolt detection frame coordinates to obtain the bolt recognition result of the highway guardrail at the current position.

[0014] The inventive concept of this application is: Existing deep learning-based target detection algorithms detect the position and category of existing targets on an image by inputting image features. However, the cross-sectional shape of the corrugated guardrail is a curve, and the bolts fixed on it are also distributed in a curve, resulting in the bolts photographed from the front of the high-speed camera not being on the same horizontal plane. Some bolts are severely blocked and missing in the image, which has a significant impact on the actual accuracy of highway guardrail bolt detection. In addition, the images collected by the high-speed camera are fixed with high resolution, and will be resized and downsampled before being sent to the model for inference. Whether bilinear interpolation or nearest neighbor interpolation is used, the bolt features, which are already small targets, will face significant loss. Therefore, neither the traditional manually designed feature detection method nor the deep learning method can be used directly in the above scenario, and a good coordination between detection efficiency and accuracy has been achieved, which brings challenges to model recognition and detection in actual complex scenarios. The bolts on the guardrail come in a variety of sizes and shapes. Fusion of different levels of semantic information has a significant impact on the decoupling of multiple detection heads, which requires the neck network to be more efficient. In the YOLO family, the neck usually uses feature pyramid networks and their variants to fuse multi-level features. A common strategy for multi-scale feature extraction is to use top-down and bottom-up feature pyramid networks. However, current information fusion methods have obvious defects. When cross-layer information fusion is performed, the traditional feature pyramid structure cannot transfer information losslessly, which leads to the loss or degradation of features. This situation hinders the ability of non-adjacent layers to effectively integrate information, thereby limiting YOLO's performance in information fusion.

[0015] Based on the above reasons, this application fixes two upper and lower high-speed cameras on a horizontal stabilizer so that the equipment can be adapted to different vehicles, and then adjusts the upper and lower cameras to shoot the upper and lower angles of the same position to capture images of highway guardrail bolts from different directions, thereby solving the problem that the bolts are not in the same plane, and then constructs the original image dataset of highway guardrail bolts. The original image dataset is sent to the yolov8 network to detect the distribution of the bolt group. After the bolt group is cut on the original image, the original features of the bolt are retained to the greatest extent as a new bolt model dataset. After the new bolt model dataset is detected by the constructed yolov8-bolt model, the detection frame and category information are used to deduplicate the bolts of the upper and lower cameras to achieve the purpose of finding and filling gaps. In addition, in order to improve the detection accuracy of the model, the neck structure of the second detection model of the present application adopts the ASFF-GD (Adaptive Spatial Fusion-Gathering Distribution) neck structure. The adaptive spatial fusion network in the progressive feature pyramid is introduced in the early stage of the neck structure to facilitate direct exchange between non-adjacent levels, thereby alleviating the semantic information gap between feature maps at different levels. This integration aims to improve the recognition accuracy of the model. The progressive feature pyramid method is adopted to assign different spatial weights to different features through adaptive spatial feature fusion. This improvement emphasizes the importance of key-level feature maps and reduces the impact of contradictory information from different objects on feature fusion, ensuring that feature semantic information from different levels is effectively aggregated throughout the progressive fusion process. In the later stage of the neck structure, an aggregation-distribution mechanism is introduced to achieve efficient information exchange in YOLO. By globally fusing multi-level features and injecting global information into higher levels, feature information is extracted and fused using convolution-based and attention-based modules respectively. This significantly enhances the information fusion capability of the neck without significantly increasing latency, further promoting information flow, and thus improving the performance of the model between different object sizes.

[0016] The present invention has the beneficial effects: 1. The present invention uses two high-speed cameras, one above and one below, to shoot the bolts on the corrugated guardrail, thereby solving the blind spot problem of a single camera caused by the bolts not being in the same plane. The first detection model is used to detect the bolt group in the guardrail, and these bolts are cut out as a new data set. Subsequently, the second detection model is used to detect the cut data set, minimizing the significant loss of bolt features caused by resizing and downsampling. The detection results of the upper and lower cameras are deduplicated to avoid missed bolt detection due to the blind spots of the upper and lower cameras, thereby effectively checking for leaks and filling gaps to ensure the accuracy of the number of bolts.

[0017] 2. The neck structure of the second detection model adopts the ASFF-GD neck structure, which consists of the adaptive spatial fusion and aggregation-distribution mechanism of the asymptotic feature pyramid. In the early stage, the progressive feature pyramid method is used to gradually fuse features to ensure that the semantic information of features at different levels can be effectively aggregated during the entire fusion process, which helps to alleviate the semantic information differences between different feature maps. The improvement of adaptive spatial fusion highlights the importance of key-level feature maps and reduces the impact of contradictory information from different objects, aiming to improve the recognition accuracy of the model. In the later stage, the aggregation-distribution mechanism is introduced to achieve efficient information exchange in YOLO, further enhancing the information fusion ability of the neck without significantly increasing the delay, thereby improving the performance of the model between different object sizes. By using a unified module to collect and fuse information at each level and distribute it to different levels, it not only avoids the information loss inherent in the traditional structure, but also enhances the information fusion ability of the neck without significantly increasing the delay, and improves the robustness and accuracy of the model in actual complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flow chart of a highway guardrail bolt detection method based on hierarchical detection provided by an embodiment of the present invention; Figure 2 A schematic diagram of the dual-camera distribution provided in an embodiment of the present invention; Figure 3 A schematic diagram of a neck structure provided by an embodiment of the present invention; Figure 4 A schematic diagram of an adaptive spatial fusion process provided by an embodiment of the present invention; Figure 5 A schematic diagram of the composition of the aggregation distribution mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0021] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0022] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness.One of ordinary skill in the art will recognize that various changes and modifications may be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0023] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0024] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0025] Example 1 like Figure 1 As shown, on the one hand, the present invention provides a highway guardrail bolt detection method based on hierarchical detection, which specifically includes the following steps: S1, respectively acquiring a first highway guardrail image and a second highway guardrail image captured by an upper camera and a lower camera at the same position of the highway guardrail; In this application, both the upper camera and the lower camera are high-speed cameras, which are fixed on custom stabilizers respectively so as to adapt the cameras to different vehicles, and triggers at fixed distances are set on the vehicles to trigger the upper and lower cameras in real time to shoot images of highway guardrails in an exposure manner, such as Figure 2 As shown, when installing the upper and lower cameras, ensure that all bolts on the guardrail are fully included during shooting.

[0026] S2, inputting the first highway guardrail image and the second highway guardrail image into the trained first detection model respectively, identifying and cutting out the bolt group from the first highway guardrail image and the second highway guardrail image respectively, to obtain a first bolt image to be detected and a second bolt image to be detected; Specifically, the first detection model uses the YOLOv8 model to perform preliminary processing on the image, extract the image of the area where the bolt is located from the image taken by the camera, and remove redundant background in the image so that it can be input into the second detection model for accurate identification.

[0027] S3, using the trained second detection model, respectively identifying the bolt type and state of each bolt in the first bolt image to be detected and the second bolt image to be detected, to obtain a first recognition result set and a second recognition result set, wherein the recognition result set includes a bolt recognition result and a bolt detection frame coordinate corresponding to each bolt; The second training model adopts the YOLOv8-bolt model with an improved neck structure, wherein the YOLOv8-bolt model includes an input layer, a backbone network, a neck structure, and a head output layer. The neck structure adopts an adaptive spatial fusion network with an asymptotic feature pyramid, and introduces an aggregation-distribution mechanism in the adaptive spatial fusion network.

[0028] It can be understood that both detection models in this application are based on the YOLOv8 model. The first detection model performs preliminary detection and clipping on the highway guardrail image captured by the camera, and provides the clipped data set for the second detection model. The training process of the first detection model is: About 20,000 highway guardrail bolt images collected by the upper camera and the lower camera are obtained as training sets and verification sets, with a resolution of 1199*1919, to construct the first detection model image data set. According to the bolt distribution, the bolt distribution of the highway guardrail bolt images in the first detection model image data set is divided into five categories, namely: guardrail part with columns and splicing lines, guardrail part with columns but no splicing lines, guardrail part with no columns but splicing lines, guardrail part with no columns and no splicing lines, and the entire guardrail. After classification and annotation, they are sent to the yolov8 network for training. The trained yolov8 network can detect five different bolt distributions. According to the distribution of bolts, the images of highway guardrail bolts were cut out and manually reviewed, and then divided into four categories according to the bolts and types for classification and labeling. Specifically, the image data sets of connecting bolts, splicing bolts, contour marks, and bolt falling off were obtained, and about 20,000 images of four major types of bolts in different postures and occlusion conditions were obtained. For the image data set of the second detection model mentioned above, the convolutional neural network in the internal structure of YOLOv8-bolt was used to extract the feature information of the image, and the feature information set of highway guardrail bolts was constructed. The bolt feature information set was used as the training set, and the bolt category was used as the label to train the model to obtain the trained second detection model.

[0029] like Figure 3 As shown in the figure, the specific composition of the second detection model includes the backbone network Backbone, the neck structure (Neck) and the head output (Head). The convolutional neural network of the backbone network extracts the image features from the bolt training image. The neck structure includes an adaptive spatial fusion network. The adaptive fusion network has three fusion modules with different spatial weights, namely ASSF-1, ASSF-2 and ASSF-3. Specifically, the features of different layers are extracted from the backbone network to obtain a multi-scale feature map set {C3, C4, C5} and its corresponding feature layer numbers are {256, 512, 1024} respectively. In order to align the dimensions and prepare for feature fusion, each multi-scale feature map in the multi-scale feature map set is upsampled by 1 × 1 convolution and bilinear interpolation in turn. Conversely, in order to downsample at the desired rate, different convolution kernels and strides are used. Specifically, two rounds of downsampling are performed by 2 × 2 convolution with a stride of 2, four rounds of downsampling are performed by 4 × 4 convolution with a stride of 4, and 8 rounds of downsampling are performed by 8 × 8 convolution with a stride of 8, and finally the set of feature maps to be fused {C'3, C'4, C'5} is obtained; The feature map set to be fused {C'3, C'4, C'5} is respectively input into the adaptive spatial fusion modules ASSF-1, ASSF-2 and ASSF-3 of the multi-level features under different spatial weights to obtain the corresponding synthetic feature set {B3, B4, B5}. Figure 3 As shown in Figure 2, three different levels of feature maps are integrated to make represents the feature vector at position from the feature layer at stage, k ∈{1,2,3}, corresponding to {C3 ,C4 ,C5}, l ∈{1,2,3}, corresponding to {B3, B4, B5}, the synthetic feature vector is obtained by adaptive spatial fusion of multi-level features, expressed as , the linear combination of the eigenvectors is as follows: (1) in , , Representation stage l The spatial weights of the three feature layers {C'3, C'4, C'5} are affected by By changing the spatial weights, we can finally get {B3, B4, B5}. In general, in order to solve the change in the number of fused features in the adaptive feature pyramid network at different stages, a stage-specific adaptive spatial fusion module allocation is introduced.

[0030] like Figure 4 As shown, the neck structure adopts a gathering-distribution mechanism and is divided into: 1. Gathering High-level feature alignment module High-FAM, It includes two pooling layers avgpool1 and avgpool2. Based on the feature layer size of B5, avgpool1 reduces the feature layer size of the synthetic feature B4 to the same as B5, and avgpool2 reduces the feature layer size of the synthetic feature B3 to the same as B5. Specifically, the resize of the input feature is avgpool reduces the feature size to the smallest size within the feature group ; Since the transformer fusion module extracts high-level information, the pooling operation helps to aggregate information and reduces the computational requirements of subsequent steps in the transformer module. Add the number of C channels to derive F hf ; The high-level information fusion module includes several stacked transformer fusion modules (the number of transformer fusion modules is represented by L) and a split module (Split), which includes three steps: (1) the information derived from the high-level feature alignment module is F hf Combined with the transformer fusion module F f , (2) Through a 1×1 convolution operation, F f The number of C channels of B4 is reduced to the sum of the number of C channels of B5. (3) Through the split operation, the C channel dimension is divided into F M4 and F M5 , and merged with the corresponding current level feature (B4 or B5). The formula is: (2) Specifically, each transformer fusion module includes a multi-head attention block (Multi-HeadAttention) and a feed-forward network (Feed-Forword), and residual connections are used between modules. To configure the multi-head attention block, the head size of the key K and the query Q is assigned to D (e.g., 16) channels, and the value V = 2D (e.g., 32) channels. To speed up reasoning, the speed-unfriendly operator layer normalization is replaced with batch normalization for each convolution, and all GELU activations are replaced with ReLU. This minimizes the impact of the transformer module on the model speed, aiming to balance speed and computational overhead.

[0031] The residual is calculated by: hf The output after inputting into the multi-head attention block is the same as that from F hf After the residual calculation is performed, it is input into the feedforward network; then the input of the feedforward network and the output of the feedforward network are calculated again to obtain F f .

[0032] 2. For F M4 , F M5 The following distribution process is performed, and the distribution (Inject) includes: In order to more effectively inject global information into different layers, attention operations are used to fuse information, such as Figure 5 As shown. Each synthesized feature is used as the local feature input of the current layer, expressed as F local ={B4, B5}, and the output of the split module is used as the global feature input of the current layer, expressed as F global ={ F M4 , F M5}; right F global Each feature in is calculated using two different convolutions, and we get and , by F local Input into the convolution calculation module and get ; By , and Perform attention fusion calculation and according to F global The size of is scaled using average pooling (Avgpool) or bilinear interpolation (bilinear) and , to ensure correct alignment. Finally, the fused features are obtainedF out At the end of each attention fusion calculation, RepBlock is used to further F out Extract and fuse information, and finally get the output Ni of the neck structure. Then the formula is: (3) Among them, B i Take B4 or B5, N i Take N4 and N5, from Figure 4 It can be seen that there are two information distribution modules (Inject) in the distribution mechanism of the neck structure. The two Injects correspond to B4 and B5. N4 and N5 represent the outputs corresponding to B4 or B5, and B3 is directly output as N3. When it is finally input to the head output layer (Head), after decoding N3, N4 and N5, the classification results and detection boxes of the bolts can be output.

[0033] It can be seen from Table 1 that the detection results of the YOLOv8-bolt model with improved neck structure proposed in this application are compared with the existing algorithms.

[0034] Table 1 Evaluation indicators of detection results of each algorithm S4. After deduplication of the bolt recognition images in the first recognition result set and the second recognition result set using the bolt detection frame coordinates, the bolt recognition result of the highway guardrail at the current position is obtained, and the deduplication bolt recognition result is mapped back to the first highway guardrail image to obtain the final bolt recognition frame and category. The specific deduplication process is: According to the first recognition result set, the bolt detection frame coordinates corresponding to the m bolts are marked as , according to the second recognition result set, the bolt detection frame coordinates corresponding to the n bolts are marked as ,in, λ is the detection frame; Respectively F up and F down The maximum values ​​of the horizontal and vertical coordinates in the coordinates of each detection frame constitute the relative coordinates and , then F up and F down The coordinates of the remaining detection boxes in are mapped to the same coordinate system by subtracting the corresponding relative coordinates, and we get , , the mapping calculation formula is as follows: (4) Then F 上 and F 下 The IoU calculation is performed on all the bolt detection frame coordinates in the image to determine whether the corresponding bolts are the same bolt. If they are not the same bolt, the current bolt detection frame coordinates are added to the F 上 The calculation method is as follows: (5) in, is the preset coincidence factor, =0.2. In the final recognition result, the first bolt image to be detected is used as the original image by default in this application, and the deduplicated image can be F 上 Generate csv statistics table and test result graph, and display them in real time through the front-end interface. Example 2 Based on the detection method of Example 1, this embodiment provides a highway guardrail bolt detection system based on hierarchical detection, including: A camera image acquisition module, comprising an upper camera and a lower camera installed on the same vehicle and photographing the same position of a highway guardrail from different directions; A data acquisition module, used to respectively acquire a first highway guardrail image and a second highway guardrail image captured by an upper camera and a lower camera at the same position of the highway guardrail; A first detection module is used to input the first highway guardrail image and the second highway guardrail image into the trained first detection model, respectively, identify and cut out the bolt group from the first highway guardrail image and the second highway guardrail image, and obtain a first bolt image to be detected and a second bolt image to be detected; A second detection module is used to use the trained second detection model to identify the type and state of each bolt in the first bolt image to be detected and the second bolt image to be detected, respectively, to obtain a first recognition result set and a second recognition result set, wherein the recognition result set includes a bolt recognition result and a bolt detection frame coordinate corresponding to each bolt; The deduplication module is used to deduplicate the bolt recognition images in the first recognition result set and the second recognition result set by using the bolt detection frame coordinates to obtain the bolt recognition result of the highway guardrail at the current position.

[0035] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. According to the technical essence of the present invention, within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A highway guardrail bolt detection method based on hierarchical detection, characterized in that: The specific steps include: S1, respectively acquiring a first highway guardrail image and a second highway guardrail image captured by an upper camera and a lower camera at the same position of the highway guardrail; S2, inputting the first highway guardrail image and the second highway guardrail image into the trained first detection model respectively, identifying and cutting out the bolt group from the first highway guardrail image and the second highway guardrail image respectively, to obtain a first bolt image to be detected and a second bolt image to be detected; S3, using the trained second detection model, respectively identifying the bolt type and state of each bolt in the first bolt image to be detected and the second bolt image to be detected, to obtain a first recognition result set and a second recognition result set, wherein the recognition result set includes a bolt recognition result and a bolt detection frame coordinate corresponding to each bolt; S4. After deduplication of the bolt recognition images in the first recognition result set and the second recognition result set using the bolt detection frame coordinates, the bolt recognition result of the highway guardrail at the current position is obtained.

2. A highway guardrail bolt detection method based on hierarchical detection according to claim 1, characterized in that: The first detection model uses the YOLOv8 model, and the training process of the first detection model is: S21, obtaining a number of highway guardrail bolt images captured by the upper camera and the lower camera to construct a first detection model image data set, classifying and annotating the highway guardrail bolt images in the first detection model image data set according to the bolt distribution, and then sending them to the yolov8 network to detect the bolt distribution; S22. Cut out a bolt group from the highway guardrail bolt image according to the bolt distribution, classify and label each bolt in the bolt group, and store them in a corresponding classified image dataset as data for training the second detection model. The classified image dataset includes: connecting bolts, splicing bolts, contour marks, and bolt-off image datasets.

3. A highway guardrail bolt detection method based on hierarchical detection according to claim 1, characterized in that: The second detection model adopts the YOLOv8-bolt model, which includes an input layer, a backbone network, a neck structure, and a head output layer. The neck structure adopts an adaptive spatial fusion network with an asymptotic feature pyramid, and introduces an aggregation-distribution mechanism in the adaptive spatial fusion network.

4. A highway guardrail bolt detection method based on hierarchical detection according to claim 3, characterized in that: The fusion process of the adaptive spatial fusion network is: S31, extract features of different layers from the backbone network, and obtain a multi-scale feature map set {C3, C4, C5} and its corresponding feature layer numbers {n, 2n, 4n} respectively; S32, for each multi-scale feature map in the multi-scale feature map set, sequentially use 1×1 convolution and bilinear interpolation to upsample the multi-scale feature map, apply 2×2 convolution with a stride of 2 for two rounds of downsampling, apply 4×4 convolution with a stride of 4 for four rounds of downsampling, and apply 8×8 convolution with a stride of 8 for 8 rounds of downsampling, and finally obtain the feature map set to be fused {C'3, C'4, C'5}; S33. The feature graph set to be fused {C'3, C'4, C'5} is obtained by adaptive spatial fusion of multi-level features under different spatial weights to obtain the corresponding synthetic feature set {B3, B4, B5}.

5. A highway guardrail bolt detection method based on hierarchical detection according to claim 4, characterized in that: Aggregation in the aggregation-distribution mechanism includes: High-level feature alignment module High-FAM, It is used to reduce the feature layer size of each synthetic feature of the synthetic feature set {B3, B4, B5} to the same size and then export it through the channel , ; A high-level information fusion module, including several stacked transformer fusion modules, is used to derive the information from the high-level feature alignment module. Combine to get , ; The split module is used to split F f Divided into F along the channel dimension M4 and F M5 , .

6. A highway guardrail bolt detection method based on hierarchical detection according to claim 5, characterized in that: High-level feature alignment module High-FAM It includes two pooling layers avgpool1 and avgpool2. avgpool1 reduces the feature layer size of the synthetic feature B4 to the same as B5, and avgpool2 reduces the feature layer size of the synthetic feature B3 to the same as B5.

7. A highway guardrail bolt detection method based on hierarchical detection according to claim 5, characterized in that: Each transformer fusion module consists of a multi-head attention block and a feed-forward network. hf The output after inputting into the multi-head attention block is the same as that from F hf After residual calculation, it is input into the feedforward network; The feedforward network consists of two 1x1 convolutional layers. A deep convolutional layer is introduced between the two 1x1 convolutional layers. The expansion factor of the feedforward network is set to 2. All GELU activations are replaced with ReLU activations. The input and output of the feedforward network are calculated again to obtain F. f .

8. A highway guardrail bolt detection method based on hierarchical detection according to claim 5, characterized in that: The distribution in the aggregation-distribution mechanism includes: The attention operation is used to fuse information, and each synthesized feature is input as the local feature of the current layer, which is expressed as ={B4, B5}, and the output of the split module is used as the global feature input of the current layer, expressed as ={ F M4 , F M5 }; right Each feature in is calculated using two different convolutions, and we get and , by Input into the convolution calculation module and get ; By , and Perform attention fusion calculation and according to The size of , using average pooling or bilinear interpolation to scale and , and finally get the fused features At the end of each attention fusion calculation, RepBlock is used to further The information is extracted and fused, and finally the output of the neck structure is obtained.

9. A highway guardrail bolt detection method based on hierarchical detection according to claim 1, characterized in that: The specific process of deduplication in step S4 is as follows: According to the first recognition result set, the bolt detection frame coordinates corresponding to the m bolts are marked as , according to the second recognition result set, the bolt detection frame coordinates corresponding to the n bolts are marked as ,in, λ is the detection frame; Respectively F up and F down The maximum values ​​of the horizontal and vertical coordinates in the coordinates of each detection frame constitute the relative coordinates and , then F up and F down The coordinates of the remaining detection boxes in are mapped to the same coordinate system by subtracting the corresponding relative coordinates, and we get , ; Then F 上 and F 下 The IoU calculation is performed on all the bolt detection frame coordinates in the image to determine whether the corresponding bolts are the same bolt. If they are not the same bolt, the current bolt detection frame coordinates are added to the F 上 The calculation method is as follows: in, is the preset overlap factor.

10. A highway guardrail bolt detection system based on hierarchical detection, characterized in that: include: A camera image acquisition module, comprising an upper camera and a lower camera installed on the same vehicle and photographing the same position of a highway guardrail from different directions; A data acquisition module, used to respectively acquire a first highway guardrail image and a second highway guardrail image captured by an upper camera and a lower camera at the same position of the highway guardrail; A first detection module is used to input the first highway guardrail image and the second highway guardrail image into the trained first detection model, respectively, identify and cut out the bolt group from the first highway guardrail image and the second highway guardrail image, and obtain a first to-be-detected bolt image and a second to-be-detected bolt image; The second detection module is used to use the trained second detection model to identify the type and state of each bolt in the first bolt image to be detected and the second bolt image to be detected, respectively, to obtain a first recognition result set and a second recognition result set, wherein the recognition result set includes a bolt recognition result and a bolt detection frame coordinate corresponding to each bolt; The deduplication module is used to deduplicate the bolt recognition images in the first recognition result set and the second recognition result set by using the bolt detection frame coordinates to obtain the bolt recognition result of the highway guardrail at the current position.

Images