Unmanned aerial vehicle image bridge deck crack real-time detection method and system based on BCYOLO lightweight coupling head model

Abstract

The invention discloses an unmanned aerial vehicle image bridge deck crack real-time detection method and system based on a BCYOLO lightweight coupling head model, belongs to the technical field of computer vision, unmanned aerial vehicle application technologies and edge AI hardware, and solves the problem that when an existing convolutional neural network is used for processing complex geometric structures and local slender and curled structures, the defects of the existing convolutional neural network are overcome. And the characteristics cannot be effectively extracted by adopting a standard convolution kernel, so that the capturing capability of local structure characteristics is insufficient. The method comprises the following steps: acquiring a bridge surface crack image and preprocessing to obtain a data set; a data set is adopted to train the constructed BCYOLO network; and the unmanned aerial vehicle collects high-resolution images or video data of different positions of the bridge and inputs the high-resolution images or video data into the trained BCYOLO network for bridge deck crack real-time detection. The method is used for bridge deck crack real-time detection.

Description

Technical Field

[0001] A method and system for real-time detection of bridge deck cracks using drone images based on a BCYOLO lightweight coupling head model are provided, which are used for real-time detection of bridge deck cracks and belong to the fields of computer vision, drone application technology and edge AI hardware technology. Background Art

[0002] With the development of economy, China's road and bridge industry has made remarkable progress, and most modern bridges are made of concrete. However, over time, cracks often appear on the surface of concrete, which not only affects the structural health of the bridge, but also may endanger the safety of pedestrians. Therefore, timely detection and treatment of cracks are crucial in bridge maintenance. Due to the increase in the number of concrete bridges, the workload of inspection and maintenance is also increasing. At present, most bridge damage detection relies on visual inspection or non-destructive testing, which requires high professional knowledge and skills of workers, especially when working at heights in bad weather. Therefore, it is particularly important to propose a reasonable and effective bridge crack detection method. In recent years, deep learning has made significant progress in the field of image recognition and detection, but its application in bridge crack detection is still challenging. The multi-scale, variability and tubular structure of cracks make it difficult for current methods to deal with them effectively. For example, it is difficult for the model to capture small and slender cracks, and it is also difficult to identify wider and irregular cracks. Texture, illumination changes and other noises in complex backgrounds also interfere with the detection results, increasing the probability of false positives and false negatives. Therefore, improving the generalization ability of the model in multi-scale crack recognition and robustness in complex backgrounds has become a key issue.

[0003] The existing method of bridge surface crack detection takes cracks as the target object. Based on the input image, the computer determines whether it contains cracks. If so, a bounding box is used to locate the cracks in the image. This work combines the improved one-stage network YOLOv8 model BCYOLO (Bridge Crack You Once Look Once) with the NPU edge computing device RK3588 to design a set of detection equipment for surface cracks in tubular topology bridges. The YOLO series network adopts a three-part structure (trunk, neck and detection head). Although the YOLOv8 algorithm is already quite efficient, its performance in complex backgrounds, low pixels and multi-scale tubular targets is still insufficient, mainly due to the relatively low quality of features extracted by the detection network.

[0004] In summary, the prior art has the following technical problems:

[0005] 1. Insufficient feature extraction: When dealing with complex geometric structures and local elongated curly structures, the existing convolutional neural network (CNN) cannot effectively extract these features using standard convolution kernels, resulting in insufficient ability to capture local structural features;

[0006] 2. Fusion of target and background: In the actual detection environment, when the target (such as bridge deck cracks) is highly integrated with the background, the existing technology cannot effectively distinguish the target from the background, resulting in a decrease in recognition accuracy;

[0007] 3. Background noise interference: Background noise in complex environments can easily interfere with the detection model, especially in the context of high similarity, making it difficult for the model to correctly classify and locate cracks;

[0008] 4. High computational complexity: High-precision models are usually accompanied by high computational complexity and are not suitable for use in resource-constrained environments, such as real-time drone detection, which limits the scope of application of the model;

[0009] 5. Real-time and bandwidth limitations: Under poor network conditions or insufficient bandwidth, existing technologies rely on the transmission of a large number of pictures and videos, which can easily lead to loss of real-time performance and cannot meet the needs of real-time detection. Summary of the invention

[0010] The purpose of the present invention is to provide a real-time detection method and system for bridge deck cracks using drone images based on the BCYOLO lightweight coupling head model, so as to solve the problem that the existing convolutional neural network cannot effectively extract these features when processing complex geometric structures and local slender curly structures using standard convolution kernels, resulting in insufficient ability to capture local structural features.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is:

[0012] A real-time detection method for bridge deck cracks using UAV images based on a BCYOLO lightweight coupling head model includes the following steps:

[0013] Step 1, collecting bridge surface crack images and preprocessing them to obtain a data set;

[0014] Step 2: Use the dataset to train the constructed BCYOLO network;

[0015] Step 3: The drone collects high-resolution images or video data at different locations of the bridge and inputs them into the trained BCYOLO network for real-time detection of bridge deck cracks.

[0016] Furthermore, the BCYOLO network in step 2 is based on the YOLOv8 network, replacing the backbone structure with a C2f_dynamic snake convolution backbone structure composed of C2f_dynamic snake convolution modules, and replacing the YOLOv8 original detection head with a decoupling head carrying an attention mechanism.

[0017] Furthermore, the C2f_dynamic snake convolution backbone structure includes a convolution layer, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a SPPF layer and a SimAM module connected in sequence.

[0018] Further, the C2f_dynamic serpentine convolution module includes a first convolution layer, a split layer, an even-numbered dynamic serpentine convolution layer connected in sequence, and a splicing layer that receives the output of each even-numbered dynamic serpentine convolution layer and adds the input of the dynamic serpentine convolution layer of the previous layer to splice the output of the first convolution layer and the split layer, and a second convolution layer that receives the output of the splicing layer, wherein, among the even-numbered dynamic serpentine convolution layers, starting from the second odd-numbered dynamic serpentine convolution layer, the input of the odd-numbered dynamic serpentine convolution layer is the result of adding the output of the previous even-numbered dynamic serpentine convolution layer to the input of the previous odd-numbered dynamic serpentine convolution layer;

[0019] The dynamic snake-shaped convolution layer performs deformation learning on the input features, including sequentially performing two-dimensional convolution and multiple offset processing, and then performing snake-shaped convolution processing on the input features and the results after deformation learning to obtain the output result;

[0020] The SimAM module includes generating a three-dimensional weight for each channel of the feature map output by the SPPF layer, and fusing the three-dimensional weight with the output of the SPPF layer by point-by-point multiplication;

[0021] The SPPF layer includes a convolutional layer, three two-dimensional maximum pooling layers, a splicing layer and a convolutional layer connected in sequence;

[0022] Each convolution layer in the C2f_dynamic snake-shaped convolution backbone structure includes a two-dimensional convolution layer, a two-dimensional batch normalization layer and an activation function layer SiLU which are connected in sequence.

[0023] Furthermore, the decoupling head carrying the attention mechanism has a CBAM module, that is, on the basis of the original detection head of YOLOv8, the CBAM module is merged into two parallel branches for receiving the features of the previous layer. The CBAM module includes a channel attention module for receiving the features of the previous layer, and a spatial attention module for receiving the result of adding the output of the channel attention module to the input. The output of the spatial attention module after adding the input is the output of the CBAM module.

[0024] Furthermore, the two-dimensional convolution in the serpentine convolution layer uses a 3×3 convolution kernel with a dilation coefficient of 1, and the formula for the coordinate set of the two-dimensional convolution is:

[0025] K={(x+i,y+j)∣i,j∈{-1,0,1}}

[0026] Where K represents the coordinate set of the two-dimensional convolution, (x+i,y+j) represents the coordinate of the two-dimensional convolution, i represents the position offset of the two-dimensional convolution pixel point (x,y) on the X axis, and j represents the position offset of the two-dimensional convolution pixel point (x,y) on the Y axis;

[0027] After the two-dimensional convolution processing, multiple offset processing is performed to obtain the offset sum, and the dynamic snake convolution coordinates are calculated based on the input features of the dynamic snake convolution layer based on the offset sum. The formula is:

[0028]

[0029] in, represents the center coordinate in the two-dimensional convolution, c = {0, 1, 2, 3, 4} represents the horizontal distance from the offset point to the center coordinate, Δx represents the offset δ on the X-axis, Δy represents the offset δ on the Y-axis, {δ|δ∈[-1, 1]}.

[0030] Furthermore, the SimAM module is based on the mean μ and variance σ of each channel of the feature map output by the SPPF layer 2 Calculate the energy function E;

[0031] Then, the energy function E is transformed nonlinearly to generate three-dimensional weights, and the three-dimensional weights are fused with the output of the SPPF layer by point-by-point multiplication;

[0032] The formula for the mean μ is:

[0033]

[0034] Variance σ 2 The formula is:

[0035]

[0036] The formula of energy function E is:

[0037]

[0038] The formula of the fused feature map is:

[0039]

[0040] Among them, X is the input feature map, that is, the feature map output by the SPPF layer, ⊙ is the dot product operation, λ is the hyperparameter, Q is the number of channels, t represents the target neuron, and x m is the mth target neuron, sigmoid represents the sigmoid function, Represents 3D weights.

[0041] Furthermore, the drone is a DJI M300 drone, and the trained BCYOLO network is transplanted into the RK3588 system of the drone.

[0042] A real-time detection system for bridge deck cracks using drone images based on a BCYOLO lightweight coupling head model, comprising a memory and a processor, wherein the memory stores a computer program for execution by the processor to implement the real-time detection method for bridge deck cracks using drone images based on the BCYOLO lightweight coupling head model.

[0043] Compared with the prior art, the advantages of the present invention are:

[0044] The present invention can efficiently and accurately detect bridge deck cracks in complex and changeable environments, significantly improves the accuracy and robustness of the detection system, and expands its application scope. It is particularly suitable for the needs of real-time detection of drone images, and has broad application prospects and practical significance, which are specifically embodied in the following aspects:

[0045] 1. The present invention expands the perception range of the model and enhances the feature extraction capability of complex geometric structures and slender curly structures by designing the C2f_dynamic snake convolution module and integrating dynamic snake convolution, thereby solving the problem of insufficient local feature extraction of standard convolution kernels in the prior art;

[0046] Second, the present invention introduces the SimAM module (simulated attention mechanism), which enhances the focus on the target, suppresses background interference, and improves the recognition accuracy when the target and background are highly fused, by calculating the importance score of each position on the feature map and adjusting the response, thereby solving the problem of reduced recognition accuracy caused by the fusion of the target and the background in the prior art;

[0047] 3. The present invention adopts a lightweight decoupling head that combines channel and spatial attention mechanisms to adaptively select and concentrate the most representative image features, reduce the interference of background noise, and improve the accuracy of classification and positioning, thereby solving the problem of difficulty in classification and positioning caused by background noise interference in complex environments in the prior art;

[0048] 4. The present invention adopts the BCYOLO network to perform multi-scale feature extraction strategy to effectively extract features of different scales and levels. Without significantly increasing the computational complexity, the performance of the detector is enhanced, making it suitable for resource-constrained environments, thereby solving the problem of high computational complexity of high-precision models in the prior art.

[0049] 5. The present invention avoids the timeliness loss caused by the transmission of a large number of pictures and videos by installing Rockchip RK3588 on the DJI M300 and combining it with NPU for on-site processing, thereby ensuring the real-time detection capability of the system under poor network conditions or insufficient bandwidth, thereby solving the problem of real-time loss under poor network conditions or insufficient bandwidth in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 Schematic diagram of the structure of the BCYOLO network in the present invention, where C2F represents the baseline network layer;

[0052] Figure 2 Schematic diagram of sampling comparison between standard convolution and dynamic snake convolution in the present invention;

[0053] Figure 3 Schematic diagram of a dynamic snake-shaped convolutional layer in the present invention;

[0054] Figure 4 Schematic diagram of coordinate calculation and receptive field of dynamic snake convolution in the present invention, wherein the left figure is coordinate calculation and the right figure is receptive field;

[0055] Figure 5 Schematic diagram of the structure of the C2f_dynamic serpentine convolution module in the C2f_dynamic serpentine convolution backbone structure of the present invention;

[0056] Figure 6 Schematic diagram of the structure of each convolution layer in the C2f_dynamic snake-shaped convolution backbone structure of the present invention;

[0057] Figure 7 It is a schematic diagram of the structure of the SPPF layer in the present invention;

[0058] Figure 8 It is a structural schematic diagram of the SimAM module in the present invention;

[0059] Fig. 9 It is a schematic diagram of the structure of the original detection head of YOLOv8 in the present invention and the improved decoupling head in the present invention;

[0060] Fig.10 It is a structural schematic diagram of the bridge deck crack detection system in the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] It should be noted that in the structural description of the present invention, although some layers or modules are described identically, their connection positions are different and the received inputs and outputs are also different.

[0063] In order to solve the above problems, the present invention proposes a real-time detection method for bridge cracks based on drone images of BCYOLO lightweight coupling head model. BCYOLO network is an optimized lightweight deep learning model, and integrates a lightweight coupling head with attention mechanism. Combined with drone images, BCYOLO network can play an important role in bridge detection. Drones can quickly cover large areas of bridges, improve detection efficiency, and flexibly adjust flight altitude and angle to obtain the best detection perspective, while reducing the work of personnel in high-risk environments and improving detection safety. The introduction of edge computing devices further improves the real-time and stability of detection. Edge computing performs calculations and processing at a location close to the data source, reducing data transmission delays and bandwidth requirements. Edge computing devices combined with BCYOLO network can process image data on the bridge surface in real time to achieve efficient and accurate crack detection. Edge computing devices have high-performance processing capabilities and low-latency response. By transferring computing tasks from the cloud to edge devices, the present invention shows good application prospects in areas with poor network conditions or involving data security. In the case of insufficient bandwidth, it effectively avoids the problem of timeliness loss by reducing the transmission of a large number of pictures and videos, ensuring data reliability and transmission efficiency.

[0064] Firstly, the crack images on the bridge surface are collected and preprocessed to obtain the data set;

[0065] Then use the data set to train the constructed BCYOLO network;

[0066] The BCYOLO network is based on the YOLOv8 network, replacing the backbone structure with a C2f_dynamic snake convolution backbone structure composed of C2f_dynamic snake convolution modules, and replacing the YOLOv8 original detection head with a decoupling head carrying an attention mechanism.

[0067] Specifically:

[0068] Considering the tubular and multi-scale characteristics of bridge deck cracks, a multi-view feature fusion strategy is proposed to enhance the focus on important features from multiple perspectives. Different from the standard convolution kernel that only extracts local features, the deformable convolution can adapt to the geometric deformation of different objects, and a network backbone structure consisting of C2f_dynamic snake convolution modules is designed. Figure 2 As shown, the overall structure of the dynamic snake convolution presents a thin-walled shape, which is crucial for focusing on the tubular structure of the bridge deck crack. Traditional standard convolution operates on a fixed grid, where each sample point is weighted by the convolution kernel, which is not suitable for detecting unique structures in the target.

[0069] like Figure 1 As shown, the C2f_dynamic snake convolution backbone structure includes a sequentially connected convolution layer, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, an SPPF layer, and a SimAM module. The C2f_dynamic snake convolution module in the C2f_dynamic snake convolution backbone structure is an improvement on the C2f layer in the backbone structure of the YOLOv8 network.

[0070] like Figure 5 As shown, the C2f_dynamic serpentine convolution module includes a first convolution layer, a split layer, an even-numbered dynamic serpentine convolution layer connected in sequence, and a splicing layer that receives the output of each even-numbered dynamic serpentine convolution layer and adds the input of the dynamic serpentine convolution layer of the previous layer to splice the output of the first convolution layer and the split layer, and a second convolution layer that receives the output of the splicing layer, wherein the even-numbered dynamic serpentine convolution layer starts from the second odd-numbered dynamic serpentine convolution layer, and the input of the odd-numbered dynamic serpentine convolution layer is the result of adding the output of the previous even-numbered dynamic serpentine convolution layer to the input of the previous odd-numbered dynamic serpentine convolution layer;

[0071] The structure of the dynamic snake convolution layer is as follows Figure 3 As shown in , a continuity constraint is introduced in the design of the convolution kernel, allowing each convolution position to choose its oscillation direction based on the previous position. This ensures the continuity of the receptive field while maintaining flexibility similar to the multi-scale structure of the bridge deck cracks. The dynamic snake convolution layer performs deformation learning on the input features, including sequentially performing two-dimensional convolution and multiple offset processing, and then performing snake convolution processing on the input features and the results after deformation learning to obtain the output result; as shown in Figure 4 As shown, specifically:

[0072] The two-dimensional convolution in the snake convolution layer uses a 3×3 convolution kernel with a dilation coefficient of 1, so the formula for the coordinate set of the two-dimensional convolution is:

[0073] K={(x+i,y+j)∣i,j∈{-1,0,1}}

[0074] Where K represents the coordinate set of the two-dimensional convolution, (x+i,y+j) represents the coordinate of the two-dimensional convolution, i represents the position offset of the two-dimensional convolution pixel point (x,y) on the X axis, and j represents the position offset of the two-dimensional convolution pixel point (x,y) on the Y axis;

[0075] After the two-dimensional convolution processing, multiple offset processing is performed to obtain the offset sum, and the dynamic snake convolution coordinates are calculated based on the input features of the dynamic snake convolution layer based on the offset sum. The formula is:

[0076]

[0077] in, Represents the center coordinate in the two-dimensional convolution, c = {0, 1, 2, 3, 4} represents the horizontal distance from the offset point to the center coordinate, Δx represents the offset δ on the X-axis, Δy represents the offset δ on the Y-axis, {δ|δ∈[-1, 1]}. That is, the first convolution layer with a 1×1 convolution kernel is used to adjust the number of channels in the input feature map, and the feature map is further processed by a technical feature segmentation different from the 1×1 convolution (i.e., the input feature map is divided into multiple sub-feature maps by convolution operations or other methods for more refined feature extraction). After processing, by repeatedly applying dynamic snake convolution layers, not only the perceptual range of the model is expanded, but also the diversity of information flow is enhanced by adding jump connections. This structural optimization is particularly effective in enhancing the model's recognition of edge details and complex textures, because it can produce more detailed gradient distributions, thereby capturing broader and more hierarchical information during feature extraction. This design enables the BCYOLO network to show greater flexibility and accuracy in processing objects with complex boundaries and textures.

[0078] Complex backgrounds usually affect the model's effect on target feature extraction. A common solution is to add an attention mechanism, but most attention mechanisms will lead to a significant increase in network parameters. A key feature of the SimAM module is its lightweight, non-parametric design. Compared with other attention mechanisms, it adds almost no additional parameters. The SimAM module includes generating three-dimensional weights for each channel of the feature map output by the SPPF layer, and fusing the three-dimensional weights with the output of the SPPF layer by point-by-point multiplication; specifically:

[0079] like Figure 8 As shown in the figure, the SimAM module is based on the mean μ and variance σ of each channel of the feature map output by the SPPF layer 2Calculate the energy function E; then generate three-dimensional weights by performing nonlinear transformation on the energy function E, and fuse the three-dimensional weights with the output of the SPPF layer by point-by-point multiplication;

[0080] The formula for the mean μ is:

[0081]

[0082] Variance σ 2 The formula is:

[0083]

[0084] The formula of energy function E is:

[0085]

[0086] The formula of the fused feature map is:

[0087]

[0088] Among them, X is the input feature map, that is, the feature map output by the SPPF layer, ⊙ is the dot product operation, λ is the hyperparameter, Q is the number of channels, t represents the target neuron, and x m is the mth target neuron, sigmoid represents the sigmoid function, Represents 3D weights.

[0089] like Figure 7 As shown, the SPPF layer includes a convolutional layer, three two-dimensional maximum pooling layers, a splicing layer and a convolutional layer connected in sequence;

[0090] like Figure 6 As shown, each convolution layer in the C2f_dynamic snake-shaped convolution backbone structure includes a two-dimensional convolution layer, a two-dimensional batch normalization layer and an activation function layer SiLU which are connected in sequence.

[0091] The decoupling head carrying the attention mechanism has a CBAM module, that is, on the basis of the original detection head of YOLOv8, the CBAM module is respectively merged into two parallel branches for receiving the features of the previous layer. The CBAM module includes a channel attention module for receiving the features of the previous layer, and a spatial attention module for receiving the result of adding the output of the channel attention module to the input. The output of the spatial attention module after adding the input is the output of the CBAM module.

[0092] Object detection usually includes two subtasks: object localization and classification. In object detection, the conflict between classification and regression tasks is a well-known problem. In the detector of YOLOv8, the detection head performs these two subtasks simultaneously on the same convolutional layer. Although this design has the advantages of simple structure and fast speed, the features of different subtasks tend to interfere with each other. In order to avoid this interference, the detection head can be decoupled. For this purpose, a decoupled detection head with CBAM (Convolutional BlockAttention Module) attention (decoupled head with attention mechanism) is designed and compared with the original detection head of YOLOv8. Fig. 9 shown.

[0093] The original detection head only outputs predictions through cubic convolution, which has some deficiencies in bridge deck crack detection, mainly the problem of feature interference between classification and regression tasks. In contrast, by merging the CBAM module into each of the two parallel branches, the calculation of the classification and regression subtasks (Loss_cls and Loss_box) is enhanced. This allows each branch to better learn the required features for the task based on the loss function. Each subtask branch is processed by CA (ChannelAttention) and SA (SpartialAttention) modules, guiding the detection head to focus on the effective information of the feature map according to the task attributes. Subsequent experimental results show that compared with the original detection head, the problem of mutual interference between classification and regression features in previous bridge deck crack detection is solved, enabling each sub-branch module to focus on the features of interest more effectively.

[0094] In the YOLOv8 network, a backbone structure composed of C2f_Dsconv is first designed. The standard convolution that is only weighted by the convolution kernel at each sampling point is replaced by a dynamic snake convolution structure that is more suitable for multi-scale tubular targets, which enhances the feature extraction capability of cracks, a variable structure. At the same time, a SimAM (Simple Attention Module) module is added to the last layer of the backbone. Adding it at the very end allows the attention mechanism to see the feature map of the entire backbone network. Without additional parameters, more feature information of cracks at the same position in different channels can be extracted, with a global view. Special convolution branches are designed for classification and regression tasks. Each branch can better learn the features required for its own task according to the loss function, thereby effectively alleviating the conflict between classification and regression tasks. In addition, through lightweight design, the model reduces the computational complexity and resource consumption while maintaining efficient feature extraction capabilities, which is suitable for efficient and real-time bridge deck crack detection in practical applications. The drone system equipped with RK3588, combined with peripherals such as OV5640 camera and display, uses NPU to perform real-time identification and detection of bridge deck cracks, realizing the overall development of the bridge deck crack detection system.

[0095] Finally, high-resolution images or video data at different locations of the bridge are collected by drones and input into the trained BCYOLO network for real-time detection of bridge deck cracks.

[0096] RK3588 can be used as an embedded device or applied to edge computing. RK3588 adopts 8nm advanced process and 8-core 64-bit architecture, with high performance and low power consumption. Bridge deck crack detection is a key link in bridge maintenance and safety management. The neural processing unit (NPU) in the RK3588 edge AI device can be equipped with the above-mentioned BCYOLO network to significantly improve detection efficiency and accuracy. The RK3588 device transfers data processing tasks from the central server to the edge node close to the bridge site, so that crack detection data can be processed locally in real time without transmitting large amounts of data to the remote cloud for calculation, thereby reducing data transmission delays.

[0097] Through the NPU in the RK3588 edge AI device, combined with the BCYOLO network, the crack detection device is able to pre-process and filter data locally and transmit only important information to the central server, which not only reduces bandwidth consumption, but also reduces the cost and complexity of data transmission. In addition, the BCYOLO network uses advanced image processing and deep learning algorithms to perform real-time analysis of bridge deck cracks, automatically identify and provide high-precision detection results. Because data is processed locally, edge computing reduces the number and distance of sensitive data transmission on the network and improves data security.

[0098] During deployment, first configure the Debian system for RK3588. The trained BCYOLO network based on the torch format needs to be converted to the general onnx format, and then converted to the rknn format. The converted model also needs to be retrained with the training data set to reduce the negative impact on the model performance during the format conversion. Quantization is performed on the NPU to convert the float16 data into int8 data. Configure the Debian system for RK3588, call the load_rknn interface to input the processed int8 data, and call the inference interface to detect and identify the bridge crack information collected by the OV5640 camera.

[0099] We introduced the DJI Matrice 300 drone into the system to further improve the flexibility and efficiency of bridge deck crack detection. The DJI Matrice 300 drone, equipped with a high-resolution OV5640 camera, ensures that clear bridge deck images and video data can be collected during flight. Configure the flight path and shooting parameters of the drone to cover the entire bridge deck area to ensure that data is collected without omission. The DJI Matrice 300 drone flies to different locations on the bridge, uses the OV5640 camera to collect high-resolution image and video data, and transmits the collected data to the DJI Matrice 300 drone equipped with the RK3588 edge AI device (that is, the RK3588 system) for processing in real time through wireless transmission. On the RK3588 device, call the load_rknn interface to input the processed int8 data, and call the inference interface to detect and identify the bridge deck crack information collected by the drone. The cracks are analyzed and identified in real time through the BCYOLO network to provide high-precision detection results. This process realizes the functions of collecting images, calculating and uploading at the same time. The drone collects images in real time and transmits them to the RK3588 device for processing. The crack detection results can be immediately uploaded to the central server via the wireless network to achieve real-time monitoring and management. Only important information is transmitted to the central server, which reduces bandwidth consumption, data transmission costs and complexity. Fig.10 shown.

[0100] This improved solution not only improves the flexibility and coverage of the system by introducing the DJI Matrice 300 drone, but also improves the efficiency and accuracy of bridge deck crack detection. Using the DJI Matrice 300 drone with the RK3588 edge AI device transplanted with the BCYOLO network, the bridge deck crack detection equipment can pre-process and filter data locally. In areas with poor network conditions or involving data security, this technology shows good application prospects. In the case of insufficient bandwidth, it effectively avoids the problem of timeliness loss by reducing the transmission of a large number of pictures and videos, ensuring data reliability and transmission efficiency.

Claims

1. A real-time detection method for bridge deck cracks using drone images based on the BCYOLO lightweight coupling head model, characterized in that: The steps include: Step 1, collecting bridge surface crack images and preprocessing them to obtain a data set; Step 2: Use the dataset to train the constructed BCYOLO network; Step 3: The drone collects high-resolution images or video data at different locations of the bridge and inputs them into the trained BCYOLO network for real-time detection of bridge deck cracks.

2. According to claim 1, a real-time detection method for bridge deck cracks using drone images based on a BCYOLO lightweight coupling head model is characterized in that: The BCYOLO network in step 2 is based on the YOLOv8 network, replacing the backbone structure with a C2f_dynamic snake convolution backbone structure composed of C2f_dynamic snake convolution modules, and replacing the YOLOv8 original detection head with a decoupling head carrying an attention mechanism.

3. The method for real-time detection of bridge deck cracks using drone images based on the BCYOLO lightweight coupling head model according to claim 2 is characterized by: The C2f_dynamic snake convolution backbone structure includes a convolution layer, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a convolution layer, a C2f_dynamic snake convolution module, a SPPF layer and a SimAM module, which are connected in sequence.

4. The method for real-time detection of bridge deck cracks using drone images based on the BCYOLO lightweight coupling head model according to claim 3 is characterized by: The C2f_dynamic serpentine convolution module includes a first convolution layer, a split layer, an even-numbered dynamic serpentine convolution layer connected in sequence, and a splicing layer that receives the output of each even-numbered dynamic serpentine convolution layer and adds the input of the dynamic serpentine convolution layer of the previous layer to splice the output of the first convolution layer and the split layer, and a second convolution layer that receives the output of the splicing layer, wherein, among the even-numbered dynamic serpentine convolution layers, starting from the second odd-numbered dynamic serpentine convolution layer, the input of the odd-numbered dynamic serpentine convolution layer is the result of adding the output of the previous even-numbered dynamic serpentine convolution layer to the input of the previous odd-numbered dynamic serpentine convolution layer; The dynamic snake-shaped convolution layer performs deformation learning on the input features, including sequentially performing two-dimensional convolution and multiple offset processing, and then performing snake-shaped convolution processing on the input features and the results after deformation learning to obtain the output result; The SimAM module includes generating a three-dimensional weight for each channel of the feature map output by the SPPF layer, and fusing the three-dimensional weight with the output of the SPPF layer by point-by-point multiplication; The SPPF layer includes a convolutional layer, three two-dimensional maximum pooling layers, a splicing layer and a convolutional layer connected in sequence; Each convolution layer in the C2f_dynamic snake-shaped convolution backbone structure includes a two-dimensional convolution layer, a two-dimensional batch normalization layer and an activation function layer SiLU which are connected in sequence.

5. The method for real-time detection of bridge deck cracks using drone images based on the BCYOLO lightweight coupling head model according to claim 4 is characterized by: The decoupling head carrying the attention mechanism has a CBAM module, that is, on the basis of the original detection head of YOLOv8, the CBAM module is respectively merged into two parallel branches for receiving the features of the previous layer. The CBAM module includes a channel attention module for receiving the features of the previous layer, and a spatial attention module for receiving the result of adding the output of the channel attention module to the input. The output of the spatial attention module after adding the input is the output of the CBAM module.

6. The method for real-time detection of bridge deck cracks using drone images based on the BCYOLO lightweight coupling head model according to claim 5 is characterized by: The two-dimensional convolution in the snake convolution layer uses a 3×3 convolution kernel with a dilation coefficient of 1, so the formula for the coordinate set of the two-dimensional convolution is: K={(x+i,y+j)∣i,j∈{-1,0,1}} Where K represents the coordinate set of the two-dimensional convolution, (x+i,y+j) represents the coordinate of the two-dimensional convolution, i represents the position offset of the two-dimensional convolution pixel point (x,y) on the X axis, and j represents the position offset of the two-dimensional convolution pixel point (x,y) on the Y axis; After the two-dimensional convolution processing, multiple offset processing is performed to obtain the offset sum, and the dynamic snake convolution coordinates are calculated based on the input features of the dynamic snake convolution layer based on the offset sum. The formula is: in, represents the center coordinate in the two-dimensional convolution, c = {0, 1, 2, 3, 4} represents the horizontal distance from the offset point to the center coordinate, Δx represents the offset δ on the X-axis, Δy represents the offset δ on the Y-axis, {δ|δ∈[-1, 1]}.

7. The method for real-time detection of bridge deck cracks using drone images based on the BCYOLO lightweight coupling head model according to claim 6 is characterized by: The SimAM module is based on the mean μ and variance σ of each channel of the feature map output by the SPPF layer 2 Calculate the energy function E; Then, the energy function E is transformed nonlinearly to generate three-dimensional weights, and the three-dimensional weights are fused with the output of the SPPF layer by point-by-point multiplication; The formula for the mean μ is: Variance σ 2 The formula is: The formula of energy function E is: The formula of the fused feature map is: Among them, X is the input feature map, that is, the feature map output by the SPPF layer, ⊙ is the dot product operation, λ is the hyperparameter, Q is the number of channels, t represents the target neuron, and x m is the mth target neuron, sigmoid represents the sigmoid function, Represents 3D weights.

8. The method for real-time detection of bridge deck cracks using drone images based on the BCYOLO lightweight coupling head model according to claim 1 is characterized by: The drone is a DJI M300 drone, and the trained BCYOLO network is transplanted into the RK3588 system of the drone.

9. A real-time detection system for bridge deck cracks using drone images based on a BCYOLO lightweight coupling head model, comprising a memory and a processor, characterized in that: The storage stores a computer program for execution by the processor to implement the method according to any one of claims 1 to 8.

Images