Bridge disease image acquisition device based on multi-view unmanned aerial vehicle and disease detection method

Abstract

The invention discloses a bridge disease image acquisition device based on a multi-view unmanned aerial vehicle and a disease detection method, the device comprises a top shooting system and a bottom shooting system, three binocular high-definition interactive cameras are shared, and the bottom shooting system can enable homonymy points P1 and P2 of P to appear on imaging planes of the binocular high-definition cameras at the same time. It is guaranteed that the binocular high-definition camera has the same shooting area for the detection plane, the top shooting system can conduct horizontal shooting and vertical shooting, and the top shooting system and the bottom shooting system form a multi-view shooting system. The disease detection method comprises the following steps: S1, shooting images of disease points on the surface of a bridge from different angles through a binocular high-definition interaction camera; s2, carrying out pixel correction on the images shot in the S1, uniformly correcting the images at the inclined visual angle into equivalent vertical orthographic views, and preprocessing the images; and S3, constructing a YOLO-FGE disease detection model, identifying the disease type of the bridge and accurately positioning the position of the bridge.

Description

Technical Field

[0001] This invention belongs to the field of bridge inspection technology, and in particular relates to a bridge defect image acquisition device and defect detection method based on multi-view UAV. Background Technology

[0002] With the rapid development of urbanization and infrastructure construction, bridges, as an important component of transportation networks, directly impact public safety and economic development through their safety and durability. However, traditional bridge defect detection methods mainly rely on manual visual inspection or ground-based equipment, which are not only inefficient and costly but also pose safety hazards, such as the risks of working at heights and blind spots in the inspection. Especially at the bottom of bridges, on curved surfaces of piers, or in environments with insufficient lighting, traditional methods struggle to obtain complete, high-definition images of defects, resulting in insufficient detection accuracy and the inability to promptly identify damage such as cracks, corrosion, and spalling.

[0003] While drones have been applied to bridge inspection, they still have many limitations. For example, images captured by single-camera drones are incomplete and have difficulty processing distortions caused by curved surfaces. When photographing bottom defects, the image clarity is low due to limitations in viewing angle and lighting, making it impossible to form multi-dimensional stereoscopic analysis. Furthermore, although existing image processing algorithms can perform basic defect identification on bridge images, they are not comprehensive enough in extracting complex damage features, especially in terms of detection speed and accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a bridge defect image acquisition device and defect detection method based on multi-view UAVs, so as to improve the speed and accuracy of bridge defect detection.

[0005] The technical solution adopted in this invention is a bridge defect image acquisition device based on multi-view UAV, including a UAV body. The internal logic control of the UAV body includes a main control module, an image and point cloud shooting module, and a result display module. The main control module is equipped with a YOLO-FGE defect detection model. The image and point cloud shooting module preprocesses the captured bridge defect images, and the result display module displays the detection results of the YOLO-FGE defect detection model.

[0006] Furthermore, a camera system is installed on the top and bottom of the drone main unit.

[0007] The bottom shooting system includes a first camera and a second camera. During shooting, the first camera and the second camera can be rotated by the camera horizontal rotation motor, so that the corresponding points P1 and P2 of P appear simultaneously on the imaging plane of the binocular high-definition camera.

[0008] The top-mounted shooting system includes three cameras, capable of both horizontal and vertical shooting.

[0009] The technical solution of the present invention also includes a bridge defect detection method based on multi-view UAVs, which acquires bridge defect images using a bridge defect image acquisition device based on multi-view UAVs, and the steps include:

[0010] Step S1: Take images of the bridge surface defects from different angles;

[0011] Step S2: Perform pixel correction on the images captured in S1, uniformly correct the images from the tilted perspective to an equivalent vertical orthophoto view, and preprocess the images.

[0012] Step S3: Improve the YOLOv5 model and construct the YOLO-FGE defect detection model. Input the enhanced bridge defect image into the YOLO-FGE defect detection model to identify the type of bridge defect and accurately locate its position.

[0013] Furthermore, the specific steps of S1 are as follows:

[0014] S11, control the drone to reach the predetermined detection area, and simultaneously take pictures of the bridge surface from different angles through its No. 1 and No. 2 cameras at the bottom;

[0015] S12, while capturing images, the rangefinders at the four corners of cameras one, two, and three simultaneously measure the distance between the camera lens and four points on the bridge surface. Using the known fixed distance between the rangefinders and the measured distance difference, the included angle β between each side of the camera's ranging plane and the bridge detection surface is calculated in real time.

[0016] Furthermore, the specific steps of S2 are as follows:

[0017] S21, based on the angle β between the edge of the ranging plane and the detection surface obtained in S1, the pixel coordinates of the original image are corrected, and the image with the tilted view is uniformly corrected into an equivalent vertical orthophoto view.

[0018] S22, Enhance the corrected image.

[0019] Furthermore, in S3, the YOLO-FGE disease detection model includes a backbone network, a neck network, and a detection head. The specific detection steps are as follows:

[0020] S31, Construct the backbone network of the YOLO-FGE disease detection model as the feature extractor of the model to extract feature maps of different levels of the bridge disease image after enhancement processing. The backbone network includes 1 Stem Layer and 4 Stage Layers. The Stem Layer selects CBS convolutional blocks, and each Stage Layer contains a CBS convolutional block and an FEC2f module.

[0021] S32, select the SPPFCSPGB module as the multi-scale feature fusion module and connect it after the 4th stage layer of the backbone network;

[0022] S33, after the backbone network, add a gated multi-scale feature fusion module GMFF, which inputs the three effective feature layers output by the first three Stage Layers of the backbone network and the one effective feature layer output by the SPPFCSPGB module into GMFF for feature filtering and fusion.

[0023] S34 inputs the effective feature layer of four bridge defect information weighted into the neck network for multi-scale feature fusion and enhancement. The FEC2f module is used to replace the C3 module in the original YOLOv5 neck network. After the Concat splicing operation, the multi-dimensional collaborative attention mechanism module EMCA is added to capture more comprehensive damage features from the channel dimension, height space dimension, width space dimension and the local 3D space of the input features.

[0024] S35, input the multidimensional interactive reconstructed feature map of bridge defects into the detection head, and output the detection results of bridge damage, which include bounding box coordinates, category probability and confidence level.

[0025] Furthermore, the specific steps for feature extraction in the backbone network are as follows:

[0026] S31a, the enhanced bridge defect image is input into the backbone network. First, the defect image is converted into a feature map of a specific size through the Stem Layer module, which is the preliminary feature map of the bridge defect.

[0027] S31b allows the preliminary feature map of bridge defects to pass through 4 Stage Layers and one SPPFCSPGB layer in sequence, gradually increasing the number of channels in the preliminary feature map while gradually compressing the spatial size, resulting in 4 effective feature layers. In each Stage Layer, the feature map first passes through a CBS convolution block and then enters the FEC2f module, the core of which is FEBottleneck.

[0028] Furthermore, in the aforementioned FEBottleneck, the feature map first undergoes channel dimension transformation and fusion through a convolutional coding block. The encoded features are then input into a wavelet downsampling module, which decomposes the features into low-frequency approximation components and high-frequency detail components. These components are then concatenated to generate the frequency domain features of the bridge defects. Next, the frequency domain features of the bridge defects are input into a bottleneck structure composed of depthwise separable convolutions for thorough feature extraction, obtaining the fused frequency and spatial domain features of the bridge defects. Simultaneously, residual connections are made between the frequency domain features and the fused features. Finally, the fused frequency and spatial domain features of the bridge defects are input into a wavelet upsampling module, converting them from the frequency domain back to the spatial domain. Subsequently, a convolutional decoding block fuses the upsampled features and adjusts the number of channels to output the frequency-spatial hybrid features of the bridge defects.

[0029] Furthermore, in step S32, the frequency-spatial hybrid features of bridge defects are input into SPPFCSPGB, and an effective feature layer is output, as follows:

[0030] First, the frequency-spatial hybrid features of bridge defects output from the 4th layer of the backbone network are divided into one residual connection branch and three BBS branches. The BBS branch with a convolution kernel size of 3×3 is multiplied by the output of a BBS with a convolution kernel size of 1×1 after passing through the sigmoid function, thus forming a gating mechanism. The gating-filtered features are input into a BBS with a convolution kernel size of 1×1 and then concatenated with the residual connection branch to obtain the residual gating feature map.

[0031] Then, the residual gated feature map is subjected to max pooling operation in parallel using multiple pooling kernels of different sizes to capture contextual information of different ranges in the residual gated feature map; after the max pooling operation is performed by multiple pooling kernels of different sizes, the output results of the multiple max pooling operations are concatenated to obtain the context residual gated feature map.

[0032] Finally, the context residual gated feature map is sequentially fed into CBS with kernel sizes of 1×1 and 3×3 to further extract context information, and then residually concatenated with one of the four parallel branches of the output of the fourth layer in the backbone network with a kernel size of 1×1 in the BBS branch. After context information fusion through a CBS with a kernel size of 1×1, the output result of the SPPFCSPGB module is obtained.

[0033] Furthermore, the specific steps in S33 are as follows:

[0034] S33a, input the four effective feature layers into a 1×1 convolution to adjust the number of channels, then upsample the smaller-scale features to ensure that all features are aligned in spatial dimensions, and then stitch the adjusted four feature layers along the channel dimension to form a preliminary multi-scale information fusion feature map of bridge defects.

[0035] S33b inputs the preliminary multi-scale information fusion features of bridge defects into four branches simultaneously. The gated branch generates channel weights through a 1×1 Conv and a Sigmoid activation function. The other three convolutional branches are input into parallel convolutional layers. The outputs of the three convolutional branches are concatenated to form a comprehensive multi-scale fusion feature map of bridge defects. The comprehensive multi-scale fusion feature map is multiplied by the weights generated in the gated screening stage to obtain a multi-scale gated feature map.

[0036] S33c inputs the comprehensive multi-scale fusion feature map of bridge defects into the CBS convolutional block for local feature extraction and normalization, further refines the features, divides the refined features according to the channel size, adjusts the number of channels in each output layer through 1×1 Conv, and finally downsamples according to the target scale to obtain 4 effective feature layers after fusion and screening.

[0037] The beneficial effects of this invention are:

[0038] 1. This invention achieves comprehensive, blind-spot-free, and highly detailed data acquisition. By integrating a bottom-mounted binocular gimbal camera, a top-mounted, vertically rotatable multi-functional shooting module (camera and LiDAR), and an automatic illumination system, a multi-view, multi-modal data acquisition platform is constructed. This effectively solves the problems of detection blind spots and poor image quality inherent in traditional methods at the bottom of bridges, curved piers, and in low-light environments. It can acquire complete, high-definition images of the bridge surface and 3D point cloud data, laying a data foundation for accurate detection.

[0039] 2. This invention integrates a self-developed YOLO-FGE damage detection model. By introducing a series of innovative algorithms, including the frequency domain enhanced bottleneck structure (FEBottleneck), gated multi-scale feature fusion (GMFF), and multidimensional collaborative attention mechanism (EMCA), it significantly enhances the model's feature extraction capabilities for diseases such as cracks and spalling, especially small targets like minute cracks. It significantly improves the accuracy and robustness of disease identification, particularly excelling in detecting complex features and small targets. Experiments show that this model increases the average accuracy (mAP@0.5) of disease detection to over 82.38%, effectively reducing false positives and false negatives.

[0040] 3. This invention integrates flight control, multi-sensor scheduling, data acquisition, and intelligent analysis into one system, achieving one-click automated inspection. Compared to traditional methods that heavily rely on human experience, this invention reduces the time required for a single comprehensive bridge inspection from several days to several hours, while avoiding the risks of personnel working at heights, significantly improving safety and greatly enhancing the automation and overall efficiency of bridge inspection operations. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is an overall structural diagram of the image acquisition device of the present invention.

[0043] Figure 2 This is a schematic diagram of the internal control system of the UAV host of the present invention.

[0044] Figure 3 The present invention relates to a drone bottom shooting system and a drive system, wherein (a) is an exploded view of the rolling motor and the bottom shooting system, and (b) is an overall view.

[0045] Figure 4 This is an exploded view of the camera fixing device of the present invention, wherein (a) is an exploded view before installation and (b) is an overall view after installation.

[0046] Figure 5 This is a diagram showing the position and imaging relationship of the binocular high-definition interactive camera of the present invention.

[0047] Figure 6 This is a diagram showing the conversion relationship between images taken from different angles by the binocular high-definition interactive camera of this invention.

[0048] Figure 7 The present invention provides a top-mounted camera system and drive system for a drone, wherein (a) is a 90-degree rotating camera and (b) is a vertical camera.

[0049] Figure 8 This is a diagram of the YOLO-FGE disease detection model architecture of the present invention.

[0050] Figure 9 This is a schematic diagram of the FEC2f module of the present invention.

[0051] Figure 10 This is a diagram of the SPPFCSPGB module architecture of the present invention.

[0052] Figure 11 This is a diagram of the GMFF module architecture of the present invention.

[0053] Figure 12 This is a diagram of the EMCA module architecture of the present invention.

[0054] Figure 13 This is a comparison chart of the detection results of the YOLO-FGM model of this invention with other models, where (a) is the loss curve and (b) is the average accuracy curve.

[0055] Figure 14 The images show the visualization results of the YOLO-FGE bridge defect detection model of this invention, where (a) is the detection result of a single type of defect and (b) is the detection result of multiple types of defects.

[0056] In the diagram, 1. UAV main unit, 2. Main unit horizontal rotation motor, 3. Propeller, 4. Propeller mounting rod, 5. UAV support leg, 6. UAV support foot, 7. Connecting crossbar, 8. Camera horizontal rotation motor, 9. Main unit rotation connecting rod, 10. Vertical rolling motor, 11. Rangefinder, 12. Camera 1, 13. Laser emitter, 14. Automatic sensor fill light, 15. Camera mounting bracket, 16. Camera 2, 17. Rolling motor mounting hole, 18. Camera mounting bracket, 19. Camera gimbal, 20. Fixing bolt 1, 21. Fixing bolt 2, 22. Camera 3, 23. LiDAR. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example

[0059] Embodiments of the present invention provide a bridge defect image acquisition device based on multi-view UAV, the internal logic of which adopts... Figure 2 The method shown includes a main control module, an image and point cloud acquisition module, and a result display module, used to acquire bridge defect images and 3D point cloud data. The main control module is equipped with a YOLO-FGE defect detection model based on the improved YOLOv5 model. The image and point cloud acquisition module preprocesses the captured bridge defect images, and the processed images are input into the YOLO-FGE defect detection model for detection. The result display module is used to display the detection results of the YOLO-FGE defect detection model.

[0060] The bridge defect image acquisition device includes a drone host 1, such as Figure 1 As shown, propeller fixing rods 4 are installed at the four corners of the drone host 1, and propellers 3 are installed on the propeller fixing rods 4 to provide power for the drone's flight. Four drone support legs 5 are installed at the bottom of the drone host 1, and drone support feet 6 are provided below the drone support legs 5 to provide good stability for the drone's take-off and landing. A connecting crossbar 7 is also provided between the drone support legs 5 on both sides to stabilize the drone body.

[0061] A bottom shooting system is installed below the drone host 1. The bottom shooting system includes a host horizontal rotation motor 2, which can rotate the shooting system 360°. The host horizontal rotation motor 2 is fixed to the center of the bottom of the drone host 1 by threads.

[0062] Below the main unit's horizontal rotation motor 2, there is a main unit rotation connecting rod 9, such as... Figure 3 As shown, (a) is an exploded view of the rolling motor and the shooting system, and (b) is an overall view of the bottom shooting system and drive system of the UAV. One end of the main unit rotating connecting rod 9 is connected to the main unit horizontal rotating motor 2, and the other end is fixed with a vertical rolling motor 10. A camera mounting bracket 15 is installed on the vertical rolling motor 10, and the camera mounting bracket 15 is fixed to the vertical rolling motor 10 through the rolling motor mounting hole 17 and bolts.

[0063] Both ends of the camera mounting bracket 15 are equipped with camera horizontal rotation motors 8, and a laser emitter 13 and an automatic sensor fill light 14 are arranged in the middle. The laser emitter 13 is positioned above the automatic sensor fill light 14.

[0064] A camera mounting head 19 is located in front of the camera horizontal rotation motor 8. The installation and connection process is as follows: Figure 4 As shown, (a) is an exploded view before installation, and (b) is a view after installation. The camera mounting head 19 has a sliding groove on its inner side. The camera mounting bracket 18 is secured to the inner side of the camera mounting head 19 through the sliding groove and fixed by fixing bolt 21. The camera mounting bracket 18 is used to hold the camera. Camera 12 and Camera 16 are respectively fixed to the two camera mounting brackets 18 by fixing bolt 20.

[0065] Rangefinders 11 are installed at the four corners of both the first camera 12 and the second camera 16 to simultaneously measure the distance between the camera lens and four points on the bridge surface.

[0066] Both camera 12 and camera 16 are binocular high-definition interactive cameras. The laser emitter 13 emits a laser beam (point P) towards the detection plane. During imaging, cameras 12 and 16 rotate via the camera horizontal rotation motor 8, causing points P1 and P2 (corresponding to P) to appear simultaneously on the imaging plane of the binocular high-definition camera. This ensures that the binocular high-definition camera has the same imaging area on the detection plane, allowing for the acquisition of images from different angles of the same affected area. The imaging relationship diagram is shown below. Figure 5 As shown.

[0067] When the ambient light is insufficient, the light-sensing element in the automatic fill light 14 is activated, and the automatic fill light 14 automatically provides supplemental light to the detection surface so that the brightness of the detection surface meets the shooting requirements.

[0068] The top of the drone host 1 is also equipped with a multi-functional shooting system consisting of a binocular high-definition interactive camera (camera 3 22) and a lidar 23. Its installation method is the same as that of the shooting system at the bottom, only the direction is reversed. An automatic sensor fill light 14 is also set below the camera 3 22, and rangefinders 11 are also set at the four corners.

[0069] like Figure 7 As shown, the camera system on top of the drone can take both horizontal and vertical shots, forming a multi-view camera system with the camera system at the bottom to obtain richer images of bridge surface defects. Figure 7 In the image, (a) shows a 90-degree rotation shot, and (b) shows a vertical shot. When a higher-level health assessment of bridge surface defects is required through three-dimensional reconstruction, the lidar module of the top-mounted camera system can be used to perform a full-range scan of the bridge, acquire three-dimensional point cloud data of the bridge, and then perform multimodal data fusion with the image data captured by the binocular high-definition interactive camera.

[0070] When surface defects exist at the bottom of a bridge, and the area is often poorly lit, traditional drone cameras have limited shooting angles, making it impossible to capture complete images of the bridge's bottom damage. The top-mounted shooting system of this invention uses a rolling motor to rotate a multi-functional shooting module consisting of a high-definition camera and a LiDAR sensor 90° vertically. This allows for easy acquisition of image and point cloud data of the bridge's bottom, providing reliable data for assessing the damage.

[0071] Embodiments of the present invention also provide a bridge defect detection method based on multi-view UAVs, which acquires bridge defect images using a bridge defect image acquisition device based on multi-view UAVs, and the steps include:

[0072] Step S1: Using a binocular high-definition interactive camera, images of surface defects (such as concrete spalling and cracks) on the bridge are captured from different angles. The specific steps are as follows:

[0073] S11: Control the drone to reach the designated inspection area and simultaneously capture images of the bridge surface from different angles using its No. 1 camera 12 and No. 2 camera 16 on its bottom, ensuring overlap in the fields of view of the two cameras. When photographing the bottom area of ​​the bridge, activate the shooting system on top of the drone and the power drive of the flight submodule to ensure the acquisition of complete bridge defect image data. Depending on the actual inspection needs of the bridge, the lidar 23 on top of the drone can be activated simultaneously to obtain high-precision three-dimensional point cloud data.

[0074] S12, while capturing images, the rangefinders 11 at the four corners of the camera (in this invention, the drone is equipped with three binocular high-definition interactive cameras, namely camera 12, camera 16, and camera 22, with a rangefinder 11 at each of the four corners of each high-definition camera) simultaneously measure the distance between the camera lens and four points on the bridge surface. Using the known fixed distance between the rangefinders and the measured distance differences, the angle β between each side of the camera's ranging plane and the bridge detection surface is calculated in real time. The conversion relationship of the high-definition camera shooting at different angles is as follows: Figure 6 As shown.

[0075] Taking camera 12 as an example, the relative distance between two adjacent counterclockwise rangefinders 11 is denoted as h. 112 h 123 h 134 h 141 , where h 112 = h 134 h 123 = h 141 The above is the inherent known data; the distances between the four rangefinders 11 in camera 12 and the detection surface are denoted as l. 11 l 12 l 13 l 14 The ranging plane of camera 12 is denoted as d. 11 d 12 d 13 d 14 The four sides h of the first ranging plane 112 h 123 h 134 h 141 The angle between the bridge surface being inspected and the surface being inspected is denoted as β. 112 β 123 β 134 β 141 , where edge d 11 d 12 Angle β with the detection surface 112 The calculation formula is as follows:

[0076] ;

[0077] Other included angles β 123 β 134 β 141 The calculation method is the same as the method described above, and the same method is used for the other cameras.

[0078] Step S2 involves pixel correction of the images captured in S1 to eliminate image distortion caused by the shooting angle and the curved structure of the bridge. Images from tilted angles are uniformly corrected to equivalent vertical orthographic views, and the images are preprocessed. The specific steps are as follows:

[0079] S21, based on the angle between the edge of the ranging plane and the detection surface obtained in S1, corrects the pixel coordinates of the original image, such as the first ranging plane d. 11 d 12 d 13 d 14 In the middle, along the edge d 11 d 12 Pixel coordinates in the projection direction Its corrected pixel coordinates The corrected relation is:

[0080] ;

[0081] The pixel correction method is the same for other side projection directions.

[0082] In this invention, the relationship between the camera and the detection surface during actual shooting can be categorized into three scenarios: 1) both cameras are tilted, 2) both cameras are perpendicular, and 3) one camera is tilted and the other is perpendicular. When the detection structure is curved (e.g., photographing a cylindrical bridge pier), tilted shooting occurs, and pixel correction can be performed using the aforementioned pixel correction relationship. During perpendicular shooting, the angle between the ranging plane and the detection surface is zero, eliminating the need for pixel correction. In actual measurement, the angle between the camera and the detection surface can be adjusted according to the specific circumstances, but it is essential to ensure that the fields of view of the binocular high-definition cameras overlap, and to shoot the same area from multiple angles.

[0083] S22 performs enhancement processing on the corrected image, such as rotation, cropping, stretching, and grayscale transformation, to provide the model with more robust input.

[0084] Step S3 involves improving the YOLOv5 (You Only Look Once) model to construct a YOLO-FGE defect detection model. The enhanced bridge defect image is input into the YOLO-FGE defect detection model to identify the types of bridge defects and accurately locate their positions. The YOLO-FGE defect detection model includes a backbone network, a neck network, and a detection head. The backbone network directly affects the subsequent networks' ability to understand defect image information. The neck network receives defect features extracted by the backbone network, performs multi-scale feature fusion and enhancement, and then passes the processed defect features to the detection head. The detection head is the model's "predictor," receiving the multi-scale fused feature map from the neck network and outputting the final damage detection results (boundary box coordinates, class probability, and confidence score). The architecture diagram of the YOLO-FGE defect detection model is shown below. Figure 8 As shown, the specific steps of the detection are as follows:

[0085] S31, construct the backbone network of the YOLO-FGE disease detection model as the feature extractor of the model, extract feature maps of different levels of the bridge disease image after enhancement processing. The backbone network includes 1 Stem Layer and 4 Stage Layers. The Stem Layer selects CBS (Convolution + Batch Normalization + SiLU) convolutional blocks. Each Stage Layer contains a CBS convolutional block and an FEC2f (Frequency domain enhancement of cross stage partial feature fusion) module.

[0086] The CBS module is a standard convolutional block responsible for basic bridge defect feature extraction and channel number adjustment. The FEC2f module, through its multi-branch structure and dense skip connections, enables smoother gradient flow and integrates features from different "depths" and "receptive fields," allowing the backbone network to capture a variety of information from details to semantics. To improve the model's multi-scale feature perception and edge preservation capabilities, as well as its stability in complex or noisy environments, this invention employs wavelet transform and depthwise separable convolutions to construct a frequency-domain enhanced bottleneck structure, FEBottleneck, replacing the standard convolutional bottleneck structure in existing technologies.

[0087] The specific steps for feature extraction in the backbone network are as follows:

[0088] In step S31a, the enhanced bridge defect image is input into the backbone network. First, the Stem Layer module converts the defect image into a feature map of a specific size (e.g., a shape of 640×640×3). After passing through the Stem Layer module, a preliminary feature map of 320×320×64 is extracted from the original defect image (the specific size can be set according to specific needs). This preliminary feature map of the bridge defect is then used for deeper feature extraction. In this invention, the design of the Stem Layer optimizes computational efficiency.

[0089] S31b passes the initial feature map of bridge defects through four Stage Layers and one SPPFCSPGB layer, progressively increasing the number of channels while gradually compressing the spatial size, resulting in four effective feature layers (one effective feature layer from each of the first three Stage Layers, and one effective feature layer from SPPFCSPGB). In each Stage Layer, the feature map first passes through a CBS convolutional block and then enters the FEC2f module. The architecture of FEC2f is as follows... Figure 9 As shown, the core of this module is FEBottleneck.

[0090] In FEBottleneck, feature maps are first transformed and fused through a 1×1 convolutional coding block to optimize feature representation. Subsequently, the encoded features are input into a wavelet downsampling module, which explicitly decomposes the features into low-frequency approximation components (LL) and high-frequency detail components (including horizontal high-frequency components (LH), vertical high-frequency components (HL), and diagonal high-frequency components (HH)). This frequency-domain decomposition mechanism allows the backbone network to operate directly in the frequency domain, thus processing multi-scale information more accurately. Specifically, low-frequency components are beneficial for target localization and category recognition, while high-frequency components are crucial for capturing detailed features such as edges and textures.

[0091] Subsequently, the low-frequency approximate components and high-frequency detail components are concatenated to generate frequency domain features of bridge defects. These features are then input into a bottleneck structure composed of depthwise separable convolutions for thorough feature extraction, thereby obtaining fused frequency and spatial domain features of bridge defects. Simultaneously, residual connections are made between the frequency domain features and the fused features to ensure that detailed information is not lost during propagation in the deep network, providing a more accurate foundation for subsequent upsampling reconstruction.

[0092] Finally, the frequency and spatial domain fusion features of bridge defects are input into the wavelet upsampling module to convert them from the frequency domain back to the spatial domain. Then, the upsampled features are fused through a 1×1 convolutional decoding block, and the number of channels is adjusted to output the frequency-spatial hybrid features of bridge defects.

[0093] The feature extraction process of the FEBottleneck module can be represented by the following formula:

[0094] ;

[0095] ;

[0096] ;

[0097] In the formula, The frequency-spatial hybrid features extracted from the previous FEBottleneck module. For wavelet transform, The frequency domain characteristics of bridge defects are obtained by splicing together the data after wavelet transform. For depthwise separable convolutions with a kernel size of 3×3, The frequency-spatial hybrid characteristics of bridge defects. This is the inverse wavelet transform. The frequency-spatial hybrid features extracted by the current FEBottleneck module, For splicing operations, This represents a CBS convolution block with a kernel size of 1×1.

[0098] S32, the SPPFCSPGB (Spatial pyramid pooling fast cross stage partial with gate bottleneck) module is selected as the multi-scale feature fusion module, connected after the 4th stage layer of the backbone network. The architecture diagram of SPPFCSPGB is as follows. Figure 10 As shown, the frequency-spatial hybrid features of bridge defects are input into SPPFCSPGB, and an effective feature layer is output, as follows:

[0099] First, the frequency-spatial hybrid features of bridge defects output from layer 4 of the backbone network are divided into four parallel branches: one residual connection branch and three BBS (BottConv + Batch Normalization + SiLU) branches. The 3×3 BBS branch is multiplied by the output of a 1×1 BBS branch after passing through a sigmoid function, thus forming a gating mechanism. The gated features are then input into a 1×1 BBS branch and concatenated with the residual connection branch to obtain the residual gated feature map. Finally, the residual gated feature map is subjected to max pooling operations in parallel using multiple pooling kernels of different sizes to capture contextual information of different ranges within the residual gated feature map, enabling the neck network to simultaneously perceive both local details and global context of the bridge defect image.

[0100] After performing max pooling operations with multiple pooling kernels of different sizes, the outputs of these operations are concatenated to obtain a context residual gated feature map. This map is then passed sequentially to CBS (Convolutional Baseline Layout) kernels of 1×1 and 3×3 sizes to further extract contextual information. It is then residually concatenated with one of the four parallel branches of the fourth layer output from the backbone network (BBS branch with a 1×1 kernel) to ensure smooth feature gradients. Finally, the contextual information is fused through a CBS kernel of 1×1, resulting in the output of the SPPFCSPGB module, which is an effective feature layer.

[0101] S33 adds a gated multi-scale feature fusion (GMFF) module after the backbone network. This GMFF takes the three effective feature layers output from the first three Stage Layers of the backbone network and the one effective feature layer output from the SPPFCSPGB module, and inputs them into the GMFF for feature selection and fusion. This improves the model's ability to fuse effective features at multiple scales, enhances the effective features extracted by the backbone network, and suppresses noise and other interference. The architecture of GMFF is as follows: Figure 11 As shown. The specific steps are as follows:

[0102] S33a: The four effective feature layers are each input into a 1×1 convolution (1×1 Conv) to adjust the number of channels, ensuring that the channel ratio is the same and that the feature dimensions are the same. Then, the smaller-scale features are upsampled to match the largest scale, ensuring that all features are aligned in spatial dimensions. The adjusted four feature layers are then fused to form a large feature map with high resolution, which is the preliminary multi-scale information fusion feature map of bridge defects. The formula is as follows:

[0103] ;

[0104] in, This is a preliminary multi-scale information fusion feature map of bridge defects. , , and These are four effective feature layers at different scales. For 2x upsampling, For 4x upsampling, For 8x upsampling, For feature splicing, For convolution operations, the subscript indicates the size of the convolution kernel.

[0105] S33b inputs the preliminary multi-scale information fusion features of bridge defects into four branches simultaneously. The gated branch, after passing through a 1×1 Conv, generates channel weights (between 0 and 1) using a Sigmoid activation function. The other three convolutional branches are input into parallel convolutional layers to capture local context at different scales: 3×3 Conv: small to medium receptive field, extracting fine details; 5×5 Conv: medium receptive field, balancing local and global contexts; 7×7 Conv: large receptive field, capturing broader context. The outputs of the three convolutional branches are concatenated to form a comprehensive multi-scale fusion feature map of bridge defects. This comprehensive multi-scale fusion feature map is multiplied by the weights generated in the gating stage to obtain a multi-scale gating feature map, achieving channel-level filtering to eliminate redundancy that may occur in the preliminary multi-scale information aggregation stage. The formula is as follows:

[0106] ;

[0107] ;

[0108] in, This is a preliminary multi-scale information fusion feature map of bridge defects. For copy operation, , , and All are copied. , For multi-scale gated feature maps, For activation function, For convolution operations, the subscript indicates the size of the convolution kernel.

[0109] S33c inputs the comprehensive multi-scale fused feature map of bridge defects into the CBS convolutional block for local feature extraction and normalization, further refining the features. Then, the refined features are split according to channel size. Each output layer adjusts the number of channels using a 1×1 Conv. Finally, downsampling is performed according to the target scale to match the original input structure, resulting in four fused and filtered effective feature layers. This provides efficient and rich defect feature information for further multi-scale feature fusion and enhancement of the neck network. The formula is as follows:

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] ;

[0115] In the formula, For 2x downsampling, For 4x downsampling, For 8x downsampling, , , and The four branches are derived from the refined multi-scale gated feature map and divided according to channel size. , , and These are the four effective feature layers after fusion and downsampling. For convolution operations, the subscript indicates the size of the convolution kernel. This indicates that the block has passed through a CBS convolutional block. This is a splitting operation.

[0116] In summary, the first three Stage layers and SPPFCSPGB in the backbone network output a total of four effective feature layers. Then, the GMFF module fuses and filters the extracted bridge defect information to obtain four weighted effective feature layers of bridge defect information.

[0117] S34, the effective feature layer weighted by the four bridge defect information is input into the neck network for multi-scale feature fusion and enhancement. In this invention, the overall architecture of the neck network of the YOLO-FGE network follows the design of the YOLOv5 network. The improvement lies in replacing the C3 module in the original YOLOv5 neck network with the FEC2f module, and adding a multidimensional collaborative attention mechanism module EMCA (Enhanced Multidimensional Collaborative Attention) after the Concat splicing operation. This captures more comprehensive damage features from the channel dimension, height space dimension, width space dimension, and the local 3D space of the input features, thereby improving the detection accuracy of the defect detection model for complex scenes and small target damage. The architecture of EMCA is as follows: Figure 12 As shown.

[0118] In EMCA, the feature space of the first branch remains unchanged. 3D attention weights are obtained by calculating the similarity between each pixel in the feature map and its neighboring pixels. These 3D attention weights are then multiplied by the original feature map to achieve weighted adjustment. The second and third branches are rotated along the H-axis and W-axis, respectively, to obtain width and height branches. The fourth branch remains spatially unchanged as a channel branch. The rotated second, third, and fourth branches are then concatenated and input into a compression transformation module consisting of average pooling and standard pooling for feature compression, thereby suppressing redundant information and reducing computational complexity. The compressed features are then input into a 1×k convolution activation transformation module to achieve feature interaction across width, height, and channel dimensions, and multidimensional interactive attention weights are obtained using the sigmoid function. These multidimensional interactive attention weights are multiplied by the rotated second, third, and fourth branches to achieve multidimensional interactive feature weighting. Subsequently, the second and third branches, after multidimensional feature weighting, are inversely rotated to obtain a feature map consistent with the input feature space. Finally, the four feature-weighted branches are fused to obtain a reconstructed feature map of the bridge's multidimensional interactive features.

[0119] S35 inputs the multidimensional interactive reconstructed feature map of bridge defects into the detection head, and outputs the bridge damage detection results, namely bounding box coordinates, class probability, and confidence score, such as... Figure 14 As shown. In this invention, the detection head of the YOLO-FGE network retains the original detection head of YOLOv5.

[0120] Experimental verification

[0121] A self-built concrete bridge surface damage dataset (containing seven common concrete bridge surface defects: weathering erosion, cracks, voids, exposed rebar, concrete separation, concrete spalling, and honeycomb pitting) was used to train the YOLO-FGE model. The loss curve during training and the average accuracy curve for the bridge surface defects are shown below. Figure 13 As shown, (a) is the loss curve and (b) is the average accuracy curve. The model of this invention was trained for 300 epochs, and the training loss of the model tended to converge after the 220th epoch. From the average accuracy curve of bridge surface defects, it can be seen that YOLO-FGE outperforms classic models such as YOLOv12 in detecting weathering erosion, cracks, voids, and concrete separation. Its performance in detecting surface defects with difficult-to-distinguish damage boundaries, such as exposed rebar, concrete separation, and honeycomb pitting, is comparable to the advanced YOLOv12 model.

[0122] Table 1 Comparison of bridge surface defect detection results between YOLO-FGE model and other classic models

[0123] Table 1 compares the bridge surface defect detection results of the YOLO-FGE model with other classic models. It can be seen that the YOLO-FGE network significantly improves the F1 score and average accuracy while maintaining a manageable increase in computational cost and parameter count. Compared to the original YOLOv5-s, YOLO-FGE improves the F1 score by 14.49% and the average accuracy (mAP@0.5) by 9.82%, both outperforming the advanced YOLOv12 model. In terms of damage detection efficiency, the YOLO-FGE model achieves a good detection efficiency of 128 FPS (Frames Per Second), meeting the 30 FPS requirement for real-time detection.

[0124] Figure 14 The visualization results of the YOLO-FGE model for detecting single-type and multi-type defects on the surface of concrete bridges are shown. (a) shows the detection results for single-type defects, and (b) shows the detection results for multi-type defects. It can be seen that the YOLO-FGE model can accurately outline the area where the defect is located and give the corresponding confidence score in the detection of both fine cracks and complex scenarios with multiple defects, demonstrating good practicality.

[0125] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0126] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A bridge defect image acquisition device based on multi-view UAV, comprising a UAV body (1), characterized in that, The internal logic control of the main body of the UAV (1) includes a main control module, an image and point cloud shooting module and a result display module. The main control module is equipped with a YOLO-FGE defect detection model. The image and point cloud shooting module preprocesses the captured bridge defect images. The result display module displays the detection results of the YOLO-FGE defect detection model.

2. The bridge defect image acquisition device based on multi-view UAV according to claim 1, characterized in that, The drone host (1) is equipped with a shooting system on its top and bottom. The bottom shooting system includes a first camera (12) and a second camera (16). During shooting, the first camera (12) and the second camera (16) can be rotated by the camera horizontal rotation motor (8) so that the corresponding points P1 and P2 of P appear simultaneously on the imaging plane of the binocular high-definition camera. The top-mounted shooting system includes a third camera (22) capable of both horizontal and vertical shooting.

3. A method for bridge defect detection based on multi-view UAVs, characterized in that, The method of acquiring bridge defect images using a bridge defect image acquisition device based on a multi-view UAV as described in any one of claims 1 to 2 includes the following steps: Step S1: Take images of the bridge surface defects from different angles; Step S2: Perform pixel correction on the images captured in S1, uniformly correct the images from the tilted perspective to an equivalent vertical orthophoto view, and preprocess the images. Step S3: Improve the YOLOv5 model and construct the YOLO-FGE defect detection model. Input the enhanced bridge defect image into the YOLO-FGE defect detection model to identify the type of bridge defect and accurately locate its position.

4. The bridge defect detection method based on multi-view UAV according to claim 3, characterized in that, The specific steps of S1 are as follows: S11, control the drone to arrive at the predetermined detection area, and simultaneously take pictures of the bridge surface from different angles through the No. 1 camera (12) and No. 2 camera (16) at its bottom; S12, while capturing images, the rangefinders (11) at the four corners of the first camera (12), the second camera (16) and the third camera (22) simultaneously measure the distance between the camera lens and four points on the bridge surface. Using the known fixed distance between the rangefinders (11) and the measured distance difference, the included angle β between each side of the camera ranging plane and the bridge detection surface is calculated in real time.

5. A bridge defect detection method based on multi-view UAV according to claim 3, characterized in that, The specific steps of S2 are as follows: S21, based on the angle β between the edge of the ranging plane and the detection surface obtained in S1, the pixel coordinates of the original image are corrected, and the image with the tilted view is uniformly corrected into an equivalent vertical orthophoto view. S22, Enhance the corrected image.

6. The bridge defect detection method based on multi-view UAV according to claim 3, characterized in that, In S3, the YOLO-FGE disease detection model includes a backbone network, a neck network, and a detection head. The specific detection steps are as follows: S31, Construct the backbone network of the YOLO-FGE disease detection model as the feature extractor of the model to extract feature maps of different levels of the bridge disease image after enhancement processing. The backbone network includes 1 Stem Layer and 4 Stage Layers. The Stem Layer selects CBS convolutional blocks, and each Stage Layer contains a CBS convolutional block and an FEC2f module. S32, select the SPPFCSPGB module as the multi-scale feature fusion module and connect it after the 4th stage layer of the backbone network; S33, after the backbone network, add a gated multi-scale feature fusion module GMFF, which inputs the three effective feature layers output by the first three Stage Layers of the backbone network and the one effective feature layer output by the SPPFCSPGB module into GMFF for feature filtering and fusion. S34 inputs the effective feature layer of four bridge defect information weighted into the neck network for multi-scale feature fusion and enhancement. The FEC2f module is used to replace the C3 module in the original YOLOv5 neck network. After the Concat splicing operation, the multi-dimensional collaborative attention mechanism module EMCA is added to capture more comprehensive damage features from the channel dimension, height space dimension, width space dimension and the local 3D space of the input features. S35, input the multidimensional interactive reconstructed feature map of bridge defects into the detection head, and output the detection results of bridge damage, which include bounding box coordinates, category probability and confidence level.

7. A bridge defect detection method based on multi-view UAV according to claim 6, characterized in that, The specific steps for feature extraction in the backbone network are as follows: S31a, the enhanced bridge defect image is input into the backbone network. First, the defect image is converted into a feature map of a specific size through the Stem Layer module, which is the preliminary feature map of the bridge defect. S31b allows the preliminary feature map of bridge defects to pass through four Stage Layers and one SPPFCSPGB layer in sequence, gradually increasing the number of channels in the preliminary feature map while gradually compressing the spatial size, resulting in four effective feature layers. In each Stage Layer, the feature map first passes through a CBS convolutional block and then enters the FEC2f module, the core of which is FEBottleneck.

8. A bridge defect detection method based on multi-view UAV according to claim 7, characterized in that, In the aforementioned FEBottleneck, the feature map first undergoes channel dimension transformation and fusion through a convolutional coding block. The encoded features are then input into a wavelet downsampling module, which decomposes the features into low-frequency approximation components and high-frequency detail components. These components are then concatenated to generate the frequency domain features of the bridge defects. Next, the frequency domain features of the bridge defects are input into a bottleneck structure composed of depthwise separable convolutions for thorough feature extraction, obtaining a fused frequency and spatial domain feature of the bridge defects. The frequency domain features and the fused feature are then residually connected. Finally, the fused frequency and spatial domain feature of the bridge defects is input into a wavelet upsampling module, converting it back from the frequency domain to the spatial domain. A convolutional decoding block then fuses the upsampled features and adjusts the number of channels to output the frequency-spatial hybrid feature of the bridge defects.

9. A bridge defect detection method based on multi-view UAV according to claim 6, characterized in that, In step S32, the frequency-spatial hybrid features of bridge defects are input into SPPFCSPGB, and an effective feature layer is output, as follows: First, the frequency-spatial hybrid features of bridge defects output from the 4th layer of the backbone network are divided into one residual connection branch and three BBS branches. The BBS branch with a convolution kernel size of 3×3 is multiplied by the output of a BBS with a convolution kernel size of 1×1 after passing through the sigmoid function, thus forming a gating mechanism. The gating-filtered features are input into a BBS with a convolution kernel size of 1×1 and then concatenated with the residual connection branch to obtain the residual gating feature map. Then, the residual gated feature map is subjected to max pooling operation in parallel using multiple pooling kernels of different sizes to capture contextual information of different ranges in the residual gated feature map; after the max pooling operation is performed by multiple pooling kernels of different sizes, the output results of the multiple max pooling operations are concatenated to obtain the context residual gated feature map. Finally, the context residual gated feature map is sequentially fed into CBS with kernel sizes of 1×1 and 3×3 to further extract context information, and then residually concatenated with one of the four parallel branches of the output of the fourth layer in the backbone network with a kernel size of 1×1 in the BBS branch. After context information fusion through a CBS with a kernel size of 1×1, the output result of the SPPFCSPGB module is obtained.

10. A bridge defect detection method based on multi-view UAV according to claim 6, characterized in that, The specific steps in S33 are as follows: S33a, input the four effective feature layers into a 1×1 convolution to adjust the number of channels, then upsample the smaller-scale features to ensure that all features are aligned in spatial dimensions, and then stitch the adjusted four feature layers along the channel dimension to form a preliminary multi-scale information fusion feature map of bridge defects. S33b inputs the preliminary multi-scale information fusion features of bridge defects into four branches simultaneously. The gated branch generates channel weights through a 1×1 Conv and a Sigmoid activation function. The other three convolutional branches are input into parallel convolutional layers. The outputs of the three convolutional branches are concatenated to form a comprehensive multi-scale fusion feature map of bridge defects. The comprehensive multi-scale fusion feature map is multiplied by the weights generated in the gated screening stage to obtain a multi-scale gated feature map. S33c inputs the comprehensive multi-scale fusion feature map of bridge defects into the CBS convolutional block for local feature extraction and normalization, further refines the features, divides the refined features according to the channel size, adjusts the number of channels in each output layer through 1×1 Conv, and finally downsamples according to the target scale to obtain 4 effective feature layers after fusion and screening.

Images