A bridge defect detection system based on BIM model

Abstract

The present invention discloses a bridge defect detection system based on a BIM model, which relates to the technical field of bridge detection and comprises: a detection data acquisition module, a detection data processing module, a comprehensive judgment module and a terminal interaction module; the detection data acquisition module is used to plan the collection position and collection method according to the BIM model, perform laser scanning on the bridge to be detected to obtain point cloud data of a first target position set, photograph the bridge to be detected to obtain full-structure image data, select vibration sampling points and obtain cable vibration data of the bridge to be detected through a vibration sensor; the detection data processing module is used to pre-process the point cloud data of the first target position set, the full-structure image data and the cable vibration data, and perform feature extraction through a three-branch convolutional neural network; the comprehensive judgment module is used to perform feature fusion processing through an improved Transformer network to obtain deep fusion features, and perform classification tasks according to the deep fusion features to obtain defect detection results.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge defect detection, and in particular to a bridge defect detection system based on a BIM model. Background Art

[0002] Bridges are a vital component of transportation infrastructure. Regular inspection and maintenance are crucial after their completion and commissioning. Manual inspection of large bridges suffers from high rates of missed defects and inefficiencies. Thanks to breakthroughs in information technology, artificial intelligence, drone technology, and BIM modeling have been gradually applied to bridge defect detection. However, due to the diverse nature of bridge types, there is a lack of targeted and comprehensive inspection methods for suspension bridges. Therefore, developing a defect detection system specifically for suspension bridges is a promising research area.

[0003] Currently, Chinese invention patent application number 202210419345.5 discloses a BIM-based bridge inspection and management system. The application includes a CPU module, which enables coordinated adjustments during bridge inspection. The CPU module includes a deployment unit, a detection unit, and a management unit. The deployment unit includes a registration module and a statistics module. The signal output end of the registration module is connected to the signal receiving end of the statistics module, and the signal output end of the statistics module is connected to the signal receiving end of the scheduling module. The CPU module enables coordinated adjustments during bridge inspection. The scheduling module transmits information via a communication module, allowing inspectors to understand the inspection time and sequence, thereby improving efficiency. The scheduling information is timestamped by a time module and stored by a storage module, facilitating future inquiries about the bridge's maintenance cycle. However, this method lacks targeted inspection of different bridge parts and comprehensive defect assessment. Summary of the Invention

[0004] The technical problem addressed by this invention is that traditional methods lack targeted inspection of different bridge components. In particular, cable-stayed bridges rely solely on images, lacking comprehensive multi-source analysis. Furthermore, due to obstruction by suspension and stay cables, the flight paths used by traditional drones for data collection are unsuitable. Furthermore, the utilization of multi-source data is low, and the lack of adaptive weighting for defect detection hinders the rational integration of multi-source features.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A bridge defect detection system based on a BIM model, comprising: a detection data acquisition module, a detection data processing module, a comprehensive judgment module, and a terminal interaction module;

[0007] The inspection data acquisition module includes a point cloud data acquisition unit, an image data acquisition unit, and a vibration data acquisition unit. These units are used to plan the acquisition locations and methods according to the BIM model, perform laser scanning on the bridge to be inspected to obtain point cloud data of the first target location set, photograph the bridge to be inspected to obtain full-structure image data, select vibration sampling points, and obtain cable vibration data of the bridge to be inspected through vibration sensors.

[0008] The detection data processing module includes a preprocessing unit and a feature extraction unit, which are used to preprocess the point cloud data, full-structure image data and cable vibration data of the first target part set to obtain a point cloud input vector, an image input vector and vibration frequency input data, and perform feature extraction through a three-branch convolutional neural network to obtain point cloud output features, image output features and vibration frequency output features;

[0009] The comprehensive judgment module includes a feature fusion unit and a decision classification unit. It is used to perform feature fusion processing on the point cloud output features, image output features, and vibration frequency output features through an improved Transformer network to obtain deep fusion features. The classification task is performed based on the deep fusion features to obtain defect detection results.

[0010] The terminal interaction module includes a display interaction unit and a log storage unit, which is used to provide a display interface to the user and receive user operation instructions, receive user adjustments and settings for the system parameters of the detection data acquisition module, detection data processing module, and comprehensive judgment module, generate a detection log based on the defect detection results output by the comprehensive judgment module, and store the output data of the detection data acquisition module, detection data processing module, and comprehensive judgment module.

[0011] Preferably, the point cloud data acquisition unit is used to perform laser scanning on the bridge to be inspected to obtain point cloud data of the first target part set, and the processing logic includes:

[0012] Retrieve the BIM model of the bridge to be inspected, establish a geodetic coordinate system based on the BIM model, traverse the BIM model of the bridge to be inspected based on a preset defect location reference set to obtain target locations that require laser scanning, and record the spatial coordinate data corresponding to all target locations that require laser scanning to obtain a first target location set;

[0013] The preset defect location reference set includes bridge towers, main cables, stiffening beams, anchors and saddles;

[0014] Obtaining component information data corresponding to each part of the first target part set through the BIM model, wherein the component information data includes the size, geometry, spatial coordinates, and spatial orientation of each part of the first target part set;

[0015] The BIM model is divided and annotated based on a preset scanning boundary expansion coefficient and component information data to obtain an extended laser scanning boundary. A UAV laser scanning route is planned based on the extended laser scanning boundary to obtain a first acquisition route. The UAV is commanded to fly along the first acquisition route, and data is collected by a 3D laser radar carried by the UAV to obtain point cloud data of the first target part set.

[0016] The preset scanning boundary expansion coefficients include a bridge tower area expansion coefficient, a main cable area expansion coefficient, a stiffening beam area expansion coefficient, an anchorage area expansion coefficient, and a saddle area expansion coefficient.

[0017] Preferably, the image data acquisition unit is used to photograph the bridge to be inspected to obtain full-structure image data, and the processing logic includes:

[0018] Retrieve the BIM model of the bridge to be inspected, and layer it based on the spatial height of the BIM model of the bridge to be inspected to obtain the bridge deck layer, bridge bottom layer, tower beam layer and cable layer. Set the corresponding drone flight trajectory and acquisition angle for each layer respectively. Set the flight trajectory of the bridge deck layer to a parallel strip route, and the shooting angle to be 45° downward and vertical. Set the flight trajectory of the bridge bottom layer to a parallel strip route, and the shooting angle to be 45° upward and vertical. Set the flight trajectory of the tower beam layer to a vertical ascending route, and the shooting angle to be horizontal. Set the flight trajectory of the cable layer to a broken line fitting route, and the shooting angle to be horizontal.

[0019] The flight trajectories of the drones corresponding to each layer in the BIM model of the bridge to be inspected are connected in series to obtain the second acquisition route. The drone is commanded to fly according to the second acquisition route, and the full-structure image data is obtained by collecting data through the camera carried by the drone.

[0020] Preferably, the vibration data acquisition unit is used to select vibration sampling points and obtain cable vibration data of the bridge to be inspected through a vibration sensor. The processing logic includes:

[0021] Retrieve the BIM model of the bridge to be inspected, classify the cables based on their material data in the BIM model, and mark key vibration points on the BIM model. These key vibration points include the junction between the main cable and the saddle, the anchorage end of the suspender cable, and the anchorage area between the cable and the bridge tower.

[0022] Vibration sampling points are obtained by evenly selecting key vibration points based on a preset interval distance, and the vibration sampling points are collected by vibration sensors to obtain cable vibration data.

[0023] Preferably, the pre-processing unit is used to process the point cloud data of the first target part set to obtain a point cloud input vector, process the full structure image data to obtain an image input vector, and process the cable vibration data to obtain vibration frequency input data. The processing logic includes:

[0024] Based on a preset sampling step size, the point cloud data of the first target part set is sliced to obtain an actual section point cloud, the polar angle of each actual section point cloud is calculated and sorted to obtain a point cloud stacking sequence, the point cloud convex hull vector set is pushed into the stack by Chan's algorithm and the point cloud stacking sequence, the point cloud convex hull vector set is vector cross multiplication is performed on the point cloud convex hull vector set to obtain a point cloud vector outer product, the point cloud vector outer product is filtered by a preset point cloud threshold to obtain section shape feature points, the section shape feature points are connected to obtain the actual section shape curve, the BIM model pair is extracted based on the central space coordinates of each actual section shape curve to obtain the corresponding reference section shape curve, the Euclidean distance standard deviation and the mean curvature difference between the actual section shape curve and the reference section shape curve with the same central space coordinate are calculated, and the point cloud input vector is constructed based on the index of the area to which each actual section shape curve belongs, the central space coordinate, the Euclidean distance standard deviation and the mean curvature difference between the actual section shape curve and the corresponding reference section shape curve;

[0025] Performing Gaussian filtering and denoising on the full-structure image data, and performing grayscale mapping on the denoised full-structure image data to obtain an image input vector;

[0026] The cable vibration data is supplemented by bilinear interpolation, denoised by low-pass filtering, sampled based on a preset sampling frequency to obtain discrete-time vibration data, and calculated by fast Fourier transform to obtain vibration frequency input data.

[0027] Preferably, the feature extraction unit is used to perform feature extraction on the point cloud input vector to obtain point cloud output features, perform feature extraction on the image input vector to obtain image output features, and perform feature extraction on the frequency input data to obtain frequency output features through a three-branch convolutional neural network. The processing logic includes:

[0028] The feature extraction of the point cloud input vector is performed through the Capsule-Network network branch. The calculation logic is as follows: the point cloud input vector is continuously convolved through the series convolution layer to obtain the primary capsule layer input data, and the primary capsule layer performs local feature encoding on the primary capsule layer input data to obtain a multi-dimensional digital feature vector. The dynamic routing algorithm establishes a connection between the primary capsule layer and the advanced capsule layer, and the advanced capsule layer matches the multi-dimensional digital feature vectors corresponding to each primary capsule in the primary capsule layer to obtain the connection weight. The connection weights corresponding to each primary capsule are integrated and calculated to obtain the K-layer point cloud output features.

[0029] Preferably, feature extraction is performed on the image input vector through the GoogLeNet network branch, and the calculation logic is: convolution calculation is performed on the input vector through the multi-size convolution kernels of the parallel sub-branches in the Inception module to obtain local image features, the image spatial features of the image input vector are extracted through the maximum pooling branch of the parallel sub-branches in the Inception module, the image local features and the image spatial features are spliced in the channel dimension to obtain the Inception module output, and the image output features of the K layer are obtained by connecting the Inception modules in series and performing average pooling processing.

[0030] Preferably, the frequency input data is feature extracted through the HHT-one-dimensional convolutional neural network branch, and the calculation logic is: the frequency input data is subjected to empirical mode decomposition through the Hilbert-Huang transform algorithm to obtain an intrinsic mode function set, each function in the intrinsic mode function set is subjected to Hilbert transform processing to obtain a Hilbert spectrum, a time series spectrum feature vector is established based on the Hilbert spectrum and the time sequence, the time series spectrum feature vector is weighted summed up through the one-dimensional convolutional neural network, and the frequency output features of the K layer are obtained through ReLU function activation processing and maximum pooling processing;

[0031] Among them, the Capsule-Network network branch, the GoogLeNet network branch and the HHT-one-dimensional convolutional neural network branch are set to the same feature extraction layer number K.

[0032] Preferably, the feature fusion unit is used to perform feature fusion processing on the point cloud output features, image output features, and vibration frequency output features through an improved Transformer network to obtain deep fusion features. The processing logic includes:

[0033] An adaptive spatial attention mechanism is introduced, and the position information of the point cloud output features, image output features, and frequency output features of each extraction layer are respectively embedded through the sine-cosine encoding algorithm to obtain the features to be fused. The features to be fused include point cloud features to be fused, image features to be fused, and frequency features to be fused.

[0034] The spatial attention weight matrix is linearly transformed to generate query data, key data, and value matrix data corresponding to the point cloud features to be fused, the image features to be fused, and the vibration frequency features to be fused. The spatial attention score and spatial attention weight are calculated based on the query data, key data, and value matrix data.

[0035] Adaptively improve the spatial attention weight, introduce a weight scaling factor, multiply the spatial attention weight by the weight scaling factor to obtain the adaptive attention weight, and calculate the preliminary fusion feature based on the adaptive attention weight;

[0036] Cross-fusing the features to be fused in each feature extraction layer: linearly mapping and vector concatenating the preliminary fused features through the multi-head attention mechanism to obtain multi-head weighted features. The fully connected layer and ReLU activation in the feedforward network are used to perform dimension expansion and feature transformation on the multi-head weighted features, and deep fusion features are obtained through progressive calculation layer by layer.

[0037] The calculation expressions of the preliminary fusion features include:

[0038] ;

[0039] ;

[0040] ;

[0041] Among them, k represents the sequence number of feature extraction layers, i represents the branch sequence number, represents the weight scaling factor, e represents the natural logarithm, represents the hyperparameter, represents the amount of data corresponding to the i-th branch data in the k-th feature extraction layer, Represents the total amount of data of all branch data in the kth feature extraction layer, represents the adaptive attention weight, represents the spatial attention weight, represents the preliminary fusion features corresponding to the k-th feature extraction layer, Indicates the features to be fused corresponding to the i-th branch data in the k-th feature extraction layer.

[0042] Preferably, the decision classification unit is used to perform the classification task according to the deep fusion features to obtain the defect detection results. The processing logic includes:

[0043] Input the deep fusion features into the fully connected layer of the classifier, map the deep fusion features to the category vector space through the weight matrix and bias vector of the fully connected layer of the classifier, calculate the category probability through the Softmax function, and output the defect detection results and corresponding spatial coordinates based on the category probability;

[0044] Defect detection results include: no defects, tower tilt, main cable deformation, stiffening beam bending, anchor displacement, saddle tilt, abnormal cable vibration, component surface cracks and component surface rust

[0045] The beneficial effects of the present invention are as follows: Unlike the traditional method of performing full-structure laser scanning on the entire bridge, some areas are targetedly selected for laser scanning through a preset reference set of defective parts, which is beneficial to improving data acquisition efficiency and accuracy. The point cloud data is converted into a section shape curve, which simplifies the processing of complex point cloud data, which is beneficial to improving the calculation rate and fully reflecting the deformation of the bridge. The BIM model is used to plan image acquisition and laser scanning target areas and flight routes, which is beneficial to improving planning speed and accuracy. A parallel three-branch convolutional neural network is used to calculate data from three different sources respectively, which improves the utilization of multi-source data and is beneficial to fully obtain abstract feature data. The spatial attention mechanism and the Transformer network are used to deeply fuse different types of data, and the weight scaling factor is introduced to adaptively distribute the influence ratio of the three types of data when calculating the preliminary fusion features, which is beneficial to improving the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the basic framework of a bridge defect detection system based on a BIM model provided by one embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the basic process of a bridge defect detection system based on a BIM model provided by one embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the basic flow of a three-branch convolutional neural network provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0050] Example, refer to Figures 1 and Figure 2 , as one embodiment of the present invention, provides a bridge defect detection system based on a BIM model, comprising: a detection data acquisition module, a detection data processing module, a comprehensive judgment module and a terminal interaction module;

[0051] The inspection data acquisition module includes a point cloud data acquisition unit, an image data acquisition unit, and a vibration data acquisition unit. These units are used to plan the acquisition locations and methods according to the BIM model, perform laser scanning on the bridge to be inspected to obtain point cloud data of the first target location set, photograph the bridge to be inspected to obtain full-structure image data, select vibration sampling points, and obtain cable vibration data of the bridge to be inspected through vibration sensors.

[0052] The detection data processing module includes a preprocessing unit and a feature extraction unit, which are used to preprocess the point cloud data, full-structure image data and cable vibration data of the first target part set to obtain a point cloud input vector, an image input vector and vibration frequency input data, and perform feature extraction through a three-branch convolutional neural network to obtain point cloud output features, image output features and vibration frequency output features;

[0053] The comprehensive judgment module includes a feature fusion unit and a decision classification unit. It is used to perform feature fusion processing on the point cloud output features, image output features, and vibration frequency output features through an improved Transformer network to obtain deep fusion features. The classification task is performed based on the deep fusion features to obtain defect detection results.

[0054] The terminal interaction module includes a display interaction unit and a log storage unit, which is used to provide a display interface to the user and receive user operation instructions, receive user adjustments and settings for the system parameters of the detection data acquisition module, detection data processing module, and comprehensive judgment module, generate a detection log based on the defect detection results output by the comprehensive judgment module, and store the output data of the detection data acquisition module, detection data processing module, and comprehensive judgment module.

[0055] In this embodiment, the point cloud data acquisition unit is used to perform laser scanning on the bridge to be inspected to obtain point cloud data of the first target part set. The processing logic includes:

[0056] Retrieve the BIM model of the bridge to be inspected, establish a geodetic coordinate system based on the BIM model, traverse the BIM model of the bridge to be inspected based on a preset defect location reference set to obtain target locations that require laser scanning, and record the spatial coordinate data corresponding to all target locations that require laser scanning to obtain a first target location set;

[0057] The preset defect location reference set includes bridge towers, main cables, stiffening beams, anchors and saddles;

[0058] Obtaining component information data corresponding to each part of the first target part set through the BIM model, wherein the component information data includes the size, geometry, spatial coordinates, and spatial orientation of each part of the first target part set;

[0059] The BIM model is divided and annotated based on a preset scanning boundary expansion coefficient and component information data to obtain an extended laser scanning boundary. A UAV laser scanning route is planned based on the extended laser scanning boundary to obtain a first acquisition route. The UAV is commanded to fly along the first acquisition route, and data is collected by a 3D laser radar carried by the UAV to obtain point cloud data of the first target part set.

[0060] The preset scanning boundary expansion coefficients include a bridge tower area expansion coefficient, a main cable area expansion coefficient, a stiffening beam area expansion coefficient, an anchorage area expansion coefficient, and a saddle area expansion coefficient.

[0061] Among them, 3D laser scanning has high accuracy but slow speed, and the environment of large suspension bridges is often subject to wind interference, making it unsuitable for long-term scanning. Unlike traditional methods that perform full-structure scans of the entire bridge, a preset reference set of defective parts is used to specifically select certain areas for 3D laser scanning, which is conducive to improving data acquisition efficiency and accuracy. At the same time, the defective part set includes all types of structural parts of suspension bridges that are prone to defects. The bridge to be inspected only contains some structural parts. Through traversal processing, it is beneficial to comprehensively screen out the parts of the bridge to be inspected that require laser scanning, which is conducive to improving accuracy. In addition, the introduction of the scan boundary expansion coefficient ensures that the target structural parts are not missed while narrowing the scanning range, such as the side extension area of the stiffening beam.

[0062] In this embodiment, the image data acquisition unit is used to photograph the bridge to be inspected to obtain full-structure image data, and the processing logic includes:

[0063] Retrieve the BIM model of the bridge to be inspected, and layer it based on the spatial height of the BIM model of the bridge to be inspected to obtain the bridge deck layer, bridge bottom layer, tower beam layer and cable layer. Set the corresponding drone flight trajectory and acquisition angle for each layer respectively. Set the flight trajectory of the bridge deck layer to a parallel strip route, and the shooting angle to be 45° downward and vertical. Set the flight trajectory of the bridge bottom layer to a parallel strip route, and the shooting angle to be 45° upward and vertical. Set the flight trajectory of the tower beam layer to a vertical ascending route, and the shooting angle to be horizontal. Set the flight trajectory of the cable layer to a broken line fitting route, and the shooting angle to be horizontal.

[0064] The flight trajectories of the drones corresponding to each layer in the BIM model of the bridge to be inspected are connected in series to obtain the second acquisition route. The drone is commanded to fly according to the second acquisition route, and the full-structure image data is obtained by collecting data through the camera carried by the drone.

[0065] In this embodiment, the vibration data acquisition unit is used to select vibration sampling points and obtain cable vibration data of the bridge to be inspected through the vibration sensor. The processing logic includes:

[0066] Retrieve the BIM model of the bridge to be inspected, classify the cables based on their material data in the BIM model, and mark key vibration points on the BIM model. These key vibration points include the junction between the main cable and the saddle, the anchorage end of the suspender cable, and the anchorage area between the cable and the bridge tower.

[0067] Vibration sampling points are obtained by evenly selecting key vibration points based on a preset interval distance, and the vibration sampling points are collected by vibration sensors to obtain cable vibration data.

[0068] During the data collection process, the existing BIM model was leveraged to directly select collection locations and plan collection routes, improving data collection efficiency. Furthermore, compared to laser scanning, image acquisition is faster, allowing for full-structure image capture, facilitating the acquisition of sufficient data for the bridge under inspection. Suspension cables and stay cables are often numerous, so selecting representative vibration sampling points improves both the efficiency and accuracy of vibration data collection.

[0069] In this embodiment, the pre-processing unit is used to process the point cloud data of the first target part set to obtain a point cloud input vector, process the full-structure image data to obtain an image input vector, and process the cable vibration data to obtain vibration frequency input data. The processing logic includes:

[0070] Based on a preset sampling step size, the point cloud data of the first target part set is sliced to obtain an actual section point cloud, the polar angle of each actual section point cloud is calculated and sorted to obtain a point cloud stacking sequence, the point cloud convex hull vector set is pushed into the stack by Chan's algorithm and the point cloud stacking sequence, the point cloud convex hull vector set is vector cross multiplication is performed on the point cloud convex hull vector set to obtain a point cloud vector outer product, the point cloud vector outer product is filtered by a preset point cloud threshold to obtain section shape feature points, the section shape feature points are connected to obtain the actual section shape curve, the BIM model pair is extracted based on the central space coordinates of each actual section shape curve to obtain the corresponding reference section shape curve, the Euclidean distance standard deviation and the mean curvature difference between the actual section shape curve and the reference section shape curve with the same central space coordinate are calculated, and the point cloud input vector is constructed based on the index of the area to which each actual section shape curve belongs, the central space coordinate, the Euclidean distance standard deviation and the mean curvature difference between the actual section shape curve and the corresponding reference section shape curve;

[0071] Performing Gaussian filtering and denoising on the full-structure image data, and performing grayscale mapping on the denoised full-structure image data to obtain an image input vector;

[0072] The cable vibration data is supplemented by bilinear interpolation, denoised by low-pass filtering, sampled based on a preset sampling frequency to obtain discrete-time vibration data, and calculated by fast Fourier transform to obtain vibration frequency input data.

[0073] The index of the region to which the actual section shape curve belongs is the position number of the preset defect location reference set corresponding to the actual section shape curve. The preset defect location reference set includes bridge towers, main cables, stiffening girders, anchors, and saddles. In this embodiment, the bridge tower position is coded as 01, the main cable position is coded as 02, the stiffening girder position is coded as 03, the anchor position is coded as 04, and the saddle position is coded as 05. Using the difference between the section shape curve and the reference section shape curve in the BIM model as the point cloud input vector simplifies the processing of complex point cloud data, improves calculation speed, and fully reflects the deformation of the bridge.

[0074] In this embodiment, the feature extraction unit is used to perform feature extraction on the point cloud input vector to obtain point cloud output features, on the image input vector to obtain image output features, and on the frequency input data to obtain frequency output features through a three-branch convolutional neural network. The processing logic includes:

[0075] The feature extraction of the point cloud input vector is performed through the Capsule-Network network branch. The calculation logic is as follows: the point cloud input vector is continuously convolved through the series convolution layer to obtain the primary capsule layer input data, and the primary capsule layer performs local feature encoding on the primary capsule layer input data to obtain a multi-dimensional digital feature vector. The dynamic routing algorithm establishes a connection between the primary capsule layer and the advanced capsule layer, and the advanced capsule layer matches the multi-dimensional digital feature vectors corresponding to each primary capsule in the primary capsule layer to obtain the connection weight. The connection weights corresponding to each primary capsule are integrated and calculated to obtain the K-layer point cloud output features.

[0076] In this embodiment, feature extraction is performed on the image input vector through the GoogLeNet network branch, and the calculation logic is as follows: convolution calculation is performed on the input vector through the multi-size convolution kernels of the parallel sub-branches in the Inception module to obtain local image features, the image spatial features of the image input vector are extracted through the maximum pooling branch of the parallel sub-branch in the Inception module, the image local features and the image spatial features are spliced in the channel dimension to obtain the Inception module output, and the image output features of the K layer are obtained by connecting the Inception modules in series and performing average pooling processing.

[0077] In this embodiment, the frequency input data is feature extracted using the HHT-one-dimensional convolutional neural network branch. The calculation logic is as follows: the frequency input data is subjected to empirical mode decomposition using the Hilbert-Huang transform algorithm to obtain an intrinsic mode function set, each function in the intrinsic mode function set is subjected to Hilbert transform processing to obtain a Hilbert spectrum, a time series spectrum feature vector is established based on the Hilbert spectrum and the time sequence, the time series spectrum feature vector is weighted summed using a one-dimensional convolutional neural network, and the frequency output features of the K layer are obtained through ReLU function activation processing and maximum pooling processing;

[0078] Among them, the Capsule-Network network branch, the GoogLeNet network branch and the HHT-one-dimensional convolutional neural network branch are set to the same feature extraction layer number K.

[0079] The same number of feature extraction layers specifically refers to: K capsule layers in the Capsule Network branch, K Inception module layers in the GoogLeNet branch, and K convolutional layers in the HHT-1D convolutional neural network branch. This setting facilitates subsequent feature fusion. Furthermore, using a three-branch convolutional neural network in parallel to perform computations on data from three different sources facilitates the acquisition of fully abstracted feature data.

[0080] In this embodiment, the feature fusion unit is used to perform feature fusion processing on the point cloud output features, image output features, and vibration frequency output features through an improved Transformer network to obtain deep fusion features. The processing logic includes:

[0081] An adaptive spatial attention mechanism is introduced, and the position information of the point cloud output features, image output features, and frequency output features of each extraction layer are respectively embedded through the sine-cosine encoding algorithm to obtain the features to be fused. The features to be fused include point cloud features to be fused, image features to be fused, and frequency features to be fused.

[0082] The spatial attention weight matrix is linearly transformed to generate query data, key data, and value matrix data corresponding to the point cloud features to be fused, the image features to be fused, and the vibration frequency features to be fused. The spatial attention score and spatial attention weight are calculated based on the query data, key data, and value matrix data.

[0083] Adaptively improve the spatial attention weight, introduce a weight scaling factor, multiply the spatial attention weight by the weight scaling factor to obtain the adaptive attention weight, and calculate the preliminary fusion feature based on the adaptive attention weight;

[0084] Cross-fusing the features to be fused in each feature extraction layer: linearly mapping and vector concatenating the preliminary fused features through the multi-head attention mechanism to obtain multi-head weighted features. The fully connected layer and ReLU activation in the feedforward network are used to perform dimension expansion and feature transformation on the multi-head weighted features, and deep fusion features are obtained through progressive calculation layer by layer.

[0085] The calculation expressions of the preliminary fusion features include:

[0086] ;

[0087] ;

[0088] ;

[0089] Among them, k represents the sequence number of feature extraction layers, i represents the branch sequence number, represents the weight scaling factor, e represents the natural logarithm, represents the hyperparameter, represents the amount of data corresponding to the i-th branch data in the k-th feature extraction layer, Represents the total amount of data of all branch data in the kth feature extraction layer, represents the adaptive attention weight, represents the spatial attention weight, represents the preliminary fusion features corresponding to the k-th feature extraction layer, Indicates the features to be fused corresponding to the i-th branch data in the k-th feature extraction layer.

[0090] Among them, different types of data are deeply fused through the spatial attention mechanism and Transformer network, and the weight scaling factor is introduced to adaptively distribute the influence ratio of the three types of data when calculating the preliminary fusion features, which is conducive to improving the accuracy of the preliminary fusion features.

[0091] In this embodiment, the decision classification unit is used to perform classification tasks based on deep fusion features to obtain defect detection results. The processing logic includes:

[0092] Input the deep fusion features into the fully connected layer of the classifier, map the deep fusion features to the category vector space through the weight matrix and bias vector of the fully connected layer of the classifier, calculate the category probability through the Softmax function, and output the defect detection results and corresponding spatial coordinates based on the category probability;

[0093] Defect detection results include: no defects, bridge tower tilt, main cable deformation, stiffening beam bending, anchor displacement, saddle tilt, abnormal cable vibration, component surface cracks and component surface rust.

[0094] Those skilled in the art will appreciate that embodiments of the present invention may provide methods, systems, or computer program products. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium may be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A bridge defect detection system based on BIM model, characterized in that: include: Detection data acquisition module, detection data processing module, comprehensive judgment module and terminal interaction module; The inspection data acquisition module includes a point cloud data acquisition unit, an image data acquisition unit, and a vibration data acquisition unit. These units are used to plan the acquisition locations and methods according to the BIM model, perform laser scanning on the bridge to be inspected to obtain point cloud data of the first target location set, photograph the bridge to be inspected to obtain full-structure image data, select vibration sampling points, and obtain cable vibration data of the bridge to be inspected through vibration sensors. The detection data processing module includes a preprocessing unit and a feature extraction unit, which are used to preprocess the point cloud data, full-structure image data and cable vibration data of the first target part set to obtain a point cloud input vector, an image input vector and vibration frequency input data, and perform feature extraction through a three-branch convolutional neural network to obtain point cloud output features, image output features and vibration frequency output features; The comprehensive judgment module includes a feature fusion unit and a decision classification unit. It is used to perform feature fusion processing on the point cloud output features, image output features, and vibration frequency output features through an improved Transformer network to obtain deep fusion features. The classification task is performed based on the deep fusion features to obtain defect detection results. The terminal interaction module includes a display interaction unit and a log storage unit, which is used to provide a display interface to the user and receive user operation instructions, receive user adjustments and settings for system parameters of the detection data acquisition module, detection data processing module, and comprehensive judgment module, generate a detection log based on the defect detection results output by the comprehensive judgment module, and store the output data of the detection data acquisition module, detection data processing module, and comprehensive judgment module; The feature fusion unit is used to perform feature fusion processing on point cloud output features, image output features, and vibration frequency output features through the improved Transformer network to obtain deep fusion features. The processing logic includes: An adaptive spatial attention mechanism is introduced, and the position information of the point cloud output features, image output features, and frequency output features of each extraction layer are respectively embedded through the sine-cosine encoding algorithm to obtain the features to be fused. The features to be fused include point cloud features to be fused, image features to be fused, and frequency features to be fused. The spatial attention weight matrix is linearly transformed to generate query data, key data, and value matrix data corresponding to the point cloud features to be fused, the image features to be fused, and the vibration frequency features to be fused. The spatial attention score and spatial attention weight are calculated based on the query data, key data, and value matrix data. Adaptively improve the spatial attention weight, introduce a weight scaling factor, multiply the spatial attention weight by the weight scaling factor to obtain the adaptive attention weight, and calculate the preliminary fusion feature based on the adaptive attention weight; Cross-fusing the features to be fused in each feature extraction layer: linearly mapping and vector concatenating the preliminary fused features through the multi-head attention mechanism to obtain multi-head weighted features. The fully connected layer and ReLU activation in the feedforward network are used to perform dimension expansion and feature transformation on the multi-head weighted features, and deep fusion features are obtained through progressive calculation layer by layer. The calculation expressions of the preliminary fusion features include: ; ; ; Among them, k represents the sequence number of feature extraction layers, i represents the branch sequence number, represents the weight scaling factor, e represents the natural logarithm, represents the hyperparameter, represents the amount of data corresponding to the i-th branch data in the k-th feature extraction layer, Represents the total amount of data of all branch data in the kth feature extraction layer, represents the adaptive attention weight, represents the spatial attention weight, represents the preliminary fusion features corresponding to the k-th feature extraction layer, Indicates the features to be fused corresponding to the i-th branch data in the k-th feature extraction layer.

2. The bridge defect detection system based on a BIM model according to claim 1, characterized in that: The point cloud data acquisition unit is used to perform laser scanning on the bridge to be inspected to obtain point cloud data of the first target part set. The processing logic includes: Retrieve the BIM model of the bridge to be inspected, establish a geodetic coordinate system based on the BIM model, traverse the BIM model of the bridge to be inspected based on a preset defect location reference set to obtain target locations that require laser scanning, and record the spatial coordinate data corresponding to all target locations that require laser scanning to obtain a first target location set; The preset defect location reference set includes bridge towers, main cables, stiffening beams, anchors and saddles; Obtaining component information data corresponding to each part of the first target part set through the BIM model, wherein the component information data includes the size, geometry, spatial coordinates, and spatial orientation of each part of the first target part set; The BIM model is divided and annotated based on a preset scanning boundary expansion coefficient and component information data to obtain an extended laser scanning boundary. A UAV laser scanning route is planned based on the extended laser scanning boundary to obtain a first acquisition route. The UAV is commanded to fly along the first acquisition route, and data is collected by a 3D laser radar carried by the UAV to obtain point cloud data of the first target part set. The preset scanning boundary expansion coefficients include a bridge tower area expansion coefficient, a main cable area expansion coefficient, a stiffening beam area expansion coefficient, an anchorage area expansion coefficient, and a saddle area expansion coefficient.

3. The bridge defect detection system based on a BIM model according to claim 1, characterized in that: The image data acquisition unit is used to capture the bridge to be inspected and obtain full-structure image data. The processing logic includes: Retrieve the BIM model of the bridge to be inspected, and layer it based on the spatial height of the BIM model of the bridge to be inspected to obtain the bridge deck layer, bridge bottom layer, tower beam layer and cable layer. Set the corresponding drone flight trajectory and acquisition angle for each layer respectively. Set the flight trajectory of the bridge deck layer to a parallel strip route, and the shooting angle to be 45° downward and vertical. Set the flight trajectory of the bridge bottom layer to a parallel strip route, and the shooting angle to be 45° upward and vertical. Set the flight trajectory of the tower beam layer to a vertical ascending route, and the shooting angle to be horizontal. Set the flight trajectory of the cable layer to a broken line fitting route, and the shooting angle to be horizontal. The flight trajectories of the drones corresponding to each layer in the BIM model of the bridge to be inspected are connected in series to obtain the second acquisition route. The drone is commanded to fly according to the second acquisition route, and the full-structure image data is obtained by collecting data through the camera carried by the drone.

4. The bridge defect detection system based on a BIM model according to claim 1, characterized in that: The vibration data acquisition unit is used to select vibration sampling points and obtain the cable vibration data of the bridge to be inspected through the vibration sensor. The processing logic includes: Retrieve the BIM model of the bridge to be inspected, classify the cables based on their material data in the BIM model, and mark key vibration points on the BIM model. These key vibration points include the junction between the main cable and the saddle, the anchorage end of the suspender cable, and the anchorage area between the cable and the bridge tower. Vibration sampling points are obtained by evenly selecting key vibration points based on a preset interval distance, and the vibration sampling points are collected by vibration sensors to obtain cable vibration data.

5. The bridge defect detection system based on a BIM model according to claim 1, characterized in that: The pre-processing unit is used to process the point cloud data of the first target part set to obtain a point cloud input vector, process the full structure image data to obtain an image input vector, and process the cable vibration data to obtain vibration frequency input data. The processing logic includes: Based on a preset sampling step size, the point cloud data of the first target part set is sliced to obtain an actual section point cloud, the polar angle of each actual section point cloud is calculated and sorted to obtain a point cloud stacking sequence, the point cloud convex hull vector set is pushed into the stack by Chan's algorithm and the point cloud stacking sequence, the point cloud convex hull vector set is vector cross multiplication is performed on the point cloud convex hull vector set to obtain a point cloud vector outer product, the point cloud vector outer product is filtered by a preset point cloud threshold to obtain section shape feature points, the section shape feature points are connected to obtain the actual section shape curve, the BIM model pair is extracted based on the central space coordinates of each actual section shape curve to obtain the corresponding reference section shape curve, the Euclidean distance standard deviation and the mean curvature difference between the actual section shape curve and the reference section shape curve with the same central space coordinate are calculated, and the point cloud input vector is constructed based on the index of the area to which each actual section shape curve belongs, the central space coordinate, the Euclidean distance standard deviation and the mean curvature difference between the actual section shape curve and the corresponding reference section shape curve; Performing Gaussian filtering denoising on the full-structure image data, and performing grayscale mapping on the denoised full-structure image data to obtain an image input vector; The cable vibration data is supplemented by bilinear interpolation, denoised by low-pass filtering, sampled based on a preset sampling frequency to obtain discrete-time vibration data, and calculated by fast Fourier transform to obtain vibration frequency input data.

6. The bridge defect detection system based on a BIM model according to claim 1, characterized in that: The feature extraction unit is used to extract features from the point cloud input vector to obtain point cloud output features, extract features from the image input vector to obtain image output features, and extract features from the frequency input data to obtain frequency output features through a three-branch convolutional neural network. The processing logic includes: The feature extraction of the point cloud input vector is performed through the Capsule-Network network branch. The calculation logic is as follows: the point cloud input vector is continuously convolved through the series convolution layer to obtain the primary capsule layer input data, and the primary capsule layer performs local feature encoding on the primary capsule layer input data to obtain a multi-dimensional digital feature vector. The dynamic routing algorithm establishes a connection between the primary capsule layer and the advanced capsule layer, and the advanced capsule layer matches the multi-dimensional digital feature vectors corresponding to each primary capsule in the primary capsule layer to obtain the connection weight. The connection weights corresponding to each primary capsule are integrated and calculated to obtain the K-layer point cloud output features.

7. The bridge defect detection system based on a BIM model according to claim 6, characterized in that: The image input vector is extracted through the GoogLeNet network branch. The calculation logic is as follows: the input vector is convolved with the multi-size convolution kernels of the parallel sub-branch in the Inception module to obtain the local features of the image. The image spatial features of the image input vector are extracted through the maximum pooling branch of the parallel sub-branch in the Inception module. The local features and image spatial features are spliced in the channel dimension to obtain the Inception module output. The image output features of the K layer are obtained by connecting the Inception modules in series and performing average pooling.

8. The bridge defect detection system based on a BIM model according to claim 6, characterized in that: The frequency input data is feature extracted through the HHT-one-dimensional convolutional neural network branch. The calculation logic is as follows: the frequency input data is subjected to empirical mode decomposition using the Hilbert-Huang transform algorithm to obtain an intrinsic mode function set, each function in the intrinsic mode function set is subjected to Hilbert transform processing to obtain the Hilbert spectrum, and a time series spectrum feature vector is established based on the Hilbert spectrum and time sequence. The time series spectrum feature vector is weighted and summed using a one-dimensional convolutional neural network, and the frequency output features of the K layer are obtained through ReLU function activation processing and maximum pooling processing; Among them, the Capsule-Network network branch, the GoogLeNet network branch and the HHT-one-dimensional convolutional neural network branch are set to the same feature extraction layer number K.

9. The bridge defect detection system based on a BIM model according to claim 1, characterized in that: The decision classification unit is used to perform classification tasks based on deep fusion features to obtain defect detection results. The processing logic includes: Input the deep fusion features into the fully connected layer of the classifier, map the deep fusion features to the category vector space through the weight matrix and bias vector of the fully connected layer of the classifier, calculate the category probability through the Softmax function, and output the defect detection results and corresponding spatial coordinates based on the category probability; Defect detection results include: no defects, bridge tower tilt, main cable deformation, stiffening beam bending, anchor displacement, saddle tilt, abnormal cable vibration, component surface cracks and component surface rust.

Images