Bridge defect detection system based on BIM model

Abstract

The invention discloses a BIM model-based bridge defect detection system, which relates to the technical field of bridge detection and comprises a detection data acquisition module, a detection data processing module, a comprehensive judgment module and a terminal interaction module, the detection data acquisition module is used for planning an acquisition part and an acquisition mode according to the BIM model, performing laser scanning on a to-be-detected bridge to obtain point cloud data of a first target part set, shooting the to-be-detected bridge to obtain full-structure image data, selecting a vibration sampling point and obtaining cable vibration data of the to-be-detected bridge through a vibration sensor; the detection data processing module is used for preprocessing point cloud data, full-structure image data and cable vibration data of the first target part set, and performing feature extraction through a three-branch convolutional neural network; and the comprehensive judgment module is used for performing feature fusion processing through an improved Transform network to obtain deep fusion features, and executing a classification task according to the deep fusion features to obtain a defect detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge defect detection, and particularly relates to a bridge defect detection system based on a BIM model. Background Art

[0002] Bridges are an important part of transportation infrastructure. After being built and put into use, it is crucial to regularly detect and maintain bridges. When using manual methods to detect large bridges, there are problems of high defect omission rates and insufficient efficiency. Thanks to the innovative breakthroughs in information technology, artificial intelligence, unmanned aerial vehicle technology, and BIM model technology have gradually been applied to bridge defect detection. However, there are various types of bridges, and there is a lack of targeted and comprehensive detection methods for suspension bridges. Therefore, developing a defect detection system for suspension bridges is a research direction with prospects.

[0003] Currently, the Chinese patent application with the application number 202210419345.5 discloses a BIM-based bridge detection and management system. This application includes: a CPU module. By setting the CPU module, when detecting a bridge, overall coordination and adjustment can be carried out through the CPU module. The CPU module includes a deployment unit, a detection unit, and a management unit. The deployment unit includes a registration module and a statistics module. The signal output end of the registration module is connected to the signal receiving end of the statistics module, and the signal output end of the statistics module is connected to the signal receiving end of the schedule module. By setting the CPU module, when detecting a bridge, overall coordination and adjustment can be carried out through the CPU module, and the schedule module is transmitted through the communication module, so that detection personnel can all understand the detection time and sequence, thereby improving efficiency. The schedule information is timestamped through Time Module 1 and stored through Storage Module 1 for convenient query of the bridge maintenance cycle in the future. However, this method lacks targeted detection of different bridge parts and comprehensive defect judgment. Summary of the Invention

[0004] The technical problem solved by the present invention is that traditional methods lack targeted detection of different bridge parts. Especially for suspension bridge types, only image detection is performed, lacking multi-source comprehensive judgment. In addition, due to the occlusion of suspension cables and stay cables, the flight routes during data collection by traditional methods using unmanned aerial vehicles are not applicable. At the same time, the utilization rate of multi-source data is low, and there is a lack of adaptive adjustment of weights during defect judgment, which is not conducive to reasonably integrating multi-source features.

[0005] To solve the above technical problems, the present invention provides the following technical solutions:

[0006] A bridge defect detection system based on a BIM model, comprising: a detection data collection module, a detection data processing module, a comprehensive judgment module, and a terminal interaction module;

[0007] The detection data acquisition module includes a point cloud data acquisition unit, an image data acquisition unit, and a vibration data acquisition unit, which are used to plan the acquisition parts and acquisition methods according to the BIM model, perform laser scanning on the bridge to be detected to obtain the point cloud data of the first target part set, take pictures of the bridge to be detected to obtain the full-structure image data, select vibration sampling points, and obtain the cable vibration data of the bridge to be detected through vibration sensors;

[0008] The detection data processing module includes a preprocessing unit and a feature extraction unit, which are used to preprocess the point cloud data, full-structure image data, and cable vibration data of the first target part set to obtain a point cloud input vector, an image input vector, and a vibration frequency input data, and perform feature extraction through a three-branch convolutional neural network to obtain a point cloud output feature, an image output feature, and a vibration frequency output feature;

[0009] The comprehensive judgment module includes a feature fusion unit and a decision classification unit, which are used to perform feature fusion processing on the point cloud output feature, image output feature, and vibration frequency output feature through an improved Transform network to obtain a deep fusion feature, and perform a classification task according to the deep fusion feature to obtain a defect detection result;

[0010] The terminal interaction module includes a display interaction unit and a log storage unit, which are used to provide a display interface for the user and receive user operation instructions, receive the user's adjustment and setting of the system parameters of the detection data acquisition module, detection data processing module, and comprehensive judgment module, generate a detection log based on the defect detection result output by the comprehensive judgment module, and store the output data of the detection data acquisition module, detection data processing module, and comprehensive judgment module.

[0011] Preferably, the point cloud data acquisition unit is used to perform laser scanning on the bridge to be detected to obtain the point cloud data of the first target part set, and the processing logic includes:

[0012] Retrieve the BIM model of the bridge to be detected, establish a geodetic coordinate system based on the BIM model, traverse the BIM model of the bridge to be detected based on the preset defect part reference set to obtain the target positions that need to be laser scanned, and record the spatial coordinate data corresponding to all the target positions that need to be laser scanned to obtain the first target part set;

[0013] The preset defect part reference set includes bridge towers, main cables, stiffening girders, anchor piers, and saddles;

[0014] Obtain the component information data corresponding to each part in the first target part set through the BIM model, and the component information data includes the dimensions, geometric shapes, spatial coordinates, and spatial orientations of each part in the first target part set;

[0015] Divide and label the BIM model based on a preset scanning boundary expansion coefficient and component information data to obtain an extended laser scanning boundary. Plan the UAV laser scanning route based on the extended laser scanning boundary to obtain a first acquisition route. Command the UAV to fly according to the first acquisition route, and collect point cloud data of the first target part set through the 3D lidar carried by the UAV;

[0016] The preset scanning boundary expansion coefficients include a bridge tower area magnification coefficient, a main cable area magnification coefficient, a stiffening girder area magnification coefficient, an anchor area magnification coefficient, and a saddle area magnification coefficient.

[0017] Preferably, the image data acquisition unit is used to photograph the bridge to be detected to obtain full-structure image data. The processing logic includes:

[0018] Retrieve the BIM model of the bridge to be detected, layer it based on the spatial height of the BIM model of the bridge to be detected to obtain a deck system layer, a bridge bottom layer, a tower-girder layer, and a cable layer. Set corresponding UAV flight trajectories and acquisition angles for each layer respectively. Set the flight trajectory of the deck system layer as a parallel strip route and the shooting angle as 45° in the direction of obliquely downward and perpendicular angles. Set the flight trajectory of the bridge bottom layer as a parallel strip route and the shooting angle as 45° in the direction of obliquely upward and perpendicular angles. Set the flight trajectory of the tower-girder layer as a vertical ascending route and the shooting angle as a horizontal angle direction. Set the flight trajectory of the cable layer as a broken line fitting route and the shooting angle as a horizontal angle direction;

[0019] Connect the corresponding UAV flight trajectories of each layer in the BIM model of the bridge to be detected to obtain a second acquisition route. Command the UAV to fly according to the second acquisition route, and collect full-structure image data through the camera carried on the UAV.

[0020] Preferably, the vibration data acquisition unit is used to select vibration sampling points and obtain the cable vibration data of the bridge to be detected through vibration sensors. The processing logic includes:

[0021] Retrieve the BIM model of the bridge to be detected, classify the suspension cables based on the suspension cable material data in the BIM model, and mark the key vibration part points in the BIM model. The key vibration part points include the joint between the main cable and the saddle, the anchoring end of the suspender, and the anchorage area of the stay cable and the bridge tower;

[0022] Uniformly select the key vibration part points based on a preset interval distance to obtain vibration sampling points, and collect cable vibration data through vibration sensors for the vibration sampling points.

[0023] Preferably, the preprocessing unit is used to process the point cloud data of the first target part set to obtain a point cloud input vector, process the full-structure image data to obtain an image input vector, and process the cable vibration data to obtain a vibration frequency input data. The processing logic includes:

[0024] Slice the point cloud data of the first target part set based on a preset sampling step to obtain actual section point clouds, calculate the polar angles of each actual section point cloud respectively and sort them to obtain a point cloud stack sequence, perform stack processing through Chan's algorithm and the point cloud stack sequence to obtain a point cloud convex hull vector set, perform vector cross product calculation on the point cloud convex hull vector set to obtain a point cloud vector outer product, screen the point cloud vector outer product through a preset point cloud threshold to obtain section shape feature points, connect based on the section shape feature points to obtain an actual section shape curve, extract the corresponding reference section shape curve from the BIM model based on the central spatial coordinates of each actual section shape curve, calculate the Euclidean distance standard deviation difference and curvature difference mean between the actual section shape curve and the reference section shape curve with the same central spatial coordinates, and constitute a point cloud input vector based on the index of the region where each actual section shape curve belongs, the central spatial coordinates, the Euclidean distance standard deviation difference and curvature difference mean with the corresponding reference section shape curve;

[0025] Perform Gaussian filtering denoising processing on the full-structure image data, and perform gray mapping on the denoised full-structure image data to obtain an image input vector;

[0026] Supplement the cable vibration data through bilinear interpolation, perform denoising processing through low-pass filtering, sample based on a preset sampling frequency to obtain discrete-time vibration data, and calculate the vibration frequency input data through fast Fourier transform on the discrete-time vibration data.

[0027] Preferably, the feature extraction unit is used to extract features from the point cloud input vector, the image input vector, and the vibration frequency input data respectively through a three-branch convolutional neural network. The processing logic includes:

[0028] Extract features from the point cloud input vector through the Capsule-Network network branch. The calculation logic is: perform continuous convolutional calculation on the point cloud input vector through a series of convolutional layers to obtain the input data of the primary capsule layer, perform local feature encoding on the input data of the primary capsule layer through the primary capsule layer to obtain a multi-dimensional digital feature vector, establish a connection between the primary capsule layer and the secondary capsule layer through a dynamic routing algorithm, perform matching calculation on the multi-dimensional digital feature vectors corresponding to each primary capsule in the primary capsule layer through the secondary capsule layer to obtain connection weights, and perform integration calculation based on the connection weights corresponding to each primary capsule to obtain the point cloud output feature of the K layer.

[0029] Preferably, the feature extraction of the image input vector is performed through the GoogLeNet network branch, and the calculation logic is as follows: The multi-size convolutional kernels of the parallel sub-branches in the Inception module are used to perform convolutional calculations on the input vector respectively to obtain the local image features. The maximum pooling branch of the parallel sub-branches in the Inception module is used to extract the image spatial features of the image input vector. The local image features and the image spatial features are concatenated in the channel dimension to obtain the output of the Inception module. The image output features of K layers are obtained through the cascaded Inception modules and average pooling processing.

[0030] Preferably, the feature extraction of the vibration frequency input data is performed through the HHT-1D convolutional neural network branch, and the calculation logic is as follows: The empirical mode decomposition of the vibration frequency input data is performed through the Hilbert-Huang transform algorithm to obtain the set of intrinsic mode functions. The Hilbert transform processing is respectively performed on each function in the set of intrinsic mode functions to obtain the Hilbert spectrum. The time-series spectral feature vector is established based on the Hilbert spectrum and the time sequence. The weighted sum calculation is performed on the time-series spectral feature vector through the 1D convolutional neural network. The vibration frequency output features of K layers are obtained through the ReLU function activation processing and the maximum pooling processing;

[0031] Among them, the Capsule-Network network branch, the GoogLeNet network branch, and the HHT-1D convolutional neural network branch are set to the same feature extraction layer number K.

[0032] Preferably, the feature fusion unit is used to perform feature fusion processing on the point cloud output features, the image output features, and the vibration frequency output features through the improved Transform network to obtain the deep fusion features. The processing logic includes:

[0033] Introduce the adaptive spatial attention mechanism. The position information embedding processing is respectively performed on the point cloud output features, the image output features, and the vibration frequency output features of each extraction layer through the sine-cosine encoding algorithm to obtain the features to be fused. The features to be fused include the point cloud features to be fused, the image features to be fused, and the vibration frequency features to be fused;

[0034] The query data, key data, and value matrix data corresponding to the point cloud features to be fused, the image features to be fused, and the vibration frequency features to be fused are respectively generated through the linear transformation of the spatial attention weight matrix. The spatial attention score and the spatial attention weight are calculated based on the query data, key data, and value matrix data;

[0035] The adaptive improvement of the spatial attention weight is performed. The weight scaling factor is introduced, and the spatial attention weight is multiplied by the weight scaling factor to obtain the adaptive attention weight. The preliminary fusion features are calculated based on the adaptive attention weight;

[0036] The cross-fusion of the features to be fused in each feature extraction layer is performed through the Transformer module. The multi-head attention mechanism is used to perform linear mapping and vector splicing on the preliminarily fused features to obtain multi-head weighted features. The dimension expansion and feature transformation of the multi-head weighted features are carried out through the fully connected layer and ReLU activation in the feed-forward network. The Transformer network is calculated layer by layer to obtain deep fusion features;

[0037] The calculation expression of the preliminarily fused features includes:

[0038]

[0039] Among them, k represents the serial number of the feature extraction layer, i represents the branch serial number, represents the weight scaling factor, e represents the natural logarithm, α represents the hyperparameter, represents the data volume corresponding to the data of the i-th branch in the k-th feature extraction layer, N K represents the total data volume of all branch data in the k-th feature extraction layer, represents the adaptive attention weight, represents the spatial attention weight, represents the preliminarily fused features corresponding to the k-th feature extraction layer, represents the features to be fused corresponding to the data of the i-th branch in the k-th feature extraction layer.

[0040] Preferably, the decision classification unit is used to perform a classification task according to the deep fusion features to obtain the defect detection result. The processing logic includes:

[0041] The deep fusion features are input into the fully connected layer of the classifier. The deep fusion features are feature-mapped to the class vector space through the weight matrix and bias vector of the fully connected layer of the classifier. The class probability is calculated through the Softmax function, and the defect detection result and the corresponding spatial coordinates are output based on the class probability;

[0042] The defect detection results include: no defect, tower inclination, main cable deformation, stiffening girder bending, anchor displacement, saddle inclination, cable vibration anomaly, component surface crack, and component surface corrosion

[0043] Advantages of the present invention: Different from the traditional method of performing full-structure laser scanning on all bridges, partial areas are selectively laser-scanned according to a preset reference set of defect parts, which is conducive to improving the efficiency and accuracy of data acquisition. Converting the point cloud data into sectional shape curves simplifies the processing of complex point cloud data, which is conducive to improving the calculation speed and fully reflecting the deformation of the bridge. Planning the image acquisition and laser scanning target areas and flight routes through the BIM model is conducive to improving the planning speed and accuracy. Using a parallel three-branch convolutional neural network to calculate three different sources of data respectively improves the utilization of multi-source data and is conducive to fully obtaining abstract feature data. Through the spatial attention mechanism and the Transformer network, deep fusion of different types of data is carried out, and a weight scaling factor is introduced to adaptively allocate the influence ratio of the three types of data when calculating the preliminary fusion features, which is conducive to improving the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 FIG. is a schematic diagram of the basic framework of a bridge defect detection system based on a BIM model provided by an embodiment of the present invention;

[0045] Figure 2 FIG. is a schematic diagram of the basic process of a bridge defect detection system based on a BIM model provided by an embodiment of the present invention;

[0046] Figure 3 FIG. is a schematic diagram of the basic process of a three-branch convolutional neural network provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the following detailed description of the specific embodiments of the present invention is made with reference to the accompanying drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments.

[0048] Embodiment, referring to Figure 1 and Figure 2 , an embodiment of the present invention provides a bridge defect detection system based on a BIM model, including: a detection data acquisition module, a detection data processing module, a comprehensive judgment module, and a terminal interaction module;

[0049] The detection data acquisition module includes a point cloud data acquisition unit, an image data acquisition unit, and a vibration data acquisition unit, which are used to plan the acquisition parts and acquisition methods according to the BIM model, perform laser scanning on the bridge to be detected to obtain the point cloud data of the first target part set, take pictures of the bridge to be detected to obtain the full-structure image data, select vibration sampling points, and obtain the cable vibration data of the bridge to be detected through vibration sensors;

[0050] The detection data processing module includes a preprocessing unit and a feature extraction unit, which are used to preprocess the point cloud data, full-structure image data, and cable vibration data of the first target part set to obtain a point cloud input vector, an image input vector, and a vibration frequency input data, and perform feature extraction through a three-branch convolutional neural network to obtain a point cloud output feature, an image output feature, and a vibration frequency output feature;

[0051] The comprehensive judgment module includes a feature fusion unit and a decision classification unit, which are used to perform feature fusion processing on the point cloud output feature, image output feature, and vibration frequency output feature through an improved Transform network to obtain a deep fusion feature, and perform a classification task based on the deep fusion feature to obtain a defect detection result;

[0052] The terminal interaction module includes a display interaction unit and a log storage unit, which are used to provide a display interface to the user and receive user operation instructions, receive the user's adjustment settings of the system parameters of the detection data acquisition module, detection data processing module, and comprehensive judgment module, generate a detection log based on the defect detection result output by the comprehensive judgment module, and store the output data of the detection data acquisition module, detection data processing module, and comprehensive judgment module.

[0053] In this embodiment, the point cloud data acquisition unit is used to perform laser scanning on the bridge to be detected to obtain the point cloud data collected from the first target part, and the processing logic includes:

[0054] Retrieve the BIM model of the bridge to be detected, establish a geodetic coordinate system based on the BIM model, traverse the BIM model of the bridge to be detected based on the preset defect part reference set to obtain the target positions that need to be laser scanned, and record the spatial coordinate data corresponding to all the target positions that need to be laser scanned to obtain the first target part set;

[0055] The preset defect part reference set includes bridge towers, main cables, stiffening girders, anchor piers, and saddles;

[0056] Obtain the component information data corresponding to each part in the first target part set through the BIM model, and the component information data includes the dimensions, geometric shapes, spatial coordinates, and spatial orientations of each part in the first target part set;

[0057] Divide and label the BIM model based on the preset scanning boundary expansion coefficient and component information data to obtain an extended laser scanning boundary, plan the UAV laser scanning route based on the extended laser scanning boundary to obtain the first acquisition route, command the UAV to fly according to the first acquisition route, and perform data acquisition through the 3D lidar carried by the UAV to obtain the point cloud data of the first target part set;

[0058] The preset scanning boundary expansion coefficients include the bridge tower area magnification coefficient, the main cable area magnification coefficient, the stiffening girder area magnification coefficient, the anchor area magnification coefficient, and the saddle area magnification coefficient.

[0059] Among them, 3D laser scanning has high precision but low speed, and the environment of large suspension bridges often has wind interference, making it not suitable for long-term scanning. Different from the traditional method of full-structure scanning of the entire bridge, 3D laser scanning is selectively carried out on some areas through a preset reference set of defect parts, which is conducive to improving the efficiency and accuracy of data collection. At the same time, the defect part set contains the structural parts prone to defects of all types of suspension bridges, and the bridge to be detected only contains some structural parts. Through traversal processing, it is beneficial to comprehensively screen out the parts that need laser scanning in the bridge to be detected, which is conducive to improving the accuracy. In addition, the introduction of the scanning boundary expansion coefficient ensures that while reducing the scanning range, the target structural parts are not missed, such as the side extension area of the stiffening girder.

[0060] In this embodiment, the image data acquisition unit is used to take pictures of the bridge to be detected to obtain full-structure image data, and the processing logic includes:

[0061] Retrieve the BIM model of the bridge to be detected, layer it based on the spatial height of the BIM model of the bridge to be detected to obtain the deck layer, the bridge bottom layer, the tower-girder layer, and the cable layer, and set corresponding UAV flight trajectories and acquisition angles for each layer. The flight trajectory of the deck layer is set as a parallel strip route, and the shooting angle is set as 45° between the oblique downward and the vertical angles. The flight trajectory of the bridge bottom layer is set as a parallel strip route, and the shooting angle is set as 45° between the oblique upward and the vertical angles. The flight trajectory of the tower-girder layer is set as a vertical ascending route, and the shooting angle is set as the horizontal angle direction. The flight trajectory of the cable layer is set as a broken line fitting route, and the shooting angle is set as the horizontal angle direction;

[0062] In the BIM model of the bridge to be detected, connect the corresponding UAV flight trajectories of each layer to obtain the second acquisition route, command the UAV to fly according to the second acquisition route, and collect full-structure image data through the camera carried on the UAV.

[0063] In this embodiment, the vibration data acquisition unit is used to select vibration sampling points and obtain the cable vibration data of the bridge to be detected through vibration sensors. The processing logic includes:

[0064] Retrieve the BIM model of the bridge to be detected, classify the cables based on the cable material data in the BIM model, and mark the key vibration part points in the BIM model. The key vibration part points include the joint between the main cable and the saddle, the anchorage end of the suspension cable, and the cable-stayed cable and bridge tower anchorage area;

[0065] Uniformly select key vibration position points based on a preset interval distance to obtain vibration sampling points, and collect cable vibration data by collecting data from the vibration sampling points through vibration sensors.

[0066] Among them, during the data collection process, the existing BIM model is directly used to select the collection parts and plan the collection route, which improves the data collection efficiency. At the same time, compared with laser scanning, the image collection rate is high, and image collection of the entire structure is beneficial to fully obtain the data of the bridge to be detected. There are often a large number of suspension cables and stay cables, and selecting representative vibration sampling points improves the collection efficiency and accuracy of vibration data.

[0067] In this embodiment, the preprocessing unit is used to process the point cloud data of the first target part set to obtain a point cloud input vector, process the full-structure image data to obtain an image input vector, and process the cable vibration data to obtain a vibration frequency input data. The processing logic includes:

[0068] Slice the point cloud data of the first target part set based on a preset sampling step to obtain the actual section point cloud, calculate the polar angle of each actual section point cloud respectively and sort them to obtain a point cloud stack sequence, perform stack processing through Chan's algorithm and the point cloud stack sequence to obtain a point cloud convex hull vector set, perform vector cross product calculation on the point cloud convex hull vector set to obtain the point cloud vector outer product, screen the point cloud vector outer product through a preset point cloud threshold to obtain section shape feature points, connect based on the section shape feature points to obtain the actual section shape curve, extract the corresponding reference section shape curve from the BIM model based on the central space coordinates of each actual section shape curve, calculate the Euclidean distance standard deviation and curvature difference mean between the actual section shape curve and the reference section shape curve with the same central space coordinates, and form a point cloud input vector based on the index of the region to which each actual section shape curve belongs, the central space coordinates, the Euclidean distance standard deviation and curvature difference mean between it and the corresponding reference section shape curve;

[0069] Perform Gaussian filtering denoising processing on the full-structure image data, and perform gray mapping on the denoised full-structure image data to obtain an image input vector;

[0070] Supplement the cable vibration data through bilinear interpolation, perform denoising processing through low-pass filtering, sample based on a preset sampling frequency to obtain discrete-time vibration data, and calculate the vibration frequency input data through fast Fourier transform on the discrete-time vibration data.

[0071] Among them, the index of the region to which the actual cross-section shape curve belongs is the position number corresponding to the preset defect part reference set of the actual cross-section shape curve. The preset defect part reference set includes bridge towers, main cables, stiffening girders, anchorages, and saddles. In this embodiment, the position code of the bridge tower is 01, the position code of the main cable is 02, the position code of the stiffening girder is 03, the position code of the anchorage is 04, and the position code of the saddle is 05. Taking the difference between the cross-section shape curve and the reference cross-section shape curve in the BIM model as the point cloud input vector simplifies the processing of complex point cloud data, which is beneficial to improving the calculation speed and fully reflecting the deformation of the bridge.

[0072] In this embodiment, the feature extraction unit is used to extract features from the point cloud input vector through a three-branch convolutional neural network to obtain the point cloud output features, extract features from the image input vector to obtain the image output features, and extract features from the vibration frequency input data to obtain the vibration frequency output features. The processing logic includes:

[0073] Extract features from the point cloud input vector through the Capsule-Network network branch. The calculation logic is as follows: Continuously perform convolutional calculations on the point cloud input vector through a series of convolutional layers to obtain the input data of the primary capsule layer. Perform local feature encoding on the input data of the primary capsule layer through the primary capsule layer to obtain a multi-dimensional digital feature vector. The dynamic routing algorithm establishes a connection between the primary capsule layer and the secondary capsule layer. Through the secondary capsule layer, perform matching calculations on the multi-dimensional digital feature vectors corresponding to each primary capsule in the primary capsule layer to obtain connection weights. Based on the connection weights corresponding to each primary capsule, perform integration calculations to obtain the point cloud output features of the K layer.

[0074] In this embodiment, extract features from the image input vector through the GoogLeNet network branch. The calculation logic is as follows: Perform convolutional calculations on the input vector through the multi-size convolutional kernels of the parallel sub-branches in the Inception module to obtain the local image features. Extract the image spatial features of the image input vector through the max-pooling branch of the parallel sub-branches in the Inception module. Concatenate the local image features and the image spatial features in the channel dimension to obtain the output of the Inception module. Through a series of Inception modules and average pooling processing, obtain the image output features of the K layer.

[0075] In this embodiment, the HHT-1D convolutional neural network branch is used to extract features from the vibration frequency input data. The calculation logic is as follows: The empirical mode decomposition of the vibration frequency input data is performed by the Hilbert-Huang transform algorithm to obtain an ensemble of intrinsic mode functions. The Hilbert transform is respectively performed on each function in the ensemble of intrinsic mode functions to obtain the Hilbert spectrum. A time-series spectral feature vector is established based on the Hilbert spectrum and the time sequence. The 1D convolutional neural network performs weighted summation calculation on the time-series spectral feature vector, and through ReLU function activation processing and max pooling processing, the vibration frequency output features of K layers are obtained;

[0076] Among them, the Capsule-Network network branch, the GoogLeNet network branch, and the HHT-1D convolutional neural network branch are set to the same number of feature extraction layers K.

[0077] Among them, the same number of feature extraction layers specifically refers to: the number of capsule layers in the Capsule-Network network branch is K, the number of Inception modules in the GoogLeNet network branch is K, and the number of convolutional layers in the HHT-1D convolutional neural network branch is K. Setting the same number of feature extraction layers is beneficial to subsequent feature fusion. At the same time, using a parallel three-branch convolutional neural network to calculate three different sources of data respectively is beneficial to fully obtaining abstract feature data.

[0078] In this embodiment, the feature fusion unit is used to perform feature fusion processing on the point cloud output features, image output features, and vibration frequency output features through an improved Transform network to obtain deep fusion features. The processing logic includes:

[0079] An adaptive spatial attention mechanism is introduced. The position information of the point cloud output features, image output features, and vibration frequency output features of each extraction layer is embedded through the sine-cosine encoding algorithm to obtain the features to be fused, including the point cloud features to be fused, image features to be fused, and vibration frequency features to be fused;

[0080] The query data, key data, and value matrix data corresponding to the point cloud features to be fused, image features to be fused, and vibration frequency features to be fused are respectively generated through linear transformation by the spatial attention weight matrix, and the spatial attention score and spatial attention weight are calculated based on the query data, key data, and value matrix data;

[0081] The spatial attention weight is adaptively improved. A weight scaling factor is introduced, and the spatial attention weight is multiplied by the weight scaling factor to obtain the adaptive attention weight. The preliminary fusion features are calculated based on the adaptive attention weight;

[0082] The cross - fusion of the features to be fused in each feature extraction layer is performed through the Transformer module. The multi - head weighted features are obtained by linearly mapping and vector splicing the preliminarily fused features through the multi - head attention mechanism. The dimensionality expansion and feature transformation of the multi - head weighted features are carried out through the fully - connected layer and ReLU activation in the feed - forward network. The deep - layer fused features are obtained by the progressive calculation of the Transformer network layer by layer;

[0083] The calculation expression of the preliminarily fused features includes:

[0084]

[0085] Among them, k represents the serial number of the feature extraction layer, i represents the serial number of the branch, represents the weight scaling factor, e represents the natural logarithm, α represents the hyperparameter, represents the data volume corresponding to the data of the i - th branch in the k - th feature extraction layer, N K represents the total data volume of all branch data in the k - th feature extraction layer, represents the adaptive attention weight, represents the spatial attention weight, represents the preliminarily fused features corresponding to the k - th feature extraction layer, represents the features to be fused corresponding to the data of the i - th branch in the k - th feature extraction layer.

[0086] Among them, through the spatial attention mechanism and the Transformer network, the deep fusion of different types of data is carried out. The introduction of the weight scaling factor realizes the adaptive distribution of the influence ratios of the three types of data when calculating the preliminarily fused features, which is beneficial to improving the accuracy of the preliminarily fused features.

[0087] In this embodiment, the decision - making classification unit is used to perform a classification task according to the deep - layer fused features to obtain the defect detection result. The processing logic includes:

[0088] Input the deep - layer fused features into the fully - connected layer of the classifier. Through the weight matrix and bias vector of the fully - connected layer of the classifier, the deep - layer fused features are feature - mapped to the category vector space. The category probability is calculated through the Softmax function, and the defect detection result and the corresponding spatial coordinates are output based on the category probability;

[0089] The defect detection results include: no defect, tower inclination, main cable deformation, stiffening girder bending, anchor displacement, saddle inclination, cable vibration anomaly, component surface crack, and component surface corrosion.

[0090] Those skilled in the art should understand that the embodiments of the present invention may provide a method, a system or a computer program product. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disc. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or a plurality of processes and / or Figure 1 blocks specified in one block or a plurality of blocks.

[0091] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A bridge defect detection system based on BIM model, characterized in that: include: Detection data acquisition module, detection data processing module, comprehensive judgment module and terminal interaction module; The detection data acquisition module includes a point cloud data acquisition unit, an image data acquisition unit and a vibration data acquisition unit, which are used to plan the acquisition position and acquisition method according to the BIM model, perform laser scanning on the bridge to be detected to obtain point cloud data of the first target position set, photograph the bridge to be detected to obtain full-structure image data, select vibration sampling points and obtain cable vibration data of the bridge to be detected through a vibration sensor; The detection data processing module includes a preprocessing unit and a feature extraction unit, which are used to preprocess the point cloud data, the full-structure image data and the cable vibration data of the first target part set to obtain a point cloud input vector, an image input vector and a vibration frequency input data, and extract features through a three-branch convolutional neural network to obtain point cloud output features, image output features and vibration frequency output features; The comprehensive judgment module includes a feature fusion unit and a decision classification unit, which is used to perform feature fusion processing on the point cloud output features, image output features, and vibration frequency output features through an improved Transform network to obtain deep fusion features, and perform classification tasks based on the deep fusion features to obtain defect detection results; The terminal interaction module includes a display interaction unit and a log storage unit, which are used to provide a display interface to the user and receive user operation instructions, receive user adjustments and settings for the system parameters of the detection data acquisition module, the detection data processing module, and the comprehensive judgment module, generate a detection log based on the defect detection results output by the comprehensive judgment module, and store the output data of the detection data acquisition module, the detection data processing module, and the comprehensive judgment module.

2. A bridge defect detection system based on a BIM model as claimed in claim 1, characterized in that: The point cloud data acquisition unit is used to perform laser scanning on the bridge to be inspected to obtain point cloud data collected by the first target part, and the processing logic includes: Retrieve the BIM model of the bridge to be inspected, establish a geodetic coordinate system based on the BIM model, traverse the BIM model of the bridge to be inspected based on a preset defective part reference set to obtain the target position to be laser scanned, and record the spatial coordinate data corresponding to all the target positions to be laser scanned to obtain a first target part set; The preset defective part reference set includes bridge towers, main cables, stiffening beams, anchors and cable saddles; Acquire component information data corresponding to each part of the first target part set through the BIM model, wherein the component information data includes the size, geometric shape, spatial coordinates and spatial orientation of each part of the first target part set; Based on the preset scanning boundary expansion coefficient and component information data, the BIM model is divided and annotated to obtain an extended laser scanning boundary, based on the extended laser scanning boundary, a UAV laser scanning route is planned to obtain a first acquisition route, the UAV is commanded to fly according to the first acquisition route, and data is collected by a 3D laser radar carried by the UAV to obtain point cloud data of the first target part set; The preset scanning boundary expansion coefficients include a bridge tower area expansion coefficient, a main cable area expansion coefficient, a stiffening beam area expansion coefficient, an anchor area expansion coefficient and a saddle area expansion coefficient.

3. A bridge defect detection system based on a BIM model as claimed in claim 1, characterized in that: The image data acquisition unit is used to photograph the bridge to be inspected to obtain full-structure image data. The processing logic includes: Retrieve the BIM model of the bridge to be inspected, and layer the model based on the spatial height of the BIM model of the bridge to be inspected to obtain the bridge deck layer, bridge bottom layer, tower beam layer and cable layer. Set the corresponding drone flight trajectory and acquisition angle for each layer, set the flight trajectory of the bridge deck layer to a parallel strip route, and the shooting angle to be 45° downward and vertical, set the flight trajectory of the bridge bottom layer to a parallel strip route, and the shooting angle to be 45° upward and vertical, set the flight trajectory of the tower beam layer to a vertical ascending route, and the shooting angle to a horizontal angle, and set the flight trajectory of the cable layer to a broken line fitting route, and the shooting angle to a horizontal angle; In the BIM model of the bridge to be inspected, the flight trajectories of the drones corresponding to each layer are connected in series to obtain the second acquisition route, and the drone is commanded to fly according to the second acquisition route. The camera on the drone is used to collect data to obtain full-structure image data.

4. The bridge defect detection system based on the BIM model according to claim 1, characterized in that: The vibration data acquisition unit is used to select vibration sampling points and obtain the cable vibration data of the bridge to be tested through the vibration sensor. The processing logic includes: Retrieve the BIM model of the bridge to be inspected, classify the cables based on the cable material data in the BIM model, and mark the key vibration points in the BIM model, including the junction between the main cable and the saddle, the anchorage end of the cable, and the anchorage area between the cable and the bridge tower; Based on the preset interval distance, key vibration points are evenly selected to obtain vibration sampling points, and the vibration data of the cable are obtained by collecting data from the vibration sampling points through vibration sensors.

5. The bridge defect detection system based on the BIM model as claimed in claim 1, characterized in that: The preprocessing unit is used to process the point cloud data of the first target part set to obtain a point cloud input vector, process the full structure image data to obtain an image input vector, and process the cable vibration data to obtain vibration frequency input data. The processing logic includes: Based on the preset sampling step length, the point cloud data of the first target part set is sliced ​​to obtain the actual section point cloud, the polar angle of each actual section point cloud is calculated and sorted to obtain the point cloud stacking sequence, the point cloud convex hull vector set is pushed into the stack by Chan's algorithm and the point cloud stacking sequence, the point cloud convex hull vector set is vector cross multiplication is calculated to obtain the point cloud vector outer product, the point cloud vector outer product is screened by the preset point cloud threshold to obtain the section shape feature points, the section shape feature points are connected to obtain the actual section shape curve, the BIM model pair is extracted based on the central space coordinates of each actual section shape curve to obtain the corresponding reference section shape curve, the Euclidean distance standard deviation and the curvature difference mean between the actual section shape curve and the reference section shape curve of the same central space coordinate are calculated, and the point cloud input vector is formed based on the index of the area to which each actual section shape curve belongs, the central space coordinate, and the Euclidean distance standard deviation and the curvature difference mean between the corresponding reference section shape curve; Performing Gaussian filtering and denoising on the full-structure image data, and performing grayscale mapping on the denoised full-structure image data to obtain an image input vector; The cable vibration data is supplemented by the bilinear interpolation method, denoised by low-pass filtering, sampled based on the preset sampling frequency to obtain discrete-time vibration data, and calculated by fast Fourier transform to obtain the vibration frequency input data.

6. The bridge defect detection system based on the BIM model according to claim 1, characterized in that: The feature extraction unit is used to extract features from the point cloud input vector to obtain point cloud output features, extract features from the image input vector to obtain image output features, and extract features from the frequency input data to obtain frequency output features through three-branch convolutional neural networks. The processing logic includes: The feature extraction of point cloud input vector is performed through the Capsule-Network network branch. The calculation logic is as follows: the point cloud input vector is continuously convolved through the series convolutional layers to obtain the primary capsule layer input data, and the primary capsule layer performs local feature encoding on the primary capsule layer input data to obtain a multi-dimensional digital feature vector. The dynamic routing algorithm establishes a connection between the primary capsule layer and the advanced capsule layer, and the multi-dimensional digital feature vectors corresponding to each primary capsule in the primary capsule layer are matched and calculated through the advanced capsule layer to obtain the connection weight. The connection weights corresponding to each primary capsule are integrated and calculated to obtain the K-layer point cloud output features.

7. A bridge defect detection system based on a BIM model as claimed in claim 6, characterized in that: The feature extraction of the image input vector is performed through the GoogLeNet network branch. The calculation logic is as follows: the input vector is convolved by the multi-size convolution kernels of the parallel sub-branch in the Inception module to obtain the local features of the image, and the image spatial features of the image input vector are extracted by the maximum pooling branch of the parallel sub-branch in the Inception module. The local features and image spatial features of the image are spliced ​​in the channel dimension to obtain the output of the Inception module. The image output features of the K layer are obtained by the serial connection of the Inception modules and the average pooling process.

8. A bridge defect detection system based on a BIM model as claimed in claim 6, characterized in that: The frequency input data is feature extracted through the HHT-one-dimensional convolutional neural network branch. The calculation logic is as follows: the frequency input data is empirically decomposed through the Hilbert-Huang transform algorithm to obtain the intrinsic mode function set, each function in the intrinsic mode function set is subjected to Hilbert transform processing to obtain the Hilbert spectrum, and the time series spectrum feature vector is established based on the Hilbert spectrum and the time sequence. The time series spectrum feature vector is weighted and summed through the one-dimensional convolutional neural network, and the frequency output features of the K layer are obtained through ReLU function activation processing and maximum pooling processing; Among them, the Capsule-Network network branch, the GoogLeNet network branch and the HHT-one-dimensional convolutional neural network branch are set to the same feature extraction layer number K.

9. The bridge defect detection system based on the BIM model according to claim 1, characterized in that: The feature fusion unit is used to perform feature fusion processing on the point cloud output features, image output features, and frequency output features through the improved Transform network to obtain deep fusion features. The processing logic includes: An adaptive spatial attention mechanism is introduced, and the position information of the point cloud output features, image output features and frequency output features of each extraction layer are respectively embedded through the sine-cosine coding algorithm to obtain the features to be fused. The features to be fused include point cloud features to be fused, image features to be fused and frequency features to be fused. Through the linear transformation of the spatial attention weight matrix, the query data, key data and value matrix data corresponding to the point cloud features to be fused, the image features to be fused and the vibration frequency features to be fused are respectively generated, and the spatial attention score and the spatial attention weight are calculated based on the query data, key data and value matrix data; The spatial attention weight is adaptively improved by introducing a weight scaling factor. The adaptive attention weight is obtained by multiplying the spatial attention weight by the weight scaling factor. The preliminary fusion feature is calculated based on the adaptive attention weight. The features to be fused in each feature extraction layer are cross-fused through the Transformer module. The preliminary fused features are linearly mapped and vectorized through the multi-head attention mechanism to obtain multi-head weighted features. The multi-head weighted features are dimensionally expanded and transformed through the fully connected layer and ReLU activation in the feedforward network. The Transformer network calculates layer by layer to obtain deep fusion features. The calculation expressions of the preliminary fusion features include: Among them, k represents the sequence number of feature extraction layers, i represents the branch sequence number, represents the weight scaling factor, e represents the natural logarithm, α represents the hyperparameter, represents the amount of data corresponding to the i-th branch data in the k-th feature extraction layer, N K represents the total amount of data of all branch data in the kth feature extraction layer, represents the adaptive attention weight, represents the spatial attention weight, represents the preliminary fusion features corresponding to the kth feature extraction layer, Indicates the features to be fused corresponding to the i-th branch data in the k-th feature extraction layer.

10. The bridge defect detection system based on the BIM model according to claim 1, characterized in that: The decision classification unit is used to perform classification tasks based on deep fusion features to obtain defect detection results. The processing logic includes: The deep fusion features are input into the fully connected layer of the classifier, and the deep fusion features are mapped to the category vector space through the weight matrix and bias vector of the fully connected layer of the classifier. The category probability is calculated through the Softmax function, and the defect detection results and corresponding space coordinates are output based on the category probability; The defect detection results include: no defects, bridge tower tilt, main cable deformation, stiffening beam bending, anchor displacement, saddle tilt, abnormal cable vibration, component surface cracks and component surface rust.

Images