A bridge pier verticality intelligent detection method and system based on deep learning

Abstract

The application discloses a kind of bridge pier verticality intelligent detection method and system based on deep learning, it is related to image processing technical field, obtain the initial point cloud data obtained after scanning to the object to be detected, and the object to be detected includes bridge pier;Target point cloud data belonging to the object to be detected in initial point cloud data is identified by pre-trained point cloud segmentation model;Edge error points in target point cloud data are identified and rejected, and processing point cloud data is obtained;According to model fitting result, the center axis of the object to be detected is determined according to processing point cloud data to the object to be detected;According to the verticality index of the object to be detected is generated according to center axis;Wherein, point cloud segmentation model includes NAM, PTFM and CM;NAM is used to represent multi-scale context and local details, PTFM is used to capture long-range dependency information based on local details, and CM is used to connect NAM and PTFM.It improves detection accuracy and efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly to a bridge pier verticality intelligent detection method and system based on deep learning. BACKGROUND

[0002] In production and construction scenarios, it is necessary to ensure the verticality of manufacturing objects, such as in wall and bridge pier column construction scenarios, the verticality of walls and bridge pier columns needs to be ensured. For this purpose, a level, a total station or a manual plumb line can be used for verticality detection.

[0003] However, the level, total station or manual plumb line method can only obtain limited point data, and cannot cover the entire object. Moreover, the measurement result depends on manual operation, and there is a large subjective error and poor repeatability. With the increase of the height and structural complexity of the manufacturing object, manual measurement is not only time-consuming and laborious, but also difficult to find local deviations in time, and it is difficult to meet the quality detection requirements of high precision and high efficiency.

[0004] In summary, how to improve the precision and efficiency of verticality detection is a problem that needs to be solved by the technical personnel in the field. SUMMARY

[0005] The purpose of the present application is to provide a bridge pier verticality intelligent detection method based on deep learning, which can solve the technical problem of how to improve the precision and efficiency of verticality detection to some extent. The present application also provides a bridge pier verticality intelligent detection system based on deep learning.

[0006] In order to achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0007] A bridge pier verticality intelligent detection method based on deep learning, comprising:

[0008] Obtaining initial point cloud data obtained after scanning a to-be-detected object, the to-be-detected object comprising a bridge pier;

[0009] Identifying target point cloud data belonging to the to-be-detected object in the initial point cloud data through a pre-trained point cloud segmentation model;

[0010] Identifying and removing edge error points in the target point cloud data to obtain processed point cloud data;

[0011] Model fitting is performed on the to-be-detected object according to the processed point cloud data, and a center axis of the to-be-detected object is determined according to the model fitting result;

[0012] Generating a verticality index of the to-be-detected object according to the center axis;

[0013] The point cloud segmentation model comprises a neighborhood aggregation module, a key point conversion module and a connection module; the neighborhood aggregation module is used for representing multi-scale context and local details, the key point conversion module is used for capturing remote dependency information based on local details, and the connection module is used for connecting the neighborhood aggregation module and the key point conversion module.

[0014] Preferably, the initial point cloud data obtained after scanning the object to be detected is acquired, comprising:

[0015] The original point cloud data obtained after scanning the object to be detected by a three-dimensional laser scanner is acquired.

[0016] The original point cloud data is subjected to coordinate normalization in the vertical direction of the main shaft to obtain normalized point cloud data.

[0017] The normalized point cloud data is subjected to denoising to obtain denoised point cloud data.

[0018] The point cloud resolution of the denoised point cloud data is uniformly adjusted to a target interval to obtain initial point cloud data.

[0019] Preferably, the edge error points in the target point cloud data are identified and removed to obtain processed point cloud data, comprising:

[0020] The target point cloud data is unfolded into a two-dimensional image along the main shaft direction to obtain two-dimensional projection point cloud data.

[0021] The two-dimensional projection point cloud data is constructed into a two-dimensional grid image.

[0022] The boundary region of the two-dimensional grid image is identified.

[0023] In the two-dimensional projection point cloud data, points with a distance value less than a set distance value from the boundary region are screened as edge error points.

[0024] The edge error points are removed from the two-dimensional projection point cloud data to obtain processed point cloud data.

[0025] Preferably, the object to be detected is subjected to model fitting according to the processed point cloud data, and the central axis of the object to be detected is determined according to the model fitting result, comprising:

[0026] The cross section of the object to be detected is identified.

[0027] In response to the cross section of the object to be detected being cylindrical, a cylindrical fitting algorithm is used to calculate the fitting column model parameters of the processed point cloud data.

[0028] The processed point cloud data is sampled to obtain sampling points.

[0029] generating a fitting distance of the sampling point to a fitting cylindrical surface according to the fitting cylindrical model parameters;

[0030] constructing a cylindrical model parameter objective function based on the least square fitting and the fitting distance;

[0031] generating current cylindrical model parameters based on the cylindrical model parameter objective function;

[0032] determining whether to end iteration;

[0033] if iteration is continued, returning to execute the step of sampling the point cloud data to obtain sampling points;

[0034] if iteration is ended, obtaining a center axis based on the current cylindrical model parameters.

[0035] Preferably, the architecture of the point cloud segmentation model is composed of an encoder and a decoder; the encoder is composed of a down-sampling layer and a network basic block, and the network basic block is composed of the neighborhood aggregation module, the key point conversion module and the connection module; the decoder is composed of an up-sampling layer and a multi-layer perceptron.

[0036] Preferably, the data processing process of the neighborhood aggregation module comprises:

[0037] ;

[0038] ;

[0039] ;

[0040] wherein, represents the output of the neighborhood aggregation module; represents the number of NA heads; represents the total number of NA heads; represents adjacent aggregated features; represents adjacent features; represents point-by-point generation; represents a learnable reduction function; represents the spatial coordinates corresponding to the local neighborhood centered at the i-th point; represents the semantic feature set corresponding to the local neighborhood centered at the i-th point; represents a connection operation; k represents a distinguishing symbol; represents a multi-layer perceptron; represents a normalized exponential function.

[0041] Preferably, the data processing process of the connection module comprises:

[0042] ;

[0043] wherein, denotes the connection result of the connection module; denotes the learnable parameter; denotes the sigmoid function; denotes the output result of the neighborhood aggregation module; denotes the output result of the key point conversion module.

[0044] A bridge pier verticality intelligent detection system based on deep learning, comprising:

[0045] A first acquisition module is configured to acquire initial point cloud data obtained after scanning a to-be-detected object, wherein the to-be-detected object comprises a bridge pier.

[0046] A first identification module is configured to identify target point cloud data belonging to the to-be-detected object in the initial point cloud data by using a pre-trained point cloud segmentation model.

[0047] A first processing module is configured to identify and remove edge error points in the target point cloud data to obtain processed point cloud data.

[0048] A first fitting module is configured to perform model fitting on the to-be-detected object according to the processed point cloud data, and determine a central axis of the to-be-detected object according to the model fitting result.

[0049] A first generation module is configured to generate a verticality index of the to-be-detected object according to the central axis.

[0050] Preferably, the point cloud segmentation model comprises a neighborhood aggregation module, a key point conversion module and a connection module; the neighborhood aggregation module is configured to represent multi-scale context and local details, the key point conversion module is configured to capture remote dependency information based on local details, and the connection module is configured to connect the neighborhood aggregation module and the key point conversion module.

[0051] Preferably, the first acquisition module comprises:

[0052] A first acquisition unit is configured to acquire original point cloud data obtained after a three-dimensional laser scanner scans a to-be-detected object.

[0053] A first processing unit is configured to perform coordinate normalization on the original point cloud data by taking a main shaft as a vertical direction to obtain normalized point cloud data.

[0054] A first denoising unit is configured to perform denoising on the normalized point cloud data to obtain denoised point cloud data.

[0055] A first adjustment unit is configured to uniformly adjust the point cloud resolution of the denoised point cloud data to a target interval to obtain initial point cloud data.

[0056] Preferably, the first processing module comprises:

[0057] a first projection unit, configured to spread the target point cloud data along the main axis direction into a two-dimensional image to obtain two-dimensional projection point cloud data;

[0058] a first construction unit, configured to construct the two-dimensional projection point cloud data into a two-dimensional grid image;

[0059] a first identification unit, configured to identify a boundary region of the two-dimensional grid image;

[0060] a first screening unit, configured to screen, in the two-dimensional projection point cloud data, a point with a distance value less than a set distance value from the boundary region as an edge error point;

[0061] a first rejection unit, configured to reject the edge error point in the two-dimensional projection point cloud data to obtain processed point cloud data.

[0062] The application provides a bridge pier verticality intelligent detection method based on deep learning, obtains initial point cloud data obtained after scanning a to-be-detected object, and the to-be-detected object comprises a bridge pier; a pre-trained point cloud segmentation model is used to identify target point cloud data belonging to the to-be-detected object in the initial point cloud data; edge error points in the target point cloud data are identified and rejected to obtain processed point cloud data; a model is fitted to the to-be-detected object according to the processed point cloud data, and a central axis of the to-be-detected object is determined according to a model fitting result; a verticality index of the to-be-detected object is generated according to the central axis; wherein the point cloud segmentation model comprises a neighborhood aggregation module, a key point conversion module and a connection module; the neighborhood aggregation module is used to represent multi-scale context and local details, the key point conversion module is used to capture remote dependency information based on local details, and the connection module is used to connect the neighborhood aggregation module and the key point conversion module. In the application, since the neighborhood aggregation module is used to represent multi-scale context and local details, the key point conversion module is used to capture remote dependency information based on local details, and the connection module is used to connect the neighborhood aggregation module and the key point conversion module, the point cloud segmentation model can realize high-precision automatic segmentation of to-be-detected objects of different sizes under complex background noise, provide reliable target point cloud for verticality detection, and enhance the applicability of verticality detection; the edge error points in the target point cloud data are identified and rejected, the influence of the error points can be excluded, only the stable point cloud data in the core region is retained for verticality detection, and the detection precision is improved; the whole process is automatically performed without human intervention, and the detection efficiency is high. A bridge pier verticality intelligent detection system based on deep learning provided by the application also solves the corresponding technical problems. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0064] Figure 1 A flowchart of a bridge pier verticality intelligent detection method based on deep learning provided by the embodiments of the present application.

[0065] Figure 2 A structural schematic diagram of a point cloud segmentation model.

[0066] Figure 3 A working schematic diagram of a key point transformer.

[0067] Figure 4 A circular bridge pier column point cloud diagram.

[0068] Figure 5 A rectangular bridge pier column point cloud diagram.

[0069] Figure 6 A structural schematic diagram of a bridge pier verticality intelligent detection system based on deep learning provided by the embodiments of the present application.

[0070] Figure 7 A structural schematic diagram of an electronic device provided by the embodiments of the present application.

[0071] Figure 8 Another structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0073] Please refer to Figure 1 , Figure 1 A flowchart of a bridge pier verticality intelligent detection method based on deep learning provided by the embodiments of the present application.

[0074] The bridge pier verticality intelligent detection method based on deep learning provided by the embodiments of the present application can include the following steps:

[0075] Step S101: obtaining initial point cloud data obtained after scanning a to-be-detected object.

[0076] In actual applications, the verticality of the object to be detected can be detected through image processing, that is, initial point cloud data obtained after scanning the object to be detected is acquired, and the type of the object to be detected can be determined according to the application scenario, such as a wall, a pier, a table and chairs.

[0077] In an example embodiment, the initial point cloud data can be acquired by means of a three-dimensional laser scanner, that is, in the process of acquiring the initial point cloud data obtained after scanning the object to be detected, the original point cloud data obtained after the object to be detected is scanned by the three-dimensional laser scanner can be acquired; the original point cloud data is normalized in coordinates with the main shaft as the vertical direction, such as the Z-axis as the vertical direction, to obtain normalized point cloud data, which is convenient for subsequent verticality calculation; the normalized point cloud data is denoised, such as using a Statistical Outlier Removal (SOR) algorithm to filter out isolated points and measurement errors, to obtain denoised point cloud data; the point cloud resolution of the denoised point cloud data is uniformly adjusted to a target interval, such as using a Voxel Grid Filter to uniformly adjust the point cloud resolution to an interval of 1 cm, etc., to obtain the initial point cloud data, so as to reduce the amount of calculation while retaining the geometric features of the object to be detected.

[0078] In a specific application scenario, in the process of acquiring the original point cloud data obtained after the object to be detected is scanned by the three-dimensional laser scanner, in order to collect as much point cloud data as possible, the object to be detected can be scanned in all directions by using a high-precision three-dimensional laser scanner to obtain complete surface point cloud data, and multiple stations can be arranged during the scanning process to ensure that there is no dead angle coverage, and each point has a millimeter-level positioning accuracy, so that the original point cloud data fully reflects the shape and inclination of the object to be detected, and provides basic data for subsequent verticality detection.

[0079] Step S102: identifying target point cloud data belonging to the object to be detected in the initial point cloud data by using a pre-trained point cloud segmentation model; the point cloud segmentation model includes a neighborhood aggregation module, a key point conversion module and a connection module; the neighborhood aggregation module is used to represent multi-scale context and local details, the key point conversion module is used to capture remote dependency information based on local details, and the connection module is used to connect the neighborhood aggregation module and the key point conversion module.

[0080] In actual applications, the initial point cloud data obtained after scanning the object to be detected may carry point clouds that affect the perpendicularity detection, such as background noise. In order to avoid the influence of these point clouds on the subsequent perpendicularity detection, a point cloud segmentation model pre-trained can be used to identify the target point cloud data belonging to the object to be detected in the initial point cloud data, that is, the point cloud data that can be used for the perpendicularity detection of the object to be detected is extracted by means of the point cloud segmentation model. The target point cloud data is automatically identified by means of the point cloud segmentation model, avoiding the error of manually selecting the region of the object to be detected, and improving the reliability and efficiency of the processing.

[0081] In an exemplary embodiment, the point cloud segmentation model of the present application can include a neighbor aggregation module (NAM) for local perception, a transformer module (PTFM) for global modeling, and a connector module (CM) for connecting the NAM and the PFTM. The NAM effectively learns complex local features of different scales through improved NA (normalized attention) operations and a multi-head mechanism, and represents multi-scale context and local details. The PTFM uses a set of learnable keypoints to effectively perform global attention and capture long-range dependency information based on local details. The CM serves as a fusion of high and low frequency signals, connecting the NAM and the PTFM, thereby merging the advantages of the two blocks by adaptively assigning mixed weights for local and global environments.

[0082] In a specific application scenario, the architecture of the point cloud segmentation model can be composed of an encoder and a decoder, as shown in Figure 2 The encoder is composed of a down-sampling layer and a network base block, and the network base block is composed of a NAM, a PTFM and a CM. The decoder is composed of an up-sampling layer and a multi-layer perceptron. The features are transmitted through the skip connection between the encoder and the decoder. Finally, three fully connected layers with softmax are used to predict the classification scores of each point, and the segmentation result is determined by the highest score class.

[0083] As shown in Figure 2 The NAM is composed of a multi-head neighbor aggregation (MNA) and a multi-layer perceptron (MLP). The details of the MNA are as shown in Figure 2is shown. In MNA, we adopt the set abstraction layer of PointNet++ network as the basic model of the neighborhood aggregation (NA) operation because it is simple and effective. As described in equation (1), the set abstraction layer of PointNet++ contains two steps: 1) local feature extraction and 2) local feature aggregation.

[0084] (1)

[0085] where F N is the neighboring feature; F NA is the neighboring aggregated feature. and denote the spatial coordinates and semantic feature set of the local neighborhood centered at the ith point, respectively; denotes the concatenation operation.

[0086] We make simple and intuitive enhancements while preserving its main structure to address the following issues. 1) In the local feature extraction step, the positional embedding represents the relative position of the centroid in its neighborhood. Combined with the absolute feature of the neighboring points, this asymmetry can create a semantic gap. 2) In the local feature aggregation step, max-pooling can lead to the loss of local details.

[0087] Therefore, we first introduce a symmetric feature extractor, in which both the positional embedding and the semantic feature are represented based on the relative information of the centroid within the local neighborhood. This symmetric extractor can make up for the semantic gap between the positional embedding and the semantic feature. Our improvement can be expressed as follows:

[0088] (2)

[0089] Subsequently, we enhance the local feature by amplifying more important features. Specifically, we capture the importance of features through a softmax function. Then, we use a learnable reduction function φ to aggregate the local neighborhood information to the centroid as a pooling operation. The reduction function φ is a linear mapping that reduces the neighborhood dimension k to 1. This method not only allows the neural network to learn the most significant local features but also covers the entire local neighborhood, thereby mitigating the loss of local details due to the use of only max-pooling. Our improvement can be expressed as follows:

[0090] (3)

[0091] where denotes point-wise generation, and φ denotes the learnable reduction function.

[0092] Finally, based on the designed NA operation, a multi-head neighborhood aggregation is constructed. Compared with the typical NA operation focusing on the features of a limited scale, the method of the present application contains more NA heads, and each head is expected to focus on local features of different scales. In implementation, the neighborhood search range of each head F Nai is twice that of the previous head F Nai−1 . The improvement of the present application can be summarized as follows

[0093] (4)

[0094] Through model ablation experiments, it is found that embedding the above MNA block in the basic point cloud segmentation network PointNet++ can improve the model segmentation accuracy (mIoU) from 67.6% to 83.1%, an increase of 15.5%.

[0095] In other words, the data processing process of NAM includes:

[0096] ;

[0097] ;

[0098] ;

[0099] wherein, denotes the output of the NAM; denotes the number of NA heads; denotes the total number of NA heads; denotes the adjacent aggregated features; denotes the adjacent features; denotes the point-by-point generation; denotes the learnable reduction function; denotes the spatial coordinates corresponding to the local neighborhood centered at the i-th point; denotes the semantic feature set corresponding to the local neighborhood centered at the i-th point; denotes the connection operation; k denotes the distinguishing symbol; denotes the multilayer perceptron composed of BN (batch normalization layer), linear (linear mapping), GELU (activation function) and linear (linear mapping); denotes the normalized exponential function.

[0100] In specific application scenarios, as shown in Figure 2 , the key point conversion module (PTFM) is composed of a key point transformer (KPT) operation and a multilayer perceptron (MLP), see Figure 2The module is used to capture the remote dependency information based on local details. The working schematic of the module is shown in Figure 3 , Figure 3 The middle dot represents the neighborhood centroid, and the other dots represent the key points. The module works in two stages: the first stage is key point search and neighborhood aggregation, and the second stage is global attention on aggregated key points. The specific working process is as follows: first, the input point cloud shape of the entire scene is depicted by only a few key points, so these key points must be able to fully represent local details. In solving the problem of local detail representation, the neighborhood aggregation (NA) operation can effectively collect information from the local neighborhood of the centroid. Therefore, before searching for key points, we will perform an NA operation once, so that the key points can fully represent local details. Therefore, in general, the role of PTFM can be described as how to use a few key points to represent the entire shape. Through model ablation experiments, it is found that embedding the above PTFM module in the basic point cloud segmentation network PointNet++ can improve the model segmentation accuracy (mIoU) from 67.6% to 79.3%, an increase of 11.7%; embedding the above MNA and PTFM modules in the basic point cloud segmentation network PointNet++ at the same time can improve the model segmentation accuracy (mIoU) from 67.6% to 85.7%, an increase of 18.1%.

[0101] In specific application scenarios, as shown in Figure 2 , the connector module (CM) is composed of a connection strategy and an MLP, which inputs the initial features into the NAM to obtain local features, and then inputs them into the PTFM to obtain global features. By combining the local and global features in the connector module, a comprehensive representation is obtained, which can be described as: . That is, the data processing process of the CM includes: ;

[0102] wherein, represents the connection result of the CM; represents a learnable parameter; represents a sigmoid function; represents the output result of the NAM; represents the output result of the PTFM.

[0103] Through model ablation experiments, it is found that embedding the above MNA, PTFM and CM modules in the basic point cloud segmentation network PointNet++ at the same time can improve the model segmentation accuracy (mIoU) from 67.6% to 87.3%, an increase of 19.7%.

[0104] In this way, the point cloud segmentation model automatically focuses on the geometric pattern and spatial structure of the object point cloud to be detected in the feature learning process, can effectively filter out redundant information such as the ground, support, and obstacles, and only retain the object region to be detected. Compared with manual cropping or rule-based extraction relying on artificial ROI (Return on Investment), the method can adapt to different cross-sections and different heights of objects to be detected, has good robustness to shadows, occlusions, and background clutter in the construction site, and realizes intelligent extraction without manual interaction or heuristic rules, and supports batch recognition of a large number of object point clouds to be detected.

[0105] It should be noted that the training process of the point cloud segmentation model can be flexibly determined according to actual needs, for example, the training point cloud data of the object to be detected can be obtained, the training point cloud data can be preprocessed by coordinate transformation, denoising, and downsampling, and a training data set can be obtained; the preprocessed training data set is manually segmented for semantic labeling, assuming that the object to be detected is a pier column, the semantic labeling result can include 3 categories of pier columns, ground, and other background noise, wherein different segmentation data of different categories can be represented by different labels, as shown in Figure 4 and Figure 5 , Figure 4 and Figure 5 , wherein k1 represents the ground and other backgrounds, k2 represents the pier column, and k3 represents the cap beam; 8:2 is used to divide the training set and the validation set, and data augmentation is performed by rotation, scaling, noise addition, and reordering. In the network training process, one device is used for data training, for example, a device with 24 GB of memory is used to train the point cloud segmentation model, and the following parameters are set in the training process: the network structure is 4 layers of X-Conv convolution modules + up-sampling modules, the optimizer is Adam, the initial learning rate is 0.001, the decay rate is 0.8 (every 5000 iterations), the batch size is 8, the number of label categories is 3 (object to be detected, ground, and other backgrounds), the weight initialization method is uniform distribution, and the loss function is cross-entropy loss function (with class weight). 24000 points are fixed in each batch to input into the network, the batch size is 8, and 100 times per batch. CrossEntropy loss with label smoothing, AdamW optimizer, 0.01 initial learning rate, and 10-4 weight decay (with cosine decay) are used to optimize all models. The number of training rounds is set to 100 rounds, and finally an overall accuracy of more than 92% is achieved on the validation set, and the IoU (Intersection over Union) index reaches more than 87% on the pier column category.

[0106] Step S103: identifying and removing the edge error points in the target point cloud data to obtain the processed point cloud data.

[0107] In actual application, after obtaining the target point cloud data by means of the point cloud segmentation model, the target point cloud data may have errors, and in order to further remove the error points, the edge error points in the target point cloud data can be identified and removed to obtain the processed point cloud data, so as to apply the processed point cloud data for the verticality detection in subsequent application.

[0108] In the exemplary embodiment, in the process of identifying and removing the edge error points in the target point cloud data to obtain the processed point cloud data, the extracted target point cloud data can be detected for the boundary contour by using the image processing method, and the points too close to the boundary are removed, that is, the target point cloud data can be unfolded into a two-dimensional image along the main axis direction, such as being projected to the main plane for the rectangular cross section, being unfolded into a flattened image in the circumferential direction for the circular cross section, and the like, to obtain the two-dimensional projection point cloud data. Assuming that the target point cloud data is , and the main axis direction is the Z axis, the two-dimensional projection point cloud data is ; the two-dimensional projection point cloud data is constructed into a two-dimensional grid image I(x, y). Since in the point cloud image, the area near the edge often has a large deviation or mixed noise points due to the laser incidence angle or the object corner, in the present embodiment, the edge point data within a certain range is excluded before fitting, and only the points of the middle main part of the object to be detected are retained for model fitting, so that the interference of irregular points on the boundary on the fitting result can be avoided, and the fitted central axis is more accurate and reliable, that is, the boundary region B of the two-dimensional grid image is identified, such as by using the Canny edge detection method to identify the boundary region. In the two-dimensional projection point cloud data, the points with a distance value less than a set distance value from the boundary region are selected as the edge error points, and the edge error points are removed in the two-dimensional projection point cloud data to obtain the processed point cloud data , and the set distance value may be a value between 5-10 cm.

[0109] Step S104: model fitting is performed on the object to be detected according to the processed point cloud data, and the central axis of the object to be detected is determined according to the model fitting result.

[0110] Step S105: the verticality index of the object to be detected is generated according to the central axis.

[0111] In actual application, after obtaining the accurate processed point cloud data, the model fitting can be performed on the object to be detected according to the processed point cloud data, and the central axis of the object to be detected is determined according to the model fitting result; finally, the verticality index of the object to be detected is generated according to the central axis.

[0112] In the example embodiment, in the process of fitting a model to the object to be detected according to the point cloud data and determining the central axis of the object to be detected according to the model fitting result, the cross section of the object to be detected can be identified; in response to the cross section of the object to be detected being cylindrical, a cylindrical fitting algorithm is used to calculate the fitting cylinder model parameters of the processed point cloud data, such as using a RANSAC (Random Sample Consensus) cylindrical fitting algorithm for calculation, and taking the cylindrical surface fitting as an example, the minimum sampling number is 3 points of fitting cylinder model parameters (r, c, v), wherein r is the radius of the cylindrical surface; c is any point on the axis; v is the axis direction unit vector; the processed point cloud data is sampled to obtain sampling points; the sampling points are fitted to the fitting cylindrical surface according to the fitting cylinder model parameters to obtain the fitting distance of the sampling points to the fitting cylindrical surface . , , the two vertical lines refer to taking the absolute value, and one vertical line represents a segmentation mark. If (λ refers to the maximum allowable error of allowing a point to approach the cylindrical surface), it is considered that is an inlier of the model. For laser point cloud data with a precision of millimeter level, λ = 5-10 mm; based on the least square fitting and the fitting distance, a cylinder model parameter objective function is constructed ; the current cylinder model parameters are generated based on the cylinder model parameter objective function; it is determined whether to end the iteration; if the iteration continues, the step of sampling the processed point cloud data to obtain the sampling points is returned; if the iteration ends, the central axis is obtained based on the current cylinder model parameters, such as setting the iteration number to 1000 and the distance tolerance to 1 cm by using the RANSAC algorithm, and finally obtaining the space vector of the fitting central axis. In this way, the robustness of the RANSAC model fitting is enhanced by the multi-point sampling and multiple fitting strategy, and the influence of the single random sampling error on the final axis determination is reduced. The combination of boundary avoidance and multi-point fitting greatly improves the accuracy and stability of the vertical axis extraction. In addition, in response to the cross section of the object to be detected being rectangular, a vertical main plane can be fitted based on the processed point cloud data, and the intersection line of the vertical main plane is taken as the central axis.

[0113] In the example embodiment, the perpendicularity index can be a vertical direction deviation angle or a top offset of the object to be detected. Assuming that the central axis direction vector is v and the Z-axis direction unit vector z = (0, 0, 1), the included angle calculation formula is: ; the vertical direction deviation angle (90°-θ) or the top offset is taken as the perpendicularity index, wherein H is the height of the object to be detected.

[0114] It should be noted that after obtaining the perpendicularity index, the perpendicularity index can also be visualized and reported, such as generating a three-dimensional color deviation map, that is, representing the local offset of each point with color, blue representing perpendicularity, and red representing large deviation; such as outputting a PDF format detection report including fields such as object number to be detected, measurement time, deviation angle, offset, and whether qualified; such as uploading to a construction quality acceptance system and displaying in association with a BIM model.

[0115] The application provides a bridge pier perpendicularity intelligent detection method based on deep learning, initial point cloud data obtained after scanning a to-be-detected object is acquired, the to-be-detected object including a bridge pier; a target point cloud data belonging to the to-be-detected object in the initial point cloud data is identified through a pre-trained point cloud segmentation model; edge error points in the target point cloud data are identified and removed to obtain processed point cloud data; model fitting is performed on the to-be-detected object according to the processed point cloud data, and a central axis of the to-be-detected object is determined according to a model fitting result; a perpendicularity index of the to-be-detected object is generated according to the central axis; wherein the point cloud segmentation model includes a neighborhood aggregation module, a key point conversion module and a connection module; the neighborhood aggregation module is used to represent multi-scale context and local details, the key point conversion module is used to capture remote dependency information based on local details, and the connection module is used to connect the neighborhood aggregation module and the key point conversion module. In the application, since the neighborhood aggregation module is used to represent multi-scale context and local details, the key point conversion module is used to capture remote dependency information based on local details, and the connection module is used to connect the neighborhood aggregation module and the key point conversion module, so that the point cloud segmentation model can realize high-precision automatic segmentation of to-be-detected objects of different sizes under complex background noise, provide reliable target point cloud for perpendicularity detection, and enhance the applicability of perpendicularity detection; and the edge error points in the target point cloud data need to be identified and removed, so that the influence of the error points can be excluded, only the stable point cloud data in the core area is retained for perpendicularity detection, and the detection precision is improved; and the whole process is automatically performed without human intervention, so that the detection efficiency is high.

[0116] Please refer to Figure 6 , Figure 6 The structure diagram of a bridge pier perpendicularity intelligent detection system based on deep learning provided by the embodiment of the application is shown.

[0117] The bridge pier perpendicularity intelligent detection system based on deep learning provided by the embodiment of the application can include:

[0118] The first acquisition module 101 is configured to acquire initial point cloud data obtained after scanning a to-be-detected object;

[0119] The first identification module 102 is configured to identify target point cloud data belonging to the object to be detected from the initial point cloud data by using a pre-trained point cloud segmentation model.

[0120] The first processing module 103 is configured to identify and remove edge error points in the target point cloud data, to obtain processed point cloud data.

[0121] The first fitting module 104 is configured to perform model fitting on the object to be detected according to the processed point cloud data, and determine a central axis of the object to be detected according to the model fitting result.

[0122] The first generation module 105 is configured to generate a perpendicularity index of the object to be detected according to the central axis.

[0123] The point cloud segmentation model includes NAM, PTFM and CM; the NAM is used to represent multi-scale context and local details, the PTFM is used to capture long-range dependency information based on local details, and the CM is used to connect the NAM and the PTFM.

[0124] The embodiment of the present application provides a bridge pier perpendicularity intelligent detection system based on deep learning, and the first acquisition module can comprise:

[0125] The first acquisition unit is configured to acquire original point cloud data obtained by scanning the object to be detected by a three-dimensional laser scanner.

[0126] The first processing unit is configured to perform coordinate normalization on the original point cloud data by taking the main shaft as a vertical direction, to obtain normalized point cloud data.

[0127] The first denoising unit is configured to perform denoising on the normalized point cloud data, to obtain denoised point cloud data.

[0128] The first adjustment unit is configured to uniformly adjust the point cloud resolution of the denoised point cloud data to a target interval, to obtain initial point cloud data.

[0129] The embodiment of the present application provides a bridge pier perpendicularity intelligent detection system based on deep learning, and the first processing module can comprise:

[0130] The first projection unit is configured to expand the target point cloud data into a two-dimensional image along the direction of the main shaft, to obtain two-dimensional projection point cloud data.

[0131] The first construction unit is configured to construct the two-dimensional projection point cloud data into a two-dimensional grid image.

[0132] The first identification unit is configured to identify a boundary region of the two-dimensional grid image.

[0133] The first screening unit is configured to screen, in the two-dimensional projection point cloud data, a point with a distance value less than a set distance value from the boundary region as an edge error point.

[0134] The first elimination unit is configured to eliminate edge error points in the two-dimensional projection point cloud data to obtain processed point cloud data.

[0135] The first fitting module can include:

[0136] The first identification unit is configured to identify the cross section of the object to be detected.

[0137] The first fitting unit is configured to, in response to the cross section of the object to be detected being cylindrical, calculate fitting column model parameters of the processed point cloud data using a cylindrical fitting algorithm, sample the processed point cloud data to obtain sample points, generate fitting distances of the sample points to a fitting cylindrical surface according to the fitting column model parameters, construct a column model parameter objective function based on least squares fitting and the fitting distances, generate current column model parameters based on the column model parameter objective function, and determine whether to end iteration.

[0138] The first fitting module can include:

[0139] The data processing process of the NAM includes:

[0140] ;

[0141] ;

[0142] ;

[0143] wherein, represents the output of the NAM; represents the number of NA heads; represents the total number of NA heads; represents adjacent aggregated features; represents adjacent features; represents point-by-point generation; represents a learnable reduction function; represents the spatial coordinates corresponding to the local neighborhood centered on the ith point; represents the semantic feature set corresponding to the local neighborhood centered on the ith point; denotes a connection operation; k denotes a distinguisher; denotes a multi-layer perceptron; denotes a normalized exponential function.

[0144] The embodiment of the present application provides a kind of bridge pier verticality intelligent detection system based on deep learning, the data processing process of CM includes:

[0145] ;

[0146] Wherein, denotes the connection result of the CM; denotes learnable parameter; denotes sigmoid function; denotes the output result of the NAM; denotes the output result of the PTFM.

[0147] The present application also provides an electronic device and a computer readable storage medium, which both have the corresponding effects of the bridge pier verticality intelligent detection method based on deep learning provided by the embodiment of the present application. Please refer to Figure 7 , Figure 7 The structure diagram of an electronic device provided by the embodiment of the present application.

[0148] The electronic device provided by the embodiment of the present application includes a memory 201 and a processor 202, the memory 201 stores a computer program, and the processor 202 implements the steps of the bridge pier verticality intelligent detection method based on deep learning as described in any of the above embodiments when executing the computer program.

[0149] Please refer to Figure 8The electronic device provided in another embodiment of the present application can further include: an input port 203 connected to the processor 202, configured to transmit an externally input command to the processor 202; a display unit 204 connected to the processor 202, configured to display a processing result of the processor 202 to the outside world; and a communication module 205 connected to the processor 202, configured to realize communication between the electronic device and the outside world. The display unit 204 can be a display panel, a laser scanning display, or the like. The communication module 205 can adopt a communication mode including but not limited to a Mobile High-Definition Link (MHL) technology, a Universal Serial Bus (USB) technology, a High-Definition Multimedia Interface (HDMI) technology, a wireless connection technology such as a WIreless Fidelity (WiFi) technology, a Bluetooth communication technology, a low-power Bluetooth communication technology, and an IEEE 802.11s-based communication technology.

[0150] The computer readable storage medium provided in the embodiment of the present application stores a computer program, and the computer program is executed by a processor to implement the steps of the bridge pier verticality intelligent detection method based on deep learning described in any of the above embodiments.

[0151] The computer readable storage medium involved in the present application includes a Random Access Memory (RAM), a memory, a Read-Only Memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable magnetic disk, a Compact Disc Read-Only Memory (CD-ROM), or any other form of storage medium known in the technical field.

[0152] The related parts of the bridge pier verticality intelligent detection system based on deep learning, the electronic device, and the computer readable storage medium provided in the embodiment of the present application are described in detail in the corresponding part of the bridge pier verticality intelligent detection method based on deep learning provided in the embodiment of the present application, and will not be described here. In addition, the parts of the above technical solutions provided in the embodiment of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail, so as to avoid excessive description.

[0153] It is also to be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" can include a combination of two or more components, and the term "an element" can include comparable reference to a plurality of elements. Additionally, the term "or" as used herein means any one member of a logical disjunction (i.e., it is equivalent to "or" and "or else") and not a logical exclusion. Also, the terms "comprise," "comprising," "include," "including," and the like mean "including but not limited to." Furthermore, the terms "first," "second," "third," etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0154] The above description of disclosed embodiments provides examples, and is not intended to be limiting. Numerous modifications of the embodiments, as defined herein, will be apparent to those skilled in the art, and will be encompassed within the spirit or scope of the application. Accordingly, the scope of the application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A deep learning-based intelligent detection method for bridge pier verticality, characterized in that, include: Acquire initial point cloud data after scanning the object to be detected, the object to be detected including bridge piers; The target point cloud data belonging to the object to be detected in the initial point cloud data is identified by a pre-trained point cloud segmentation model. Edge error points in the target point cloud data are identified and removed to obtain processed point cloud data. The object to be detected is fitted with a model based on the processed point cloud data, and the central axis of the object to be detected is determined based on the model fitting result. The perpendicularity index of the object to be tested is generated based on the central axis. The point cloud segmentation model includes a neighborhood aggregation module, a key point conversion module, and a connection module. The neighborhood aggregation module is used to represent multi-scale context and local details, the key point conversion module is used to capture remote dependency information based on local details, and the connection module is used to connect the neighborhood aggregation module and the key point conversion module. The process involves identifying and removing edge error points from the target point cloud data to obtain processed point cloud data, including: The target point cloud data is unfolded into a two-dimensional image along the main axis to obtain two-dimensional projected point cloud data; The two-dimensional projected point cloud data is used to construct a two-dimensional raster image; Identify the boundary regions of the two-dimensional raster image; In the two-dimensional projected point cloud data, points whose distance value from the boundary region is less than a set distance value are selected as edge error points; The edge error points are removed from the two-dimensional projected point cloud data to obtain processed point cloud data; The process of fitting a model to the object to be detected based on the processed point cloud data, and determining the central axis of the object to be detected based on the model fitting result, includes: Identify the cross-section of the object to be detected; In response to the fact that the cross-section of the object to be detected is cylindrical, the cylindrical fitting algorithm is used to calculate the fitting cylindrical model parameters of the processed point cloud data; The processed point cloud data is sampled to obtain sampling points; Based on the parameters of the fitted cylinder model, the fitting distance from the sampling point to the surface of the fitted cylinder is generated; Based on the least squares fit and the fitted distance, a target function for the column model parameters is constructed. Generate the current column model parameters based on the objective function of the column model parameters; Determine whether to end the iteration; If the iteration continues, return to the step of sampling the processed point cloud data to obtain sample points; If the iteration ends, the center axis is obtained based on the current column model parameters; The data processing procedure of the neighborhood aggregation module includes: ； ； ； in, This represents the output of the neighborhood aggregation module; Indicates the NA header number; Indicates the total number of NA headers; Indicates adjacent aggregation characteristics; Indicates adjacent features; This indicates point-by-point generation; This represents a learnable reduction function; This represents the spatial coordinates of the local neighborhood centered at point i. This represents the semantic feature set corresponding to the local neighborhood centered at point i; Indicates a join operation; 'k' represents the delimiter; This represents a multilayer perceptron; This represents the normalized exponential function; The data processing procedure of the connection module includes: ； in, This indicates the connection result of the connection module; Indicates learnable parameters; Represents the sigmoid function; This represents the output result of the neighborhood aggregation module; This indicates the output result of the key point conversion module.

2. The method according to claim 1, characterized in that, Acquire the initial point cloud data obtained after scanning the object to be detected, including: Obtain the raw point cloud data of the object to be inspected after scanning it with a 3D laser scanner; The original point cloud data is normalized by taking the principal axis as the vertical direction to obtain normalized point cloud data. The normalized point cloud data is denoised to obtain denoised point cloud data. The point cloud resolution of the denoised point cloud data is uniformly adjusted to the target spacing to obtain the initial point cloud data.

3. The method according to any one of claims 1 to 2, characterized in that, The point cloud segmentation model architecture consists of an encoder and a decoder; the encoder consists of a downsampling layer and a network base block, and the network base block consists of the neighborhood aggregation module, the key point conversion module and the connection module; the decoder consists of an upsampling layer and a multilayer perceptron.

4. A deep learning-based intelligent detection system for bridge pier verticality, characterized in that, include: The first acquisition module is used to acquire the initial point cloud data obtained after scanning the object to be detected, the object to be detected including bridge piers; The first identification module is used to identify target point cloud data belonging to the object to be detected in the initial point cloud data through a pre-trained point cloud segmentation model; The first processing module is used to identify and remove edge error points in the target point cloud data to obtain processed point cloud data. The first fitting module is used to perform model fitting on the object to be detected based on the processed point cloud data, and to determine the central axis of the object to be detected based on the model fitting result. The first generation module is used to generate the verticality index of the object to be detected based on the central axis. The point cloud segmentation model includes a neighborhood aggregation module, a key point conversion module, and a connection module. The neighborhood aggregation module is used to represent multi-scale context and local details, the key point conversion module is used to capture remote dependency information based on local details, and the connection module is used to connect the neighborhood aggregation module and the key point conversion module. The first processing module includes: The first projection unit is used to unfold the target point cloud data into a two-dimensional image along the main axis direction to obtain two-dimensional projected point cloud data. The first construction unit is used to construct a two-dimensional raster image from the two-dimensional projected point cloud data; The first recognition unit is used to identify the boundary region of the two-dimensional raster image; The first filtering unit is used to filter points in the two-dimensional projected point cloud data whose distance value from the boundary region is less than a set distance value as edge error points. The first elimination unit is used to eliminate the edge error points in the two-dimensional projected point cloud data to obtain processed point cloud data. The first fitting module includes: The first identification unit is used to identify the cross-section of the object to be detected; The first fitting unit is configured to: respond to the fact that the cross-section of the object to be detected is cylindrical, calculate the fitting cylindrical model parameters of the processed point cloud data using a cylindrical fitting algorithm; sample the processed point cloud data to obtain sampling points; generate the fitting distance from the sampling points to the surface of the fitted cylinder based on the fitting cylindrical model parameters; construct a cylindrical model parameter objective function based on least squares fitting and the fitting distance; generate the current cylindrical model parameters based on the cylindrical model parameter objective function; determine whether to end the iteration; if the iteration continues, return to the step of sampling the processed point cloud data to obtain sampling points; if the iteration ends, obtain the central axis based on the current cylindrical model parameters. The data processing procedure of the neighborhood aggregation module includes: ； ； ； in, This represents the output of the neighborhood aggregation module; Indicates the NA header number; Indicates the total number of NA headers; Indicates adjacent aggregation characteristics; Indicates adjacent features; This indicates point-by-point generation; This represents a learnable reduction function; This represents the spatial coordinates of the local neighborhood centered at point i. This represents the semantic feature set corresponding to the local neighborhood centered at point i; Indicates a join operation; 'k' represents the delimiter; This represents a multilayer perceptron; This represents the normalized exponential function; The data processing procedure of the connection module includes: ； in, This indicates the connection result of the connection module; Indicates learnable parameters; Represents the sigmoid function; This represents the output result of the neighborhood aggregation module; This indicates the output result of the key point conversion module.

5. The system according to claim 4, characterized in that, The first acquisition module includes: The first acquisition unit is used to acquire the original point cloud data obtained by the 3D laser scanner after scanning the object to be inspected; The first processing unit is used to perform coordinate normalization on the original point cloud data with the principal axis as the vertical direction to obtain normalized point cloud data. The first denoising unit is used to denoise the normalized point cloud data to obtain denoised point cloud data. The first adjustment unit is used to uniformly adjust the point cloud resolution of the denoised point cloud data to the target spacing to obtain the initial point cloud data.

Images