A method and system for bridge crack detection and three-dimensional visualization of cracks

Abstract

The application discloses a bridge crack detection and three-dimensional visualization method and system, and belongs to the technical field of target detection. A target detection model is obtained by collecting two-dimensional images of a bridge disease related area by a robot. The target detection model is preliminarily extracted based on a YOLOv2 algorithm. Pixel-level segmentation of the target includes pixel-by-pixel segmentation of the target detection model based on a U-net neural network. Three-dimensional reconstruction includes using a nerf neural network to establish a three-dimensional model using multi-angle two-dimensional images, and adding the target detection model in the pixel-level segmentation stage into the three-dimensional model. The application identifies a target using two-dimensional images and establishes a three-dimensional model. The target object in the two-dimensional image is labeled in the three-dimensional model, three-dimensional visualization of the target model is realized, and a comprehensive crack detection and disease reconstruction system is established. Bridge disease information can be accurately mastered, bridge maintenance experts can be assisted to timely find bridge safety hazards, and accidents can be prevented.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge detection and monitoring, and in particular to a method and system for bridge crack detection and three-dimensional visualization of cracks. Background Art

[0002] With the rapid development of science and technology, more and more countries have begun to attach importance to the construction of infrastructure. Especially since the beginning of the 20th century, with the development of science and technology and the people's demand for daily life quality, the country's investment in infrastructure construction has continued to increase. The technological content and cost of infrastructure have gradually increased, and the categories involved have become more extensive. In particular, investment in transportation construction has become a key development priority for each country. Among them, bridges play an important role in transportation that cannot be ignored. They not only ensure the normal production and travel of people, but also serve as a national defense guarantee. They symbolize a country's scientific and technological level and economic capabilities, and are important infrastructure that the country vigorously develops.

[0003] However, as more and more bridges are built, bridge maintenance becomes an important factor affecting whether the bridges can continue to be used. During the service life of a bridge, which can be as long as 50, 60 or even 100 years, it will be affected by the long-term fatigue effect and mutation effect of the load. At the same time, corrosion from the external environment, aging of the material itself, and accidental factors such as earthquakes, avalanches, and floods will also cause the diseases inside the bridge to gradually accumulate, eventually causing the bridge structure to be damaged, durability to decrease, reliability to reduce, and even catastrophic accidents to occur.

[0004] At present, most bridges are reinforced concrete structures. Since reinforced concrete structures themselves are prone to cracks, and are subjected to long-term squeezing by vehicles and erosion by wind, sand and rain, cracks can easily develop deeper, reducing the bearing capacity of concrete. At the same time, large cracks not only affect the appearance and service life of the building, but also cause steel corrosion, reduced durability of components, poor impermeability and other safety problems, thereby affecting the safety performance of the bridge structure and even causing exposed steel bars, leading to major accidents and endangering the lives and property of the people.

[0005] Traditional bridge inspection methods require inspectors to visually observe bridge defects, which is time-consuming and inefficient. Furthermore, inspection results are easily affected by the inspector's subjective factors. Furthermore, manual inspections require significant manpower, material, and financial resources. Furthermore, some bridge areas are difficult for humans to access, making effective inspection impossible. This leads to discrepancies between inspection results and actual bridge conditions. Furthermore, research on the visualization of defects in bridge inspections is limited, demonstrating the shortcomings of a single research approach and limited research content. In intelligent bridge inspection research, there is still significant room for improvement in apparent defect recognition technology. Furthermore, establishing a professional bridge defect data management and analysis system remains a long way to go.

[0006] In traditional bridge inspection, there are often the following defects:

[0007] First, the information is fragmented, and the collected bridge defect information cannot be effectively combined. The inspection object is often limited to a single structure, making it difficult to link the detected information with the overall bridge structure. Therefore, how to combine the collected image data with the collected individual defect information and the bridge as a whole has gradually become a new research direction in the field of bridge inspection.

[0008] Second, the existing bridge defect detection industry lacks quantitative analysis methods. Bridge defect identification software can only analyze the location and shape of cracks, and it is difficult to quantitatively measure the specific size of the cracks. Moreover, the detection speed is difficult to achieve real-time transmission, and the detection level is seriously affected by the surrounding environmental factors.

[0009] Third, local crack identification is not combined with the overall bridge. The intelligent inspection research institutions on the market generally have the limitation of insufficient integrity. The bridge defects they detect lack integrity, the detected defects are often limited to a single component, the detection results are difficult to use for overall evaluation, and it is impossible to effectively view the distribution of various defects on the outer surface of the bridge. It is difficult to evaluate the overall safety of the bridge with only a single crack identification algorithm. Summary of the Invention

[0010] The purpose of the present invention is to overcome the defects and shortcomings of the existing technology and provide a method and system for bridge crack detection and three-dimensional visualization of cracks, combining cracks with the three-dimensional model of the bridge to achieve crack visualization, and then creating an intelligent system to facilitate users to view detection data.

[0011] To achieve the above object, the present invention provides the following technical solutions:

[0012] A method for bridge crack detection and three-dimensional crack visualization includes the following steps:

[0013] S1, using the robot to collect two-dimensional images of the bridge disease-related areas;

[0014] S2, uses the single-stage target detection YOLOv2 algorithm to roughly extract the crack-related areas to improve the efficiency of crack recognition;

[0015] S3 uses pixel-level semantic segmentation to achieve accurate and complete separation of cracks and background. The semantic segmentation network is based on U-net.

[0016] S4, uses the nerf neural network to reconstruct the bridge 3D using a large number of 2D images of bridge structures;

[0017] S5, adding the crack segmentation image to the three-dimensional bridge model to achieve three-dimensional visualization of the bridge cracks in the bridge model.

[0018] After collecting images of the cracked area of ​​the bridge, the target detection method is first used through YOLOv2 to quickly detect the crack area from the original image, reducing the workload of subsequent pixel-level semantic segmentation. Then, convolutional feature extraction and pixel-level semantic segmentation are performed on the crack area. Finally, in order to connect the disease with the entire bridge and better monitor the deformation of the bridge, the NeRF network is used and a three-dimensional reconstruction part is added. This can reflect the main bridge diseases in the bridge structure.

[0019] The above-mentioned bridge detection method may also have the following additional technical features:

[0020] By changing the encoder part of the U-net network to VGG16 and changing the upsampling part of the decoder accordingly, this crack detection system can significantly improve the detection accuracy even with smaller datasets. The following describes the detection network model:

[0021] The crack detection network, implemented using VGG16, detects cracks, dividing the image into a prediction matrix and outputting the matrix after detection. The network initially passes the image input through convolutional and max pooling layers, halving the image's width and height with each pooling layer. Furthermore, in the overall implementation, neither model incorporates fully connected layers, which significantly reduces the number of parameters.

[0022] The semantic segmentation network is based on the U-net, whose structure is shown in the figure. The first part of the U-net network is feature extraction, similar to VGG, using VGG16 for feature extraction. The second part is the upsampling part, which mainly increases the image resolution. Except for the last upsampling, all other upsampling operations in this model use deconvolution.

[0023] In addition, to connect the damage to the bridge as a whole and better monitor bridge deformation, the NeRF network was used and a 3D reconstruction component was added. The NeRF network is a scene representation method based on the implicit field representation of the illumination model, which is used to generate different perspectives of a scene.

[0024] The network is mainly divided into the following steps:

[0025] (1) Acquisition of bridge outer surface information: Acquire two-dimensional image information of the bridge outer surface through relevant equipment.

[0026] The process of obtaining image information requires that the bridge surface is unobstructed. In order to avoid the impact of light and shadow changes caused by weather and different sun positions over time on data quality, the time for obtaining two-dimensional images should be as concentrated as possible.

[0027] (2) Obtaining the camera position: Using the structure-from-motion algorithm, the change of each camera coordinate system relative to the world coordinate system is obtained from the image in step (1), and the viewing angle (θ, φ) is obtained.

[0028] (3) Training the network: Use the volume rendering network to render the result corresponding to the required perspective. By constructing a loss function with the input image, the back-propagation algorithm is applied to save the bridge information into the neural network radiation field network. The process is to first connect the target pixel with the center of the camera of the corresponding perspective to obtain a ray. At the intersection of this ray and the scene, a point is sampled to obtain the 3D coordinates of the corresponding point. Then, the 3D coordinates of these points, together with the perspective, form 5D coordinates (x, y, z, θ, φ) and input them into the neural network radiation field network to obtain the color and density of the corresponding image. The obtained color and density are integrated along the ray to obtain the color of the target pixel, and then the rendering result of the entire image is obtained. The result is combined with the input image to construct a loss function. Since the rendering process and the neural network radiation field network operation process are both differentiable, the constructed loss function is used for error back-propagation to update the parameters of the neural network radiation field network so that the shape and color information of the bridge are saved in the neural network radiation field network.

[0029] (4) Output results: Since the shape and color information of the bridge are saved in the neural network radiation field network, the final three-dimensional model is rendered by sampling points through the volume rendering network and then querying the information of these points in the volume neural network radiation field network.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] This invention uses a robot to collect two-dimensional images of bridge defect-related areas to obtain a target detection model. Initial target extraction is performed using the YOLOv2 algorithm. Pixel-level segmentation involves pixel-by-pixel segmentation of the target detection model using a U-net neural network. 3D reconstruction uses a NeRF neural network to build a 3D model using multi-angle two-dimensional images. The target detection model from the pixel-level segmentation phase is then incorporated into the 3D model. By combining YOLOv2, U-net, and NeRF network models, a bridge crack detection system for crack identification and 3D reconstruction of defects is established. Through initial positioning using YOLOv2 and pixel-level semantic segmentation using a U-net network, real-time, high-precision crack detection is achieved. A comprehensive crack detection and defect reconstruction system is also established. This system enables personnel to accurately obtain bridge defect information without leaving their homes, assisting bridge maintenance experts in promptly identifying and preventing potential safety hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1is a flow chart of a bridge crack detection method according to a first embodiment of the present invention;

[0033] Figure 2 It is composed of the crack detection network of the YOLOv2 neural network;

[0034] Figure 3 It is a schematic diagram of the network structure of the U-net neural network;

[0035] Figure 4 This is the original image of the crack;

[0036] Figure 5 is the effect diagram of the identified cracks;

[0037] Figure 6 It is a visualization picture of the crack on the 3D model. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] A method and system for bridge crack detection and three-dimensional crack visualization, comprising the following steps:

[0040] S1, collect two-dimensional images of the bridge disease-related areas to obtain the target detection model;

[0041] Specifically, drones are used to collect images of various parts of the bridge, including piers, beams, bridge decks and their connections.

[0042] S2, uses single-stage target detection YOLOv2 to roughly extract crack-related areas, improve crack recognition efficiency, and optimize the target detection model;

[0043] S3 uses pixel-level semantic segmentation to achieve accurate and complete separation of cracks and background, and extracts the target detection model;

[0044] S4, using the nerf neural network to use a large number of two-dimensional images of bridge structures to perform three-dimensional reconstruction of the bridge based on the target detection model to obtain a three-dimensional model;

[0045] S5, adding the crack segmentation image to the three-dimensional bridge model to achieve three-dimensional visualization of the bridge cracks in the three-dimensional bridge model.

[0046] Among them, S2 uses the YOLOv2 network to determine the crack area and preliminarily segment the initial crack information, specifically including:

[0047] S21, the front part of the network is superimposed with multiple convolutional layers and maximum pooling layers. In this process, the convolutional neural network is used to extract image features, which increases the depth of the feature map, and the maximum pooling is used for downsampling to reduce the resolution of the feature map;

[0048] In this embodiment, the collected images are 512x512 resolution, RGB three-channel images. The convolution uses a standard two-dimensional convolution with a kernel size of 3x3, a horizontal and vertical stride of 1, a padding of 1, and an exponential growth of 2. The convolution operation does not reduce the resolution of the feature map, but the number of channels in the feature map will continue to increase. The pooling kernel of the maximum pooling layer is 2x2, with a horizontal and vertical stride of 2. With each maximum pooling layer, the length and width of the feature map are halved, but the number of channels in the feature map does not change.

[0049] S22 introduces nonlinearity, and obtains high-level features from low-level image edge and color features; the latter part is the fully connected layer that obtains the predicted object category and border information from the extracted high-level features.

[0050] In this embodiment, the nonlinear function uses the RELU activation function, which is placed after the convolution kernel operation to operate on each output value after convolution. By adding the RELU activation function, nonlinear operations can be added to linear convolution operations, thereby increasing the network's expression and fitting capabilities.

[0051] S23, add batchNorm to alleviate gradient disappearance

[0052] In this embodiment, BatchNorm is added between the convolutional layer and the RELU activation function. BatchNorm normalizes the input data of each batch to stabilize the input data distribution of each channel, which can make the model more stable, improve the generalization ability of the model, and prevent overfitting.

[0053] S24, better finetuning. Since the network parameters are based on ImageNet, the learning rate of the pre-trained part of the network is reduced in order to achieve faster convergence. After pre-training on ImageNet at a resolution of 224x224 for several rounds, the images are upsampled to a resolution of 448x448 and then transitioned to a resolution of 448x448 during fine-tuning.

[0054] By introducing transition training of sampled data from ImageNet, the model can adapt to the subsequent training of target detection datasets more quickly, speeding up the training process.

[0055] S25 introduces a pre-set anchor size and adjusts it to achieve better results. The anchor size is not manually set, but refers to the size of a pre-set box. The model predicts the adjustment parameters for this box, such as size change and position change. After applying the adjustment parameters, the final box is obtained by clustering the box sizes in the markup using the k-means clustering algorithm.

[0056] In this embodiment, the length and width of all annotation boxes are recorded, and the length and width are used as the horizontal and vertical coordinates of the two-dimensional coordinates, which are then mapped to a two-dimensional Cartesian coordinate system. Then, the k-means clustering algorithm is used to obtain several groups of coordinates with more points clustered around them. These coordinates are converted into length and width as the preset anchor size.

[0057] According to the above steps S21-S25, the crack detection network process is as follows Figure 2 As shown in the figure, conv represents two-dimensional convolution, conv3-512-2 represents a convolution layer with a convolution kernel size of 3x3, a number of 512, and a stride of 2, BN represents batch normalization, RELU represents the RELU activation function, Sigmoid represents the Sigmoid activation function, and maxpool2-2 represents a pooling layer with a pooling kernel of 2 and a stride of 2.

[0058] Finally, the S2 stage can obtain the position of the box where the crack is located from the input image, which is the result of rough extraction. The image located in the box will serve as the input of the S3 stage.

[0059] Among them, S3 performs pixel-by-pixel semantic segmentation on the image, and the steps include:

[0060] S31, the encoder part of the U-net network is changed to VGG16, and the upsampling part of the decoder is changed accordingly. The crack detection network is implemented using VGG16, and its function is to detect the target of cracks. The image prediction matrix is ​​output after detection;

[0061] At step S32, the network starts to input image data of (3, 512, 512), that is, a color RGB image with a width of 512 and a height of 512. After passing through 5 convolutional layers and a maximum pooling layer, the width and height of the image are halved each time through the pooling layer; after each stage of convolutional layer, the image resolution remains unchanged, and the channel depth is doubled.

[0062] In this embodiment, the input image is the output of the S2 stage, which includes a picture of cracks.

[0063] S34 is used to improve the image resolution. Except for the last upsampling, the rest of the upsampling parts of this model use deconvolution operations.

[0064] The upsampling operation specifically refers to the amplified feature map obtained through bilinear interpolation. Each upsampling operation is concatenated with the downsampled feature map of the corresponding resolution. Then, convolution and activation functions are performed, followed by another upsampling operation until the feature map's resolution returns to its original size. Finally, a classification head is used to determine the probability of each class for each pixel, and the class with the highest probability is selected as the output for that pixel.

[0065] According to the above steps S31-S34, the process structure of the U-net neural network is as follows Figure 3 As shown in the figure, conv represents two-dimensional convolution, conv3-512-2 represents a convolution layer with a convolution kernel size of 3x3, a number of 512, and a stride of 2, BN represents batch normalization, RELU represents the RELU activation function, Sigmoid represents the Sigmoid activation function, maxpool2-2 represents a pooling layer with a pooling kernel of 2 and a stride of 2, and ⊕ represents splicing in the channel direction.

[0066] Through the processing in stage S3, the pixels where the cracks are located can be obtained, achieving pixel-level crack segmentation. The identified pixels are then marked in the original image and used as input for the next stage, stage S4.

[0067] S4 uses the NeRF network and adds a 3D reconstruction component to connect the disease with the entire bridge, thus better monitoring bridge deformation.

[0068] S41, obtaining the structural information of the outer surface of the bridge. During the process of collecting the two-dimensional image information, the bridge surface must be unobstructed. In order to avoid environmental influences, the time for obtaining the two-dimensional image should be as concentrated as possible.

[0069] In this embodiment, the output of the S4 stage is the output of the S3 stage, and its original source is the data collected in the S1 stage.

[0070] S42, obtaining the camera position, and obtaining the change of each camera coordinate system relative to the world coordinate system from the image in step (1) by using the motion recovery structure algorithm, and then obtaining the viewing angle (θ, φ).

[0071] The world coordinate system is typically located at the center of the 3D reconstructed object. This can be achieved by photographing the object, for example, by aligning the camera's line of sight with the center of the object being reconstructed. The perspectives (θ, φ) here refer to two angles in spherical coordinates. Determining a point in spherical coordinates requires two angles and a distance from the coordinate center. Here, only the camera angle is required, so only two angles are needed.

[0072] S43, connecting the target pixel with the center of the camera of the corresponding viewing angle to obtain a ray, and sampling a point where the ray intersects the scene to obtain the three-dimensional coordinates of the corresponding point.

[0073] To align with camera capture, perspective projection is used. In this example, the front section distance is set to 0.05, and the back section distance is set to 50. The aspect ratio is consistent with that of the camera output image. The target pixel location directly uses the front section as the pixel's plane. Sampling is performed between the front and back sections. In this example, 100 points are uniformly sampled.

[0074] S44, the three-dimensional coordinates of these points together with the viewing angle form 5D coordinates (x, y, z, θ, φ) and input them into the neural network radiation field network to obtain the color and density of the corresponding image, and these colors and densities are integrated along the ray to obtain the color of the target pixel, and then the rendering result of the entire image is obtained.

[0075] S45, construct a loss function with the result and the input image, perform error back propagation, and update the parameters of the neural network radiation field network so that the shape and color information of the bridge are saved in the neural network radiation field network.

[0076] The loss function in this embodiment is the L2 loss function.

[0077] S46, since the shape and color information of the bridge are saved in the neural network radiation field network, points are sampled through the volume rendering network, and then the information of these points in the volume neural network radiation field network is queried to render the final three-dimensional model.

[0078] The query process is the same as the process of obtaining a pixel point during training, except that gradient tracking is not required.

[0079] Although this specification is described according to implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0080] Therefore, the above description is only a preferred embodiment of the present application and is not intended to limit the scope of implementation of the present application; that is, all equivalent modifications made according to the scope of the claims of the present application are within the scope of protection of the claims of the present application.

Claims

1. A method for bridge crack detection and three-dimensional visualization of cracks, characterized in that: The following steps are involved: S1, using the robot to collect two-dimensional images of the bridge disease-related areas; S2, uses the single-stage target detection YOLOv2 algorithm to roughly extract the crack-related areas to improve the efficiency of crack recognition; S3 uses pixel-level semantic segmentation to achieve accurate and complete separation of cracks and background. The semantic segmentation network is based on U-net. S4, uses the nerf neural network to reconstruct the bridge 3D using a large number of 2D images of bridge structures; S5, adding the crack segmentation image to the three-dimensional bridge model to achieve three-dimensional visualization of the bridge cracks in the bridge model.

2. A system based on the bridge crack detection and crack 3D visualization method according to claim 1, characterized in that: This includes a target detection model that uses a robot to collect two-dimensional images of bridge damage-related areas, and three network models generated by combining YOLOv2, U-net, and NeRF; The target detection model uses the YOLOv2 algorithm to quickly detect crack areas from the original image, perform preliminary positioning, and build an initial extraction network model; Use U-net to perform convolutional feature extraction and pixel-level semantic segmentation on the crack area of ​​the target detection model, and build a network model after pixel segmentation; Through the NeRF network, a three-dimensional reconstruction part is added to construct a three-dimensional model.

3. The system according to claim 2, characterized in that The specific steps of using the YOLOv2 network to determine the crack area, preliminarily segment the initial crack information, and build the initial extraction network model are as follows: S21, the front part of the network is superimposed with multiple convolutional layers and maximum pooling layers. In this process, the convolutional neural network is used to extract image features, which increases the depth of the feature map, and the maximum pooling is used for downsampling to reduce the resolution of the feature map; S22, introduces nonlinearity, and obtains high-level features from low-level image edge and color features; The latter part is the fully connected layer that obtains the predicted object category and border information from the extracted high-level features; S23, adding batchNorm to alleviate gradient disappearance; S24, better finetuning by introducing transition training on ImageNet upsampled data; S25 introduces a pre-set anchor size and adjusts it based on it to achieve better results; the size of the anchor is not set manually, but the k-means clustering algorithm is used to cluster the sizes of the boxes in the mark to obtain the initial extracted network model.

4. The system according to claim 2, wherein: U-net performs pixel-by-pixel semantic segmentation on the image to build a pixel-by-pixel segmented network model. The specific steps include: S31, the encoder part of the U-net network is changed to VGG16, and the upsampling part of the decoder is changed accordingly. The crack detection network is implemented using VGG16, and its function is to detect the target of cracks. The image prediction matrix is ​​output after detection; S32, the network starts to input image data, passing through the convolution layer and the maximum pooling layer. Each time it passes through the pooling layer, the width and height of the image are halved; S33, similar to VGG, uses VGG16 as the feature extraction part; S34 is used to improve the image resolution. Except for the last upsampling, the rest of the upsampling parts of this model use deconvolution operations to obtain the network model after pixel segmentation.

5. The system according to claim 2, wherein: The NeRF network is used and a 3D reconstruction component is added to connect the defects with the entire bridge, better monitor bridge deformation, and construct a 3D model. The specific steps include: S41, acquiring the structural information of the bridge's outer surface. During the acquisition of the two-dimensional image information, the bridge surface must be unobstructed. To avoid environmental influences, the time for acquiring the two-dimensional image should be as concentrated as possible. S42, obtaining the camera position, and obtaining the change of each camera coordinate system relative to the world coordinate system from the image in step (1) by using the motion recovery structure algorithm, and then obtaining the viewing angle (θ, φ); S43, connecting the target pixel with the center of the camera of the corresponding viewing angle to obtain a ray, sampling a point where the ray intersects the scene to obtain the three-dimensional coordinates of the corresponding point; S44, the 3D coordinates of these points together with the viewing angle form 5D coordinates (x, y, z, θ, φ) and input them into the neural network radiation field network to obtain the color and density of the corresponding image, and these obtained colors and densities are integrated along the ray to obtain the color of the target pixel, thereby obtaining the rendering result of the entire image; S45, constructing a loss function with the result and the input image, performing error back propagation, and updating the parameters of the neural network radiation field network so that the shape and color information of the bridge are saved in the neural network radiation field network; S46, since the shape and color information of the bridge are saved in the neural network radiation field network, points are sampled through the volume rendering network, and then the information of these points in the volume neural network radiation field network is queried to render the final three-dimensional model.

Images