A pavement crack extraction method based on neural radiance field and semantic segmentation network

Abstract

The present invention discloses a pavement crack extraction method based on a neural radiation field and a semantic segmentation network, and specifically relates to the field of road engineering technology. The pavement image data of a research area is acquired by an unmanned aerial vehicle and preprocessed to form a required image data set. The preprocessed image data set is three-dimensionally reconstructed by combining a motion recovery structure multi-view stereo vision algorithm and a neural radiation field algorithm to obtain a pavement point cloud model and a point cloud data set. A semantic segmentation network is constructed and the point cloud data set is trained. The pavement crack information is extracted and calculated according to the semantic segmentation result. Compared with the prior art, the detection cost is reduced, the real-time detection requirements are met, and the current status of point cloud processing automation, poor expressiveness and insufficient application is significantly improved. In addition, the road scene with cracks can be intuitively and accurately characterized, thereby providing an underlying platform and data support for the digital representation of pavement information.

Description

Technical Field

[0001] The present invention relates to the technical field of road engineering, and in particular to a road surface crack extraction method based on a neural radiation field and a semantic segmentation network. Background Art

[0002] At present, the detection of pavement cracks in the industry is mainly performed by professional technicians and high-cost facilities. However, in some areas, roads are located in special areas with harsh conditions. The cost of manual detection is high and it is difficult to meet real-time needs. Manual detection mainly relies on two-dimensional image recognition. Although traditional pavement crack recognition obtains a binary image of the crack through the threshold method, extracts and analyzes the binary features, and then describes the crack characteristics, the generalization ability of this pavement crack recognition method is low, and cracks cannot be segmented through general algorithms. Morphological operations and geometric calculations are required to quantify the cracks based on the crack morphology. The results of crack quantification depend on the accuracy of the crack image and further processing.

[0003] In addition, the method based on two-dimensional images cannot calculate the crack depth, which is crucial for road maintenance and repair, so 3D information of the road surface is needed. The acquisition of 3D information of the road surface is achieved by three-dimensional stereo imaging technology, and most of the current three-dimensional stereo imaging technology is completed by laser scanning systems. However, the hardware requirements and costs of using laser scanning systems are very high, and the operation is not easy and the data acquisition is difficult.

[0004] In addition, it is still challenging to achieve high-precision crack segmentation from the massive point cloud data obtained from the 3D model. On the one hand, the acquired point cloud data has uneven density distribution and partial missing data, which increases the difficulty of automatically extracting crack geometry information. On the other hand, complex road conditions lead to uncertainty and incompleteness in crack segmentation, reducing the accuracy of the model.

[0005] At present, the extraction algorithms for road point clouds are mainly divided into two categories: ① Traditional empirical threshold method: including plane grid method and facet grid method. However, these methods are limited by fixed parameters and require manual settings, making them difficult to apply to complex and diverse road conditions. ② Semantic segmentation algorithm: This method is applied to the classification and segmentation of road scenes. It converts irregular point cloud data into image information, uses convolutional neural networks to segment pixels without setting empirical thresholds, and further projects three-dimensional point clouds to multi-view images, thereby constructing a convolutional neural network for multi-view images. Although this algorithm converts irregular three-dimensional point cloud data into regular form, it cannot fully utilize the spatial and structural information between point clouds, and in the process of reprojecting the segmentation results into three-dimensional space, errors will inevitably occur due to different dimensions and division methods. Summary of the invention

[0006] In order to solve the problems of low efficiency, high labor and time cost and insufficient automation of current pavement crack detection methods, the present invention provides a pavement crack extraction method based on neural radiation field and semantic segmentation network, which not only reduces the detection cost and meets the real-time detection needs, but also significantly improves the current status of point cloud processing automation, poor expressiveness and insufficient application. In addition, it can also intuitively and accurately characterize road scenes with cracks, thereby providing an underlying platform and data support for the digital representation of pavement information.

[0007] The technical solutions provided are as follows:

[0008] A pavement crack extraction method based on neural radiation field and semantic segmentation network obtains pavement image data of the study area through unmanned aerial vehicles and preprocesses it to form the required image dataset. The preprocessed image dataset is 3D reconstructed by combining the motion recovery structure multi-view stereo vision algorithm and the neural radiation field algorithm to obtain the pavement point cloud model and point cloud dataset. By constructing a semantic segmentation network and training the point cloud dataset, the pavement crack information is extracted and calculated according to the semantic segmentation results.

[0009] Preferably, the reconstruction process of the motion recovery structure multi-view stereo vision algorithm recovers the camera extrinsic parameters and describes the relationship that must be satisfied between the corresponding points of the two images according to the epipolar geometry constraints, then decomposes the intrinsic matrix to obtain the relative motion parameters of the camera, and then optimizes the matching relationship between the newly added camera posture and the existing camera and the three-dimensional point cloud.

[0010] Preferably, the neural radiation field algorithm uses a multi-layer perceptron neural network to implicitly express the scene. The neural radiation field input includes the ray angle in the preprocessed image data set and the 3D position on the ray in the scene. The output includes the color information and spatial volume density corresponding to the 3D position. The view is synthesized along the emission light query information and the ray angle and the 3D position, color information and spatial volume density information on the ray in the scene are used to generate a new perspective image during rendering. The static scene is described as a continuous five-dimensional vector function. The formula is as follows:

[0011] F(θ):(x,d)→(C(r),σ) (1)

[0012] Where: F(θ) is a neural network; x is the spatial coordinate of the point; d is the viewing direction of the input image; C is the color information emitted from the position of each point in the direction of d; σ is the volume density of each point;

[0013] For a known scene and observation angle, the view corresponding to F(θ) needs to rely on the numerical integration method to approximate a real volume rendering process. By calculating the difference between the rendered image and the real image, comparing the loss function and continuously iterating and optimizing the rendering results, a high-realistic real-world point cloud model is generated on this basis, as shown in formula (2):

[0014]

[0015] Where: t n and t f are the near and far boundaries of the light, respectively; T(t) is the cumulative transmittance, indicating that the light from t n to f The probability of not being blocked by particles; c(r(t)) is the color of the point; σ(r(t)) is the density of the point, and C(r) represents the rendering function.

[0016] Preferably, the semantic segmentation network PointNet++ is used as the skeleton, and a multi-scale feature fusion strategy is introduced in the spatial feature extraction structure. By adopting multiple parallel branch networks and convolution and pooling operations of different sizes, combined with local polar coordinate representation and attention pooling fusion modules, local and global features of point cloud data sets are extracted from multiple scales, and an encoder-decoder structure is introduced to extract the crack edge contour in the pavement point cloud model;

[0017] Local polar coordinates construct a local feature that is invariant along the z axis in the polar coordinate system. i and its surrounding k neighboring points Convert to polar coordinates, calculate the geometric distance between points in the neighborhood and determine the centroid of the neighborhood Next, determine the local direction and set the initial azimuth angle φ i ,θ i Subtract the local azimuth α i , β i , get the relative azimuth

[0018] The attention pooling fusion module is to more accurately reflect the spatial information of the point cloud and retain the complex geometric structure of the point cloud. Therefore, the original coordinate attention mechanism is integrated with the aggregation attention pooling mechanism. The attention pooling fusion module uses parallel pooling operations in the same direction to perform feature aggregation, inputs the local features and global features of the point cloud, geometric distance and feature distance, considers the enhancement of spatial features from the perspective of geometric distance and feature distance, and then aggregates the maximum feature value and neighborhood mean. The process is as follows:

[0019] First, based on the feature vector g(i) of any point i and the feature vector g(k) of the kth neighboring point around it, the feature distance is calculated by using the L1 norm. The formula is as follows:

[0020]

[0021] Where: |·| is the L1 norm; Ave(·) is the mean function; g(i) and g(k) represent the eigenvectors of the i-th and k-th neighboring points, respectively; is the characteristic distance;

[0022] Then according to the geometric distance and feature distance The relationship between the neighboring points determines the attention weight, and the geometric distance Distance to feature Take the negative number, perform weighted summation through the normalized exponential function softmax, merge the results, and get the distance feature Then the distance feature With local features Fusion obtains the fused distance feature And perform weight learning, the formula is as follows:

[0023]

[0024] Where: and is the geometric distance and characteristic distance; and is the distance feature and the local feature; ⊕ is the merge operation;

[0025] Finally, the aggregated local features are calculated, and the convolution function and activation function are used to automatically learn the attention weights to select important features and remove unimportant features. and local features Perform weighted summation to calculate the local average feature f iLAve , as shown in formula (6); then by calculating the local maximum feature f among k neighboring points iLmax ; Finally, the above two features are combined as the local aggregation feature f iL , as shown in formula (7):

[0026]

[0027] Among them, K represents the total number of points in the point domain.

[0028] Preferably, the network structure of the semantic segmentation network adopts a hierarchical structure. The encoder part in the encoder-decoder structure adopts the convolutional neural network PointNet as the basic model. The decoder part adopts a jump connection method to splice the features corresponding to the points in the encoder part. The final semantic segmentation training results are saved in txt format.

[0029] Preferably, based on the semantic segmentation training results, the trained crack point cloud segmentation results are added to the pavement point cloud model and displayed spatially, and the pavement change information is displayed according to the transverse and longitudinal sections of the pavement; after the crack spatial morphology is generated, the crack surface contour line and the crack center line are used to quantify the crack calculation results, the outer contour line of the irregular crack is fitted by the surface, the crack center line is fitted based on the least squares method and regression analysis method, and then the crack geometric parameters are calculated based on the crack contour line and the crack center line.

[0030] The present invention has the following advantages:

[0031] The pavement image data of the study area is acquired through UAVs and preprocessed to form the required image data set. The preprocessed image data set is 3D reconstructed by combining the structure from motion multi-view stereo vision algorithm and the neural radiation field algorithm to obtain the pavement point cloud model and point cloud data set. A semantic segmentation network is constructed and the point cloud data set is trained. The pavement crack information is extracted and calculated based on the semantic segmentation results. Compared with the existing technology, the detection cost is reduced, the real-time detection needs are met, and the current status of point cloud processing automation, poor expressiveness and insufficient application is significantly improved. In addition, it can intuitively and accurately characterize road scenes with cracks, thereby providing an underlying platform and data support for the digital representation of pavement information. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flow chart of a pavement crack extraction method based on a neural radiation field and a semantic segmentation network provided in this embodiment;

[0033] Figure 2 The Point-NeRF reconstructed point cloud model diagram provided in this embodiment;

[0034] Figure 3 A relative azimuth angle calculation diagram provided for this embodiment;

[0035] Figure 4 A diagram of the network segmentation structure provided for this embodiment;

[0036] Figure 5 A visualization diagram of the semantic segmentation results provided in this embodiment;

[0037] Figure 6 The pavement fitting and crack three-dimensional map provided in this embodiment;

[0038] Figure 7 A road space visualization diagram provided for this embodiment;

[0039] Figure 8 A contour line diagram of the crack surface provided for this embodiment;

[0040] Fig. 9 A crack centerline diagram is provided for this embodiment. DETAILED DESCRIPTION

[0041] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] like Figure 1 As shown, this embodiment provides a road crack extraction method based on neural radiation field and semantic segmentation network, which obtains road surface image data of the study area through a drone equipped with an RTK lens, and pre-processes the collected image data according to the needs, including cropping and labeling, to finally form an image data set, and reconstructs the pre-processed image data set by combining the motion recovery structure multi-view stereo vision algorithm and the neural radiation field algorithm to obtain a road point cloud model (such as Figure 2 As shown in the figure, as well as the pavement point cloud dataset, a semantic segmentation network is constructed and the point cloud dataset is trained, and the pavement crack information is extracted and calculated according to the semantic segmentation results.

[0043] The reconstruction process of the structure-from-motion multi-view stereo vision algorithm recovers the camera extrinsics and describes the relationship that must be satisfied between the corresponding points of the two images according to the epipolar geometry constraints. It then decomposes the intrinsic matrix to obtain the relative motion parameters of the cameras and optimizes the matching relationship between the newly added camera posture and the existing camera and 3D point cloud.

[0044] The neural radiance field algorithm uses a multi-layer perceptron neural network to implicitly express the scene. The neural radiance field input includes the ray angle in the preprocessed image data set and the 3D position on the ray in the scene. Its output includes the color information and spatial volume density corresponding to the 3D position. It synthesizes the view by querying the information along the emitted light and generates a new perspective image by using the ray angle and the 3D position, color information and spatial volume density information on the ray in the scene during rendering. The static scene is described as a continuous five-dimensional vector function. The formula is as follows:

[0045] F(θ):(x,d)→(C(r),σ) (1)

[0046] Where: F(θ) is a neural network; x is the spatial coordinate of the point; d is the viewing direction of the input image; C is the color information emitted from the position of each point in the direction of d; σ is the volume density of each point;

[0047] For a known scene and observation angle, the view corresponding to F(θ) needs to rely on the numerical integration method to approximate a real volume rendering process. By calculating the difference between the rendered image and the real image, comparing the loss function and continuously iterating and optimizing the rendering results, a high-realistic real-world point cloud model is generated on this basis, as shown in formula (2):

[0048]

[0049] Where: t n and t f are the near and far boundaries of the light, respectively; T(t) is the cumulative transmittance, indicating that the light from t n to f The probability of not being blocked by particles; c(r(t)) is the color of the point; σ(r(t)) is the density of the point, and C(r) represents the rendering function.

[0050] Preferably, a semantic segmentation network is constructed to segment the acquired pavement point cloud dataset to obtain a point cloud dataset of pavement cracks. The semantic segmentation network PointNet++ is used as the skeleton, and a multi-scale feature fusion strategy is introduced in the spatial feature extraction structure. By adopting multiple parallel branch networks and convolution and pooling operations of different sizes, combined with local polar coordinate representation and attention pooling fusion modules, local and global features of point cloud datasets are extracted from multiple scales. An encoder-decoder structure is introduced to extract the crack edge contour in the pavement point cloud model.

[0051] The local polar coordinates are used to reduce the influence of model rotation on the extraction of geometric features. The local polar coordinates construct a local feature that is invariant along the z-axis in the polar coordinate system. i and its surrounding k neighboring points Convert to polar coordinates, calculate the geometric distance between points in the neighborhood and determine the centroid of the neighborhood Next, determine the local direction and set the initial azimuth angle φ i ,θ i Subtract the local azimuth α i , β i , get the relative azimuth like Figure 3 As shown;

[0052] The attention pooling fusion module is to more accurately reflect the spatial information of the point cloud and retain the complex geometric structure of the point cloud. Therefore, the original coordinate attention mechanism is integrated with the aggregation attention pooling mechanism. The attention pooling fusion module uses parallel pooling operations in the same direction to perform feature aggregation, inputting the local features and global features, geometric distances, and feature distances of the cloud, and considers the enhancement of spatial features from the perspective of geometric distances and feature distances, and then aggregates the maximum value of the feature and the neighborhood mean, such as Figure 4 As shown, the process is as follows:

[0053] First, based on the feature vector g(i) of any point i and the feature vector g(k) of the kth neighboring point around it, the feature distance is calculated by using the L1 norm. The formula is as follows:

[0054]

[0055] Where: |·| is the L1 norm; Ave(·) is the mean function; g(i) and g(k) represent the eigenvectors of the i-th and k-th neighboring points, respectively; is the characteristic distance;

[0056] Then according to the geometric distance and feature distance The relationship between the neighboring points determines the attention weight, and the geometric distance Distance to feature Take the negative number, perform weighted summation through the normalized exponential function softmax, merge the results, and get the distance feature Then the distance feature With local features Fusion obtains the fused distance feature And perform weight learning, the formula is as follows:

[0057]

[0058] Where: and is the geometric distance and characteristic distance; and is the distance feature and the local feature; ⊕ is the merge operation;

[0059] Finally, the aggregated local features are calculated, and the convolution function and activation function are used to automatically learn the attention weights to select important features and remove unimportant features. and local features Perform weighted summation to calculate the local average feature f iLAve , as shown in formula (6); then by calculating the local maximum feature f among k neighboring points iLmax; Finally, the above two features are combined as the local aggregation feature f iL , as shown in formula (7):

[0060]

[0061] Among them, K represents the total number of points in the point domain.

[0062] The network structure of the semantic segmentation network adopts a hierarchical structure. The encoder part of the encoder-decoder structure uses the convolutional neural network PointNet as the basic model. The decoder part uses a skip connection method to splice the features corresponding to the points in the encoder part. The final semantic segmentation training results are saved in txt format. The visualization effect is shown in the figure below. Figure 5 shown.

[0063] Preferably, the road surface plane fitted by the RANSAC algorithm is as follows Figure 6 As shown in (a), Figure 6 (b) reflects the shape of the pavement cracks. The crack point cloud data set in the trained pavement point cloud model is processed again, including the estimation of pavement cracks, the generation of crack space map, and the extraction of crack surface contour lines and center lines; according to the semantic segmentation training results, the trained crack point cloud segmentation results are added to the pavement point cloud model and displayed spatially, and the pavement change information is displayed according to the horizontal and longitudinal sections of the pavement, such as Figure 7 As shown in the figure; after the crack space morphology is generated, the crack surface contour line and the crack center line are used to quantify the crack calculation results, and the outer contour line of the irregular crack is fitted by the surface, as shown in the figure. Figure 8 As shown in (the horizontal and vertical axes represent the length and width of the crack after the projection magnification ratio), the center line of the crack is fitted based on the least squares method and regression analysis method, as shown in Fig. 9 As shown (the horizontal and vertical coordinates represent the length and width of the crack after the projection magnification ratio), and then the crack geometric parameters are calculated according to the crack contour line and the crack center line. The final calculation results are shown in Table 1;

[0064] Table 1 Calculation results of crack geometric parameters

[0065]

[0066] From the calculation results in the table, the error between the crack length calculation result of the method of the present invention and the actual measurement result is within 0.3m, and the error rate is within 8%. The error between the crack width calculation result and the actual measurement result is within 0.02m, and the error rate is within 11%. The error between the crack depth calculation result and the actual measurement result is within 0.7cm, and the error rate is within 20%. Overall, the calculation results have a high correlation with the actual measurement results, and the error rate is low.

[0067] The crack length is measured by accumulating the Euclidean distance between each pair of adjacent point clouds. The formula is as follows:

[0068]

[0069] The crack width is calculated based on the crack centerline. A vertical line is drawn through the centerline at the observation section of the crack. The distance between the intersection of the vertical line and the crack edge line is the crack width at the observation point. The formula is as follows:

[0070]

[0071] The crack depth is obtained by fitting the pavement plane according to the RANSAC plane fitting algorithm, and the distance from the crack point cloud to the fitted pavement is calculated as the depth of the crack. The formula is as follows:

[0072]

[0073] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A pavement crack extraction method based on neural radiation field and semantic segmentation network, which uses drones to obtain pavement image data of the study area and preprocesses it to form the required image data set, characterized by: The preprocessed image dataset is 3D reconstructed by combining the structure-from-motion multi-view stereo vision algorithm and the neural radiance field algorithm to obtain a pavement point cloud model and a point cloud dataset. A semantic segmentation network is constructed and the point cloud dataset is trained to extract and calculate pavement crack information based on the semantic segmentation results. The reconstruction process of the structure-from-motion multi-view stereo vision algorithm recovers the camera extrinsics and describes the relationship that must be satisfied between the corresponding points of the two images according to the epipolar geometry constraints, then decomposes the intrinsic matrix to obtain the relative motion parameters of the camera, and then optimizes the matching relationship between the newly added camera posture and the existing camera and 3D point cloud; The neural radiance field algorithm uses a multi-layer perceptron neural network to implicitly express the scene. The neural radiance field input includes the ray angle in the preprocessed image data set and the 3D position on the ray in the scene. Its output includes the color information and spatial volume density corresponding to the 3D position. It synthesizes the view by querying the information along the emitted light and generates a new perspective image by using the ray angle and the 3D position, color information and spatial volume density information on the ray in the scene during rendering. The static scene is described as a continuous five-dimensional vector function. The formula is as follows: F(θ):(x,d)→(C(r),σ) (1) Where: F(θ) is a neural network; x is the spatial coordinate of the point; d is the viewing direction of the input image; C is the color information emitted from the position of each point in the direction of d; σ is the volume density of each point; For a known scene and observation angle, the view corresponding to F(θ) needs to rely on the numerical integration method to approximate a real volume rendering process. By calculating the difference between the rendered image and the real image, comparing the loss function and continuously iterating and optimizing the rendering results, a high-realistic real-world point cloud model is generated on this basis, as shown in formula (2): Where: t n and t f are the near and far boundaries of the light, respectively; T(t) is the cumulative transmittance, indicating that the light from t n to f The probability of not being blocked by particles; c(r(t)) is the color of the point; σ(r(t)) is the density of the point, and C(r) represents the rendering function.

2. The pavement crack extraction method based on neural radiation field and semantic segmentation network according to claim 1 is characterized by: With the semantic segmentation network PointNet++ as the framework, a multi-scale feature fusion strategy is introduced in the spatial feature extraction structure. By adopting multiple parallel branch networks and convolution and pooling operations of different sizes, combined with local polar coordinate representation and attention pooling fusion modules, local and global features of point cloud datasets are extracted from multiple scales. An encoder-decoder structure is introduced to extract the crack edge contour in the pavement point cloud model. Local polar coordinates construct a local feature that is invariant along the z axis in the polar coordinate system. i and its surrounding k neighboring points Convert to polar coordinates, calculate the geometric distance between points in the neighborhood and determine the centroid of the neighborhood Next, determine the local direction and set the initial azimuth angle φ i ,θ i Subtract the local azimuth α i , β i , get the relative azimuth The attention pooling fusion module uses parallel pooling operations in the same direction to aggregate features. It inputs the local features and global features, geometric distances, and feature distances of the point cloud, considers the enhancement of spatial features from the perspective of geometric distances and feature distances, and then aggregates the maximum feature value and the neighborhood mean. The process is as follows: First, based on the feature vector g(i) of any point i and the feature vector g(k) of the kth neighboring point around it, the feature distance is calculated by using the L1 norm. The formula is as follows: Where: |·| is the L1 norm; Ave(·) is the mean function; g(i) and g(k) represent the eigenvectors of the i-th and k-th neighboring points, respectively; is the characteristic distance; Then according to the geometric distance and feature distance The relationship between the neighboring points determines the attention weight, and the geometric distance Distance to feature Take the negative number, perform weighted summation through the normalized exponential function softmax, merge the results, and get the distance feature Then the distance feature With local features Fusion obtains the fused distance feature And perform weight learning, the formula is as follows: Where: and is the geometric distance and characteristic distance; and It is the distance feature and the local feature; For the merge operation; Finally, the aggregated local features are calculated, and the convolution function and activation function are used to automatically learn the attention weights to select important features and remove unimportant features. and local features Perform weighted summation to calculate the local average feature f iLAve , as shown in formula (6); then by calculating the local maximum feature f among k neighboring points iLmax ; Finally, the above two features are combined as the local aggregation feature f iL , as shown in formula (7): Among them, K represents the total number of points in the point domain.

3. The pavement crack extraction method based on neural radiation field and semantic segmentation network according to claim 2 is characterized by: The encoder part of the encoder-decoder structure uses the convolutional neural network PointNet as the basic model, and the decoder part uses a jump connection method to splice the features corresponding to the points in the encoder part. The final semantic segmentation training results are saved in txt format.

4. The pavement crack extraction method based on neural radiation field and semantic segmentation network according to claim 1 is characterized by: According to the semantic segmentation training results, the trained crack point cloud segmentation results are added to the pavement point cloud model and displayed spatially, and the pavement change information is displayed according to the horizontal and vertical sections of the pavement; After generating the spatial morphology of the crack, the crack calculation results are quantified by the crack surface contour line and the crack center line. The outer contour line of the irregular crack is fitted by the surface, and the crack center line is fitted based on the least squares method and regression analysis method. Then, the crack geometric parameters are calculated based on the crack contour line and the crack center line.

Images