Three-dimensional quantification method for road diseases

Abstract

The invention discloses a three-dimensional quantification method for road diseases. The method comprises the following steps: acquiring a road disease image; performing two-dimensional segmentation on the road disease image by adopting a two-dimensional segmentation network; performing depth prediction on the segmented road disease image to obtain a depth map; obtaining 3D point cloud reconstruction of the disease area according to the depth map and preset camera parameters; and calculating feature data of the point cloud to complete three-dimensional quantification of the road disease, wherein the feature data comprises depth data and volume data. By adopting the method, the three-dimensional quantitative data of the road diseases can be intuitively obtained.

Description

Technical Field

[0001] The invention relates to the field of road surface disease detection, and in particular to a three-dimensional quantification method for road diseases. Background Art

[0002] At present, traditional image-based pavement damage detection technology requires a lot of manpower and material resources and has low detection accuracy. Existing technologies generally detect and locate damage through two-dimensional information, and use two-dimensional information to assess the severity of damage, but cannot obtain three-dimensional quantitative information such as the volume and depth of the damage. Directly obtaining three-dimensional information is generally done through three-dimensional laser scanners, depth cameras, etc., which are expensive to build and maintain. Therefore, it is crucial to efficiently and cheaply detect pavement damage and obtain three-dimensional information of the damage.

[0003] Patent document with application number 202410184663.7 discloses a road pavement anomaly detection method and inspection system based on drones. This method collects road surface images by manually controlling the drone to fly at a predetermined route and altitude, and uses the Yolo detection model to identify the collected road surface disease images, thereby realizing drone-based road surface anomaly detection and positioning. However, this method only realizes two-dimensional detection and positioning of diseases, and cannot obtain more intuitive three-dimensional quantitative data of diseases. Therefore, the inventor of the present invention proposes a three-dimensional quantification method for road diseases to solve the problem that road diseases cannot be well quantified in three dimensions. Summary of the invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide a three-dimensional quantification method for road damage, which can intuitively obtain three-dimensional quantification data of road damage.

[0005] Based on this, the present invention provides a three-dimensional quantification method for road damage, the method comprising:

[0006] Collect road damage images;

[0007] Using a two-dimensional segmentation network to perform two-dimensional segmentation on the road damage image;

[0008] Performing depth prediction on the segmented road damage image to obtain a depth map;

[0009] Obtaining 3D point cloud reconstruction of the diseased area according to the depth map and preset camera parameters;

[0010] The characteristic data of the point cloud is calculated to complete the three-dimensional quantification of road damage, and the characteristic data includes: depth data and volume data.

[0011] The step of using a two-dimensional segmentation network to perform two-dimensional segmentation on the road damage image includes:

[0012] The two-dimensional segmentation network comprises: a plurality of feature coupling encoders arranged in parallel, a dual-branch feature association module correspondingly connected to each of the feature coupling encoders, and a lightweight feature decoder connected to each of the dual-branch feature association modules;

[0013] The lightweight feature decoder is composed of Residual-MLP and MLP-Mixer, wherein the Residual-MLP is correspondingly connected to the dual-branch feature association module, and the outputs of the Residual-MLP are connected to the MLP-Mixe, wherein the number of the feature coupling encoder, the dual-branch feature association module and the Residual-MLP is the same.

[0014] Among them, the feature coupling encoder is composed of a CNN branch and a Transformer branch. The CNN branch and the Transformer branch are used to extract the global features and local features of road diseases respectively. In the feature coupling encoder, after each extraction of the global features and the local features, feature merging is performed to fuse the feature information and pass it to the next feature coupling encoder.

[0015] Wherein, the dual-branch feature association module includes:

[0016] The attention mechanism is used to introduce channel compression and expansion operations to weight the feature channels. The feature T i and feature Z i is defined as and Finally, the fusion feature G i , branch characteristics and Fusion is performed to obtain the interrelated features F of the CNN branch and the Transformer branch i , where the feature T i and the characteristic Z i From the CNN branch and the Transformer branch respectively, the fusion feature G i By the characteristic T i and the characteristic Z i It is obtained by feature concatenation, where i represents the i-th feature coupling encoder.

[0017] The MLP-Mixer consists of two MLP blocks, namely Token-MLP-block and Channel-MLP-block, each of which contains two fully connected layers and an independent nonlinear activation, namely GELU.

[0018] Among them, the multi-layer features from the feature coupling encoder and output by the dual-branch feature association module are called F i , modify the multi-layer feature F i The feature channel dimension is increased and passed through the MLP layer. A 1×1 convolution kernel is inserted into the side branch to ensure the dimension matching while retaining the spatial information for feature merging. In this step, the fused feature obtained after the convolution operation and the MLP layer is called Pi. In the second step, the fused feature Pi is upsampled to 1 / 4 and concatenated to obtain the feature F. The merged feature F is encoded by 8×8 convolution feature. The two-dimensional feature patch is mapped to one-dimensional tokens through linear projection to construct a "Channels × Tokens" table. Finally, the "Channels × Tokens" table is used as the input of MLP-Mixer and is serially passed through Token-MLP and Channel-MLP for spatial and channel feature modeling. The fusion feature P is obtained. i After concatenation, convolution, flattening, etc., the data is input into MLP-Mixer and passed through the MLP layer for upsampling to predict the segmentation mask M with a resolution of H×W×N. cls , where N cls Expressed as the number of predicted categories.

[0019] The step of performing depth prediction on the segmented road damage image to obtain a depth map includes:

[0020] The segmented road damage image is input into an encoder, wherein the encoder is composed of a first number of integrated SEs and convolution modules, and each time the road damage image passes through a convolution module, the size is halved while the number of channels is increased;

[0021] The decoder consists of a deconvolution network corresponding to the first number of layers. The input of each layer is the result of upsampling the output of the previous layer and fusion with all feature maps of the same size in the encoder network. The final output is a depth map with the same size as the input image.

[0022] Wherein, obtaining 3D point cloud reconstruction of the diseased area according to the depth map and preset camera parameters includes:

[0023] According to the depth map and the camera intrinsic parameters, the three-dimensional coordinates of the pothole point cloud in the camera coordinate system are obtained by calculation;

[0024] The three-dimensional coordinates in the camera coordinate system are converted into the three-dimensional coordinates in the world coordinate system by using the camera extrinsic parameters;

[0025] The reconstructed defect point cloud is obtained by fusing multiple depth maps according to the image pose.

[0026] Wherein, the method further comprises:

[0027] Calculate the reconstruction depth of all pothole point cloud data in each diseased area, and calculate the maximum value and average value of the reconstruction depth as the maximum reconstruction depth and the average reconstruction depth of the pothole area;

[0028] The volume of the damaged point cloud is calculated using a convex hull-based calculation method to quantify the pothole volume.

[0029] The method is applied to a drone.

[0030] In the present invention, first, a road damage image is collected; a two-dimensional segmentation network is used to perform two-dimensional segmentation on the road damage image; a depth map is obtained by performing depth prediction on the segmented road damage image; a 3D point cloud reconstruction of the damage area is obtained based on the depth map and preset camera parameters; and the feature data of the point cloud is calculated to complete the three-dimensional quantification of the road damage, wherein the feature data includes: depth data and volume data. The present invention finally realizes the three-dimensional quantification of road damage through two-dimensional segmentation and three-dimensional reconstruction, greatly improving the reconstruction efficiency of road damage, and does not require the use of expensive machines such as three-dimensional laser scanners, and can realize the three-dimensional quantification of road damage at a relatively low cost, and the three-dimensional quantification effect is real and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 is a flow chart of a three-dimensional quantification method for road damage provided by an embodiment of the present invention;

[0033] Figure 2 is a schematic diagram of a two-dimensional segmentation network provided by an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of a characteristic coupling encoder provided by an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of a TBFA-Module provided in an embodiment of the present invention;

[0036] Figure 5 is a schematic diagram of an H-MLP Decoder provided by an embodiment of the present invention;

[0037] Figure 6 is a schematic diagram of the SE principle provided by an embodiment of the present invention;

[0038] Figure 7 This is a flow chart of adding SE into a residual network provided by an embodiment of the present invention;

[0039] Figure 8 is a schematic diagram of a decoder provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Figure 1 is a schematic diagram of a three-dimensional quantification method for road damage provided by an embodiment of the present invention, the method comprising:

[0042] S101. Collect road damage images;

[0043] Drones can be used to collect images of road damage.

[0044] The assembly of a drone specifically includes: the assembly of the frame, motor, electronic speed controller, flight control, GPS, gimbal, camera, data transmission radio and image transmission radio.

[0045] Specific steps for debugging the drone: install the mission planner software on a portable notebook, debug the flight control and test the GPS, data transmission radio and image transmission radio; debugging includes drone control debugging, gimbal control debugging, and drone horizontal debugging.

[0046] The mission planner ground control station is used to realize the fixed-point flight of the UAV. By specifying the flight altitude, controlling the waypoint, and setting the hovering time, the UAV can fly according to the specified track, thereby improving the intelligence of UAV road disease detection and greatly saving manpower.

[0047] S102. Perform two-dimensional segmentation on the road damage image using a two-dimensional segmentation network;

[0048] The step of using a two-dimensional segmentation network to perform two-dimensional segmentation on the road damage image includes:

[0049] Figure 2is a schematic diagram of a two-dimensional segmentation network provided by an embodiment of the present invention, wherein the two-dimensional segmentation network comprises: a plurality of feature coupling encoders arranged in parallel, a dual-branch feature association module correspondingly connected to each of the feature coupling encoders, and a lightweight feature decoder connected to both of the dual-branch feature association modules;

[0050] The lightweight feature decoder is composed of Residual-MLP and MLP-Mixer, wherein the Residual-MLP is correspondingly connected to the dual-branch feature association module, and the outputs of the Residual-MLP are connected to the MLP-Mixe, wherein the number of the feature coupling encoder, the dual-branch feature association module and the Residual-MLP is the same.

[0051] Among them, the feature coupling encoder is composed of a CNN branch and a Transformer branch. The CNN branch and the Transformer branch are used to extract the global features and local features of road diseases respectively. In the feature coupling encoder, after each extraction of the global features and the local features, feature merging is performed to fuse the feature information and pass it to the next feature coupling encoder.

[0052] Specifically, Figure 3 is a schematic diagram of a feature coupling encoder provided in an embodiment of the present invention, a feature coupling encoder Feature Coupling Encoder (FC-Encoder) based on a fusion architecture of CNN and Transformer;

[0053] The feature coupling encoder is composed of a CNN branch and a Transformer branch in parallel. The CNN branch is used to extract the position features of similar areas in the crack image, and the Transformer branch is used to learn the global information of the crack, and then couple the detailed local features with the overall global features to enhance the global perception of features and the ability to capture local details. Figure 3 a, b, and c in the figure are the CNN branch, the multi-scale feature fusion structure, and the Transformer branch, respectively.

[0054] Wherein, the dual-branch feature association module includes:

[0055] The attention mechanism is used to introduce channel compression and expansion operations to weight the feature channels. The feature T i and feature Z i is defined as and Finally, the fusion feature G i , branch characteristics and Fusion is performed to obtain the interrelated features F of the CNN branch and the Transformer branch i , where the feature T i and the characteristic Z i From the CNN branch and the Transformer branch respectively, the fusion feature G i By the characteristic T i and the characteristic Z i The specific implementation process can be described as follows:

[0056]

[0057]

[0058] k=F ex (O i )=σ(D2δ(D1)O i ) (3)

[0059]

[0060]

[0061] In the above formula, F sq (·) indicates global average pooling of features; F ex In (·), the feature channels are compressed and expanded through two fully connected layers; D represents the fully connected layer; δ represents the ReLU activation function; σ represents the Sigmoid activation function; Conv(C in , C out ) represents the feature dimensions of the input and output in this convolution operation; Concat represents the concatenation of multiple features.

[0062] The MLP-Mixer consists of two MLP blocks, namely Token-MLP-block and Channel-MLP-block, each of which contains two fully connected layers and an independent nonlinear activation, namely GELU.

[0063] Among them, the multi-layer features from the feature coupling encoder and output by the dual-branch feature association module are called F i , modify the multi-layer feature F i The feature channel dimension is obtained by passing through the MLP layer, and a 1×1 convolution kernel is inserted into the lateral branch to ensure the dimension matching while retaining the spatial information for feature merging. In this step, the fused feature obtained through the convolution operation and the MLP layer is called P i In the second step, the fusion feature P iThe two-dimensional feature patch is mapped to one-dimensional tokens through linear projection to construct a "Channels×Tokens" table. Finally, the "Channels×Tokens" table is used as the input of MLP-Mixer and serially passed through Token-MLP and Channel-MLP for spatial and channel feature modeling to obtain the fusion feature P. i After concatenation, convolution, flattening, etc., the data is input into MLP-Mixer and passed through the MLP layer for upsampling to predict the segmentation mask M with a resolution of H×W×N. cls , where N cls Expressed as the number of predicted categories.

[0064] In the present invention, N cls =2, indicating the predicted background and disease category. The decoder can be expressed as:

[0065]

[0066]

[0067]

[0068]

[0069] U *,i =X *,i +D2σ(D1LayerNorm(X) *,i ),fori=1...C (10)

[0070] Y j,* =U j,* +D4σ(D3LayerNorm(X) j,* ),fori=1...S (11)

[0071] M=Linear(C,N cls )(Y+X) (12)

[0072] Here σ represents the nonlinear activation unit GELU; C and S represent the feature dimension and number of patches after linear projection mapping; Linear(C in ,C out )(·) indicates that C in and C outThey serve as the input and output feature dimensions of the linear layer respectively; Upsample represents the upsampling process, during which the image size will become larger; Flatten represents the flattening operation, that is, converting the two-dimensional feature patch into one-dimensional Tokens to form an input form that can be received by MLP-Mixer; D represents the fully connected layer. Figure 4 is a schematic diagram of a TBFA-Module provided in an embodiment of the present invention, which can be referred to Figure 4 .

[0073] Figure 5 This is a schematic diagram of the H-MLP Decoder provided by an embodiment of the present invention. In the figure, H represents the height of the picture; W represents the width of the picture; C represents the number of channels; Layer Norm represents layer normalization, which helps to normalize the distribution of input data and improve the stability and learning ability of the network. Skip connections are made before Layer and after MLP to improve the generalization ability of the network.

[0074] Road damage can be potholes, cracks, etc. Given a type of damage image to train, the network can learn to segment this type of damage. Let's take cracks as an example.

[0075] MFAF-Net is a pixel-level binary segmentation. Given a crack image and its corresponding groundtruth, the pothole image is serially captured through a feature coupling encoder (FC-Encoder) to capture feature information at different levels. In the FC-Encoder, CNN-Encoder and Transformer-Enconer are used in parallel to extract the global and local features of the crack. In the FC-Encoder, each time after extracting the global and local features, the features are merged to fuse the feature information for backward transmission. Through convolution downsampling, the feature map size gradually becomes smaller while containing more receptive field information, so that the context information in the disease features can be transmitted. At the same time, the global information and local information in the same scale level are fused through the double-branch feature association module (TBFA-Module). By inserting the channel attention module, the convolution feature channels are modeled respectively to obtain strong correlation information representing the global and local features, and then the fused feature information at the current level is obtained. In this way, more detailed feature map fusion features are obtained. The number of TBFA-Modules can be four, and finally the fusion information from different levels of the four TBFA-Modules is reorganized through residual MLP. After upsampling to 1 / 4 of the original image, feature fusion is performed in the channel and spatial scale directions, and the deconvolution filter is used to adjust the feature map size to the original image size, thereby generating a defect (crack) prediction map, and calculating its Sigmoid loss and Dice loss with the ground truth.

[0076] S103. Performing depth prediction on the segmented road damage image to obtain a depth map;

[0077] The step of performing depth prediction on the segmented road damage image to obtain a depth map includes:

[0078] The segmented road damage image is input into an encoder, wherein the encoder is composed of a first number of integrated SEs and convolution modules, and each time the road damage image passes through a convolution module, the size is halved while the number of channels is increased;

[0079] The decoder consists of a deconvolution network corresponding to the first number of layers. The input of each layer is the result of upsampling the output of the previous layer and merging it with all feature maps of the same size in the encoder network. The final output is a depth map with the same size as the input image.

[0080] The segmented image is used for depth prediction using a codec architecture. The encoder consists of 5 integrated SE and convolution modules; the decoder uses a 5-layer corresponding deconvolution network. The depth prediction model is iterated and optimized by a loss function.

[0081] Conv-1 conv2-4 Conv-5 SE-1 … SE-5 BN … BN Relu … Relu Max pool … Max pool

[0082] When the input image of the encoder network is a color image captured by a 512*512 camera, the size of the input image is halved each time it passes through a convolution module, and the number of channels increases. The image size output by each module is 256*256; 128*128; 64*64; 32*32; 16*16. The basic architecture of the encoder uses the resnet18 network. The resnet18 residual network can avoid the gradient vanishing problem in deep networks. The SE attention mechanism is added after each convolution layer (see the figure below). It is a channel-based attention mechanism. The model can learn the importance of each channel based on the SE mechanism, thereby improving the performance of the model.

[0083] Figure 6 It is a schematic diagram of the SE principle provided by an embodiment of the present invention, wherein Ftr: represents one or a group of standard convolution operations; Fsq represents compression, that is, performing global average pooling to change the size of the feature map; Fex represents excitation, through which the attention weights of each channel are obtained; Fscale: weighted to the previous feature map channel by channel through multiplication.

[0084] SE mainly consists of two parts, compression and excitation; the process idea of ​​the model is as follows:

[0085] 1. Input feature map, size H*W*C;

[0086] 2. Compress spatial features and perform global average pooling to obtain a 1*1*C feature map;

[0087] 3. Through the operation learning of the FC fully connected layer, a feature map with channel attention is obtained, with a dimension of 1*1*C;

[0088] 4. Features of channel attention Figure 1 *1*C, the original input feature map H*W*C, multiply the weight coefficient channel by channel, and finally output the feature map with channel attention;

[0089] Figure 7 : is a flow chart of adding SE into the residual network provided by an embodiment of the present invention, such as Figure 7 As shown in the figure: after inputting the feature map, global average pooling is performed based on the width and height of the feature map to reduce the spatial feature dimension to 1×1, as shown in formula (13); then two fully connected layers and nonlinear activation functions are used to establish connections between channels, as shown in formula (14); then the normalized weights are obtained through the Sigmoid activation function, and finally, they are weighted channel by channel to each channel of the original feature map by multiplication to complete the recalibration of the original features by channel attention, as shown in formula (15).

[0090] Among them, the first fully connected layer (FC) reduces the number of channels and can reduce the amount of calculation.

[0091]

[0092]

[0093]

[0094] u c represents the input feature map; D represents the fully connected layer; l represents a 1×1×C feature vector formed by directly compressing the W×H×C feature map containing global information; l c represents the cth element of l; represents the vector form of vector l after passing through two fully connected layers and an activation function; σ represents the sigmoid activation function; x c Represents the channel-attention-weighted feature map of the final output.

[0095] Figure 8 : is a schematic diagram of a decoder provided by an embodiment of the present invention. The decoder can be composed of 5 deconvolution networks. The input of each layer is the result of upsampling the output of the previous layer and merging it with all feature maps of the same size in the encoder network. The final output is a depth map Depth Image with the same size as the input image.

[0096] S104. Obtaining 3D point cloud reconstruction of the diseased area according to the depth map and preset camera parameters;

[0097] Wherein, obtaining 3D point cloud reconstruction of the diseased area according to the depth map and preset camera parameters includes:

[0098] According to the depth map and the camera intrinsic parameters, the three-dimensional coordinates of the pothole point cloud in the camera coordinate system are obtained by calculation;

[0099] The three-dimensional coordinates in the camera coordinate system are converted into the three-dimensional coordinates in the world coordinate system by using the camera extrinsic parameters;

[0100] The reconstructed defect point cloud is obtained by fusing multiple depth maps according to the image pose.

[0101] Specifically, the point (x, y) on the image coordinate system and the point (Xc, Yc, Zc) on the camera coordinate system can obtain the following relationship based on the principle of similar triangles:

[0102] Xc / x=Yc / y=Zc / f;

[0103] Among them, f represents the focal length of the drone camera; Zc represents the z-axis value of the camera coordinate, that is, the distance from the target to the camera, which is the pixel value in the depth map.

[0104] The matrix form is as follows:

[0105]

[0106] After obtaining the depth map and the internal parameters of the drone camera, the three-dimensional coordinates of the pothole point cloud in the camera coordinate system can be calculated by the above formula. Where f represents the focal length of the drone camera.

[0107] Use the drone camera external parameters to transform the point (Xc, Yc, Zc) in the camera coordinate system to the world coordinate system (Xs, Ys, Zs). The formula is:

[0108]

[0109] is the vector representation of a point in the camera coordinate system, is the vector representation of this point in the world coordinate system,

[0110] The matrix form is as follows:

[0111]

[0112] Among them, R represents rotation and T represents translation; the two parameters R and T are called camera extrinsics. After obtaining the world coordinates of the defect point cloud, multiple depth maps are fused according to the image pose to obtain the reconstructed defect point cloud.

[0113] S105. Calculate feature data of the point cloud to complete the three-dimensional quantification of road damage, wherein the feature data includes: depth data and volume data.

[0114] Specifically, the point cloud obtained is the point cloud of the segmented diseased area, and there is no need to perform operations such as boundary extraction.

[0115] Calculate the reconstruction depth of all the pothole point cloud data in each defect (pothole) area, and take their maximum value and average value as the maximum reconstruction depth and average reconstruction depth of the pothole area. The depth data of the defect (pothole) can be obtained.

[0116] The volume of the defect (pothole) point cloud is calculated using a convex hull-based calculation method; the volume of the pothole is quantified, and the volume information of the defect (pothole) is obtained.

[0117] Convex hull: refers to the smallest convex polygon or convex polyhedron that contains all the points in the point cloud.

[0118] The method is applied to unmanned aerial vehicles. The trained segmentation and reconstruction model is placed on a development board carried by the unmanned aerial vehicle, and finally the automatic segmentation and reconstruction of road damage is realized;

[0119] The Nvidia Jetson nano development board is used as the core processor, and is equipped with Tensor RT and Deepstream model acceleration frameworks to ensure that the model can run efficiently and in real time to achieve real-time detection.

[0120] In the present invention, first, a road damage image is collected; a two-dimensional segmentation network is used to perform two-dimensional segmentation on the road damage image; a depth map is obtained by performing depth prediction on the segmented road damage image; a 3D point cloud reconstruction of the damage area is obtained based on the depth map and preset camera parameters; and the feature data of the point cloud is calculated to complete the three-dimensional quantification of the road damage, wherein the feature data includes: depth data and volume data. The present invention finally realizes the three-dimensional quantification of road damage through two-dimensional segmentation and three-dimensional reconstruction, greatly improving the reconstruction efficiency of road damage, and does not require the use of expensive machines such as three-dimensional laser scanners, and can realize the three-dimensional quantification of road damage at a relatively low cost, and the three-dimensional quantification effect is real and reliable.

[0121] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A three-dimensional quantification method for road damage, characterized in that: include: Collect road damage images; Using a two-dimensional segmentation network to perform two-dimensional segmentation on the road damage image; Performing depth prediction on the segmented road damage image to obtain a depth map; Obtaining 3D point cloud reconstruction of the diseased area according to the depth map and preset camera parameters; The characteristic data of the point cloud is calculated to complete the three-dimensional quantification of road damage, and the characteristic data includes: depth data and volume data.

2. The three-dimensional quantification method of road damage according to claim 1, characterized in that: The two-dimensional segmentation of the road damage image using a two-dimensional segmentation network comprises: The two-dimensional segmentation network comprises: a plurality of feature coupling encoders arranged in parallel, a dual-branch feature association module correspondingly connected to each of the feature coupling encoders, and a lightweight feature decoder connected to each of the dual-branch feature association modules; The lightweight feature decoder is composed of Residual-MLP and MLP-Mixer, wherein the Residual-MLP is correspondingly connected to the dual-branch feature association module, and the outputs of the Residual-MLP are connected to the MLP-Mixe, wherein the number of the feature coupling encoder, the dual-branch feature association module and the Residual-MLP is the same.

3. The three-dimensional quantification method of road damage according to claim 2, characterized in that: The feature coupling encoder is composed of a CNN branch and a Transformer branch. The CNN branch and the Transformer branch are used to extract the global features and local features of road diseases respectively. In the feature coupling encoder, after each extraction of the global features and the local features, feature merging is performed to fuse the feature information and pass it to the next feature coupling encoder.

4. The three-dimensional quantification method of road damage according to claim 2, characterized in that: The dual-branch feature association module includes: The attention mechanism is used to introduce channel compression and expansion operations to weight the feature channels. While performing feature coupling encoding, the feature T from the CNN branch and the Transformer branch is extracted from the feature coupling encoder. i and Z i , for T i and Z i Perform feature concatenation to obtain fusion feature G i , the feature T after the attention mechanism i and feature Z i is defined as and Finally, the fusion feature G i , branch characteristics and Fusion is performed to obtain the interrelated features F of the CNN branch and the Transformer branch i , where the feature T i and the characteristic Z i From the CNN branch and the Transformer branch respectively, the fusion feature G i By the characteristic T i and the characteristic Z i It is obtained by feature concatenation, where i represents the i-th feature coupling encoder.

5. The three-dimensional quantification method for road damage according to claim 2, characterized in that: The MLP-Mixer consists of two MLP blocks, namely Token-MLP-block and Channel-MLP-block, each of which contains two fully connected layers and an independent non-linear activation, namely GELU.

6. The three-dimensional quantification method for road damage according to claim 2, characterized in that: The multi-layer features from the feature coupling encoder and output by the dual-branch feature association module are called F i , modify the feature channel dimension of the multi-layer feature Fi and insert a 1×1 convolution kernel in the lateral branch through the MLP layer, so as to ensure the dimension matching and merge the features while retaining the spatial information. In this step, the fused feature obtained after the convolution operation and the MLP layer is called P i In the second step, the fusion feature P i It is upsampled to 1 / 4 and concatenated to obtain feature F, and the merged feature F is encoded into feature P by 8×8 convolution feature. atch, the two-dimensional feature Patch is mapped to one-dimensional Tokens through linear projection, so as to construct a "Channels×Tokens" table, and finally the "Channels×Tokens" table is used as the input of MLP-Mixer and serially passed through Token-MLP and Channel-MLP for spatial and channel feature modeling, and the obtained fusion feature P i After concatenation, convolution, and flattening, the data is input into the MLP-Mixer and passed through the MLP layer for upsampling to predict the segmentation mask M with a resolution of H×W×N. cls , where N cls Expressed as the number of predicted categories.

7. The three-dimensional quantification method of road damage according to claim 1, characterized in that: Performing depth prediction on the segmented road damage image to obtain a depth map includes: The segmented road damage image is input into an encoder, wherein the encoder is composed of a first number of integrated SEs and convolution modules, and each time the road damage image passes through a convolution module, the size is halved while the number of channels is increased; The decoder consists of a deconvolution network corresponding to the first number of layers. The input of each layer is the result of upsampling the output of the previous layer and fusion with all feature maps of the same size in the encoder network. The final output is a depth map with the same size as the input image.

8. The three-dimensional quantification method for road damage according to claim 1, characterized in that: Obtaining 3D point cloud reconstruction of the diseased area according to the depth map and preset camera parameters includes: According to the depth map and the camera intrinsic parameters, the three-dimensional coordinates of the pothole point cloud in the camera coordinate system are obtained by calculation; The three-dimensional coordinates in the camera coordinate system are converted into the three-dimensional coordinates in the world coordinate system by using the camera extrinsic parameters; The reconstructed defect point cloud is obtained by fusing multiple depth maps according to the image pose.

9. The three-dimensional quantification method of road damage according to claim 1, characterized in that: The method further comprises: Calculate the reconstruction depth of all pothole point cloud data in each diseased area, and calculate the maximum value and average value of the reconstruction depth as the maximum reconstruction depth and the average reconstruction depth of the pothole area; The volume of the damaged point cloud is calculated using a convex hull-based calculation method to quantify the pothole volume.

10. The three-dimensional quantification method for road damage according to claims 1 to 9, characterized in that: The method is applied to a drone.

Images