Road crack segmentation method based on convolutional neural network and transformer dual path

Abstract

The application discloses a road crack segmentation method based on a convolutional neural network and a transformer dual path, which comprises the following steps: loading a data set of road crack images based on the convolutional neural network and the transformer dual path; performing feature extraction on an initial input image based on the convolutional neural network, obtaining feature maps extracted by respective convolutional layers, and obtaining a final extracted feature map through a deep separable convolution for down-sampling; performing feature extraction on the initial input image based on the transformer, obtaining feature maps extracted by respective transformer blocks; obtaining complementary fusion features; obtaining simple fusion features; inputting the simple fusion features into a decoder for image recovery, connecting the complementary fusion features into the decoder by adopting a strategy of a skip connection, and obtaining a crack segmentation result. The complementary nature of the convolutional neural network and the transformer is utilized, the complementary fusion features and the simple fusion features are sent into the decoder for decoding together, and finally, a complete crack segmentation result is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of road quality and safety monitoring, and more particularly to a road crack segmentation method based on a convolutional neural network and a transformer dual path. BACKGROUND

[0002] Urban road pavement is easily structurally damaged by heavy truck rolling or other natural factors. Road cracks are the most common damage. Once a road crack occurs, if it is not repaired in time, it will gradually spread and produce larger and wider cracks. Cracks are also a key factor affecting the service life of a road. At present, in most parts of the world and countries, the detection of cracks is still carried out by professional technicians through manual detection. This detection method is undoubtedly time-consuming and laborious, requires a large amount of funds, and has a strong subjective factor, which is easy to cause missed detection and false detection. With the development of machine learning and deep learning, many excellent algorithms have been proposed for crack detection to realize automatic crack detection, save a large amount of manpower and resources, improve detection efficiency, and have higher detection accuracy and fewer subjective factors.

[0003] When we use traditional methods such as the crack detection method based on Gabor filtering proposed by Salman et al., the Sobel filter is used to remove noise in the gray image and the OTSU method is used to complete crack detection by Talab et al., and the method of Xu et al. first performs gray correction on road crack grouping, then calculates the saliency value of the crack at a coarse scale based on the gray sparsity and overall contrast to detect the crack. The method has poor robustness and insufficient consideration of the continuity and integrity of the crack, and thus has low detection accuracy and efficiency, and the effect is not ideal in practical application.

[0004] Therefore, it is urgent to propose a road crack segmentation method based on a convolutional neural network and a transformer dual path to solve the problems. SUMMARY

[0005] (I) Technical problem to be solved

[0006] Therefore, the present application provides a road crack segmentation method based on a convolutional neural network and a transformer dual path, which is more advantageous for segmentation of small cracks, can more completely segment cracks, improves the performance of segmentation, and effectively improves the detection accuracy and efficiency.

[0007] (II) Technical solution

[0008] To solve the above technical problems, the present application provides a road crack segmentation method based on a convolutional neural network and a transformer dual path, which includes the following steps:

[0009] Step S100, load the data set of the road crack image based on the convolutional neural network and the transformer dual path, train the dual path network;

[0010] Step S200, feature extraction is performed on the initial input image based on the convolutional neural network, feature maps extracted by each convolutional layer and a final feature map extracted by the convolutional neural network encoder through a deep separable convolution are obtained; wherein the convolutional neural network comprises a convolutional neural network encoder, the convolutional neural network encoder comprises a plurality of convolutional layers and a deep separable convolution;

[0011] Step S300, feature extraction is performed on the initial input image based on the transformer, and a feature map extracted by a transformer block in each feature extraction layer is obtained; wherein the transformer comprises a transformer encoder, the transformer encoder comprises a plurality of feature extraction layers, a transformer block is arranged in each feature extraction layer, and a feature map extracted by the transformer block in the last feature extraction layer of the transformer encoder is a final feature map extracted by the transformer encoder;

[0012] Step S400, the feature maps extracted by the intermediate convolutional layers of the convolutional neural network encoder are matched and fused with the feature maps extracted by the transformer blocks in the intermediate feature extraction layers of the transformer encoder, respectively, to obtain complementary fusion features;

[0013] Step S500, the final feature map extracted by the convolutional neural network encoder is matched and fused with the final feature map extracted by the transformer encoder, to obtain simple fusion features;

[0014] Step S600, the simple fusion features are input into the decoder for image restoration, the complementary fusion features are connected and input into the decoder by adopting the strategy of the skip connection, and a crack segmentation result is obtained.

[0015] (Three) beneficial effects

[0016] Compared with the prior art, the road crack segmentation method based on the convolutional neural network and the transformer dual path utilizes the complementarity of the convolutional neural network and the transformer, adopts the intermediate layers of the convolutional neural network encoder and the intermediate layers of the transformer encoder to respectively extract feature maps into a complementary fusion module for deep fusion, cooperates with simple fusion of the final feature map extracted by the convolutional neural network encoder and the final feature map extracted by the transformer encoder, and sends them into the decoder for decoding, so that a complete crack segmentation result is finally obtained, which is more favorable for segmentation of small cracks, can more completely segment cracks, improves the segmentation performance, and effectively improves the detection accuracy and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0017] The features and advantages of the present application will be appreciated upon reference to the following detailed description and drawings, in which:

[0018] Figure 1 is a flowchart of a road crack segmentation method based on a convolutional neural network and a transformer dual path according to a preferred embodiment of the present application;

[0019] Figure 2 is a schematic diagram of the overall framework of the road crack segmentation method based on the convolutional neural network and the transformer dual path according to a preferred embodiment of the present application;

[0020] Figure 3 is a specific structure of each convolutional layer in the convolutional neural network encoder according to a preferred embodiment of the present application, wherein the variable R represents the number of convolutional blocks in the residual block of each convolutional layer;

[0021] Figure 4 is a structure diagram of the transformer block in the transformer encoder and decoder according to a preferred embodiment of the present application;

[0022] Figure 5 is a structure diagram of the full convolution high-low frequency attention in the transformer block according to a preferred embodiment of the present application;

[0023] Figure 6 is a structure diagram of the feedforward network in the transformer block according to a preferred embodiment of the present application;

[0024] Figure 7 is a structure diagram of the complementary fusion module according to a preferred embodiment of the present application;

[0025] Figure 8 is a structure diagram of the simple fusion module according to a preferred embodiment of the present application;

[0026] Figure 9 is an example of the feature map extracted by each intermediate layer of the convolutional neural network encoder according to a preferred embodiment of the present application;

[0027] Figure 10 is an example of the feature map extracted by each intermediate layer of the transformer encoder according to a preferred embodiment of the present application;

[0028] Figure 11 is an example of the complementary fusion feature generated by the complementary fusion module according to a preferred embodiment of the present application, wherein layer 4 represents the feature map after simple fusion;

[0029] Figure 12 is a loss change curve when training on the DeepCrack dataset when different loss functions are used according to a preferred embodiment of the present application;

[0030] Figure 13 is the accuracy curve of the preferred embodiment of the present application disclosed on the DeepCrack dataset compared with other methods in the training stage;

[0031] Figure 14 is the precision curve of the preferred embodiment of the present application disclosed on the DeepCrack dataset compared with other methods in the training stage;

[0032] Figure 15 is the recall curve of the preferred embodiment of the present application disclosed on the DeepCrack dataset compared with other methods in the training stage;

[0033] Figure 16 is the F1 score curve of the preferred embodiment of the present application disclosed on the DeepCrack dataset compared with other methods in the training stage;

[0034] Figure 17 is the loss value curve of the preferred embodiment of the present application disclosed on the DeepCrack dataset compared with other methods in the training stage;

[0035] Figure 18 is an example of the crack segmentation effect of the three different structures of the convolutional neural network encoder, the transformer encoder and the double-path encoder adopted by the present application disclosed by the preferred embodiment of the present application;

[0036] Figure 19 is an example of the crack segmentation effect of the present application compared with other methods disclosed by the preferred embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below. In the following description, a lot of specific details are set forth in order to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0038] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "connected", "connected" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected, it can be mechanically connected, or it can be electrically connected, it can be directly connected, or it can be indirectly connected through an intermediate medium, it can be the communication inside two elements, or it can be "driven connection", that is, power connection through various suitable ways such as belt drive, gear drive or chain wheel drive. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0039] Figure 1 is a flow diagram of a road crack segmentation method based on convolutional neural network and transformer dual path provided by an embodiment of the present application, as shown in Figure 1 The road crack segmentation method based on convolutional neural network and transformer dual path includes steps S100-S600, and the details are as follows:

[0040] Step S100, load the data set of road crack image based on convolutional neural network and transformer dual path, and train the dual path network; wherein the dual path network includes the network corresponding to the neural network and the transformer, the data set includes the training set and the test set, the data set can be obtained through the known road crack image data set, or can be obtained through the newly established data set, and the newly established data set can be obtained by using the camera to take pictures of the road surface.

[0041] In this embodiment, the data set image obtained can be subjected to data enhancement operation, specifically, the method for performing data enhancement operation on the obtained data set image is: sequentially performing size transformation, random horizontal direction flip, random vertical direction flip, random rotation of 16° and center cropping to a preset size on the images in the training set, performing size transformation and center cropping to a preset size on the images in the training set can effectively expand the data set, also can weaken the data noise, increase the model stability, and can help the convolutional neural network to learn more local features.

[0042] In the method for loading the data set of road crack image based on convolutional neural network and transformer dual path and training the dual path network of the present application, the network is optimized by using Adam optimizer, the initial learning rate is set to 0.001, the scheduler with warm restart cosine annealing is selected to dynamically adjust the network learning rate, the initial learning rate is returned for the first time at the 15th batch, and then each regression will be expanded by 2 times based on the previous time, and the network is trained for a total of 80 batches, and the size of each batch is set to 4.

[0043] In the method for training the dual path network in step S100 of the present application, the loss function used is:

[0044] Loss(x, y) = BCE Loss(x, y) + Dice Loss(x, y);

[0045] wherein, BCE_Loss(x, y) = -(1-y)log(1-x)-ylog(x), x represents a predicted value, y represents a true value, and ε represents a smoothing factor, which prevents the denominator from being predicted as zero and also plays a role in smoothing the loss and gradient.

[0046] Specifically, taking a convolutional neural network as an example, the transformer is the same, in this embodiment, Binary Cross-Entropy loss (BCE_Loss) and Dice loss (Dice_Loss) are used as the loss function of the convolutional neural network to optimize the convolutional neural network in the training stage. Crack segmentation only needs to distinguish cracks and background, so it belongs to a binary classification problem, and BCE_Loss can be used as part of the loss function: BCE_Loss(x, y) = -(1-y)log(1-x)y log(x), x represents a predicted value, and y represents a true value; However, there is another problem in crack segmentation, i.e. imbalance between positive and negative samples, and in an image, the crack region and the background are extremely imbalanced, and the binary cross-entropy loss is to predict each pixel separately, and then take the average to get the prediction result of all pixels, which will cause the crack region to be disturbed by a large amount of background, which is not conducive to segmentation, and Dice_loss will make the convolutional neural network pay more attention to the crack region during the training process, effectively alleviating the problem caused by the imbalance between positive and negative samples. Therefore, in order to better exploit the foreground information and highlight the crack region, the Dice loss is added to the BCE_Loss: x represents a predicted value, y represents a true value, and ε represents a smoothing factor, which prevents the denominator from being predicted as zero and also plays a role in smoothing the loss and gradient.

[0047] Step S200, based on the convolutional neural network, feature extraction is performed on the initial input image, and the feature maps extracted by each convolutional layer and the feature maps finally extracted by the convolutional neural network encoder through a deep separable convolution A are obtained; wherein the convolutional neural network comprises a convolutional neural network encoder, and the convolutional neural network encoder comprises a plurality of convolutional layers and a deep separable convolution A, in this embodiment, the convolutional layer of each layer in the convolutional neural network encoder is as shown in the figure, and the convolutional neural network encoder comprises a convolutional layer 1, a convolutional layer 2, a convolutional layer 3, a convolutional layer 4 and a deep separable convolution A arranged in sequence. Figure 3

[0048] ​Specifically, the initial input image is input into the convolutional neural network encoder to extract image features of the initial input image. In the convolutional neural network encoder, the initial input image sequentially passes through four convolutional layers in the convolutional neural network encoder, i.e., a convolutional layer 1, a convolutional layer 2, a convolutional layer 3, and a convolutional layer 4. The convolutional layer 1, the convolutional layer 2, the convolutional layer 3, and the convolutional layer 4 gradually correspond to the extraction of feature maps of different scales. In the convolutional neural network encoder, after passing through each convolutional layer, the feature map extracted by the convolutional layer is down-sampled by two times, and the number of channels C is expanded by two times. The size of the feature map extracted by each convolutional layer is sequentially from the convolutional layer 1 to the convolutional layer 4. and H is the height of the feature image, and W is the width of the feature image. In the embodiment, the number of channels C is set to 32.

[0049] The convolutional layer in the convolutional neural network is provided with a convolution block. The convolution block is a basic unit, which is composed of a depth separable convolution, a batch normalization, and a GELU activation function. The use of the depth separable convolution has less parameter quantity and calculation amount than the standard convolution operation, but the feature extraction capability is not weakened at all. Therefore, the extraction of the features of the input image can be completed by stacking the convolution blocks. A plurality of stacked convolution blocks form a residual block in the convolutional layer.

[0050] In the embodiment, the method of the step S200 specifically includes the following steps.

[0051] In each convolutional layer of the convolutional neural network encoder, a convolution block is used to perform a rough feature extraction on the current input image, so that each position in the input image no longer has single information, but establishes a local interaction, so that each position in the input image has local texture and semantic information. The convolutional neural network encoder includes four convolutional layers, i.e., the convolutional layer 1, the convolutional layer 2, the convolutional layer 3, and the convolutional layer 4, and a depth separable convolution A connected in sequence. The current input image corresponding to the first convolutional layer 1 is the initial input image. The current input image corresponding to the second convolutional layer 2, the third convolutional layer 3, and the fourth convolutional layer 4 is the feature map output by the previous convolutional layer. The current input image corresponding to the depth separable convolution A is the feature map output by the fourth convolutional layer 4.

[0052] The down-sampling is completed by the depth separable convolution in each convolutional layer. Specifically, the depth separable convolution 1 corresponding to the convolutional layer 1 completes the down-sampling of the initial input image. The depth separable convolution 2 corresponding to the convolutional layer 2 completes the down-sampling of the feature map output by the convolutional layer 1. The depth separable convolution 3 corresponding to the convolutional layer 3 completes the down-sampling of the feature map output by the convolutional layer 2. The depth separable convolution 4 corresponding to the convolutional layer 4 completes the down-sampling of the feature map output by the convolutional layer 3.

[0053] The number of convolution blocks stacked in the residual block is R for feature extraction; wherein the first layer convolution layer 1, the second layer convolution layer 2, the third layer convolution layer 3, and the fourth layer convolution layer 4 correspond to the number of convolution blocks R in the residual block, respectively taking values of 2, 2, 4, and 2;

[0054] The features extracted by the last convolution block in the residual block are connected with the input of the convolution layer in the channel direction, and then the features are aggregated and reduced in dimension through a point-by-point convolution, to obtain the feature map extracted by each convolution layer of the convolutional neural network encoder; specifically, the convolution blocks in each convolution layer are stacked from top to bottom in the residual block, and the last convolution block in the residual block corresponds to the lowermost convolution block; the features extracted by the last convolution block in the convolution layer 1 are connected with the input of the convolution layer 1 in the channel direction, and then the features are aggregated and reduced in dimension through a point-by-point convolution, to obtain the feature map extracted by the convolution layer 1; the features extracted by the last convolution block in the convolution layer 2 are connected with the input of the convolution layer 2 in the channel direction, and then the features are aggregated and reduced in dimension through a point-by-point convolution, to obtain the feature map extracted by the convolution layer 2; the features extracted by the last convolution block in the convolution layer 3 are connected with the input of the convolution layer 3 in the channel direction, and then the features are aggregated and reduced in dimension through a point-by-point convolution, to obtain the feature map extracted by the convolution layer 3; the features extracted by the last convolution block in the convolution layer 4 are connected with the input of the convolution layer 4 in the channel direction, and then the features are aggregated and reduced in dimension through a point-by-point convolution, to obtain the feature map extracted by the convolution layer 4;

[0055] The feature map of the fourth layer convolution layer 4 is down-sampled through a depth separable convolution to obtain the final feature map extracted by the convolutional neural network encoder, to facilitate the fusion with the final feature map extracted by the transformer encoder.

[0056] Step S300, feature extraction is performed on the initial input image based on the transformer, and feature maps extracted by each transformer block are obtained; wherein the transformer includes a transformer encoder, the transformer encoder includes a plurality of feature extraction layers arranged in sequence, such as a first feature extraction layer, a second feature extraction layer, a third feature extraction layer and a fourth feature extraction layer, and a transformer block is arranged in each feature extraction layer; specifically, the first feature extraction layer includes a block embedding layer and a transformer block 1, the second feature extraction layer includes a block merging layer 1 and a transformer block 2, the third feature extraction layer includes a block merging layer 2 and a transformer block 3, the fourth feature extraction layer includes a block merging layer 3 and a transformer block 4, the transformer block 1 is between the block embedding layer and the block merging layer 1, the transformer block 2 is between the block merging layer 1 and the block merging layer 2, the transformer block 3 is between the block merging layer 2 and the block merging layer 3, the transformer block 4 is after the block merging layer 3, and the feature map extracted by the transformer block 4 in the last feature extraction layer of the transformer encoder is the feature map finally extracted by the transformer encoder; the initial input image is input into the transformer encoder, and the image feature extraction operation of each feature extraction layer in the transformer encoder is sequentially performed to obtain the feature maps extracted by the transformer block 1, the transformer block 2, the transformer block 3 and the transformer block 4, and the transformer blocks corresponding to each feature extraction layer in the transformer encoder are as shown in Figure 4

[0057] Specifically, the method of step S300 includes the following steps:

[0058] Step S310, divide the initial input image into a plurality of 4x4 pixel size image blocks through the block embedding layer, and obtain the divided initial input image; at this time, the initial input image will also be projected to a higher dimension, and the initial input image will extract a feature map with more sufficient semantic information in the transformer encoder;

[0059] ​Step S320, the current input image enters the first residual block in the transformer block, and the first image feature output by the first residual block in the transformer block is obtained by calculating the full convolution high-low frequency attention; wherein, each transformer block is mainly composed of two residual blocks of the first residual block and the second residual block, at this time, the current input image corresponding to the transformer block 1 is the initial input image divided, the current input image corresponding to the transformer block in the second to fourth feature extraction layers is the output image after the feature map extracted by the transformer block in the previous layer is processed by the block merging layer in the current feature extraction layer, that is, the current input image corresponding to the transformer block 2 in the second feature extraction layer is the output image after the feature map extracted by the transformer block 1 is processed by the block merging layer 1 in the second feature extraction layer, the current input image corresponding to the transformer block 3 in the third feature extraction layer is the output image after the feature map extracted by the transformer block 2 is processed by the block merging layer 2 in the third feature extraction layer, the current input image corresponding to the transformer block 4 in the fourth feature extraction layer is the output image after the feature map extracted by the transformer block 3 is processed by the block merging layer 3 in the fourth feature extraction layer, and the full convolution high-low frequency attention in the transformer block is as shown in Figure 5 .

[0060] Step S330, the first image feature output by the first residual block in the transformer block is taken as the input of the second residual block in the transformer block, and the second image feature output by the second residual block in the transformer block is obtained after the feedforward network and the batch normalization operation; the input feature is projected to a high-dimensional space by using the feedforward network, and then is changed back to a low-dimensional space, so as to learn more abstract features; the feedforward network in the transformer block is as shown in Figure 6 .

[0061] Step S340, the second image feature output by the second residual block in the transformer block is processed by the batch normalization, and is added with the first image feature output by the first residual block in the transformer block element by element, as the output image of the transformer block; wherein, when the current input image is the initial input image divided, the output image of the transformer block is the feature map extracted by the transformer block 1; when the current input image is the output image after the feature map extracted by the transformer block 1 is processed by the block merging layer 1 in the second feature extraction layer, the output image of the transformer block is the feature map extracted by the transformer block 2; when the current input image is the output image after the feature map extracted by the transformer block 2 is processed by the block merging layer 2 in the third feature extraction layer, the output image of the transformer block is the feature map extracted by the transformer block 3; when the current input image is the output image after the feature map extracted by the transformer block 3 is processed by the block merging layer 3 in the fourth feature extraction layer, the output image of the transformer block is the feature map extracted by the transformer block 4;

[0062] Step S350, the feature map extracted by the transformer block 1 is preprocessed by using the deep separable convolution and the batch normalization, the image blocks in the feature map extracted by the transformer block 1 are merged two by two through the block merging layer 1, the scale conversion of the feature map extracted by the transformer block 1 is completed, the feature extraction is performed on the scale-converted feature map extracted by the transformer block 1 through the transformer block 2, and the feature map extracted by the transformer block 2 is obtained; wherein the transformer encoder includes a plurality of feature extraction layers arranged in sequence, such as a first feature extraction layer, a second feature extraction layer, a third feature extraction layer and a fourth feature extraction layer, and the transformer block is arranged in each feature extraction layer; specifically, the first feature extraction layer includes the block embedding layer and the transformer block 1, the second feature extraction layer includes the block merging layer 1 and the transformer block 2, the third feature extraction layer includes the block merging layer 2 and the transformer block 3, and the fourth feature extraction layer includes the block merging layer 3 and the transformer block 4; the feature extraction is performed on the scale-converted feature map extracted by the transformer block 1 through the transformer block 2, and the feature map extracted by the transformer block 2 is obtained, and the method can be obtained through steps S320-S340, and the current input image is the scale-converted feature map extracted by the transformer block 1.

[0063] Step S360, the feature map extracted by the transformer block 2 is preprocessed by using the deep separable convolution and the batch normalization, the image blocks in the feature map extracted by the transformer block 2 are merged two by two through the block merging layer 2, the scale conversion of the feature map extracted by the transformer block 2 is completed, the feature extraction is performed on the scale-converted feature map extracted by the transformer block 2 through the transformer block 3, and the feature map extracted by the transformer block 3 is obtained; wherein the feature extraction is performed on the scale-converted feature map extracted by the transformer block 2 through the transformer block 3, and the feature map extracted by the transformer block 3 is obtained, and the method can be obtained through steps S320-S340, and the current input image is the scale-converted feature map extracted by the transformer block 2.

[0064] Step S370, the feature map extracted by the transformer block 3 is preprocessed by using the deep separable convolution and the batch normalization, the image blocks in the feature map extracted by the transformer block 3 are merged two by two through the block merging layer 3, the scale conversion of the feature map extracted by the transformer block 3 is completed, the feature extraction is performed on the scale-converted feature map extracted by the transformer block 3 through the transformer block 4, and the feature map extracted by the transformer block 4 is obtained; wherein the feature extraction is performed on the scale-converted feature map extracted by the transformer block 3 through the transformer block 4, and the feature map extracted by the transformer block 4 is obtained, and the method can be obtained through steps S320-S340, and the current input image is the scale-converted feature map extracted by the transformer block 3.

[0065] In one embodiment, the step S320, the first residual block in the transformer block into which the current input image enters, is obtained by a method for calculating full convolution high-low frequency attention to obtain a first image feature output by the first residual block in the transformer block, and specifically includes the following steps:

[0066] The current input image X is divided in the channel direction according to the proportion a, the high-frequency branch obtains (1-a)N h , and the low-frequency branch obtains aN h , N h is the number of heads, and self-attention is calculated for the high-frequency branch and the low-frequency branch respectively.

[0067] In the high-frequency branch, the current input image X obtains queries Q, keys K and values V of each head through a depth separable convolution: Q=C q (X), K=C k (X), and V=C v (X), and the queries Q, the keys K and the values V are calculated through scaled dot-product attention:

[0068]

[0069]

[0070] Wherein, C q , C k , and C v respectively represent the depth separable convolution for obtaining the queries Q, the keys K and the values V; D h is the hidden layer dimension, SA h represents the result of scaled dot-product self-attention of one head, MSA represents that the calculation results of multiple heads are connected in the channel direction, and a high-frequency feature is obtained through a depth separable convolution aggregation; the number of groups of the queries Q, the keys K and the values V is the number of heads (1-a)N h assigned to the high-frequency branch.

[0071] In the low-frequency branch, the current input image X is first obtained through a global average pooling operation to obtain a low-frequency component, the low-frequency component is obtained through a depth separable convolution to obtain keys K' and values V' of multiple heads, and queries Q' are directly obtained from the current input image X through a depth separable convolution; the queries Q', the keys K' and the values V' are calculated through scaled dot-product attention, and then the calculation results of multiple heads are connected in the channel direction, and a low-frequency feature is obtained through a depth separable convolution aggregation, and the number of groups of the queries Q', the keys K' and the values V' is the number of heads ΔN h; wherein the depth separable convolution in the low frequency branch to obtain the query Q', the key K' and the value V' is the same as the depth separable convolution in the high frequency branch to obtain the query Q, the key K and the value V; the formula for calculating the query Q', the key K' and the value V' in the low frequency branch through the scaled dot-product attention is the same as the formula for calculating the query Q, the key K and the value V in the high frequency branch through the scaled dot-product attention;

[0072] The obtained high frequency feature and low frequency feature are spliced in the channel direction to extract the high and low frequency features of the current input image X.

[0073] The current input image X and the high and low frequency features after batch normalization are element-wise added to serve as the first image feature output by the first residual block in the transformer block.

[0074] In one embodiment, the step S330, taking the first image feature output by the first residual block in the transformer block as the input of the second residual block in the transformer block, and obtaining the second image feature output by the second residual block in the transformer block through the feedforward network and the batch normalization operation, is specifically operated as follows:

[0075] The input is first upgraded through a point-by-point convolution, then nonlinearly transformed through a GELU activation function, then the features in the high-dimensional space are learned through a depth separable convolution, again through a GELU activation function and a Dropout layer, and finally through a point-by-point convolution to return to the original dimension to obtain the feature image obtained through the feedforward network. After the feature image obtained through the feedforward network is batch-normalized, the second image feature output by the second residual block in the transformer block is obtained.

[0076] Step S400, match and fuse the feature maps extracted by the intermediate convolution layers of the convolutional neural network encoder with the feature maps extracted by the transformer blocks in the intermediate feature extraction layers of the transformer encoder to obtain complementary fusion features. Specifically, match and fuse the feature maps extracted by the second convolution layer 2 of the convolutional neural network encoder with the feature maps extracted by the transformer block 1 in the first feature extraction layer of the transformer encoder to obtain the first complementary fusion feature. Match and fuse the feature maps extracted by the third convolution layer 3 of the convolutional neural network encoder with the feature maps extracted by the transformer block 2 in the second feature extraction layer of the transformer encoder to obtain the second complementary fusion feature. Match and fuse the feature maps extracted by the fourth convolution layer 4 of the convolutional neural network encoder with the feature maps extracted by the transformer block 3 in the third feature extraction layer of the transformer encoder to obtain the third complementary fusion feature. Fuse the features generated by the intermediate layers of the convolutional neural network encoder and the transformer encoder to enhance the crack information carried by them.

[0077] The method of step S400 is specifically operated as follows:

[0078] The feature map x extracted by the middle convolutional layer of the convolutional neural network encoder i The feature map y extracted by the transformer block in the middle feature extraction layer of the transformer encoder i The feature map x of the convolutional neural network encoder part input into the complementary fusion module i And the feature map y of the transformer encoder part i First, the channel attention adjusts the weight of each channel, so that the proportion of channels beneficial to segmentation is larger, and the influence of channels without benefit and redundancy on segmentation is reduced; the two part feature maps after the channel attention The correlation enhanced feature map is obtained by matrix multiplication To enhance the correlation of the two part feature maps; the two part feature maps after the channel attention are connected in the channel direction And the correlation enhanced feature map Preliminary fusion is performed to obtain the preliminary fusion feature map f i The preliminary fusion feature map f i Is aggregated by a residual convolution module to obtain a complementary fusion feature; the complementary fusion module is as shown in Figure 7 In this embodiment, the complementary fusion module is set to three, which are complementary fusion module 1, complementary fusion module 2 and complementary fusion module 3. The feature map extracted by the convolutional layer 2 in the convolutional neural network encoder is matched with the feature map extracted by the transformer block 1 in the first feature extraction layer in the transformer encoder and input into the complementary fusion module 1. The feature map extracted by the convolutional layer 3 in the convolutional neural network encoder is matched with the feature map extracted by the transformer block 2 in the second feature extraction layer in the transformer encoder and input into the complementary fusion module 2. The feature map extracted by the convolutional layer 4 in the convolutional neural network encoder is matched with the feature map extracted by the transformer block 3 in the third feature extraction layer in the transformer encoder and input into the complementary fusion module 3.

[0079] Further, the method of aggregating the preliminary fusion feature map by a residual convolution module to obtain a complementary fusion feature, specifically operates as follows:

[0080] The preliminary fusion feature map is first reduced in dimension by a combination of an inverted deep separable convolution, batch normalization and a GELU activation function in the residual convolution module to obtain a reduced feature map, and then the reduced feature map is aggregated by two combinations of a deep separable convolution, batch normalization and a GELU activation function to obtain a complementary fusion feature; wherein the residual convolution module includes one combination 1 of an inverted deep separable convolution, batch normalization and a GELU activation function, and two combinations 2 of a deep separable convolution, batch normalization and a GELU activation function, the deep separable convolution, batch normalization and GELU activation function in combination 2 are sequentially connected in series, the inverted deep separable convolution, batch normalization and GELU activation function in combination 1 are sequentially connected in series, combination 1 and the two combinations 2 are sequentially connected in series, and a standard deep separable convolution first performs a depth convolution and then performs a point-by-point convolution, while an inverted deep separable convolution first performs a point-by-point convolution and then performs a depth convolution, which further reduces the amount of calculation when dimension reduction is required.

[0081] Step S500, match and fuse the feature map finally extracted by the convolutional neural network encoder and the feature map finally extracted by the transformer encoder to obtain simple fusion features.

[0082] The method of step S500 specifically operates as follows:

[0083] The feature map finally extracted by the convolutional neural network encoder and the feature map finally extracted by the transformer encoder are matched and input into a simple fusion module. For the feature map finally extracted by the convolutional neural network encoder and the feature map finally extracted by the transformer encoder input into the simple fusion module, the weights of each channel are adjusted through channel attention, and then the connected feature map is obtained by connecting in the channel direction. Then, an inverted deep separable convolution, batch normalization and a GELU activation function are used to aggregate and reduce the dimension of the connected feature map to obtain simple fusion features; wherein the inverted deep separable convolution, batch normalization and GELU activation function are sequentially connected in series, and the simple fusion module is as shown in Figure 8 .

[0084] Step S600, input the simple fusion features into the decoder for image restoration, and connect the complementary fusion features into the decoder by adopting a skip connection strategy to obtain a crack segmentation result; the complementary fusion features are transmitted into the decoder through a skip connection method, so that the decoder can obtain more information when restoring the image, so as to better restore the crack features.

[0085] The method of step S600 specifically operates as follows:

[0086] The sub-pixel convolution in the decoder is used to up-sample the input simple fusion feature to obtain a feature up-sampled by the sub-pixel convolution. It does not need to expand the size of the input as linear interpolation does, and good results can be achieved using a very small convolution kernel, which greatly reduces the parameter quantity and the calculation amount. Moreover, it is not simply zero-padded as transposed convolution is, but uses pixel recombination, that is, the recovered image contains richer and more accurate information by using the features of each channel. Then each up-sampling operation is up-sampled by two times. The decoder mainly consists of sub-pixel convolution, inverted deep separable convolution and transformer block. Specifically, the decoder includes sub-pixel convolution 1, three combinations 3 composed of inverted separable convolution, transformer block and sub-pixel convolution, combination 4, combination 5 and inverted separable convolution 1. The sub-pixel convolution, combination 3, combination 4, combination 5 and inverted separable convolution are sequentially connected in series. The inverted separable convolution, transformer block and sub-pixel convolution in combination 3, combination 4 and combination 5 are sequentially connected in series. Further, combination 3 includes inverted separable convolution 2, transformer block 5 and sub-pixel convolution 2 connected in series, combination 4 includes inverted separable convolution 3, transformer block 6 and sub-pixel convolution 3 connected in series, and combination 5 includes inverted separable convolution 4, transformer block 7 and sub-pixel convolution 4 connected in series. In this embodiment, the sub-pixel convolution 1 in the decoder is used to up-sample the input simple fusion feature to obtain a feature up-sampled by the sub-pixel convolution 1.

[0087] The complementary fusion feature corresponding to the layer of the receiving encoder is connected with the feature up-sampled by the sub-pixel convolution in the channel direction by using the strategy of jump connection. The complementary fusion feature, the feature up-sampled by the sub-pixel convolution and the feature down-sampled by the inverted deep separable convolution are fused and down-sampled to obtain a down-sampled feature. Then the down-sampled feature is decoded by the transformer block and up-sampled by the matched sub-pixel convolution. This cycle is repeated three times to obtain the final feature up-sampled by the last sub-pixel convolution. In this embodiment, when the complementary fusion feature is the third complementary fusion feature, the third complementary fusion feature is connected with the feature up-sampled by the sub-pixel convolution 1 in the channel direction by using the strategy of jump connection. The third complementary fusion feature, the feature up-sampled by the sub-pixel convolution 1 and the feature down-sampled by the inverted deep separable convolution 2 are fused and down-sampled to obtain a down-sampled feature. Then the down-sampled feature is decoded by the transformer block 5 and up-sampled by the matched sub-pixel convolution 2 to obtain the feature up-sampled by the sub-pixel convolution 2. When the complementary fusion feature is the second complementary fusion feature, the second complementary fusion feature is connected with the feature up-sampled by the sub-pixel convolution 1 in the channel direction by using the strategy of jump connection. The second complementary fusion feature, the feature up-sampled by the sub-pixel convolution 1 and the feature down-sampled by the inverted deep separable convolution 3 are fused and down-sampled to obtain a down-sampled feature. Then the down-sampled feature is decoded by the transformer block 6 and up-sampled by the matched sub-pixel convolution 3 to obtain the feature up-sampled by the sub-pixel convolution 3. When the complementary fusion feature is the first complementary fusion feature, the first complementary fusion feature is connected with the feature up-sampled by the sub-pixel convolution 1 in the channel direction by using the strategy of jump connection. The first complementary fusion feature, the feature up-sampled by the sub-pixel convolution 1 and the feature down-sampled by the inverted deep separable convolution 4 are fused and down-sampled to obtain a down-sampled feature. Then the down-sampled feature is decoded by the transformer block 7 and up-sampled by the matched sub-pixel convolution 4 to obtain the final feature up-sampled by the sub-pixel convolution 4.

[0088] The final feature after upsampling of the last sub-pixel convolution is obtained by an inverted deep separable convolution to obtain a crack segmentation result; the final feature after upsampling of the sub-pixel convolution 4 is obtained by an inverted deep separable convolution 1 to obtain a crack segmentation result.

[0089] In this embodiment, as shown in Figure 2 For the entire network, a pyramid structure design is adopted to extract features of different scales in layers. The initial input image is an RGB image with a size of 3xHxW, and both decoders are divided into 4 layers. In the convolutional neural network encoder, the feature map is down-sampled by two times and the channel number is expanded by two times after each layer, and the size of the feature map from top to bottom is and In the transformer encoder, the initial input image is first passed through the block embedding layer, so that the size becomes 1 / 4 of the input image. Therefore, in order to make the sizes of the feature maps for complementary fusion consistent, the features extracted by the second layer of the convolutional neural network encoder are complementarily fused with the features extracted by the first layer of the transformer encoder, and so on. The features extracted by the last layer of the convolutional neural network encoder are changed in size again by a deep separable convolution, so as to be simply fused with the features extracted by the last layer of the transformer encoder. The simply fused features and the complementarily fused features are passed through the decoder to gradually restore the initial input image size, thereby obtaining a crack segmentation result with a size of 1xHxW; wherein the channel number C is set to 32, the height H of the image is set to 512, and the width W of the image is set to 512.

[0090] As shown in Figure 9 to Figure 11 As shown in

[0091] The road crack segmentation method based on the convolutional neural network and the transformer dual-path of the present application utilizes the complementarity of the convolutional neural network and the transformer, adopts the features extracted by the intermediate layers of the convolutional neural network encoder and the intermediate layers of the transformer encoder to enter the complementary fusion module for deep fusion, cooperates with the simple fusion of the features extracted by the last layer of the convolutional neural network encoder and the features extracted by the last layer of the transformer encoder, and sends them into the decoder for decoding to finally obtain a complete crack segmentation result. This is more conducive to the segmentation of small cracks, can more completely segment the cracks, improves the performance of segmentation, and effectively improves the detection accuracy and efficiency.

[0092] The application adopts accuracy (Acc), precision (Pre), recall (Re), F1 score (F1) and mean intersection over union (mIoU) as evaluation indexes,

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] Wherein, true positive (TP) represents that the real class of the pixel is crack, and it is correctly identified as crack; false positive (FP) means that the real class of the pixel is background, and it is wrongly identified as crack; false negative (FN) means that the real class of the pixel is crack, and it is wrongly identified as background. True negative (TN) represents that the real class of the pixel is background, and it is correctly identified as background.

[0099] To verify the rationality and effectiveness of the road crack segmentation method based on convolutional neural network and transformer dual path proposed in the application, six different loss functions commonly used for segmentation, Cross-Entropy loss (CE_Loss), BCE_Loss, Dice_Loss, Tversky loss (Tv_Loss), Focal loss (FL) and Generalized Focal loss (GFL) and two kinds of combined loss functions BCE_Loss+Dice_Loss and FL+Dice_Loss are applied to the application and tested on the DeepCrack dataset. The best one is selected. Figure 12 The loss value change curve of different loss functions in the training stage of DeepCrack dataset. Table 1 is the test result of the application using different loss functions on DeepCrack dataset.

[0100] Table 1

[0101]

[0102]

[0103] The application is tested on three different characteristic public road crack datasets DeepCrack, CrackForest and CrackTree260, and the effect of the application reaches the current advanced level. The comprehensive effect of the application method is higher than that of seven different advanced methods Unet, FCN16s, SegNet, RCF, HED, DeepCrack and Deeplabv3+ respectively, and the crack segmentation effect comparison is as shown in the following table 2. Figure 19 Figure 13 The accuracy transformation curve of the training stage of various methods on the DeepCrack dataset. Figure 14 The precision transformation curve of the training stage of various methods on the DeepCrack dataset. Figure 15 The recall transformation curve of the training stage of various methods on the DeepCrack dataset. Figure 16 The F1 score transformation curve of the training stage of various methods on the DeepCrack dataset. Figure 17 The loss value transformation curve of the training stage of various methods on the DeepCrack dataset. Table 2 is the test result of various methods on the DeepCrack dataset. Table 3 is the test result of various methods on the CrackForest dataset. Table 4 is the test result of various methods on the CrackTree260 dataset.

[0104] Table 2

[0105] Model Pre (%) Re (%) F1(％) mloU (%) Unet 87.71 72.29 79.26 83.32 FCN16s 88.81 71.88 79.45 81.33 SegNet 75.45 78.11 76.76 79.08 RCF 88.87 51.64 65.32 69.52 HED 82.97 64.40 72.51 75.36 DeepCrack 82.33 84.66 83.48 84.93 Deeplabv3+ 88.93 72.41 79.82 82.84 Proposed 89.31 82.33 85.68 87.04

[0106] Table 3

[0107] Model Pre (%) Re (%) F1(％) mloU (%) Unet 78.00 58.59 66.91 73.28 FCN16s 73.08 71.59 72.33 77.95 SegNet 76.23 66.13 70.28 76.05 RCF 53.31 74.76 62.24 69.42 HED 50.40 71.41 59.10 67.29 DeepCrack 67.61 72.02 69.75 73.45 Deeplabv3+ 71.88 66.07 68.85 73.41 Proposed 78.47 81.29 79.85 82.56

[0108] Table 4

[0109] Model Pre (%) Re (%) F1(％) mloU (%) Unet 86.64 54.95 67.25 76.85 FCN16s 79.44 35.38 48.96 67.34 SegNet 84.10 56.37 67.50 76.49 RCF 70.97 1.57 3.07 49.43 HED 81.77 17.34 28.62 57.15 DeepCrack 79.88 63.10 70.51 77.25 Deeplabv3+ 72.21 28.88 41.25 63.65 Proposed 80.55 70.76 75.34 80.43

[0110] In addition, we also carried out sufficient ablation experiments. Table 5 is the result of the ablation experiment. In order to prove the effectiveness of the complementary fusion idea of the application, we compared the performance of the double-path encoder structure adopted by the application with the performance of the convolutional neural network encoder, the transformer encoder and the double-path encoder structure not adopted. The crack segmentation effect comparison is as shown in the following table 5. Figure 18

[0111] Table 5

[0112] Model Pre (%) Re (%) F1(％) mloU (%) without CNN 90.17 78.50 83.94 85.54 Without Transformer 89.15 80.52 84.61 86.26 Proposed 89.31 82.33 85.68 87.04

[0113] ​​For the division of high and low frequencies in the full convolution in the application, multiple groups of parameters are tested, and the best combination is obtained. Table 6 is the experimental results when the high and low frequency division ratio a takes different values.

[0114] Table 6

[0115] α Pre (%) Re (%) F1(％) mloU (%) {0.4,0.5,0.3,0.2} 86.93 80.45 83.56 85.87 {0.4,0.4,0.4,0.4} 88.46 76.91 82.28 84.99 {0.5,0.5,0.5,0.5} 88.81 72.63 79.91 84.10 {0.4,0.3,0.2,0.1} 89.31 82.33 85.68 87.04 {0.5,0.4,0.3,0.2} 88.89 77.66 82.89 85.48

[0116] In order to verify that the complementary fusion module used in the application is effective, we also carried out ablation experiments. Table 7 is the ablation experiment results of the complementary fusion module.

[0117] Table 7

[0118] CFM1 CFM2 CFM3 Pre (%) Re (%) F1(％) mloU (%) 87.45 77.43 82.14 84.16 √ 91.1 75.91 82.81 85.11 √ 88.93 76.7 82.36 84.76 √ 88.61 74.2 80.77 83.74 √ √ 88.46 77.38 82.55 84.9 √ √ 90.65 73.22 81.01 83.92 √ √ 89.67 78.11 83.49 85.7 √ √ √ 89.31 82.33 85.68 87.04

[0119] In order to better verify the generalization and adaptability of the application, we carried out cross-dataset verification. The training and testing use different datasets, and the application also shows good performance. Table 8 is the results of cross-dataset verification.

[0120] Table 8

[0121]

[0122]

[0123] In summary, the road crack segmentation method based on convolutional neural network and transformer dual path in the application utilizes the complementarity of convolutional neural network and transformer, adopts the intermediate layers of the convolutional neural network encoder and the intermediate layers of the transformer encoder to extract feature maps into the complementary fusion module for deep fusion, and cooperates with the simple fusion of the feature maps finally extracted by the convolutional neural network encoder and the feature maps finally extracted by the transformer encoder, and then sends them into the decoder for decoding, to finally obtain the restored crack segmentation result. It is more beneficial for the segmentation of small cracks, can more completely segment the cracks, improves the segmentation performance, and effectively improves the detection accuracy and efficiency.

[0124] Although the embodiments of the application are described in conjunction with the drawings, various modifications and variations can be made by those skilled in the art without departing from the spirit and scope of the application, and such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A road crack segmentation method based on a convolutional neural network and a transformer dual-path, characterized in that, Comprising the following steps: Step S100, load the data set of the road crack image based on the convolutional neural network and the transformer dual path, train the dual path network; Step S200, feature extraction is performed on the initial input image based on the convolutional neural network, feature maps extracted by each convolutional layer and a final feature map extracted by the convolutional neural network encoder through a deep separable convolution are obtained; wherein the convolutional neural network comprises a convolutional neural network encoder, the convolutional neural network encoder comprises a plurality of convolutional layers and a deep separable convolution; Step S300, feature extraction is performed on the initial input image based on the transformer, and a feature map extracted by each transformer block is obtained; wherein the transformer comprises a transformer encoder, the transformer encoder comprises a plurality of feature extraction layers, each feature extraction layer is provided with a transformer block, and a feature map extracted by the transformer block in the last feature extraction layer of the transformer encoder is a final feature map extracted by the transformer encoder; Wherein each transformer block is mainly composed of two residual blocks, i.e. a first residual block and a second residual block, the current input image enters the first residual block in the transformer block, and a first image feature output by the first residual block in the transformer block is obtained by calculating full convolution high-low frequency attention; comprising the following steps: The current input image X is divided in the channel direction by a proportion a, the high-frequency branch will get (1-a)N h , the low-frequency branch will get aN h , N h is the number of heads, and the self-attention is calculated for the high-frequency branch and the low-frequency branch respectively; In the high-frequency branch, the current input image X is obtained through a depth separable convolution to obtain the query Q, the key K and the value V of each head: Q = C q (X), K = C k (X), V = C v (X), the query Q, the key K and the value V are calculated through the scaled dot-product attention: wherein C q , C k , C v respectively represent a depth separable convolution obtained by querying Q, key K and value V; D h is a hidden layer dimension, SA h represents a result of a scaled dot-product self-attention of one head, MSA represents connecting the calculation results of multiple heads in the channel direction, and a high-frequency feature is obtained through a depth separable convolution aggregation; the group number of the query Q, the key K and the value V is the number of heads allocated to the high-frequency branch (1- a)N h ; In the low-frequency branch, the current input image X is first subjected to a global average pooling operation to obtain its low-frequency component, and the low-frequency component is subjected to a depth separable convolution to obtain keys K' and values V' of multiple heads, and a query Q' is obtained by directly subjecting the current input image X to a depth separable convolution; the query Q', the keys K' and the values V' are calculated through scaled dot-product attention, and then the calculation results of the multiple heads are connected in the channel direction, and a low-frequency feature is obtained through a depth separable convolution aggregation; the number of groups of the query Q', the keys K' and the values V' is the number of heads αN assigned to the low-frequency branch h ​ The obtained high frequency feature and low frequency feature are spliced in the channel direction to extract the high-low frequency feature of the current input image X; The current input image X and the high-low frequency feature after batch normalization are added element by element, and the first image feature output by the first residual block in the transformer block is obtained; Step S400, the feature maps extracted by the intermediate convolutional layers of the convolutional neural network encoder are matched and fused with the feature maps extracted by the transformer blocks in the intermediate feature extraction layers of the transformer encoder, and complementary fusion features are obtained; The method of step S400 is specifically operated as follows: The feature maps extracted by the intermediate convolutional layers of the convolutional neural network encoder and the feature maps extracted by the transformer blocks in the intermediate feature extraction layers of the transformer encoder are matched and input into the complementary fusion module. For the feature maps of the convolutional neural network encoder part and the feature maps of the transformer encoder part input into the complementary fusion module, first, adjust the weight of each channel through channel attention, then perform matrix multiplication operation on the two parts of feature maps after channel attention to obtain correlation enhanced feature maps, and finally, connect the two parts of feature maps after channel attention with the correlation enhanced feature maps in the channel direction to obtain preliminary fusion feature maps, and then aggregate the preliminary fusion feature maps through a residual convolution module to obtain complementary fusion features; Step S500, the final feature map extracted by the convolutional neural network encoder is matched and fused with the final feature map extracted by the transformer encoder, and simple fusion features are obtained; The method of step S500 is specifically operated as follows: The feature map finally extracted by the convolutional neural network encoder is matched with the feature map finally extracted by the transformer encoder and input into a simple fusion module, for the feature map finally extracted by the convolutional neural network encoder and the feature map finally extracted by the transformer encoder input into the simple fusion module, the weights of each channel are adjusted through channel attention, then the connected feature map is obtained by connecting in the channel direction, and then the connected feature map is aggregated and reduced in dimension by using an inverted deep separable convolution, batch normalization and a GELU activation function, to obtain simple fusion features; In step S600, the simple fusion features are input into a decoder for image restoration, and the complementary fusion features are connected and input into the decoder by adopting a strategy of a skip connection, to obtain a crack segmentation result. 2.The road crack segmentation method based on a convolutional neural network and a transformer dual-path according to claim 1, wherein, In the method for training the dual-path network in step S100, a loss function used is: Loss(x, y) = BCE Loss(x, y) + Dice Loss(x, y); where BCE_Loss(x, y) = -(1 - y)log(l - x) - ylog(x), x represents a predicted value, y represents a true value, and ε represents a smoothing factor. 3.The road crack segmentation method based on convolutional neural network and transformer dual-path according to claim 1, characterized in that, The method in step S200 specifically includes the following steps: In each convolutional layer of the convolutional neural network encoder, a convolutional block is used to perform rough feature extraction on the current input image, wherein the convolutional neural network encoder includes four convolutional layers, i.e., a convolutional layer 1, a convolutional layer 2, a convolutional layer 3 and a convolutional layer 4, and a deep separable convolution; Down-sampling is completed through the deep separable convolution in each convolutional layer; Different numbers of convolutional blocks are stacked in the residual block to perform feature extraction, and the number is R; The features extracted by the last convolutional block in the residual block are connected with the input of the convolutional layer in the channel direction, and the features are aggregated and reduced in dimension again through a point-by-point convolution, to obtain the feature map extracted by each convolutional layer of the convolutional neural network encoder; The feature map of the fourth convolutional layer 4 is down-sampled through a deep separable convolution to obtain the feature map finally extracted by the convolutional neural network encoder. 4.The road crack segmentation method based on convolutional neural network and transformer dual-path according to claim 1, wherein, The method in step S300 includes the following steps: In step S310, the initial input image is divided into a plurality of image blocks with a size of 4x4 pixels through a block embedding layer, to obtain the divided initial input image; In step S320, the current input image enters a first residual block in a transformer block, and a first image feature output by the first residual block in the transformer block is obtained by calculating a fully convolutional high-low frequency attention; each transformer block mainly includes two residual blocks, i.e., the first residual block and a second residual block; In step S330, the first image feature output by the first residual block in the transformer block is taken as the input of the second residual block in the transformer block, and a second image feature output by the second residual block in the transformer block is obtained after a feedforward network and a batch normalization operation; In step S340, the second image feature output by the second residual block in the transformer block is added element by element with the first image feature output by the first residual block in the transformer block after batch normalization processing, to obtain an output image of the transformer block; when the current input image is the divided initial input image, the output image of the transformer block is the feature map extracted by the transformer block 1. In step S350, the feature map extracted by the transformer block 1 is preprocessed by using the deep separable convolution and the batch normalization, the image blocks in the feature map extracted by the transformer block 1 are merged two by two through the block merging layer 1, the scale conversion of the feature map extracted by the transformer block 1 is completed, the feature map extracted by the transformer block 1 after the scale conversion is subjected to feature extraction through the transformer block 2, and the feature map extracted by the transformer block 2 is obtained; wherein the transformer encoder comprises a first feature extraction layer, a second feature extraction layer, a third feature extraction layer and a fourth feature extraction layer, the first feature extraction layer comprises the block embedding layer and the transformer block 1, the second feature extraction layer comprises the block merging layer 1 and the transformer block 2, the third feature extraction layer comprises the block merging layer 2 and the transformer block 3, and the fourth feature extraction layer comprises the block merging layer 3 and the transformer block 4. In step S360, the feature map extracted by the transformer block 2 is preprocessed by using the deep separable convolution and the batch normalization, the image blocks in the feature map extracted by the transformer block 2 are merged two by two through the block merging layer 2, the scale conversion of the feature map extracted by the transformer block 2 is completed, the feature map extracted by the transformer block 2 after the scale conversion is subjected to feature extraction through the transformer block 3, and the feature map extracted by the transformer block 3 is obtained. In step S370, the feature map extracted by the transformer block 3 is preprocessed by using the deep separable convolution and the batch normalization, the image blocks in the feature map extracted by the transformer block 3 are merged two by two through the block merging layer 3, the scale conversion of the feature map extracted by the transformer block 3 is completed, the feature map extracted by the transformer block 3 after the scale conversion is subjected to feature extraction through the transformer block 4, and the feature map extracted by the transformer block 4 is obtained.

5. The road crack segmentation method based on a convolutional neural network and a transformer dual-path according to claim 4, characterized in that, In step S330, the first image feature output by the first residual block in the transformer block is taken as the input of the second residual block in the transformer block, and the second image feature output by the second residual block in the transformer block is obtained through the feedforward network and the batch normalization operation, and the method specifically operates as follows: The input is first subjected to a point-by-point convolution to increase the dimension, then subjected to a GELU activation function for nonlinear transformation, then subjected to a deep separable convolution to learn the features in the high-dimensional space, then subjected to a GELU activation function again and a Dropout layer, and finally subjected to a point-by-point convolution again to return to the original dimension to obtain the feature image obtained through the feedforward network. After the feature image obtained through the feedforward network is subjected to the batch normalization, the second image feature output by the second residual block in the transformer block is obtained. 6.The road crack segmentation method based on convolutional neural network and transformer dual-path according to claim 1, wherein, The method for aggregating the feature maps through a residual convolution module to obtain complementary fusion features specifically operates as follows: In the residual convolution module, the preliminary fusion feature map is first subjected to a combination of an inverted deep separable convolution, a batch normalization and a GELU activation function to reduce the dimension to obtain a reduced feature map, and then the reduced feature map is subjected to a combination of two deep separable convolutions, batch normalizations and GELU activation functions to aggregate the reduced feature map to obtain the complementary fusion features.

7. The road crack segmentation method based on a convolutional neural network and a transformer dual-path according to claim 1, characterized in that, The method in step S600 specifically operates as follows: The input simple fusion feature is up-sampled by sub-pixel convolution in the decoder to obtain a feature up-sampled by sub-pixel convolution; wherein the decoder mainly comprises sub-pixel convolution, inverse deep separable convolution and transformer block; The complementary fusion feature corresponding to the layer of the receiving encoder is connected with the feature up-sampled by sub-pixel convolution in the channel direction by adopting a strategy of jump connection, the complementary fusion feature and the feature up-sampled by sub-pixel convolution are de-fused by inverse deep separable convolution, and dimension reduction processing is performed to obtain a feature after dimension reduction, then the feature after dimension reduction is decoded by the transformer block and sent to the matched sub-pixel convolution for up-sampling, and the cycle is repeated three times to obtain a final feature up-sampled by the last sub-pixel convolution; The final feature up-sampled by the last sub-pixel convolution is obtained by an inverse deep separable convolution to obtain a crack segmentation result.

Images