Pavement detection method based on artificial intelligence and road inspection robot

Abstract

The invention discloses a road surface detection method based on artificial intelligence and a road inspection robot, and the method employs a self-designed RCDIT network model, proposes an implicit expression method and a multi-head cross attention mechanism in the network model, guarantees the expression integrity and continuity of image blocks, and improves the detection precision. Complex interaction of image blocks and position information can be effectively processed, and the integration capability and information relevance of the model to multi-modal data are enhanced. In addition, the introduced position feedforward network further improves the abstract ability of feature representation, and is helpful for more effectively processing complex features and improving the expressive force of the model. And the RCDIT network model is combined with the road detection robot, so that the crack defects of the road surface and the road can be accurately and quickly identified and segmented, and the reliability and the accuracy of detection are greatly improved. The road detection robot can work in a 24 / 7 all-weather mode and is not limited by time and environment, manpower resources are effectively saved, and continuous monitoring on the road surface can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of road inspection robot detection, and in particular to a road surface detection method based on artificial intelligence and a road inspection robot. Background Art

[0002] Urban roads are the backbone of urban transportation and the life channel for maintaining the safe operation of the city. However, in the process of road management and maintenance, more and more road problems continue to emerge due to factors such as load flow far exceeding the design standard, old construction materials, and the impact of new construction projects. One of the main problems of roads is road pavement cracking: long-term vehicle traffic and climate change have caused road cracks of various shapes and uneven distribution on the road surface, which not only affects the comfort of driving, but also easily leads to vehicle damage and traffic accidents. These road cracks may be caused by a variety of factors, such as foundation settlement, material aging, vehicle load, etc. The appearance of road cracks not only affects the appearance of the road, but more importantly, it may accelerate the wear and deformation of the road surface, thereby increasing maintenance costs and posing safety hazards. Especially in rainy and snowy weather or low visibility conditions, these road cracks are more likely to cause traffic accidents, posing serious safety hazards to driving. Therefore, timely inspection of road pavement conditions is particularly important.

[0003] There are some shortcomings in the current manual inspection method, which are the key points that artificial intelligence and road inspection technology are trying to solve. Specifically, there are several major problems with traditional manual inspection:

[0004] (1) High human resource costs: Traditional road inspections usually rely on human resources, requiring a large number of personnel and long manual inspections. This is not only costly, but also easily affected by human factors, such as differences in subjective judgment or fatigue that leads to reduced inspection quality.

[0005] (2) Inefficiency: Manual inspections are relatively inefficient, especially for large-scale road networks, which require a lot of time and resources to complete. This results in low inspection frequency, slow feedback, and failure to detect and solve road problems in a timely manner.

[0006] (3) Limited data processing and analysis capabilities: The data collected by traditional methods are usually large and scattered, and manual analysis and processing are inefficient. This leads to limitations in data-driven road maintenance and decision-making, and it is impossible to fully utilize advanced technologies such as data mining and predictive analysis.

[0007] (4) Subjectivity and consistency issues. Inspections that rely on manual labor are easily affected by individual subjective factors, resulting in insufficient consistency and objectivity in the evaluation results. This may affect the fairness and accuracy of road maintenance.

[0008] (5) Limited ability to cope with complex environments. Traditional manual inspections often face many difficulties and safety risks when dealing with complex weather conditions (such as bad weather), nighttime, or special environments such as highways, which limits their effectiveness under specific conditions.

[0009] When performing semantic segmentation, traditional transformer structure networks often directly input the spatial tensor represented by the image block as a token into the network's transformer encoder. However, this method has the disadvantage that it cannot well guarantee the continuity and integrity of the spatial tensor represented by the image block when representing the complete image.

[0010] Traditional Transformer modules have always had pain points in network structure in terms of multimodal data processing and feature learning. Traditional Transformer modules mainly rely on the self-attention mechanism. Although they have powerful modeling capabilities within sequences, they are limited in processing multimodal data and often require additional processing steps to integrate different data sources. In addition, the computational cost of traditional modules is high, especially for long sequences and large-scale data. Summary of the invention

[0011] In view of the shortcomings of existing manual road inspections, the present invention aims to design a road surface detection method based on artificial intelligence and a road inspection robot, with the aim of solving the problem that traditional road detection algorithms in the prior art are difficult to quickly detect a variety of road conditions on roads with complex and harsh environments, resulting in the inability to achieve the real-time monitoring effect required in the project.

[0012] In order to solve the above technical problems, the technical solution of the present invention is as follows: a road surface detection method based on artificial intelligence and a road inspection robot, characterized in that the method comprises the following steps:

[0013] Step 1: Get the training dataset

[0014] The camera mounted on the road inspection robot is used to shoot videos of multiple road surfaces, and the obtained video files are subjected to video frame extraction processing, in which the frames are extracted in the form of 24 frames per second, and the images obtained by the frame extraction are resampled, and the size of the road surface images obtained by the frame extraction is adjusted to 256*256 pixels. In this way, the entire video file is processed into a road surface image file;

[0015] The obtained road surface images are screened, wherein a certain number of cracked road surface images and non-cracked road surface images are screened, and images obtained after data enhancement processing of no less than 80% of the cracked road surface images and no less than 50% of the non-cracked road surface images are respectively performed as training sets; and the remaining cracked road surface images and the remaining non-cracked road surface images are respectively performed as test sets.

[0016] Step 2: Build a road surface detection network model

[0017] The road surface detection network model is an RCDIT network model, including a block processing module, a marker, an implicit expression module, an RCDIT module, a mask image modeling head, and an adversarial training decoder;

[0018] The block processing module divides a given input image x into N original image blocks. i is an integer from 1 to N; record each original image block The corresponding position information in the original image, the original image block position information of the input image x is recorded as P(p1,p2,…,p i ,…,p N ); then, the obtained original image blocks are randomly masked according to a certain ratio, and the original image blocks at the masked positions become masked image blocks; the masked positions are represented as M, M∈P;

[0019] The tagger is an image tokenizer obtained by pre-training DVAE, which directly tokenizes an input image x into N visual tokens, denoted as z = [z1, z2, …, z i …,z N ], z∈V; V is a dictionary, V=1,2,…,|V|,|V|=8192, V contains the index of all discrete tokens, each original image block Corresponds to a visual tokenz i , mapped to a word in |V|;

[0020] The original image block at the unmasked position of the input image x and the masked image block at the masked position are input into the implicit expression module to obtain the implicit tensor of each of the original image blocks and masked image blocks of the input image x; the implicit tensor of the input image x and the category label L of the input image x are concatenated as the token sequence of the image block of the input image x, and the token sequence of the image block and the original image block position information of the input image x are input into the RCDIT module together;

[0021] The original image blocks at the unmasked positions of the input image x and the masked image blocks at the masked positions are input into the implicit expression module, and processed by the coordinate representation method, that is, the grid coordinates of the potential two-dimensional space representation are drawn by the built-in grid_sampel method of pytorch. The normalized representation method concept is used to regard the potential grid coordinates as a [-1, 1] plane rectangular coordinate system, and each image block is assigned to the grid corresponding to the position in the potential grid coordinates. The assigned positions are kept in one-to-one correspondence with the corresponding positions in the original image when the image is cut into blocks. This method is used to make all pixels of all image blocks have corresponding two-dimensional coordinates belonging to this plane rectangular coordinate system, and thus there will be corresponding image pixel values ​​and the number of channels of this pixel.

[0022] In order to ensure the continuity of image pixels during implicit expression, a cell encoding method is introduced, which can be expressed as:

[0023]

[0024] Where s is the encoding representation obtained after the cell is encoded. is the target image block, c=[c h ,c w ], including the specified High C h 、Width c w , Indicates that and c for splicing;

[0025] Then the result obtained by the coordinate representation method is processed by a multi-layer perceptron, which is composed of a linear layer and a Leaky ReLU activation function layer; a 3×3 convolution layer is immediately followed by the multi-layer perceptron; the input data in this part is an image block with a height of 64×width of 64×channel number 3, so the parameters corresponding to the linear layer in this multi-layer perceptron structure are set to: input size 64×64, hidden layer size 256, and the output result is a tensor of 1×256; after being processed by the nonlinear activation function Leaky ReLU, it enters the convolution layer, and the output result is a tensor of 64×64×3; traverse all image blocks to obtain N implicit tensors of the same size, which are concatenated with a set category label L as the token sequence of the image block input to the RCDIT module;

[0026] The RCDIT module structure consists of a first normalization layer, a multi-head self-attention module, a second normalization layer, a multi-head cross-attention module, a third normalization layer, and a multi-layer perceptron; the token sequence of the image block input to the RCDIT module is first processed by the first normalization layer and the multi-head self-attention module in sequence, and the result is added to the token sequence of the above image block to obtain a first extracted feature; the result of the first extracted feature processed by the second normalization layer and the original image block position information of the input image x are input to the multi-head cross-attention module for processing to obtain a second extracted feature; the first extracted feature and the second extracted feature are weighted summed, and the obtained feature is processed by the third normalization layer and the multi-layer perceptron in sequence to obtain the output of the RCDIT module; the principles and structures of the first normalization layer, the second normalization layer, and the third normalization layer are the same;

[0027] The principle of the multi-head self-attention module is as follows:

[0028] When calculating the attention matrix, the multi-head self-attention mechanism splits the input tensor X into h sub-tensors, each of which is the attention information learned in a different way; for each sub-tensor, a self-attention calculation is performed to obtain an output tensor O i ; Finally, the h output tensors are concatenated together to obtain the final output tensor O. The calculation formula for multi-head self-attention is:

[0029]

[0030] In the above formula, Q i , K i 、V i are the i-th subtensor X of the input tensor X respectively. i Query, key, value when performing self-attention calculation, W qi , W ki , W vi Q i , K i 、V i The learnable parameter moment, d iK K i The dimension size is used to scale the size of attention; W O is the weight parameter matrix of linear projection;

[0031] The principle of the multi-head cross attention module is as follows: the result of the first extracted feature processed by the second normalization layer and the original image block position information of the input image x are input into the multi-head cross attention module, the above two parts are input spliced, and the splicing result is regarded as a whole as the input tensor of the multi-head cross attention mechanism; then the input tensor is evenly split into two parts, X1 and X2, according to the dimension. Then one of the parts is used as the query set and the other part as the key and value set; its output is a tensor of size n×d2, and for each row vector, its attention weight for all row vectors is given; specifically, let Q=X1W Q , K=V=X2W K , the output of this module is:

[0032]

[0033] in, Both are the learned projection matrices, and d2 is the dimension size of the key or value set;

[0034] The multi-layer perceptron of the RCDIT module is a position feedforward network, which consists of two linear layers and an activation function. The first linear layer of the position feedforward network is used as the data input terminal. The function of this linear layer is to map the input feature representation to a higher-dimensional space. After the first linear layer, a nonlinear activation function layer is connected. The nonlinear activation function is a Leaky ReLU activation function, which is used to introduce nonlinear transformation and enhance the expression ability of the model. After the nonlinear activation function layer, a second linear layer is connected. The second linear layer receives the output of the nonlinear activation function layer as input and maps the input features back to a space with the same feature dimension of the input image x. The output of the second linear layer is regarded as the final output of the RCDIT module.

[0035] The mask image modeling head consists of an FC layer and a Softmax function layer. The mask image modeling head generates predicted image block information for all positions of the input image x according to the final output of the RCDIT module. For the mask processing position, the association between the mask image block information generated by the mask image modeling head and the visual token of the original image block is calculated. The mask image block information with the maximum association value is the optimal mask image block information for the desired mask position.

[0036] The maximum correlation value is calculated as:

[0037]

[0038] Where D is the total image block information, M represents the location of the mask, and x M is the masked image patch, z i Indicates the visual token to be predicted and associated; p MIN (z i |x M ) represents the minimum conditional probability of the visual token that needs to be predicted to establish an association among all the conditions of the masked image blocks.

[0039] The mask image modeling head outputs the predicted image block information of all positions of the input image x, where the mask image block information of the mask position is the above-mentioned optimal mask image block information, and the overall image feature map m is obtained, and m is used as the input of the adversarial training decoder;

[0040] The adversarial training decoder includes an adversarial sample generator, a self-attention module, and a convolutional feedforward neural network; the output result of the mask modeling head is input into the adversarial training decoder, and is input into the adversarial sample generator and the self-attention module respectively;

[0041] For input m, the adversarial sample generator uses FGSM to create perturbation η, which is expressed as follows:

[0042]

[0043] Where m is the input, It means calculating the gradient of m, θ represents the parameters of the model; J represents the loss function of the model, which is the L1 loss function; ∈ is the threshold parameter for generating perturbations; sign() is the sign function, which returns 1 for positive numbers, -1 for negative numbers, and 0 for 0; the adversarial sample is generated by calculating the gradient of the L1 loss function of the output m of the mask modeling head for the corresponding visual token in the marker, and making a small perturbation to the generated prediction result in the direction of the gradient. Finally, the adversarial sample is obtained by adding the output m of the mask modeling head to the perturbation η. Adversarial Examples This is the output of the adversarial sample generator;

[0044] For the input m, the self-attention module processes it through a self-attention mechanism:

[0045]

[0046] Where Q', K', and V' are respectively q m、W k m and W v m is obtained, where W q , W k , W v is a learnable parameter matrix; Q' represents query, K' represents key, V' represents value, and the input in self-attention is the same, all are m; d k' is the scaling factor, which is the number of channels of Q; K' in the formula T Indicates that K' is transposed;

[0047] The output of the adversarial sample generator and the output of the self-attention module are concatenated, and the result is used as the input of a convolutional feedforward neural network. The output of the convolutional feedforward neural network is the output result generated by the adversarial training decoder. The convolutional feedforward neural network outputs a grayscale image of the input image x.

[0048] The convolutional feedforward neural network is composed of a convolutional layer, a sigmoid activation function layer, a fully connected layer, and a sigmoid activation function layer connected in sequence;

[0049] Step 3: Train the road detection network model

[0050] First, the road surface detection network model in step 2 is initialized with random assignment method to learn the parameters, and the training epoch is set to 300 rounds, and the initial learning rate is set to 1e -4 , the optimizer adopts Adam optimizer;

[0051] In the training phase, a road image in the training set in step 1 is input into the road detection network model, and the L1 loss value is calculated according to the result output by the network model and the result after the pixel value of the road image is normalized. The learnable parameters of the network model are updated once according to the L1 loss value using the Adam optimizer to complete the training of a road image; the learnable parameters when the training of the previous road image is completed are used as the initial learnable parameters when the next road image is trained, and the training of one epoch is completed; the learnable parameters when the training of the previous epoch is completed are used as the initial learnable parameters when the next epoch is trained, and the training of one epoch is completed; the learnable parameters when the training of the previous epoch is completed are used as the initial learnable parameters when the next epoch is trained, and the training of the epoch reaches the preset value; after each epoch of training is completed, the road images in the training set are respectively input into the network model, and the image pixel accuracy of the epoch is calculated according to all the results output by the network model and the result after the pixel value of the corresponding real sample is normalized; the learnable parameters of the network model with the smallest L1 loss function and the highest image pixel accuracy are saved, and the training of the road detection network model is completed;

[0052] In the testing phase, the road surface images in the test set in step 1 are respectively input into the trained road surface detection network model for network model testing. According to the normalized results of all the results output by the network model and the corresponding pixel values ​​of the road surface images, the image pixel accuracy, the average image pixel accuracy, and mIoU are calculated. When the values ​​of the image pixel accuracy, the average image pixel accuracy, and mIoU fall within the intervals of (88%, 89%), (92%, 93%), and (86%, 87%), respectively, and the value change is not greater than 0.03%, the network model test is optimal, that is, the trained road surface detection network model is the optimal network model. Otherwise, the initial learning rate, the initial epoch number, and the initial learnable parameters of the optimizer are adjusted, and the training process of the training set and the testing process of the test set are repeated until the network model is tested to be optimal, and the optimal network model is obtained.

[0053] Step 4: Detection of road surface cracks

[0054] Use the camera mounted on the road inspection robot in step 1 to shoot a video of the target road surface, and use the same method to extract frames for the video, and adjust the size of the road surface image to be detected obtained by extracting frames to 256*256; place a ruler in the road surface image to be detected, and the ruler should be on the same plane as the road cracks; then input each road surface image to be detected into the optimal network model in step 3, and the network model outputs a grayscale image of the road surface image to be detected;

[0055] Then, according to the grayscale image of the road surface image to be detected output by the network model, the road cracks are quantified, and the binary mask of the road cracks is obtained by the image segmentation algorithm, that is, the cracked part of the road is marked as the foreground, and the non-defective part is marked as the background; the number of pixels in the cracked part of the road is calculated using the pixel counting method, that is, the number of foreground pixels, represented by N; the number of pixels N is multiplied by the square of the pixel size to obtain the area S of the cracked part of the road, the unit is pixel square; then, an area conversion module is used to obtain the actual area of ​​the road cracks; the area conversion module, according to the pixel area of ​​the road cracks, uses the conversion ratio between pixels and actual size, that is, the pixel scale β, to obtain the actual area of ​​the road cracks; the specific process is:

[0056] 1) Let the ruler length be L, the measured pixel length of the ruler in the image be P, and the pixel scale β is calculated by the following formula:

[0057] β=L / P

[0058] Among them, β represents the conversion ratio between pixels and actual size, that is, pixel scale, L represents the actual length of the ruler, and P represents the pixel length of the ruler in the image;

[0059] 2) Multiply the pixel area of ​​the road crack by the square of the pixel scale β to obtain the actual area of ​​the road crack S * , calculated by the following formula:

[0060] S * =S×β 2

[0061] Among them, S * represents the actual area of ​​road cracks, β represents the pixel scale, and the unit is actual length / pixel; S represents the pixel area of ​​road cracks, and the unit is pixel square;

[0062] Finally, the road hazards are classified according to the size of the road crack area: when there are no cracks on the road surface, it is a safe level; when the road crack area is less than or equal to 10 square centimeters, it is a mild hazard level; when the road crack area is greater than 10 square centimeters and less than or equal to 50 square centimeters, it is a moderate hazard level; when the road crack area is greater than 50 square centimeters, it is a severe hazard level.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] The present invention combines a road inspection robot with a road surface detection network model, and trains the network model using a road surface image dataset to obtain a trained deep learning model. This utilizes the powerful feature extraction capability and good learning ability of the neural network in the deep learning model, thereby improving the efficiency and accuracy of road surface detection by the road inspection robot. Compared with traditional manual screening methods, this greatly reduces the road surface inspection cost and improves the level of road maintenance and safety management.

[0065] The road surface detection network model is an RCDIT neural network model. The implicit expression method proposed in the RCDIT neural network model proposes innovative improvements for the continuity and integrity of images. When processing in the form of image blocks, in the process of converting each image block into a space tensor, attention is paid to the continuity and integrity of these space tensors when expressed in the neural network. The concept of latent space coordinates is introduced, and a latent space coordinate system is used to divide cells in the latent space coordinates. Instead of simply considering the center point of the cell as a centralized representation, all pixels in the cell are considered as a whole, and the discontinuous expression problem caused by the fuzzy expression at the junction between cells is solved. The integrity and continuity of the image block expression are guaranteed by implicit expression, and the generalization ability and precise expression ability of network training are significantly improved. In addition, the RCDIT module in the RCDIT neural network model can effectively handle the complex interaction of image blocks and position information by introducing a multi-head cross attention mechanism, and enhance the model's integration ability and information relevance for multimodal data. In addition, the introduced position feedforward network further improves the abstract ability of feature representation, which helps to more effectively process complex features and improve the expressiveness of the model. In summary, through these improvements, the RCDIT module, while maintaining the advantages of the traditional Transformer module, effectively makes up for its shortcomings in multimodal data integration, correlation modeling and computational efficiency, and improves the overall performance and applicability of the model.

[0066] The present invention designs a road surface detection method based on artificial intelligence and a road inspection robot. By introducing the deep learning model RCDIT, the road detection robot can accurately and quickly identify and segment road crack defects, avoiding subjective errors in manual screening, and greatly improving the reliability and accuracy of detection. In addition, the road detection robot is equipped with the RCDIT deep learning model and can work 24 / 7 all-weather, without being restricted by time and environment, effectively saving human resources, and realizing continuous monitoring of road surfaces. Compared with the prior art, the detection method of the present invention has higher efficiency, better accuracy and lower false alarm rate in the task of road crack segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The present invention is a schematic diagram of the principle flow of a road surface detection network model of an embodiment of a road surface detection method based on artificial intelligence and a road inspection robot.

[0068] Figure 2 The present invention is a schematic diagram of the principle flow of an implicit expression module of a road surface detection network model in an embodiment of a road surface detection method based on artificial intelligence and a road inspection robot.

[0069] Figure 3The present invention is a schematic diagram of the structure and principle of an RCDIT module of a road surface detection network model in an embodiment of a road surface detection method based on artificial intelligence and a road inspection robot.

[0070] Figure 4 A schematic diagram of the structure and principle of an adversarial training decoder of a road surface detection network model of an embodiment of a road surface detection method based on artificial intelligence and a road inspection robot of the present invention. DETAILED DESCRIPTION

[0071] In order to more clearly describe the technical problems, technical solutions and advantages of the present invention, the following will be described in detail with reference to the drawings and embodiments. It should be noted that these embodiments are only used to illustrate the principles and application scope of the present invention and should not be regarded as limiting the present invention.

[0072] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0073] The present invention proposes a road surface detection method based on artificial intelligence and a road inspection robot, the method comprising the following steps:

[0074] Step 1: Get the training dataset

[0075] The camera mounted on the road inspection robot is used to shoot videos of multiple road surfaces, and the obtained video files are subjected to video frame extraction, in which twenty-four frames are divided into one second, and the image obtained by frame extraction is resampled (upsample), and the size of the road surface image obtained by frame extraction is adjusted to 256*256 pixels. In this way, the entire video file is processed into a road surface image file.

[0076] The obtained road pavement images are screened, wherein 9,000 images of cracked road surfaces and 1,000 images of non-cracked road surfaces are screened out, 8,000 images of cracked road surfaces are randomly screened out from the 9,000 images of cracked road surfaces, and 500 images of non-cracked road surfaces are randomly screened out from the 1,000 images of non-cracked road surfaces, and the above 8,500 images are respectively subjected to data enhancement processing, and the 8,500 images and the images obtained by data enhancement are divided into a training set; the remaining 1,000 images of cracked road surfaces and the remaining 500 images of non-cracked road surfaces and the corresponding images obtained by data enhancement are divided into a test set. The data enhancement is to rotate each road surface image at different angles, specifically including 90° rotation, 180° rotation, 270° rotation, and horizontal and vertical flipping.

[0077] Step 2: Build a road surface detection network model

[0078] The road surface detection network model is an RCDIT network model, and its structure and principle are as follows: Figure 1 shown.

[0079] The RCDIT network model includes a block processing module, a marker, an implicit expression module, an RCDIT module, a mask image modeling head, and an adversarial training decoder.

[0080] The block processing module divides a given input image x into N original image blocks. i is an integer from 1 to N; record each original image block The corresponding position information in the original image, the original image block position information of the input image x is recorded as P(p1,p2,…,p i ,…,p N ); then, the obtained original image blocks are randomly masked according to a certain number ratio, and the original image blocks at the masked positions become masked image blocks. The masked positions are represented as M, M∈P.

[0081] As an embodiment, the block processing module evenly divides the input image x into 16 blocks in the form of a moving window, and randomly marks 5 mask image blocks in a ratio of 30%-40% of the total number of original image blocks, such as Figure 1 As shown, M∈{2,3,6,7,14}.

[0082] The tagger is an image tokenizer obtained by pre-training DVAE (Discrete Variational Autoencoder), which directly tokenizes an input image x into N visual tokens, denoted as z = [z1, z2, …, z i …,z N], z∈V. V is a dictionary, V=1,2,…,|V|,|V|=8192, V contains the index of all discrete tokens, each original image block Corresponds to a visual token z i , mapped to a word in |V|.

[0083] The original image blocks at the unmasked positions of the input image x and the masked image blocks at the masked positions are input into the implicit expression module to obtain the implicit tensors of each of the original image blocks and masked image blocks of the input image x; the implicit tensors of the input image x and the category label L of the input image x are concatenated as the token sequence of the image blocks of the input image x, and the token sequence of the image blocks is input into the RCDIT module together with the original image block position information of the input image x.

[0084] The category label L is a commonly used manually set label, which indicates the category of the processed image, and is represented here as the data set being images of the same type.

[0085] The operation principle of the implicit expression module is as follows Figure 2 As shown in the figure, the original image blocks at the unmasked positions of the input image x and the masked image blocks at the masked positions are input into the module, and processed by the coordinate representation method, that is, the grid coordinates of the potential two-dimensional space representation are drawn by the built-in grid_sampel method of pytorch. The normalized representation method concept is used to regard the potential grid coordinates as a plane rectangular coordinate system of [-1, 1], and each image block is assigned to the grid corresponding to the position in the potential grid coordinates. The assigned positions are kept in one-to-one correspondence with the corresponding positions in the original image when the image is cut into blocks. This method ensures that all pixels of all image blocks have corresponding two-dimensional coordinates belonging to this plane rectangular coordinate system, and thus have corresponding image pixel values ​​and the number of channels of this pixel.

[0086] In order to ensure the continuity of image pixels during implicit expression, a cell encoding method is introduced, which can be expressed as:

[0087]

[0088] Where s is the encoding representation obtained after the cell is encoded. is the target image block, c=[c h ,c w ], including the specified High C h 、Width c w , Indicates that and c for splicing.

[0089] The result obtained by the coordinate representation method is then processed by a multi-layer perceptron (MLP), which consists of a linear layer and a Leaky ReLU activation function layer; a 3×3 convolution layer follows the multi-layer perceptron. The input data in this part is an image block with a height of 64×width of 64×channel number 3, so the parameters corresponding to the linear layer in this multi-layer perceptron structure are set as follows: input size 64×64, hidden layer size 256, and the output result is a tensor of 1×256; after being processed by the nonlinear activation function Leaky ReLU, it enters the convolution layer, and the output result is a tensor of 64×64×3; traverse all image blocks, as an embodiment, 16 implicit tensors of the same size are obtained, which are concatenated with a set category label L as the token sequence of the image block input to the RCDIT module.

[0090] The RCDIT module structure is as follows Figure 3 As shown in the figure, it is an improvement based on the traditional transformer module, consisting of a first normalization layer, a multi-head self-attention module, a second normalization layer, a multi-head cross-attention module, a third normalization layer, and a multi-layer perceptron.

[0091] The token sequence of the image block input to the RCDIT module is first processed by the first normalization layer and the multi-head self-attention module in sequence, and the result is added to the token sequence of the above image block to obtain the first extracted feature; the result of the first extracted feature processed by the second normalization layer and the original image block position information of the input image x are input to the multi-head cross-attention module for processing to obtain the second extracted feature; the first extracted feature and the second extracted feature are weightedly summed, and the obtained features are processed by the third normalization layer and the multi-layer perceptron in sequence to obtain the output of the RCDIT module.

[0092] The structures and principles of the first, second, and third normalization layers are the same. The normalization layer helps the neural network converge faster and more stably in deep learning. For a given input, the mean and standard deviation are calculated on all its features (or channels), and then these statistics are used to normalize each feature of the input.

[0093] The specific algorithm principle of the normalization layer in the RCDIT module is as follows:

[0094] For all features in a given input, first calculate the mean and standard deviation of these features, denoted by μ and σ:

[0095]

[0096] Where n is the total number of features, f iis the value of the ith feature. The standard deviation is calculated as:

[0097]

[0098] Then use the calculated mean and standard deviation to normalize each feature. For each feature value f i , the normalized formula is:

[0099]

[0100] here represents the normalized eigenvalue.

[0101] For the normalized feature values ​​obtained, two learnable parameters are introduced, the scaling factor γ and the offset factor β. These two parameters allow the model to learn the best representation of the normalized data. The formula is expressed as:

[0102]

[0103] Among them, y i It is the final output feature value of the normalization layer.

[0104] The principle of the multi-head self-attention module is as follows:

[0105] When calculating the attention matrix, the multi-head self-attention mechanism splits the input tensor X into h sub-tensors, each of which is the attention information learned in a different way. For each sub-tensor, a self-attention calculation is performed to obtain an output tensor O i Finally, the h output tensors are concatenated together to obtain the final output tensor O. The calculation formula for multi-head self-attention is:

[0106]

[0107] In the above formula, Q i , K i 、V i are the i-th subtensor X of the input tensor X respectively. i Query, key, value when performing self-attention calculation, W qi , W ki , W vi Q i , K i 、V i The learnable parameter moment, d iK K i The dimension size of W is used to scale the attention size. O is the weight parameter matrix of the linear projection.

[0108] The principle of the multi-head cross attention module is as follows:

[0109] The multi-head cross attention mechanism is a special form of multi-head attention. The result of the first extracted feature processed by the second normalization layer and the original image block position information of the input image x are input into the multi-head cross attention module. The above two parts are spliced ​​and the spliced ​​result is regarded as a whole as the input tensor of the multi-head cross attention mechanism. Then the input tensor is evenly split into two parts, X1 and X2, according to the dimension. Then use one part as the query set and the other part as the key and value set. Its output is a tensor of size n×d2, and for each row vector, its attention weight for all row vectors is given.

[0110] Specifically, let Q = X1W Q , K=V=X2W K , the output of this module is:

[0111]

[0112] in, Both are the learned projection matrices, and d2 is the dimension size of the key or value set.

[0113] The multi-layer perceptron of the RCDIT module is a position feedforward network, which consists of two linear layers and an activation function. The first linear layer of the position feedforward network serves as the data input terminal. The function of this linear layer is to map the input feature representation to a higher-dimensional space. After the first linear layer, a nonlinear activation function layer is connected. The nonlinear activation function is a Leaky ReLU activation function, which is used to introduce nonlinear transformations and enhance the expressive power of the model. After the nonlinear activation function layer, a second linear layer is connected. The second linear layer receives the output of the nonlinear activation function layer as input and maps the input features back to a space with the same feature dimension as the input image x. The output of the second linear layer is regarded as the final output of the RCDIT module.

[0114] The mask image modeling head consists of an FC layer (fully connected layer) and a Softmax function layer. The mask image modeling head generates predicted image block information for all positions of the input image x according to the final output of the RCDIT module. For the mask processing position, the association between the mask image block information generated by the mask image modeling head and the visual token of the original image block is calculated. The mask image block information with the maximum association value is the optimal mask image block information for the required mask position.

[0115] The maximum correlation value is calculated as:

[0116]

[0117] Where D is the total image block information, M represents the location of the mask, and x M is the masked image patch, z i Indicates the visual token to be predicted and associated. MIN (z i |x M ) represents the minimum conditional probability of the visual token that needs to be predicted to establish an association among all the conditions of the masked image blocks.

[0118] The mask image modeling head outputs the predicted image block information of all positions of the input image x, where the mask image block information of the mask position is the above-mentioned optimal mask image block information, and the overall image feature map m is obtained, and m is used as the input of the adversarial training decoder.

[0119] The adversarial training decoder structure is as follows Figure 4 As shown in Figure 1, the adversarial training decoder is based on the existing transformer decoder structure, including an adversarial sample generator, a self-attention module, and a convolutional feedforward neural network. The output of the mask modeling head is input into the adversarial training decoder and then into the adversarial sample generator and the self-attention module respectively.

[0120] For the input m, the adversarial sample generator uses FGSM (Fast Gradient Sign Method) to create the perturbation η, which is expressed as follows:

[0121]

[0122] Where m is the input, Refers to calculating the gradient of m, θ represents the parameters of the model; J represents the loss function of the model, which is the L1 loss function; ∈ is the threshold parameter for generating perturbations. sign() is a sign function, which returns 1 for positive numbers, -1 for negative numbers, and 0 for 0; the adversarial sample is generated by calculating the gradient of the L1 loss function of the output m of the mask modeling head for the corresponding visual token in the marker, and making a small perturbation to the generated prediction result in the direction of the gradient. Finally, the adversarial sample is obtained by adding the output m of the mask modeling head to the perturbation η. Adversarial Examples This is the output of the adversarial sample generator. Such adversarial samples maintain the similarity of the original samples while enhancing the generalization ability of the model.

[0123] For the input m, the self-attention module processes it through a self-attention mechanism:

[0124]

[0125] Where Q', K', and V' are respectively q m、W k m and W v m is obtained, where W q , W k , W v is a learnable parameter matrix; Q' represents query, K' represents key, V' represents value, and the input in self-attention is the same, all are m; d k' is the scaling factor, which is the number of channels of Q; K' in the formula T Indicates that K' is transposed.

[0126] The output of the adversarial sample generator and the output of the self-attention module are concatenated and the result is used as the input of a convolutional feedforward neural network. The output of the convolutional feedforward neural network is the output generated by the adversarial training decoder. The convolutional feedforward neural network outputs a grayscale image of the input image x.

[0127] The convolutional feedforward neural network is composed of a convolutional layer, a sigmoid activation function layer, a fully connected layer, and a sigmoid activation function layer connected in sequence. The convolutional layer processing process is:

[0128] Z i =A i-1 *W i +b i

[0129] Among them, A i-1 is the activation output or input data of the previous layer, W i is the convolution weight of the i-th layer, b i is the bias value of the i-th layer, and * represents the convolution operation.

[0130] Activation function sigmoid processing process:

[0131] A i =σ(Z i )

[0132] Where σ represents the sigmoid activation function,

[0133] Fully connected layer processing:

[0134] Z l =W l A l-1 +b l

[0135] Among them, A l-1 is the activation function output of the previous layer, W l is the weight matrix of the fully connected layer, b l is the bias value of the fully connected layer.

[0136] The output of the fully connected layer is then processed by the activation function to obtain the convolutional feedforward neural network output:

[0137]

[0138] Step 3: Train the road detection network model

[0139] First, the road surface detection network model in step 2 is initialized with random assignment method to learn the parameters, and the training epoch is set to 300 rounds, and the initial learning rate is set to 1e -4 , the optimizer uses Adam optimizer.

[0140] In the training phase, a road image in the training set in step 1 is input into the road detection network model, and the L1 loss value is calculated based on the output of the network model and the normalized pixel value of the road image. The learnable parameters of the network model are updated once using the Adam optimizer based on the L1 loss value to complete the training of a road image; the learnable parameters when the training of the previous road image is completed are used as the initial learnable parameters when the training of the next road image is completed, and the training of an epoch is completed; the learnable parameters when the training of the previous epoch is completed are used as the initial learnable parameters when the training of the next epoch is completed, and the epoch reaches the preset value. After each epoch of training is completed, the road images in the training set are input into the network model respectively, and the image pixel accuracy of the epoch is calculated based on all the results output by the network model and the normalized pixel values ​​of the corresponding real samples. The learnable parameters of the network model with the smallest L1 loss function and the highest image pixel accuracy are saved, and the training of the road detection network model is completed.

[0141] Calculation method of image pixel accuracy:

[0142]

[0143] Among them, TP means that the true category and the predicted category in the sample are consistent and are both positive examples; TN means that the true category and the predicted category in the sample are consistent and are both negative examples; FP means that the true category in the sample is a negative example and the predicted category is a positive example; FN means that the true category in the sample is a positive example and the predicted category is a negative example.

[0144] That is: PA = number of correctly predicted pixels / total number of predicted pixels.

[0145] In the testing phase, the road surface images in the test set in step 1 are respectively input into the trained road surface detection network model for network model testing. According to the normalized results of all the results output by the network model and the corresponding pixel values ​​of the road surface images, the image pixel accuracy, the average image pixel accuracy, and mIoU (mean intersection over union) are calculated. When the values ​​of the image pixel accuracy, the average image pixel accuracy, and mIoU fall within the intervals of (88%, 89%), (92%, 93%), and (86%, 87%), respectively, and the value change is no more than 0.03%, the network model test is optimal, that is, the trained road surface detection network model is the optimal network model. Otherwise, the initial learning rate, the initial epoch number, and the initial learnable parameters of the optimizer are adjusted, and the training process of the training set and the testing process of the test set are repeated until the network model is tested to be optimal and the optimal network model is obtained.

[0146] The road surface detection network model designed by the present invention can complete the missing or low-matching pixels in the subsequent input image after training by learning the data distribution pattern in the input image, restore part of the data lost during the image transmission process, and improve the distinction between the background and the target area of ​​the image; for the quantitative calculation of road cracks, the output of the road surface detection network model after training is more conducive to obtaining accurate calculation results than the output after the pixel values ​​of the road surface image are normalized.

[0147] Step 4: Detection of road surface cracks

[0148] Use the camera mounted on the road inspection robot in step 1 to shoot a video of the target road surface, and use the same method to extract frames, and resize the road surface image to be detected to 256*256; place a ruler in the road surface image to be detected, and the ruler should be on the same plane as the road cracks. Then input each road surface image to be detected into the optimal network model in step 3, and the network model outputs a grayscale image of the road surface image to be detected.

[0149] Then, according to the grayscale image of the road surface image to be detected output by the network model, the road cracks are quantified, and the binary mask of the road cracks is obtained through the image segmentation algorithm, that is, the cracked part of the road is marked as the foreground, and the non-defective part is marked as the background. The pixel counting method is used to calculate the number of pixels in the cracked part of the road, that is, the number of foreground pixels, represented by N. The number of pixels N is multiplied by the square of the pixel size to obtain the area S of the cracked part of the road, in pixels squared. Afterwards, an area conversion module is used to obtain the actual area of ​​the road cracks. The area conversion module, based on the pixel area of ​​the road cracks, uses the conversion ratio between pixels and actual sizes, that is, the pixel scale β, to obtain the actual area of ​​the road cracks; the specific process is:

[0150] 1) Let the ruler length be L, the measured pixel length of the ruler in the image be P, and the pixel scale β is calculated by the following formula:

[0151] β=L / P

[0152] Among them, β represents the conversion ratio between pixels and actual size, that is, the pixel scale, L represents the actual length of the ruler, and P represents the pixel length of the ruler in the image.

[0153] 2) Multiply the pixel area of ​​the road crack by the square of the pixel scale β to obtain the actual area of ​​the road crack S * , calculated by the following formula:

[0154] S * =S×β 2

[0155] Among them, S * represents the actual area of ​​road cracks, β represents the pixel scale (in actual length / pixel), and S represents the pixel area of ​​road cracks (in pixels squared).

[0156] Finally, the road hazards are classified according to the size of the road crack area: when there are no cracks on the road surface, it is a safe level; when the road crack area is less than or equal to 10 square centimeters, it is a mild hazard level; when the road crack area is greater than 10 square centimeters and less than or equal to 50 square centimeters, it is a moderate hazard level; when the road crack area is greater than 50 square centimeters, it is a severe hazard level.

[0157] On the same road surface data set, the road surface detection method of the present invention is compared with the existing methods. The road surface detection method of the present invention uses the RCDIT network model as the detection model. The recognition results show that compared with the existing excellent transformer methods such as SegFormer and CrackFormer, the road surface detection method of the present invention has lower computational complexity and better performance. The results obtained by using multiple network models for road crack recognition after training are shown in Table 1.

[0158] Table 1 Recognition results of different network models

[0159]

[0160] The results shown in Table 1 show that in the processing of road crack images, the pixel accuracy and average pixel accuracy of the RCDIT network model are higher than those of the existing excellent SegFormer and CrackFormer, and the mIoU is also improved. Compared with other neural network models, the difference is more significant, which proves that the road surface detection method of the present invention has superior application performance.

[0161] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A road surface detection method based on artificial intelligence and road inspection robots, characterized in that: The method comprises the following steps: Step 1: Get the training dataset The camera mounted on the road inspection robot is used to shoot videos of multiple road surfaces, and the obtained video files are subjected to video frame extraction processing, in which the frames are extracted in the form of 24 frames per second, and the images obtained by the frame extraction are resampled, and the size of the road surface images obtained by the frame extraction is adjusted to 256*256 pixels. In this way, the entire video file is processed into a road surface image file; The obtained road surface images are screened, wherein a certain number of cracked road surface images and non-cracked road surface images are screened, and images obtained after data enhancement processing of no less than 80% of the cracked road surface images and no less than 50% of the non-cracked road surface images are respectively performed as training sets; and the remaining cracked road surface images and the remaining non-cracked road surface images are respectively performed as test sets. Step 2: Build a road surface detection network model The road surface detection network model is an RCDIT network model, including a block processing module, a marker, an implicit expression module, an RCDIT module, a mask image modeling head, and an adversarial training decoder; The block processing module divides a given input image x into N original image blocks. i is an integer from 1 to N; record each original image block The corresponding position information in the original image, the original image block position information of the input image x is recorded as P(p1,p2,…,p i ,…,p N ); then, the obtained original image blocks are randomly masked according to a certain number ratio, and the original image blocks at the masked positions become masked image blocks; the masked positions are represented as M, M∈P; The tagger is an image tokenizer obtained by pre-training DVAE, which directly tokenizes an input image x into N visual tokens, denoted as z = [z1, z2, …, z i …,z N ], z∈V; V is a dictionary, V=1,2,…,|V|,|V|=8192, V contains the index of all discrete tokens, each original image block Corresponds to a visual token z i , mapped to a word in |V|; The original image block at the unmasked position of the input image x and the masked image block at the masked position are input into the implicit expression module to obtain the implicit tensor of each of the original image blocks and masked image blocks of the input image x; the implicit tensor of the input image x and the category label L of the input image x are concatenated as the token sequence of the image block of the input image x, and the token sequence of the image block and the original image block position information of the original image x are input into the RCDIT module together; The original image blocks at the unmasked positions of the input image x and the masked image blocks at the masked positions are input into the implicit expression module, and processed by the coordinate representation method, that is, the grid coordinates of the potential two-dimensional space representation are drawn by the built-in grid_sampel method of pytorch. The normalized representation method concept is used to regard the potential grid coordinates as a [-1, 1] plane rectangular coordinate system, and each image block is assigned to the grid corresponding to the position in the potential grid coordinates. The assigned positions are kept in one-to-one correspondence with the corresponding positions in the original image when the image is cut into blocks. This method is used to make all pixels of all image blocks have corresponding two-dimensional coordinates belonging to this plane rectangular coordinate system, and thus there will be corresponding image pixel values ​​and the number of channels of this pixel. In order to ensure the continuity of image pixels during implicit expression, a cell encoding method is introduced, which can be expressed as: Where s is the encoding representation obtained after the cell is encoded. is the target image block, c=[c h ,c w ], including the specified High C h 、Width c w , Indicates that and c for splicing; Then the result obtained by the coordinate representation method is processed by a multi-layer perceptron, which is composed of a linear layer and a Leaky ReLU activation function layer; a 3×3 convolution layer is immediately followed by the multi-layer perceptron; the input data in this part is an image block with a height of 64×width of 64×channel number 3, so the parameters corresponding to the linear layer in this multi-layer perceptron structure are set to: input size 64×64, hidden layer size 256, and the output result is a tensor of 1*256; after being processed by the nonlinear activation function Leaky ReLU, it enters the convolution layer, and the output result is a tensor of 64×64×3; traverse all image blocks to obtain N implicit tensors of the same size, which are concatenated with a set category label L as the token sequence of the image block input to the RCDIT module; The RCDIT module structure consists of a first normalization layer, a multi-head self-attention module, a second normalization layer, a multi-head cross-attention module, a third normalization layer, and a multi-layer perceptron; the token sequence of the image block input to the RCDIT module is first processed by the first normalization layer and the multi-head self-attention module in sequence, and the result is added to the token sequence of the above image block to obtain a first extracted feature; the result of the first extracted feature processed by the second normalization layer and the original image block position information of the input image x are input to the multi-head cross-attention module for processing to obtain a second extracted feature; the first extracted feature and the second extracted feature are weighted summed, and the obtained feature is processed by the third normalization layer and the multi-layer perceptron in sequence to obtain the output of the RCDIT module; the principles and structures of the first normalization layer, the second normalization layer, and the third normalization layer are the same; The principle of the multi-head self-attention module is as follows: When calculating the attention matrix, the multi-head self-attention mechanism splits the input tensor X into h sub-tensors, each of which is the attention information learned in a different way; for each sub-tensor, a self-attention calculation is performed to obtain an output tensor O i ; Finally, the h output tensors are concatenated together to obtain the final output tensor O. The calculation formula for multi-head self-attention is: In the above formula, Q i , K i 、V i are the i-th subtensor X of the input tensor X respectively. i Query, key, value when performing self-attention calculation, W qi , W ki , W vi Q i , K i 、V i The learnable parameter moment, d iK K i The dimension size is used to scale the size of attention; W O is the weight parameter matrix of the linear projection; The principle of the multi-head cross attention module is as follows: the result of the first extracted feature processed by the second normalization layer and the original image block position information of the input image x are input into the multi-head cross attention module, the above two parts are input spliced, and the splicing result is regarded as a whole as the input tensor of the multi-head cross attention mechanism; then the input tensor is evenly split into two parts, X1 and X2, according to the dimension. Then one of the parts is used as the query set and the other part as the key and value set; its output is a tensor of size n×d2, and for each row vector, its attention weight for all row vectors is given; specifically, let Q=X1W Q , K=V=X2W K , the output of this module is: in, Both are the learned projection matrices, and d2 is the dimension size of the key or value set; The multi-layer perceptron of the RCDIT module is a position feedforward network, which consists of two linear layers and an activation function. The first linear layer of the position feedforward network is used as the data input terminal. The function of this linear layer is to map the input feature representation to a higher-dimensional space. After the first linear layer, a nonlinear activation function layer is connected. The nonlinear activation function is a Leaky ReLU activation function, which is used to introduce nonlinear transformation and enhance the expression ability of the model. After the nonlinear activation function layer, a second linear layer is connected. The second linear layer receives the output of the nonlinear activation function layer as input and maps the input features back to a space with the same feature dimension of the input image x. The output of the second linear layer is regarded as the final output of the RCDIT module. The mask image modeling head consists of an FC layer and a Softmax function layer. The mask image modeling head generates predicted image block information for all positions of the input image x according to the final output of the RCDIT module. For the mask processing position, the association between the mask image block information generated by the mask image modeling head and the visual token of the original image block is calculated. The mask image block information with the maximum association value is the optimal mask image block information for the desired mask position. The maximum correlation value is calculated as: Where D is the total image block information, M represents the location of the mask, and x M is the masked image patch, z i Indicates the visual token to be predicted and associated; p MIN (z i |x M ) represents the minimum conditional probability of the visual token that needs to be predicted to establish an association among all the conditions of the masked image blocks. The mask image modeling head outputs the predicted image block information of all positions of the input image x, where the mask image block information of the mask position is the above-mentioned optimal mask image block information, and the overall image feature map m is obtained, and m is used as the input of the adversarial training decoder; The adversarial training decoder includes an adversarial sample generator, a self-attention module, and a convolutional feedforward neural network; the output result of the mask modeling head is input into the adversarial training decoder, and is input into the adversarial sample generator and the self-attention module respectively; For input m, the adversarial sample generator uses FGSM to create perturbation η, which is expressed as follows: Where m is the input, Refers to calculating the gradient of m, θ represents the parameters of the model; J represents the loss function of the model, which is the L1 loss function; ∈ is the threshold parameter for generating perturbations; sign() is a sign function, which returns 1 for positive numbers, -1 for negative numbers, and 0 for 0; the adversarial sample is generated by calculating the gradient of the L1 loss function of the output m of the mask modeling head for the corresponding visual token in the marker, and slightly perturbing the generated prediction result in the direction of the gradient. Finally, the adversarial sample is obtained by adding the output m of the mask modeling head to the perturbation η. Adversarial Examples This is the output of the adversarial sample generator; For the input m, the self-attention module processes it through a self-attention mechanism: Where Q', K', and V' are respectively q m、W k m and W v m is obtained, where W q , W k , W v is a learnable parameter matrix; Q' represents query, K' represents key, V' represents value, and the input in self-attention is the same, all are m; d k' is the scaling factor, which is the number of channels of Q; K' in the formula T Indicates that K' is transposed; The output of the adversarial sample generator and the output of the self-attention module are concatenated, and the result is used as the input of a convolutional feedforward neural network. The output of the convolutional feedforward neural network is the output result generated by the adversarial training decoder. The convolutional feedforward neural network outputs a grayscale image of the input image x. The convolutional feedforward neural network is composed of a convolutional layer, a sigmoid activation function layer, a fully connected layer, and a sigmoid activation function layer connected in sequence; Step 3: Train the road detection network model First, the road surface detection network model in step 2 is initialized with a random assignment method to initialize the learnable parameters, and the training epoch is set to 300 rounds, and the initial learning rate is set to 1e -4 , the optimizer adopts Adam optimizer; In the training phase, a road image in the training set in step 1 is input into the road detection network model, and the L1 loss value is calculated according to the result output by the network model and the result after the pixel value of the road image is normalized. The learnable parameters of the network model are updated once according to the L1 loss value using the Adam optimizer to complete the training of a road image; the learnable parameters when the training of the previous road image is completed are used as the initial learnable parameters when the next road image is trained, and the training of one epoch is completed; the learnable parameters when the training of the previous epoch is completed are used as the initial learnable parameters when the next epoch is trained, and the training of one epoch is completed; the learnable parameters when the training of the previous epoch is completed are used as the initial learnable parameters when the next epoch is trained, and the training of the epoch reaches the preset value; after each epoch of training is completed, the road images in the training set are respectively input into the network model, and the image pixel accuracy of the epoch is calculated according to all the results output by the network model and the result after the pixel value of the corresponding real sample is normalized; the learnable parameters of the network model with the smallest L1 loss function and the highest image pixel accuracy are saved, and the training of the road detection network model is completed; In the testing phase, the road surface images in the test set in step 1 are respectively input into the trained road surface detection network model for network model testing. According to the normalized results of all the results output by the network model and the corresponding pixel values ​​of the road surface images, the image pixel accuracy, the average image pixel accuracy, and mIoU are calculated. When the values ​​of the image pixel accuracy, the average image pixel accuracy, and mIoU fall within the intervals of (88%, 89%), (92%, 93%), and (86%, 87%), respectively, and the value change is not greater than 0.03%, the network model test is optimal, that is, the trained road surface detection network model is the optimal network model. Otherwise, the initial learning rate, the initial epoch number, and the initial learnable parameters of the optimizer are adjusted, and the training process of the training set and the testing process of the test set are repeated until the network model is tested to be optimal, and the optimal network model is obtained. Step 4: Detection of road pavement cracks Use the camera mounted on the road inspection robot in step 1 to shoot a video of the target road surface, and use the same method to extract frames for the video, and adjust the size of the road surface image to be detected obtained by extracting frames to 256*256; place a ruler in the road surface image to be detected, and the ruler should be on the same plane as the road cracks; then input each road surface image to be detected into the optimal network model in step 3, and the network model outputs a grayscale image of the road surface image to be detected; Then, according to the grayscale image of the road surface image to be detected output by the network model, the road cracks are quantified, and the binary mask of the road cracks is obtained by the image segmentation algorithm, that is, the cracked part of the road is marked as the foreground, and the non-defective part is marked as the background; the number of pixels in the cracked part of the road is calculated using the pixel counting method, that is, the number of foreground pixels, represented by N; the number of pixels N is multiplied by the square of the pixel size to obtain the area S of the cracked part of the road, the unit is pixel square; then, an area conversion module is used to obtain the actual area of ​​the road cracks; the area conversion module, according to the pixel area of ​​the road cracks, uses the conversion ratio between pixels and actual size, that is, the pixel scale β, to obtain the actual area of ​​the road cracks; the specific process is: 1) Let the ruler length be L, the measured pixel length of the ruler in the image be P, and the pixel scale β is calculated by the following formula: β=L / P Among them, β represents the conversion ratio between pixels and actual size, that is, pixel scale, L represents the actual length of the ruler, and P represents the pixel length of the ruler in the image; 2) Multiply the pixel area of ​​the road crack by the square of the pixel scale β to obtain the actual area of ​​the road crack S * , calculated by the following formula: S * =S×β 2 Among them, S * represents the actual area of ​​road cracks, β represents the pixel scale, and the unit is actual length / pixel; S represents the pixel area of ​​road cracks, and the unit is pixel square; Finally, the road hazards are classified according to the size of the road crack area: when there are no cracks on the road surface, it is a safe level; when the road crack area is less than or equal to 10 square centimeters, it is a mild hazard level; when the road crack area is greater than 10 square centimeters and less than or equal to 50 square centimeters, it is a moderate hazard level; when the road crack area is greater than 50 square centimeters, it is a severe hazard level.

2. A road surface detection method based on artificial intelligence and road inspection robots according to claim 1, characterized in that: In step 1, the data enhancement is to rotate each road surface image at different angles, specifically including 90° rotation, 180° rotation, 270° rotation, and horizontal and vertical flipping.

3. The road surface detection method based on artificial intelligence and road inspection robots according to claim 1 is characterized in that: In step 2, mask processing is performed randomly on the obtained original image blocks according to a quantity ratio of 30%-40%.

4. The road surface detection method based on artificial intelligence and road inspection robots according to claim 1 is characterized in that: In step 2, the specific algorithm principle of the normalization layer in the RCDIT module is as follows: For all features in a given input, first calculate the mean and standard deviation of these features, denoted by μ and σ: Where n is the total number of features, f i is the value of the i-th feature; the standard deviation is calculated as: Then use the calculated mean and standard deviation to normalize each feature. For each feature value f i , the normalized formula is: here represents the normalized eigenvalue; For the normalized feature values ​​obtained, two learnable parameters are introduced, the scaling factor γ and the offset factor β. These two parameters allow the model to learn the best representation of the normalized data. The formula is expressed as: Among them, y i It is the final output feature value of the normalization layer.

5. The road surface detection method based on artificial intelligence and road inspection robots according to claim 1 is characterized in that: In step 2, the convolutional layer processing process of the convolutional feedforward neural network: Z i =A i-1 *W i +b i Among them, A i-1 is the activation output or input data of the previous layer, W i is the convolution weight of the i-th layer, b i is the bias value of the i-th layer, and * represents the convolution operation; Activation function sigmoid processing process: A i =σ(Z i ) Where σ represents the sigmoid activation function, Fully connected layer processing: Z l =W l A l-1 +b l Among them, A l-1 is the activation function output of the previous layer, W l is the weight matrix of the lth fully connected layer, b l is the bias value of the lth fully connected layer; The output of the fully connected layer is then processed by the activation function to obtain the output of the convolutional feedforward neural network.

6. The road surface detection method based on artificial intelligence and road inspection robots according to claim 1 is characterized in that: In step one, the obtained road pavement images are screened to select 9,000 images of cracked road surfaces and 1,000 images of non-cracked road surfaces. 8,000 images are randomly selected from the 9,000 images of cracked road surfaces, and 500 images are randomly selected from the 1,000 images of non-cracked road surfaces. The 8,500 images screened out are respectively subjected to data enhancement processing, and the 8,500 images and the images obtained by data enhancement are divided into a training set; the remaining 1,000 images of cracked road surfaces and the remaining 500 images of non-cracked road surfaces and the corresponding images obtained by data enhancement are divided into a test set.

Images