A method, system, device and storage medium for road surface defect detection

Abstract

The present invention discloses a road surface defect detection method, system, device and storage medium, including: obtaining a number of historical road surface image data to construct pre-training samples; constructing a road surface detection model; the road surface detection model includes a backbone network and a classifier network; based on an optimized self-supervised learning algorithm, using the pre-training samples, pre-training the backbone network in the road surface detection model to obtain a pre-trained road surface detection model; randomly obtaining some sample data from the pre-training samples and performing defect annotation on the some sample data to obtain fine-tuning training samples; and performing fine-tuning processing on the parameters of the backbone network and the classifier network in the pre-trained road surface detection model to obtain a trained road surface defect detection model; inputting the road surface image to be detected into the trained road surface defect detection model, and outputting the result, that is, obtaining the road surface defect detection result; the present invention reduces the training data volume of the road surface defect detection model and improves the detection accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and particularly relates to a road surface defect detection method, system, device and storage medium. Background Art

[0002] In real life, the situation of road surface defects is very common, and the causes of defect formation are very complex; there is no general-purpose automatic detection instrument that can be used in the existing technology, and the common detection method is still manual detection; while the manual detection method has low efficiency, low stability, and is prone to missed detection during the detection process.

[0003] Currently, defect detection algorithms based on deep learning have begun to be applied in more and more industrial detection scenarios; however, when applying the defect detection algorithm to road surface defect detection, due to the interference of natural factors such as weather, light, and foreign objects on the road surface in the road surface image acquisition, there are also large differences between the images collected in the same road section, which ultimately leads to the accuracy of the trained defect detection algorithm being difficult to meet the required accuracy; currently, measures have also been taken to increase the training samples, and at the same time, with the increase of training samples, there will be a problem of too high training sample annotation cost. Summary of the Invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a road surface defect detection method, system, device and storage medium to solve the technical problems of the existing defect detection method with low accuracy and high training sample annotation cost.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] The present invention provides a road surface defect detection method, including the following steps:

[0007] Obtain a number of historical road surface image data and construct a pre-training sample;

[0008] Construct a road surface detection model; wherein, the road surface detection model includes a backbone network and a classifier network;

[0009] Based on the optimized self-supervised learning algorithm, use the pre-training sample to pre-train the backbone network in the road surface detection model to obtain a pre-trained road surface detection model;

[0010] Randomly obtain some sample data from the pre-training sample and perform defect annotation on the part of sample data to obtain a fine-tuning training sample;

[0011] Use the fine-tuning training sample to fine-tune the parameters of the backbone network and the classifier network in the pre-trained road surface detection model to obtain a trained road surface defect detection model;

[0012] Input the road surface image to be detected into the trained road surface defect detection model, and output the result, that is, obtain the road surface defect detection result.

[0013] Furthermore, the backbone network of the road surface detection model is used to extract features from the road surface image to obtain an intermediate feature map;

[0014] The classifier network of the road surface detection model is used to classify and perform object detection on the intermediate feature map to obtain an image detection result.

[0015] Furthermore, the backbone network of the road surface detection model is a residual network; the optimized self-supervised learning algorithm is the MoCo algorithm introducing a reconstructed image branch neural network; the backbone network of the MoCo algorithm is the same as the backbone network of the road surface detection model.

[0016] Furthermore, based on the optimized self-supervised learning algorithm, using pre-training samples to pre-train the backbone network in the road surface detection model to obtain the pre-trained road surface detection model, the specific process is as follows:

[0017] Use the backbone network of the MoCo algorithm to extract features from the pre-training samples to obtain an intermediate feature map;

[0018] Use the reconstructed image branch neural network to perform image reconstruction on the intermediate feature map to obtain a reconstructed image;

[0019] Calculate the reconstruction image loss between the pre-training samples and the reconstructed image to obtain the first loss;

[0020] Use the feature extraction branch neural network of the MoCo algorithm to perform feature extraction on the intermediate feature map to obtain a number of feature vectors; the number of feature vectors corresponds to a number of historical road surface image data one by one;

[0021] Calculate the contrast loss between the number of feature vectors to obtain the second loss;

[0022] Use the first loss and the second loss to obtain a comprehensive loss;

[0023] Use the comprehensive loss function to update the model parameters of the backbone network of the MoCo algorithm by backpropagation;

[0024] Judge whether the comprehensive loss converges or the number of training iterations reaches the preset number of iterations. If so, output the model parameters of the updated backbone network of the MoCo algorithm; if not, return to continue training and iterating;

[0025] Bring the model parameters of the backbone network of the updated MoCo algorithm into the backbone network of the road surface detection model, and the pre-trained road surface detection model can be obtained.

[0026] Furthermore, the comprehensive loss is obtained by weighted summation of the first loss and the second loss.

[0027] Furthermore, the classifier network of the road surface detection model includes a classification unit and an object detection unit; among them, the classification unit is a binary classifier model, and the object detection unit is an object defect detection model based on the Faster RCNN algorithm or the YOLO algorithm.

[0028] Furthermore, the process of using the fine-tuning training samples to fine-tune the parameters of the backbone network and the classifier network in the pre-trained road surface detection model to obtain the trained road surface defect detection model is as follows:

[0029] Input the fine-tuning training samples into the backbone network of the pre-trained road surface detection model to obtain intermediate feature maps;

[0030] Input the intermediate feature maps into the classifier network in the pre-trained road surface detection model, and output the predicted classification probabilities of the fine-tuning training samples;

[0031] According to the predicted classification probabilities of the fine-tuning training samples and the defect annotations of the fine-tuning training samples, calculate the cross-entropy loss between the predicted classification probabilities of the fine-tuning training samples and the defect annotations of the fine-tuning training samples, and use the cross-entropy loss between the predicted classification probabilities of the fine-tuning training samples and the defect annotations of the fine-tuning training samples as the training loss function of the fine-tuning model;

[0032] According to the training loss function of the fine-tuning model, use the backpropagation algorithm to adjust the parameters of the backbone network and the classifier network in the pre-trained road surface detection model, so that the training loss value of the training loss function of the fine-tuning model reaches the preset convergence condition, and output the trained road surface defect detection model.

[0033] The present invention also provides a road surface defect detection system, including:

[0034] A data module for obtaining a number of historical road surface image data and constructing pre-training samples;

[0035] A modeling module for constructing a road surface detection model; among them, the road surface detection model includes a backbone network and a classifier network;

[0036] A pre-training module for pre-training the backbone network in the road surface detection model based on the optimized self-supervised learning algorithm using the pre-training samples to obtain the pre-trained road surface detection model;

[0037] A fine-tuning sample module, configured to randomly obtain partial sample data from the pre-training samples, and perform defect annotation on the partial sample data to obtain fine-tuning training samples;

[0038] A fine-tuning training module, configured to use the fine-tuning training samples to perform fine-tuning processing on the parameters of the backbone network and the classifier network in the pavement detection model after pre-training, to obtain a trained pavement defect detection model;

[0039] A prediction output module, configured to input the pavement image to be detected into the trained pavement defect detection model, and output the result, that is, obtain the pavement defect detection result.

[0040] The present invention also provides a pavement defect detection device, including:

[0041] A memory, configured to store a computer program;

[0042] A processor, configured to implement the steps of the pavement defect detection method when executing the computer program.

[0043] The present invention also provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program implements the steps of the pavement defect detection method when being executed by a processor.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] A pavement defect detection method and system provided by the present invention pre-trains the backbone network of the pavement detection model through pre-training samples based on an optimized self-supervised learning algorithm; uses the fine-tuning training samples to perform parameter fine-tuning on the backbone network and the classifier network in the pavement detection model after pre-training, to obtain a trained pavement defect detection model for defect detection; through the pre-training of the backbone network and then using fewer fine-tuning training samples for secondary fine-tuning training, it can maintain a high detection accuracy on the premise of ensuring the convergence of the model, and greatly reduce the annotation cost of training data.

[0046] Furthermore, the MoCo algorithm introducing a reconstructed image branch neural network is used for the pavement detection model, which enhances the expression of the intermediate feature map output by the backbone network of the MoCo algorithm for shape features, and effectively improves the accuracy of the detection result. Description of the Drawings

[0047] Figure 1 It is a flowchart of the pavement defect detection method described in the embodiment;

[0048] Figure 2 It is a flowchart of the pre-training process of the defect detection model in the embodiment;

[0049] Figure 3 Flow chart of the fine-tuning training process of the defect detection model in the embodiment;

[0050] Figure 4 Block diagram of the structure of the defect detection system in the embodiment. Specific implementation manners

[0051] In order to make the technical problems, technical solutions and beneficial effects solved by the present invention clearer and more understandable, the following specific embodiments are used to further elaborate on the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0052] The present invention provides a road surface defect detection method, including the following steps:

[0053] Step 1, obtain a number of historical road surface images and construct pre-training samples.

[0054] Step 2, construct a road surface detection model; wherein, the road surface detection model includes a backbone network and a classifier network.

[0055] The backbone network of the road surface detection model is used to extract features from the road surface image to obtain an intermediate feature map; it should be noted that the model or algorithm based on which the road surface image is used to extract features to obtain the intermediate feature map is not limited in the present invention; preferably, the backbone network of the road surface detection model is a residual network.

[0056] The classifier network of the road surface detection model is used to classify and perform target detection on the intermediate feature map to obtain an image detection result; preferably, the classifier network of the road surface detection model includes a classification unit and a target detection unit; wherein, the classification unit is a binary classifier model, and the target detection unit is a target defect detection model based on the Faster RCNN algorithm or the YOLO algorithm.

[0057] Step 3, based on the optimized self-supervised learning algorithm, use the pre-training samples to pre-train the backbone network in the road surface detection model to obtain a pre-trained road surface detection model; wherein, the optimized self-supervised learning algorithm is the MoCo algorithm introducing a reconstructed image branch neural network; the backbone network of the MoCo algorithm is the same as the backbone network of the road surface detection model; the specific process is as follows:

[0058] Step 31, use the backbone network of the MoCo algorithm to extract features from the pre-training samples to obtain an intermediate feature map.

[0059] Step 32, use the reconstructed image branch neural network to perform image reconstruction on the intermediate feature map to obtain a reconstructed image.

[0060] Step 33: Calculate the reconstruction image loss between the pre-training samples and the reconstructed images to obtain the first loss. The first loss is as follows:

[0061] Loss rec = F(a i , a p )

[0062] where Loss rec is the first loss, i.e., the reconstruction image loss between the pre-training samples and the reconstructed images; F(a i , a p ) is the error value between the pre-training samples and the reconstructed images; a i is the pre-training sample; a p is the reconstructed image.

[0063] Step 34: Use the feature extraction branch neural network of the MoCo algorithm to perform feature extraction processing on the intermediate feature maps to obtain a number of feature vectors. Among them, the number of feature vectors corresponds one-to-one with a number of historical road surface image data.

[0064] Step 35: Use the contrastive loss function of the MoCo algorithm to calculate the contrastive loss between the number of feature vectors to obtain the second loss.

[0065] Step 36: Use the first loss and the second loss to obtain the comprehensive loss. The comprehensive loss is obtained by weighted summation of the first loss and the second loss. The comprehensive loss is as follows:

[0066] Loss total = W1 * Loss rec + W2 * Loss moco

[0067] where Loss moco is the second loss; W1 is the weight of the first loss; W2 is the weight of the second loss.

[0068] Step 37: Use the comprehensive loss function to update the model parameters of the backbone network of the MoCo algorithm by backpropagation.

[0069] Step 38: Determine whether the comprehensive loss converges or the number of training iterations reaches the preset number of iterations. If so, output the model parameters of the updated backbone network of the MoCo algorithm; if not, return to continue the training iteration.

[0070] Step 39: Substitute the model parameters of the updated backbone network of the MoCo algorithm into the backbone network in the road surface detection model to obtain the pre-trained road surface detection model.

[0071] Step 4: Randomly obtain some sample data from the pre-training samples, and perform defect annotation on the some sample data to obtain fine-tuning training samples.

[0072] Step 5: Use the fine-tuning training samples to fine-tune the parameters of the backbone network and the classifier network in the pavement detection model after pre-training to obtain the trained pavement defect detection model; the process of fine-tuning training is as follows:

[0073] Step 51: Input the fine-tuning training samples into the backbone network of the pavement detection model after pre-training to obtain intermediate feature maps.

[0074] Step 52: Input the intermediate feature maps into the classifier network in the pavement detection model after pre-training, and output the predicted classification probabilities of the fine-tuning training samples.

[0075] Step 53: Calculate the cross-entropy loss between the predicted classification probabilities of the fine-tuning training samples and the defect annotations of the fine-tuning training samples, and use the cross-entropy loss between the predicted classification probabilities of the fine-tuning training samples and the defect annotations of the fine-tuning training samples as the training loss function of the fine-tuning model.

[0076] Step 54: According to the training loss function of the fine-tuning model, use the backpropagation algorithm to adjust the parameters of the backbone network and the classifier network in the pavement detection model after pre-training, so that the training loss value of the training loss function of the fine-tuning model reaches the preset convergence condition, and output the trained pavement defect detection model.

[0077] Step 6: Input the pavement image to be detected into the trained pavement defect detection model, and output the result, that is, obtain the pavement defect detection result.

[0078] The present invention also provides a pavement defect detection system, including a data module, a modeling module, a pre-training module, a fine-tuning sample module, a fine-tuning training module and a prediction output module.

[0079] A data module for obtaining a number of historical road surface image data and constructing pre-training samples; a modeling module for constructing a road surface detection model; wherein, the road surface detection model includes a backbone network and a classifier network; a pre-training module for pre-training the backbone network in the road surface detection model using the pre-training samples based on an optimized self-supervised learning algorithm to obtain a pre-trained road surface detection model; a fine-tuning sample module for randomly obtaining some sample data from the pre-training samples and performing defect annotation on the part of the sample data to obtain fine-tuning training samples; a fine-tuning training module for fine-tuning the parameters of the backbone network and the classifier network in the pre-trained road surface detection model using the fine-tuning training samples to obtain a trained road surface defect detection model; a prediction output module for inputting the road surface image to be detected into the trained road surface defect detection model and outputting the result, that is, obtaining the road surface defect detection result.

[0080] The present invention also provides a road surface defect detection device, including: a memory for storing a computer program; a processor for implementing the steps of the road surface defect detection method when executing the computer program.

[0081] When the processor executes the computer program, it implements the steps of the above road surface defect detection method, for example: obtaining a number of historical road surface image data and constructing pre-training samples; constructing a road surface detection model; wherein, the road surface detection model includes a backbone network and a classifier network; pre-training the backbone network in the road surface detection model using the pre-training samples based on an optimized self-supervised learning algorithm to obtain a pre-trained road surface detection model; randomly obtaining some sample data from the pre-training samples and performing defect annotation on the part of the sample data to obtain fine-tuning training samples; fine-tuning the parameters of the backbone network and the classifier network in the pre-trained road surface detection model using the fine-tuning training samples to obtain a trained road surface defect detection model; inputting the road surface image to be detected into the trained road surface defect detection model and outputting the result, that is, obtaining the road surface defect detection result.

[0082] Alternatively, when the processor executes the computer program, it implements the functions of each module in the above system. For example: a data module, which is used to obtain a number of historical road surface image data and construct pre-training samples; a modeling module, which is used to construct a road surface detection model; wherein, the road surface detection model includes a backbone network and a classifier network; a pre-training module, which is used to pre-train the backbone network in the road surface detection model based on an optimized self-supervised learning algorithm and using the pre-training samples to obtain a pre-trained road surface detection model; a fine-tuning sample module, which is used to randomly obtain some sample data from the pre-training samples and perform defect annotation on the part of the sample data to obtain fine-tuning training samples; a fine-tuning training module, which is used to fine-tune the parameters of the backbone network and the classifier network in the pre-trained road surface detection model using the fine-tuning training samples to obtain a trained road surface defect detection model; a prediction output module, which is used to input the road surface image to be detected into the trained road surface defect detection model and output the result, that is, to obtain the road surface defect detection result as described above.

[0083] Exemplarily, the computer program can be divided into one or more modules / units. The one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of completing preset functions, and the instruction segments are used to describe the execution process of the computer program in the road surface defect detection device. For example, the computer program can be divided into a data module, a modeling module, a pre-training module, a fine-tuning sample module, a fine-tuning training module, and a prediction output module. The specific functions of each module are as follows: a data module, which is used to obtain a number of historical road surface image data and construct pre-training samples; a modeling module, which is used to construct a road surface detection model; wherein, the road surface detection model includes a backbone network and a classifier network; a pre-training module, which is used to pre-train the backbone network in the road surface detection model based on an optimized self-supervised learning algorithm and using the pre-training samples to obtain a pre-trained road surface detection model; a fine-tuning sample module, which is used to randomly obtain some sample data from the pre-training samples and perform defect annotation on the part of the sample data to obtain fine-tuning training samples; a fine-tuning training module, which is used to fine-tune the parameters of the backbone network and the classifier network in the pre-trained road surface detection model using the fine-tuning training samples to obtain a trained road surface defect detection model; a prediction output module, which is used to input the road surface image to be detected into the trained road surface defect detection model and output the result, that is, to obtain the road surface defect detection result as described above.

[0084] The road surface defect detection device may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The road surface defect detection device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above are merely examples of the road surface defect detection device, and do not constitute a limitation on the road surface defect detection device. It may include more components than the above, or combine some components, or different components. For example, the road surface defect detection device may further include input / output devices, network access devices, buses, etc.

[0085] The so-called processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc. The processor is the control center of the road surface defect detection device, and connects various parts of the entire road surface defect detection device through various interfaces and lines.

[0086] The memory can be used to store the computer program and / or module. The processor realizes various functions of the road surface defect detection device by running or executing the computer program and / or module stored in the memory, and by calling the data stored in the memory.

[0087] The memory may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phone book, etc.), etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0088] The present invention also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the road surface defect detection method described above are realized.

[0089] If the modules / units integrated in the road surface defect detection system are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0090] Based on such an understanding, to implement all or part of the processes in the above road surface defect detection method, the present invention can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above road surface defect detection method can be realized. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or preset intermediate form, etc.

[0091] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0092] It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0093] For the road surface defect detection method, system, device and medium of the present invention, a large number of historical road surface image data are collected as pre-training samples; on the pre-training samples, an optimized self-supervised learning algorithm is used to pre-train the backbone network of the road surface defect detection model; a small number of samples are randomly selected from the pre-training samples and labeled to obtain training samples; the training samples are used to fine-tune the road surface defect detection model to obtain the final road surface defect detection model; the image to be detected is input into the road surface defect detection model to obtain the defect detection result; the detection method of the present invention can effectively reduce the amount of data required for training the road surface defect detection model; at the same time, improve the detection accuracy.

[0094] Embodiment

[0095] As shown in the attached Figure 1 figure, this embodiment provides a road surface defect detection method, including the following steps:

[0096] Step 1: Obtain a number of historical road surface image data and construct pre-training samples. In this embodiment, the pre-training samples are only used in the pre-training process of the backbone network in the road surface detection model; among them, in the pre-training process, there is no need to perform defect annotation on the historical road surface image data.

[0097] Step 2: Construct a road surface detection model; among them, the road surface detection model includes a backbone network and a classifier network.

[0098] In this embodiment, the backbone network of the road surface detection model is used to extract features from the road surface image to obtain an intermediate feature map; the backbone network of the road surface detection model is a residual network; the classifier network of the road surface detection model is used to perform classification and target detection processing on the intermediate feature map to obtain an image detection result; preferably, the classifier network of the road surface detection model includes a classification unit and a target detection unit; among them, the classification unit is a binary classifier model, and the target detection unit is a target defect detection model based on the Faster RCNN algorithm or the YOLO algorithm.

[0099] Step 3: Based on the optimized self-supervised learning algorithm, use the pre-training samples to pre-train the backbone network in the road surface detection model to obtain a pre-trained road surface detection model.

[0100] The self-supervised learning algorithm mainly uses auxiliary means to mine its own supervision information from a large amount of unsupervised data, and then trains the neural network through the self-supervised information of itself, so as to learn features valuable for downstream tasks; in this embodiment, the optimized self-supervised learning algorithm is a self-supervised learning algorithm based on contrast learning, which constructs positive samples (positive) and negative samples (negative); then realizes self-supervised learning by measuring the distance between the positive sample and the negative sample; among them, the core idea of the optimized self-supervised learning algorithm is: the distance between the sample and the positive sample is much greater than the distance between the sample and the negative sample.

[0101] In this embodiment, the optimized self-supervised learning algorithm is the MoCo algorithm that introduces a reconstructed image branch neural network, that is, the Momentum Contrast for Unsupervised Visual Representation Learning algorithm; the backbone network of the MoCo algorithm is the same as the backbone network of the road surface detection model, both of which are residual networks (Residual Network, ResNet); among them, the pre-training process adopts a joint training strategy; during the training process, the reconstruction image loss and the contrast loss are optimized simultaneously.

[0102] As shown in the appendix Figure 2 as follows:

[0103] Step 31: Use the backbone network of the MoCo algorithm to extract features from the pre-training samples to obtain intermediate feature maps.

[0104] Step 32: Use the reconstructed image branch neural network to perform image reconstruction processing on the intermediate feature maps to obtain reconstructed images.

[0105] In this embodiment, in order to improve the expression of road surface defect features by the pre-trained backbone network, an optimized MoCo algorithm is used as the self-supervised learning algorithm in the pre-training stage; the optimization content is mainly reflected in: adding an image reconstruction branch neural network after the backbone network of the MoCo algorithm; the input of the image reconstruction branch neural network is the intermediate feature map, and the output is the reconstructed image; preferably, the reconstructed image branch neural network includes a convolutional layer, a transposed convolutional layer, and bilinear interpolation.

[0106] In the road surface defect detection scenario, the defects in the scenario images are mainly manifested as surface cracks, etc., and the texture features of the non-defect areas are relatively single, and most of the areas on the scenario images are non-defect areas; in addition, the characteristics of the defects themselves are relatively prominent; the defect type can be determined according to the shape features of the target defects, and there are no spatial relationships and complex semantic information between the target defects; therefore, in this embodiment, by adding a branch of the reconstructed image branch neural network to the MoCo algorithm, the purpose is to enhance the expression of the shape features of the intermediate feature maps output by its backbone network.

[0107] It should be noted that: texture feature is a global feature that describes the surface properties of the target corresponding to the sample image or the sample image area; there are two types of representation methods for shape features, one is contour feature, and the other is region feature. The contour feature of the sample image mainly refers to the outer boundary of the object, while the region feature of the sample image is related to the entire shape area; the spatial relationship feature refers to the mutual spatial position or relative direction relationship between multiple targets in the sample image, and these relationships can also be divided into connection or adjacency relationships, overlapping or overlapping relationships, and inclusion or containment relationships, etc.

[0108] Step 33: Use the input sample image as the label to calculate the reconstruction image loss between the pre-training sample and the reconstructed image to obtain the first loss; where the first loss is:

[0109] Loss rec =F(a i ,a p )

[0110] where Loss rec is the first loss, that is, the reconstruction image loss between the pre-training sample and the reconstructed image; F(a i ,a p) is the error value between the pre-training sample and the reconstructed image; a i is the pre-training sample; a p is the reconstructed image.

[0111] In this embodiment, the error value F(a i , a p ) between the pre-training sample and the reconstructed image is the mean squared error MSE or structural similarity (SSIM).

[0112] Step 34: Use the feature extraction branch neural network of the MoCo algorithm to perform feature extraction processing on the intermediate feature map to obtain a number of feature vectors; wherein, the number of feature vectors corresponds to a number of historical road surface image data one by one.

[0113] Step 35: Use the contrast loss function of the MoCo algorithm to calculate the contrast loss between a number of feature vectors to obtain a second loss; in this embodiment, the contrast loss function is the loss function used in the original MoCo algorithm, and feature learning is performed by optimizing the contrast loss.

[0114] Step 36: Use the first loss and the second loss to obtain a comprehensive loss; wherein, the comprehensive loss is the sum of the weighted first loss and the second loss; the comprehensive loss is a hyperparameter, and the most suitable value needs to be searched in different application scenarios or data sets; the calculation formula of the comprehensive loss is as follows:

[0115] Loss total = W1 * Loss rec + W2 * Loss moco

[0116] where, Loss moco is the second loss; W1 is the weight of the first loss; W2 is the weight of the second loss. Among them

[0117] Step 37: Use the comprehensive loss function to update the model parameters of the backbone network of the MoCo algorithm by backpropagation;

[0118] Step 38: Determine whether the comprehensive loss converges or the number of training iterations reaches the preset number of iterations. If so, output the model parameters of the updated backbone network of the MoCo algorithm; if not, return to continue training and iterating.

[0119] Step 39: Substitute the model parameters of the updated backbone network of the MoCo algorithm into the backbone network in the road surface detection model, and the pre-trained road surface detection model is obtained.

[0120] Step 4: Randomly obtain a portion of sample data from the pre-training samples, and perform defect annotation on the portion of sample data to obtain fine-tuning training samples. In this embodiment, the fine-tuning training samples are obtained by selecting a small amount of sample data from the pre-training samples for defect annotation; it should be noted that the "small amount" here is a relative value, meaning that the amount of sample data here is much smaller than the amount of data required to train a randomly initialized defect detection model.

[0121] In this embodiment, a specific defect detection task is defined according to actual needs, and the portion of sample data is annotated; it should be noted that the scenarios for road surface defect detection include, but are not limited to, surface defect detection in scenarios such as highway road surfaces, bridges, and tunnels; here, the defect detection tasks include, but are not limited to, classification and target defect detection; among them, classification is to classify images of different types of defects into different categories, and target defect detection is to represent the defects and defect types in the image with rectangular boxes. For example, when the actual need is only to know whether there are defects in the image, rather than the position and defect category of the defects in the image, the current need can be defined as a binary classification task at this time.

[0122] Step 5: Use the fine-tuning training samples to fine-tune the parameters of the backbone network and the classifier network in the pre-trained road surface detection model to obtain the trained road surface defect detection model.

[0123] As shown in the appendix Figure 3 The process of fine-tuning training is as follows:

[0124] Step 51: Input the fine-tuning training samples into the backbone network of the pre-trained road surface detection model to obtain intermediate feature maps.

[0125] Step 52: Input the intermediate feature maps into the classifier network in the pre-trained road surface detection model to output the predicted classification probabilities of the fine-tuning training samples.

[0126] Step 53: Calculate the cross-entropy loss between the predicted classification probabilities of the fine-tuning training samples and the defect annotations of the fine-tuning training samples according to the predicted classification probabilities of the fine-tuning training samples and the defect annotations of the fine-tuning training samples, and use the cross-entropy loss between the predicted classification probabilities of the fine-tuning training samples and the defect annotations of the fine-tuning training samples as the training loss function of the fine-tuning model.

[0127] Step 54: According to the training loss function of the fine-tuning model, use the backpropagation algorithm to adjust the parameters of the backbone network and the classifier network in the pre-trained road surface detection model so that the training loss value of the training loss function of the fine-tuning model reaches the preset convergence condition, and output the trained road surface defect detection model.

[0128] In this embodiment, a classifier network is added after the pre-trained backbone network to obtain a road surface defect detection model. The classifier network is a neural network model, and its network structure includes one or more Linear layers and a Softmax layer. The backbone network is connected to the classifier and can be used as a model for classifying road surface defects after fine-tuning on training samples. Since the backbone network has been pre-trained, it can converge by fine-tuning with training samples.

[0129] In this embodiment, when assuming a binary classification task currently, a binary classifier is added after the pre-trained backbone network. When assuming a target defect detection task currently, target detection algorithms such as Faster RCNN or YOLO can be selected, and the parameters of the backbone network after pre-training are used as the initialization parameters of the backbone network in the target detection algorithm.

[0130] After assigning values to the backbone network of the defect detection algorithm, fine-tuning is performed on the defect detection model using the labeled training samples. Since the backbone network of this model has been pre-trained, the model can still converge and maintain a high accuracy even when the number of training samples is small. The embodiment of this application requires a small amount of training data during the fine-tuning stage of the defect detection model, so a large amount of annotation cost can be saved, solving the problem of high annotation cost in road surface defect detection and difficulty in actual project implementation.

[0131] Step 6: Input the road surface image to be detected into the trained road surface defect detection model, and output the result, that is, obtain the road surface defect detection result.

[0132] For the description of the relevant parts in a road surface defect detection system, device, and computer-readable storage medium provided in this embodiment, reference can be made to the detailed description of the corresponding parts in the road surface defect detection method described in this embodiment, which will not be elaborated here.

[0133] The road surface defect detection in this invention includes but is not limited to the detection of road surface defects. Similar scenarios include the defect detection of building concrete surfaces such as bridges and houses. When collecting images in the above scenarios, it is easily affected by natural environmental factors such as weather and light, resulting in unstable image quality. Affected by image quality factors, and the actual application scenario has relatively high requirements for the accuracy of road surface defect detection. It is required that the defect detection algorithm must ensure high detection accuracy and also have good generalization ability. In this invention, a common measure to improve detection accuracy and generalization ability without changing the existing detection algorithm is to increase training samples. Through pre-training of the backbone network and then using fewer fine-tuning training samples for secondary fine-tuning training, it can maintain high detection accuracy on the premise of ensuring the convergence of the model, greatly reducing the annotation cost of training data.

[0134] The above embodiments are only one of the implementation manners capable of implementing the technical solution of the present invention. The scope of protection required by the present invention is not only limited by this embodiment, but also includes any changes, substitutions and other implementation manners that are easily conceivable by any person skilled in the art within the technical scope disclosed by the present invention.

Claims

1. A road surface defect detection method, characterized in that: The following steps are involved: Obtain some historical road image data and construct pre-training samples; Constructing a road surface detection model; wherein the road surface detection model includes a backbone network and a classifier network; Based on the optimized self-supervised learning algorithm, the backbone network in the road surface detection model is pre-trained using pre-training samples to obtain a pre-trained road surface detection model. Randomly obtain part of the sample data from the pre-training samples, and perform defect labeling on the part of the sample data to obtain fine-tuning training samples; Using the fine-tuning training samples, the parameters of the backbone network and the classifier network in the pre-trained road surface detection model are fine-tuned to obtain a trained road surface defect detection model. Input the road surface image to be detected into the trained road surface defect detection model and output the result, that is, obtain the road surface defect detection result; The backbone network of the road surface detection model is a residual network; the optimized self-supervised learning algorithm is a MoCo algorithm that introduces a reconstructed image branch neural network; the backbone network of the MoCo algorithm is the same as the backbone network of the road surface detection model; Based on the optimized self-supervised learning algorithm, the backbone network in the road surface detection model is pre-trained using pre-training samples to obtain the pre-trained road surface detection model. The details are as follows: Using the backbone network of the MoCo algorithm, feature extraction is performed on the pre-training samples to obtain an intermediate feature map; Using the reconstructed image branch neural network, the intermediate feature map is reconstructed to obtain a reconstructed image; Calculate the reconstructed image loss between the pre-training sample and the reconstructed image to obtain the first loss; Using the feature extraction branch neural network of the MoCo algorithm, feature extraction processing is performed on the intermediate feature map to obtain a plurality of feature vectors; the plurality of feature vectors correspond one-to-one to a plurality of historical road surface image data; Calculate the contrast loss between several feature vectors to obtain the second loss; using the first loss and the second loss to obtain a comprehensive loss; Using the comprehensive loss, the model parameters of the backbone network of the MoCo algorithm are updated by back propagation; Determine whether the comprehensive loss converges or the number of training iterations reaches the preset number of iterations. If so, output the model parameters of the backbone network of the updated MoCo algorithm; if not, return to continue training iterations; The model parameters of the backbone network of the updated MoCo algorithm are brought into the backbone network of the road surface detection model to obtain the pre-trained road surface detection model.

2. A road surface defect detection method according to claim 1, characterized in that: The backbone network of the road surface detection model is used to extract features from road surface images to obtain intermediate feature maps; The classifier network of the road surface detection model is used to classify and perform target detection processing on the intermediate feature map to obtain an image detection result.

3. A road surface defect detection method according to claim 1, characterized in that: The comprehensive loss is obtained by weighted summing the first loss and the second loss.

4. A road surface defect detection method according to claim 1, characterized in that: The classifier network of the road surface detection model includes a classification unit and a target detection unit; wherein the classification unit is a binary classifier model, and the target detection unit is a target defect detection model based on the Faster RCNN algorithm or the YOLO algorithm.

5. A road surface defect detection method according to claim 1, characterized in that: The process of fine-tuning the parameters of the backbone network and classifier network in the pre-trained road surface detection model using fine-tuning training samples to obtain the trained road surface defect detection model is as follows: Input the fine-tuning training samples into the backbone network of the pre-trained road surface detection model to obtain the intermediate feature map; Input the intermediate feature map into the classifier network in the pre-trained road surface detection model, and output the predicted classification probability of the fine-tuned training sample; According to the predicted classification probability of the fine-tuning training sample and the defect annotation of the fine-tuning training sample, a cross entropy loss of the predicted classification probability of the fine-tuning training sample and the defect annotation of the fine-tuning training sample is calculated, and the cross entropy loss of the predicted classification probability of the fine-tuning training sample and the defect annotation of the fine-tuning training sample is used as the training loss function of the fine-tuning model; According to the training loss function of the fine-tuning model, the parameters of the backbone network and the classifier network in the pre-trained road surface detection model are adjusted through the back propagation algorithm so that the training loss value of the training loss function of the fine-tuning model reaches the preset convergence condition, and the trained road surface defect detection model is output.

6. A road surface defect detection system, characterized in that: include: The data module is used to obtain some historical road image data and construct pre-training samples; A modeling module, configured to construct a road surface detection model; wherein the road surface detection model includes a backbone network and a classifier network; A pre-training module is used to pre-train the backbone network in the road surface detection model based on the optimized self-supervised learning algorithm and using pre-training samples to obtain a pre-trained road surface detection model; A fine-tuning sample module is used to randomly obtain part of the sample data from the pre-training samples and mark defects on the part of the sample data to obtain fine-tuning training samples; A fine-tuning training module is used to fine-tune the parameters of the backbone network and the classifier network in the pre-trained road surface detection model using the fine-tuning training samples to obtain a trained road surface defect detection model; The prediction output module is used to input the road surface image to be detected into the trained road surface defect detection model and output the result, that is, to obtain the road surface defect detection result; The backbone network of the road surface detection model is a residual network; the optimized self-supervised learning algorithm is a MoCo algorithm that introduces a reconstructed image branch neural network; the backbone network of the MoCo algorithm is the same as the backbone network of the road surface detection model; Based on the optimized self-supervised learning algorithm, the backbone network in the road surface detection model is pre-trained using pre-training samples to obtain the pre-trained road surface detection model. The details are as follows: Using the backbone network of the MoCo algorithm, feature extraction is performed on the pre-training samples to obtain an intermediate feature map; Using the reconstructed image branch neural network, the intermediate feature map is reconstructed to obtain a reconstructed image; Calculate the reconstructed image loss between the pre-training sample and the reconstructed image to obtain the first loss; Using the feature extraction branch neural network of the MoCo algorithm, feature extraction processing is performed on the intermediate feature map to obtain a plurality of feature vectors; the plurality of feature vectors correspond one-to-one to a plurality of historical road surface image data; Calculate the contrast loss between several feature vectors to obtain the second loss; using the first loss and the second loss to obtain a comprehensive loss; Using the comprehensive loss, the model parameters of the backbone network of the MoCo algorithm are updated by back propagation; Determine whether the comprehensive loss converges or the number of training iterations reaches the preset number of iterations. If so, output the model parameters of the backbone network of the updated MoCo algorithm; if not, return to continue training iterations; The model parameters of the backbone network of the updated MoCo algorithm are brought into the backbone network of the road surface detection model to obtain the pre-trained road surface detection model.

7. A road surface defect detection device, characterized in that: include: memory for storing computer programs; A processor is configured to implement the steps of the road surface defect detection method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the road surface defect detection method according to any one of claims 1 to 5 are implemented.

Images