Pavement crack detection method, system and equipment based on knowledge distillation

Abstract

The invention provides a pavement crack detection method, system and equipment based on knowledge distillation, and the method comprises the steps: constructing an image data set which comprises an image data set of a source domain and an image data set of a target domain; performing data preprocessing and data expansion based on the image data set; the method specifically comprises the following steps: carrying out adaptive block enhancement and global equalization preprocessing on image data on a target domain, combining the image data with an image data set of a source domain, and carrying out data expansion to obtain an expanded image data set; based on the expanded image data set, performing knowledge distillation training on a preset student model by using a pre-trained teacher model, and taking the student model after knowledge distillation as a pavement crack detection model; and performing crack detection on a to-be-detected pavement image by using the pavement crack detection model. The pavement crack detection method based on knowledge distillation is adopted, so that the problem that an existing pavement crack detection method is low in accuracy is solved fundamentally.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to a pavement crack detection method, system and equipment based on knowledge distillation. Background Art

[0002] With economic development, my country's infrastructure construction has been continuously improved, and highways have basically covered all regions, making transportation increasingly convenient. However, as time goes by, roads will develop cracks and other diseases due to oxidation, long-term repeated stress, rain erosion and other factors. When slight cracks appear on the road surface, if maintenance is not timely, the cracks may cause serious traffic accidents. Therefore, pavement crack detection has important research significance.

[0003] Existing pavement crack detection methods include: lightweight detection networks based on quantization, lightweight detection networks based on network pruning, and lightweight detection networks based on network architecture technology. Specifically, (1) lightweight pavement crack detection networks based on quantization, the quantization process involves converting model parameters and activation functions from floating point numbers to integers or low bit width data representations. This conversion may result in information loss, especially when quantized to a lower bit width, which may affect model performance, especially in the case of a larger degree of quantization. In addition, quantization technology introduces additional complexity, including the quantization and dequantization process of the model, as well as special operations during inference, which may require more work to integrate and maintain. (2) lightweight pavement crack detection networks based on network pruning, the network pruning process introduces additional complexity, including the selection of channel pruning strategies, the adjustment of pruning hyperparameters, and model retraining. This may require more engineering and computing resources to implement. Incorrect network pruning decisions may lead to a decline in model performance, especially when pruning is too aggressive or inappropriate. Pruning may harm the performance of the model, and careful adjustment and verification are required to ensure that performance is not affected. In addition, some network pruning techniques may require special hardware support, such as accelerators that support sparse matrix multiplication. This may increase the complexity and cost of the system. (3) Lightweight pavement crack detection network based on network architecture technology. In order to reduce the size and computational complexity of the model, lightweight networks usually need to reduce the depth, width and number of parameters. This may lead to performance degradation, especially when compared with large and complex models. At the same time, lightweight networks are often limited by available hardware. In some cases, the hardware may not support certain lightweight model architectures or operations. In summary, there are two main problems with existing pavement crack detection methods: first, different hardware or software have a greater impact on the pavement crack detection model; second, useful information is easily lost when lightweight. The above problems lead to inaccurate pavement crack detection results.

[0004] Therefore, how to improve the accuracy of pavement crack detection is a problem that needs to be solved urgently in this field. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system and device for pavement crack detection based on knowledge distillation in view of the defects of the prior art. The pavement crack detection method based on knowledge distillation of the present invention calculates the mask of each crack image in the expanded image data set according to the real bounding box of the object and the existing anchor box, calculates the imitation loss based on the mask and the output features of the imitation teacher model in the local feature area close to the object, and calculates the distillation loss to balance the high-level feature layer of the teacher model and the guiding layer of the student model, so as to better transfer the knowledge from the teacher model to the student model, thereby solving the problem of low accuracy of existing pavement crack detection from the root.

[0006] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0007] A pavement crack detection method based on knowledge distillation, characterized in that it comprises the steps of:

[0008] S1. Construct an image dataset, where the image dataset includes an image dataset of a source domain and an image dataset of a target domain;

[0009] S2, based on the image data set, performing data preprocessing and data expansion; specifically comprising: performing data expansion after the image data on the target domain is subjected to adaptive block enhancement and global equalization preprocessing, and merging the image data on the source domain with the image data set on the source domain to obtain an expanded image data set;

[0010] S3, based on the expanded image data set, using the pre-trained teacher model to perform knowledge distillation training on the preset student model, and using the student model after knowledge distillation as a pavement crack detection model; comprising the following steps:

[0011] S31, calculating a mask I of each crack image in the expanded image dataset according to the real bounding box of the object and the existing anchor box;

[0012] S32, calculating the imitation loss based on the output features of the mask and the imitation teacher model in the local feature area close to the object; the specific calculation formula of the imitation loss is as follows:

[0013]

[0014] Among them, s is defined as the feature map of the guided feature layer of the student model, and t is defined as the corresponding high-level feature response map of the teacher model; for each anchor box position (i, j) close to the object on the feature map of width W and height H, (i, j) represents the adaptation function of the last hidden layer of the student model, I ij Represents the mask value at the anchor box position (i, j), C represents the number of channels of the feature map, f(s) ijc Represents the feature response of the cth channel at position (i, j) on the feature map;

[0015] S33. Calculate the distillation loss to balance the high-level feature layer of the teacher model and the guidance layer of the student model, and transfer the knowledge of the teacher model; the distillation loss is calculated as follows:

[0016]

[0017] Among them, the variable ρ represents the weight ratio of distillation loss, η box , η cls and η obj Represents that they are in L kd The weight in obj Indicates whether this batch of images contains objects. If so, it is λ obj =1, otherwise λ obj =0; They are the bounding box loss, classification loss, confidence loss, and domain loss in the previous step;

[0018] S34. Calculate the total loss of the student model based on the imitation loss and the distillation loss and in combination with the loss of the teacher model;

[0019] L stu =L+L kd +L imit ;

[0020] Where L is the total loss of the teacher model;

[0021] S35, performing knowledge distillation training on a preset student model based on the total loss of the student model, and using the student model after knowledge distillation as a pavement crack detection model;

[0022] S4. Using the pavement crack detection model to perform crack detection on the pavement image to be detected.

[0023] Specifically, in step S2, the image data in the target domain is subjected to adaptive block enhancement and global equalization preprocessing, and the specific steps are as follows:

[0024] S21, dividing the image into blocks according to the resolution size, and the block size increases from 50×50 to 500×500, with a step size of 50 each time;

[0025] S22, performing a gamma transformation on each block to correct the gray part, wherein the gamma value is set from 0.5 to 2, and the increment step is 0.1;

[0026] S23, when the block size is fixed, the variance σ is calculated for all blocks, and the Gamma value when the maximum σ is obtained after the block is enhanced is set as the final local enhancement parameter of the region;

[0027] S24, performing smoothing processing on the pixel points between adjacent blocks;

[0028] S25, CLAHE algorithm is used for global enhancement to balance the overall image.

[0029] Specifically, in step S3, YOLO v5s is used as the student model weight to be trained.

[0030] Specifically, step S31 includes the following steps:

[0031] S311. For each true bounding box of the crack image, calculate the IOU value between it and all anchor boxes to obtain a W×H×K IOU map, denoted as m IOU , where K represents the number of anchor boxes preset by YOLO v5, W represents the width of the image, and H represents the height of the image;

[0032] S312, in m IOU Find the IOU map corresponding to the maximum IOU value, denoted as M=max(m IOU ), and use this to construct a filter F, in represents a threshold factor;

[0033] S313, use filter F to filter m IOU , retain the IOU graphs larger than F and combine them using the or operator to obtain a W×H mask;

[0034] S314. In this way, all the real frames of the crack images are looped, and finally a fine-grained mask imitating the teacher model is obtained, which is recorded as I.

[0035] The present invention also proposes a pavement crack detection system based on knowledge distillation, characterized in that the pavement crack detection system executes the pavement crack detection method based on knowledge distillation, including: an image data set construction module, a data preprocessing and expansion module, a model training module, and an image crack detection module;

[0036] An image dataset construction module, wherein the image dataset includes an image dataset of a source domain and an image dataset of a target domain;

[0037] A data preprocessing and expansion module performs data expansion after the image data in the target domain is subjected to adaptive block enhancement and global equalization preprocessing, and then merges the image data set in the source domain to obtain an expanded image data set;

[0038] A model training module, based on the expanded image data set, uses a pre-trained teacher model to perform knowledge distillation training on a preset student model, and uses the student model after knowledge distillation as a pavement crack detection model;

[0039] The image crack detection module uses the pavement crack detection model to perform crack detection on the pavement image to be detected.

[0040] The present invention further provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0041] Compared with the prior art, it has the following beneficial effects:

[0042] The pavement crack detection method based on knowledge distillation of the present invention calculates the mask of each crack image in the expanded image data set according to the real bounding box of the object and the existing anchor box, calculates the imitation loss based on the mask and the output features of the imitation teacher model in the local feature area close to the object, and calculates the distillation loss to balance the high-level feature layer of the teacher model and the guiding layer of the student model, so as to better transfer the knowledge from the teacher model to the student model, that is, to use knowledge distillation to transfer the detection knowledge of the large-scale excellent teacher model to the small network model, thereby realizing the accurate detection of the small model and improving the accuracy of pavement crack detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of a pavement crack detection method based on knowledge distillation provided in an embodiment of the present invention.

[0045] Figure 2 A network architecture diagram of a pavement crack detection method based on knowledge distillation provided in an embodiment of the present invention.

[0046] Figure 3 A schematic diagram of a local feature area and a process diagram of a teacher model guiding a student model provided in an embodiment of the present invention.

[0047] Figure 4 A schematic diagram of a method for a local feature estimation region module provided in an embodiment of the present invention.

[0048] Figure 5 Schematic diagram of a pavement crack detection system based on knowledge distillation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0051] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0052] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0053] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0054] The present invention proposes a pavement crack detection method based on knowledge distillation. Figure 1 As shown, the pavement crack detection method based on knowledge distillation includes the following steps S1 to S4. At the same time, the network architecture of the present invention is as follows Figure 2 shown.

[0055] S1. Construct an image dataset, where the image dataset includes an image dataset of a source domain and an image dataset of a target domain.

[0056] The CQU-BPDD dataset contains 60,059 asphalt pavement images, with a total of eight categories, including seven types of pavement cracks, namely transverse cracks, large cracks, cracks, crack casting, longitudinal cracks, potholes, and repairs. These images are mainly taken automatically by professional on-board high-definition cameras on highways in southern my country. The shooting time and location are different, and then handed over to professionals for data annotation. The CQU-BPDD training set has 49,919 images and the test set has 10,140 images. Each pavement image corresponds to a 2×3 meter highway pavement patch with a resolution of 1200×900 and only image-level labels.

[0057] RDD2020 is a crack image dataset with position annotation for automatic road crack detection. It includes 26,620 road images collected from Japan, India, and the Czech Republic, including a training set of size 21,041, with four road disease categories, namely transverse cracks, longitudinal cracks, potholes, and cracks. The resolution of each image in the training set is 600×600, and contains an XML crack location information annotation file in the PASCALVOC format.

[0058] The CFD dataset originally consists of 118 images with a resolution of 480 × 320. Since the CFD dataset is intended to study the so-called road crack detection task, i.e., the road crack segmentation task from the perspective of computer vision, all samples are actually lesion images.

[0059] CQU-BPMDD is a large-scale asphalt pavement multi-label disease dataset released by the present invention, which can be used for detection and classification tasks. It contains 9851 crack images and 29143 normal images from highways in different regions in Southwest China, and the resolution of each image is 3692×2147. There are seven categories of crack images, namely transverse cracks, longitudinal cracks, large cracks, repairs, looseness, wave congestion, and potholes. Currently, only three categories have a large amount of data, namely longitudinal cracks, transverse cracks, and repairs. Unlike CQU-BPDD, CQU-BPMDD is a crack disease dataset in which all images are imaged using the exposure compensation method, most of which are small crack images, which are more in line with actual scenes and difficult to detect.

[0060] There are more training data in the CQU-BPDD and RDD2020 datasets, and the categories they contain are relatively similar, so these two datasets are selected as the target domain and source domain in domain adaptation training respectively. The CFD dataset contains less data and does not contain normal images. The present invention uses image restoration methods to generate normal crack images and uses them as a verification set for cross-dataset generalization capabilities. In addition, cross-dataset verification is also performed on the CQU-BPMDD dataset, which can better reflect the cross-dataset generalization capabilities of the designed model.

[0061] S2. Based on the image data set, data preprocessing and data expansion are performed; specifically, the method includes: performing data expansion after the image data on the target domain is subjected to adaptive block enhancement and global equalization preprocessing, and then merging the image data set on the source domain with the image data set on the source domain to obtain an expanded image data set.

[0062] Since the road images of CQU-BPDD are captured under different road conditions and at different times, the cracks in the road images are usually affected by the illumination, showing a serious uneven grayscale distribution. In order to enhance the road images, a simple and effective preprocessing method is proposed, namely adaptive block enhancement and global equalization. Specifically, in order to improve the local contrast, the strategy of enhancing the image by dividing it into patches is adopted.

[0063] In step S2, the image data in the target domain is subjected to adaptive block enhancement and global equalization preprocessing, and the specific steps are as follows:

[0064] S21, dividing the image into blocks according to the resolution size, and the block size increases from 50×50 to 500×500, with a step size of 50 each time;

[0065] S22, performing a gamma transformation on each block to correct the gray part, wherein the gamma value is set from 0.5 to 2, and the increment step is 0.1;

[0066] S23, when the block size is fixed, the variance σ is calculated for all blocks, and the Gamma value when the block is enhanced to obtain the maximum σ is set as the final local enhancement parameter of the region; when σ is larger, the contrast of the image is improved, and the cracks are easier to distinguish;

[0067] S24, performing smoothing processing on the pixel points between adjacent blocks;

[0068] When all patches are converted, the grayscale value conversion between adjacent patches may be discontinuous. At this time, the pixels between adjacent patches are smoothed;

[0069] S25, CLAHE algorithm is used for global enhancement to balance the overall image.

[0070] In order to further suppress the negative impact of discontinuous gray value transformation, the CLAHE algorithm is used to equalize the overall image.

[0071] The target domain crack image is processed through steps S21-S25 to obtain an adaptively enhanced image, and then the adaptively enhanced image is merged with the image dataset of the source domain to perform data expansion to obtain an expanded image dataset.

[0072] After the data in the target domain is preprocessed by the proposed adaptive block enhancement and global equalization, it is augmented together with the data in the source domain through common data augmentation methods and input into the designed student model for training. The augmentation methods mainly use Gaussian processing, image sharpening, flipping, translation, channel scaling, and image contrast transformation. Among them, the target domain CQU-BPDD is augmented using six methods, and the source domain RDD2020 randomly selects one method for augmentation.

[0073] S3, based on the expanded image data set, using the pre-trained teacher model to perform knowledge distillation training on the preset student model, and using the student model after knowledge distillation as a pavement crack detection model;

[0074] First, when performing distillation, it is usually necessary to select a lightweight model structure as the student model. Object detection networks can usually choose weights of different sizes for training. In order to obtain good detection results, weights with a large number of parameters and a more complex model structure are often selected for training. YOLO v5 is selected as the Baseline, and the main weight types of YOLO v5 are as follows: YOLO v5s, YOLO v5m, YOLO v5n, YOLO v5l, and YOLO v5x. Among them, the number of parameters and FLOPs of YOLO v5x weights are the largest. Compared with YOLO v5n and YOLO v5s, the number of parameters and FLOPs are more than ten times larger. The larger the number of network parameters, the higher the detection accuracy, and the small parameter network is more likely to fall into the local optimum than the large parameter network. Therefore, in order to realize a lightweight crack detection network, it is necessary to study these five weights separately, that is, to verify them on the CQU-BPDD dataset. The results show that due to the small number of parameters, the detection effect of YOLO v5n is far behind that of the other four weights. The main reason is that in the case of small cracks, this detection requires not only low-level spatial features, but also abstract and deep semantic information. However, YOLO v5n with shallow layers and fewer parameters cannot learn the semantic information of small cracks well, which easily leads to wrong detection. Taking into account the parameter sizes of different weights and the actual situation of the detection results, YOLO v5s is used as the student model weight to be trained, and it is used together with the YOLO v5x pre-trained by the deep domain adaptive crack detection network DDACDN for model lightweight research. Subsequent work uses knowledge distillation to transfer the detection knowledge of the pre-trained YOLO v5x weight model to the model based on YOLO v5s to achieve accurate detection of small models.

[0075] That is, in step S3, YOLO v5s is used as the student model weight to be trained, and the pre-trained YOLOv5x is used as the teacher model.

[0076] The specific steps of step S3 are as follows:

[0077] S31, calculating a mask I of each crack image in the expanded image dataset according to the real bounding box of the object and the existing anchor box;

[0078] Generally speaking, the local position feature information near the object to be detected is very important in the detection process, and this area contains rich knowledge. Because in the distillation process, this area usually reveals how a trained complex teacher model separates the ordinary image background from the object to be detected. Therefore, the method proposed in this section is based on these detection knowledge-rich areas. In order to formally define this area for calculation, the present invention uses the real bounding box and the existing anchor box to design a mask for the area, and participates in the subsequent imitation loss calculation, so that the student model imitates the teacher model's response to their high-level features. Figure 3 , Figure 4 As shown in Figure 1, the local feature region of each object is formed near the position of the object anchor box. The red and green bounding boxes on the left image are the anchor boxes at the corresponding positions. The red anchor box has the largest overlap with the ground truth bounding box, while the green anchor box is closer to the object. Figure 3 It reveals how a learned teacher model can be generalized to guide the learning of the student model. It reflects that the high-level feature response of the pre-trained teacher model contains the positioning information of the anchor box area close to the object, so the present invention performs fine-grained feature imitation around this area. Since the anchor box mechanism of the YOLO v5 detection model will be recalculated and fitted before the first training, the overlap between the anchor box and the real annotation box of the object is large, and this feature can be well formalized for the local feature area.

[0079] For the student model, in order to study the local feature area of ​​the object, the present invention calculates a mask for each crack image.

[0080] Step S31, the specific steps of calculating the mask I of each crack image include:

[0081] S311. For each true bounding box of the crack image, calculate the IOU value between it and all anchor boxes to obtain a W×H×K IOU map, denoted as m IOU , where K represents the number of anchor boxes preset by YOLO v5, W represents the width of the image, and H represents the height of the image;

[0082] S312, in m IOU Find the IOU map corresponding to the maximum IOU value, denoted as M=max(mIOU ), and use this to construct a filter F, in represents a threshold factor;

[0083] S313, use filter F to filter m IOU , retain the IOU graphs larger than F and combine them using the or operator to obtain a W×H mask;

[0084] S314. In this way, all the real frames of the crack images are looped, and finally a fine-grained mask imitating the teacher model is obtained, which is recorded as I.

[0085] S32, calculating the imitation loss based on the mask and the output features of the imitation teacher model in the local feature area close to the object;

[0086] The final mask of the crack image is obtained through step S31, which represents the local feature area close to the object, and the imitation loss needs to be calculated for it. The present invention directly calculates the feature responses of the teacher model and the student model on the last hidden layer of the student model. It is based on the following two considerations: 1. The architecture of the student model and the teacher model designed by the present invention belongs to a simplified structure type, sharing a model framework YOLOv5. During training, the number of channels of the teacher model and the student model are compatible, and the feature imitation loss can be directly calculated. 2. The teacher model is a pre-trained model with excellent performance. The high-level feature response of the teacher model close to the anchor box position of the object can be simulated on the student model with a smaller network structure. The knowledge distillation can be performed well, so that the student model can fully learn the detection knowledge of the teacher model for the local area close to the object.

[0087] When calculating the imitation loss, high-level feature maps of the student model and the teacher model are required. Specifically, define s as the feature map of the guided feature layer of the student model, and define t as the corresponding high-level feature response map of the teacher model. For each anchor box position (i, j) close to the object on the feature map of width W and height H, this section trains the student model to minimize the following loss.

[0088]

[0089] Wherein, f(*) represents the adaptation function of the last hidden layer of the student model. The present invention also needs to imitate the local feature area, calculate the loss through it, and update the network in reverse. Therefore, in order to further learn the knowledge of the teacher detection model teacher model, it is necessary to use the local area mask obtained in the previous step for further distillation. The specific calculation formula of the imitation loss is as follows:

[0090]

[0091] Among them, s is defined as the feature map of the guided feature layer of the student model, and t is defined as the corresponding high-level feature response map of the teacher model; for each anchor box position (i, j) close to the object on the feature map of width W and height H, (i, j) represents the adaptation function of the last hidden layer of the student model, I ij Represents the mask value at the anchor box position (i, j), C represents the number of channels of the feature map, f(s) ijc Represents the feature response of the cth channel at position (i, j) on the feature map.

[0092] Here, f(s) is the feature map processed by the adaptation function f from the last hidden layer of the student model, which represents the feature response of the teacher model of the cth channel at position (i, j) on the feature map. By minimizing this formula, the student model imitates the teacher model to make feature responses close to the local area of ​​the object, and gets closer and closer to the teacher model, thereby achieving the purpose of guiding the student model.

[0093] S33, calculate the distillation loss to balance the high-level feature layer of the teacher model and the guidance layer of the student model, and transfer the knowledge of the teacher model;

[0094] In the process of training the student model, in order to balance the high-level feature layer of the teacher model and the guiding layer of the student model, the present invention further designs a distillation loss. The soft labels predicted by the teacher model are calculated together with the predictions of the student model. Specifically, during the distillation process, a variable ρ is set to represent the weight ratio of the distillation loss. Usually, the classification task uses a softmax layer to output the probabilities of each category. L-DDACDN uses the YOLO v5 model. When calculating the classification loss, binary cross entropy is used to calculate each label, which reduces the complexity of the loss calculation. The distillation loss is calculated as follows:

[0095]

[0096] Among them, the variable ρ represents the weight ratio of distillation loss, η box , η cls and η obj Represents that they are in L kd The weight in obj Indicates whether this batch of images contains objects. If so, it is λ obj =1, otherwise λ obj =0; They are the bounding box loss, classification loss, confidence loss, and domain loss in the previous step; the difference is that the inputs of these four loss functions are adjusted from true values ​​and predicted values ​​to the predicted values ​​of the student model and the teacher model.

[0097] In order to improve the problem of imbalance in the number of positive and negative samples, We continue to use Focal loss to calculate. It is not difficult to see that the purpose of distillation loss is to minimize the gap between the student model and the teacher model in classification, confidence, domain adaptation, and bounding box regression, so as to better transfer knowledge from the teacher model to the student model.

[0098] S34. Calculate the total loss of the student model based on the imitation loss and the distillation loss and in combination with the loss of the teacher model;

[0099] L stu =L+L kd +L imit ;

[0100] Where L is the total loss of the teacher model;

[0101] S35. Based on the total loss of the student model, knowledge distillation training is performed on the preset student model, and the student model after knowledge distillation is used as a pavement crack detection model.

[0102] S4. Using the pavement crack detection model to perform crack detection on the pavement image to be detected.

[0103] Knowledge distillation is a method for learning end-to-end compact models. It does not require hardware or customized software to obtain acceleration, and can train small network models by distilling the knowledge of large-scale excellent models. Therefore, the present invention proposes a lightweight network architecture based on knowledge distillation.

[0104] In the present invention, firstly, the problem that large deep learning networks may not be deployed and inferred under limited storage space and computing power is analyzed, and a lightweight network architecture L-DDACDN based on knowledge distillation is proposed. The weight selection of the lightweight crack detection model is experimentally analyzed, and YOLO v5s with a smaller number of parameters is selected as the target of knowledge distillation. Secondly, in order to imitate the prediction of the teacher model at the local position close to the object anchor box, that is, to learn the high-level feature response of the area, the mask of the near-object area is designed using the real bounding box, and the imitation loss is calculated in combination with the feature response of the teacher and student models, so as to guide the student model prediction. Finally, the distillation loss is designed to balance the high-level feature layer of the teacher model and the guiding layer of the student model, so as to achieve the purpose of using the teacher model to guide the student model.

[0105] In this invention, the mask is designed by using the real bounding box annotation of the crack image and the existing anchor box, and the imitation loss is calculated by combining the high-level features of the teacher and student networks. In order to be compatible with the guided feature layer of the student network and the teacher network, the corresponding distillation loss is designed, which is calculated using the soft labels of the teacher model and the predictions of the student model, so as to better transfer knowledge from the teacher model to the student model.

[0106] Figure 5 A pavement crack detection system based on knowledge distillation is provided in an embodiment of the present invention. Figure 5 As shown, the pavement crack detection system based on knowledge distillation includes: an image data set construction module, a data preprocessing and expansion module, a model training module, and an image crack detection module.

[0107] An image dataset construction module, wherein the image dataset includes an image dataset of a source domain and an image dataset of a target domain;

[0108] A data preprocessing and expansion module performs data expansion after the image data in the target domain is subjected to adaptive block enhancement and global equalization preprocessing, and then merges the image data set in the source domain to obtain an expanded image data set;

[0109] A model training module, based on the expanded image data set, uses a pre-trained teacher model to perform knowledge distillation training on a preset student model, and uses the student model after knowledge distillation as a pavement crack detection model;

[0110] The image crack detection module uses the pavement crack detection model to perform crack detection on the pavement image to be detected.

[0111] The above-mentioned pavement crack detection system based on knowledge distillation can be implemented in the form of a computer program, which can be run on a computer device.

[0112] The computer device may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0113] The computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0114] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute a pavement crack detection method based on knowledge distillation.

[0115] The processor is used to provide computing and control capabilities to support the operation of the entire computer equipment.

[0116] The internal memory provides an environment for the operation of a computer program in a non-volatile storage medium. When the computer program is executed by a processor, the processor can execute a pavement crack detection method based on knowledge distillation.

[0117] The network interface is used to communicate with other devices over the network. Those skilled in the art will appreciate that the above-mentioned computer device structure is only a partial structure related to the present application solution, and does not constitute a limitation on the computer device to which the present application solution is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] The processor is used to run a computer program stored in the memory, and the program implements the pavement crack detection method based on knowledge distillation described in Example 1.

[0119] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0120] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0121] The present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes a pavement crack detection method based on knowledge distillation as described in Embodiment 1.

[0122] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, or any other computer-readable storage medium that can store program codes.

[0123] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0124] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0125] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0127] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A pavement crack detection method based on knowledge distillation, characterized in that: Includes steps: S1. Construct an image dataset, where the image dataset includes an image dataset of a source domain and an image dataset of a target domain; S2. Performing data preprocessing and data expansion based on the image data set; Specifically, the method includes: performing data expansion after the image data in the target domain is subjected to adaptive block enhancement and global equalization preprocessing and then merged with the image data set in the source domain to obtain an expanded image data set; S3, based on the expanded image data set, using the pre-trained teacher model to perform knowledge distillation training on the preset student model, and using the student model after knowledge distillation as a pavement crack detection model; comprising the following steps: S31, calculating a mask I of each crack image in the expanded image dataset according to the real bounding box of the object and the existing anchor box; S32, calculating the imitation loss based on the output features of the mask and the imitation teacher model in the local feature area close to the object; the specific calculation formula of the imitation loss is as follows: Among them, s is defined as the feature map of the guided feature layer of the student model, and t is defined as the corresponding high-level feature response map of the teacher model; for each anchor box position (i, j) close to the object on the feature map of width W and height H, (i, j) represents the adaptation function of the last hidden layer of the student model, I ij Represents the mask value at the anchor box position (i, j), C represents the number of channels of the feature map, f(s) ijc Represents the feature response of the cth channel at position (i, j) on the feature map; S33. Calculate the distillation loss to balance the high-level feature layer of the teacher model and the guidance layer of the student model, and transfer the knowledge of the teacher model; the distillation loss is calculated as follows: Among them, the variable ρ represents the weight ratio of distillation loss, η box , η cls and η obj Represents that they are in L kd The weight in obj Indicates whether this batch of images contains objects. If so, it is λ obj =1, otherwise λ obj =0; They are the bounding box loss, classification loss, confidence loss, and domain loss in the previous step; S34. Calculate the total loss of the student model based on the imitation loss and the distillation loss and in combination with the loss of the teacher model; L stu =L+L kd +L imit ; Where L is the total loss of the teacher model; S35, performing knowledge distillation training on a preset student model based on the total loss of the student model, and using the student model after knowledge distillation as a pavement crack detection model; S4. Using the pavement crack detection model to perform crack detection on the pavement image to be detected.

2. The method according to claim 1, characterized in that In step S2, the image data in the target domain is subjected to adaptive block enhancement and global equalization preprocessing, and the specific steps are as follows: S21, dividing the image into blocks according to the resolution size, and the block size increases from 50×50 to 500×500, with a step size of 50 each time; S22, performing a gamma transformation on each block to correct the gray part, wherein the gamma value is set from 0.5 to 2, and the increment step is 0.1; S23, when the block size is fixed, the variance σ is calculated for all blocks, and the Gamma value when the maximum σ is obtained after the block is enhanced is set as the final local enhancement parameter of the region; S24, performing smoothing processing on the pixel points between adjacent blocks; S25, CLAHE algorithm is used for global enhancement to balance the overall image.

3. The method according to claim 1, characterized in that In step S3, YOLO v5s is used as the student model weight to be trained.

4. The method according to claim 1, characterized in that: Step S31, specifically includes: S311. For each true bounding box of the crack image, calculate the IOU value between it and all anchor boxes to obtain a W×H×K IOU map, denoted as m IOU , where K represents the number of anchor boxes preset by YOLO v5, W represents the width of the image, and H represents the height of the image; S312, in m IOU Find the IOU map corresponding to the maximum IOU value, denoted as M=max(m IOU ), and construct a filter F, F = θ × M, where θ represents a threshold factor; S313, use filter F to filter m IOU , retain the IOU graphs larger than F and combine them using the or operator to obtain a W×H mask; S314. In this way, all the real frames of the crack images are looped, and finally a fine-grained mask imitating the teacher model is obtained, which is recorded as I.

5. A pavement crack detection system based on knowledge distillation, characterized in that: The pavement crack detection system executes the pavement crack detection method based on knowledge distillation as claimed in claim 1, comprising: an image data set construction module, a data preprocessing and expansion module, a model training module, and an image crack detection module; An image dataset construction module, wherein the image dataset includes an image dataset of a source domain and an image dataset of a target domain; A data preprocessing and expansion module performs data expansion after the image data in the target domain is subjected to adaptive block enhancement and global equalization preprocessing, and then merges the image data set in the source domain to obtain an expanded image data set; A model training module, based on the expanded image data set, uses a pre-trained teacher model to perform knowledge distillation training on a preset student model, and uses the student model after knowledge distillation as a pavement crack detection model; The image crack detection module uses the pavement crack detection model to perform crack detection on the pavement image to be detected.

6. A computer device, characterized in that: The device comprises a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

Images