Training method of disease detection model and disease detection method for road, bridge and tunnel

Abstract

The present application discloses a training method for a disease detection model of a road bridge and tunnel and a disease detection method. The method includes: obtaining a plurality of images of road bridge and tunnel diseases, and dividing the plurality of images into a first image set and a second image set according to the number of preset objects in the images; performing image enhancement processing on the first image set and the second image set respectively to obtain a training sample set, wherein the image enhancement processing includes: image edge clipping or adding adversarial perturbations; determining a sample label corresponding to each training sample, the sample label being used to mark the disease area and disease type; inputting a plurality of training samples into a teacher model to obtain a first prediction result output by the teacher model; and supervising training the student model based on the plurality of training samples, sample labels and the first prediction result to obtain a road bridge and tunnel disease detection model. The present application solves the technical problem in the related art that the pavement disease detection model has a large computational requirement and is not very applicable.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and more specifically, to a training method for a road, bridge, or tunnel disease detection model and a disease detection method. Background Art

[0002] With years of industrial development, my country's urbanization needs are becoming increasingly saturated, and infrastructure construction has gradually entered a situation where construction and maintenance coexist. The maintenance of roads, bridges, and tunnels incurs significant economic and human costs each year. Over time, infrastructure maintenance will become more frequent, consuming even greater expenditures in the future. Defect detection is a crucial aspect of road, bridge, and tunnel maintenance. It can detect and repair defects in a timely manner, significantly reducing the potential safety hazards associated with road, bridge, and tunnel defects.

[0003] Traditional defect detection relies heavily on deep learning networks for defect detection and classification. However, achieving high detection and classification accuracy often requires deep learning models with large parameter sizes, placing significant demands on the computing power of the equipment. However, the computing power of on-board inspection equipment is often limited. Therefore, when faced with limited computing power, we must compromise on the accuracy and efficiency of defect detection, significantly reducing detection performance. Furthermore, traditional defect detection methods only detect and classify road pavement defects and lack the ability to integrate detection of bridge and tunnel defects.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide a training method and a disease detection method for a road, bridge and tunnel disease detection model, so as to at least solve the technical problems in the related art that the pavement disease detection model requires a large amount of calculation and has poor applicability.

[0006] According to one aspect of an embodiment of the present application, a training method for a road bridge and tunnel disease detection model is provided, including: acquiring multiple road bridge and tunnel disease images, and dividing the multiple road bridge and tunnel disease images into a first image set and a second image set, wherein the number of preset objects in the road bridge and tunnel disease images in the first image set is less than the number of preset objects in the road bridge and tunnel disease images in the second image set; performing image enhancement processing on the first image set and the second image set respectively, and forming the first image set and the second image set after the image enhancement processing into a training sample set, wherein the image enhancement processing includes at least one of the following: image edge clipping, adding adversarial perturbation; determining a sample label corresponding to each training sample in the training sample set, wherein the sample label is used to mark the disease area and disease type corresponding to the training sample; inputting multiple training samples into a teacher model to obtain a first prediction result output by the teacher model; performing supervised training on a student model based on the multiple training samples, the sample labels and the first prediction result to obtain a road bridge and tunnel disease detection model, wherein both the teacher model and the student model are used to predict the disease area and disease type in the input image, and the parameter amount of the teacher model is greater than the parameter amount of the student model.

[0007] Optionally, multiple road bridge and tunnel disease images are divided into a first image set and a second image set, including: for each road bridge and tunnel disease image, detecting the number of preset objects in the road bridge and tunnel disease image according to a preset object detection algorithm, wherein the types of preset objects include: shadow areas, cracks, passing vehicles, and signs; when the number of preset objects does not exceed a preset threshold, determining that the road bridge and tunnel disease image is a first type of image, and dividing the road bridge and tunnel disease image into the first image set; when the number of preset objects exceeds a preset threshold, determining that the road bridge and tunnel disease image is a second type of image, and dividing the road bridge and tunnel disease image into the second image set.

[0008] Optionally, image enhancement processing is performed on the first image set and the second image set respectively, and the first image set and the second image set after image enhancement processing are combined into a training sample set, including: edge area cropping of a first preset proportion of road, bridge and tunnel disease images in the first image set to obtain a third image set; for the second image set, multiple adversarial perturbations corresponding to the second image set are generated based on a pre-trained adversarial generative network, and the adversarial perturbations of the second preset proportion are added to the second image set to obtain a fourth image set; the third image set and the fourth image set are combined into a training sample set.

[0009] Optionally, determining the sample label corresponding to each training sample in the training sample set includes: for each training sample in the training sample set, using a pre-trained labeling tool to label the first disease area and the first disease type corresponding to the training sample to obtain the sample label corresponding to the training sample.

[0010] Optionally, multiple training samples are input into the teacher model to obtain a first prediction result output by the teacher model, including: for each training sample, the training sample is input into the teacher model to obtain a first prediction result output by the teacher model, wherein the first prediction result includes the second disease area and the second disease type corresponding to the training sample.

[0011] Optionally, supervised training is performed on the student model based on multiple training samples, sample labels and the first prediction result, including: for each training sample, inputting the training sample into the student model to obtain a third disease area and a third disease type corresponding to the training sample output by the student model; constructing a first regression loss function based on the first disease area and the third disease area, and constructing a first classification loss function based on the first disease type and the third disease type; inputting multiple training samples into the student model for iterative training, adjusting the parameters of the student model based on the first regression loss function and the first classification loss function, and obtaining an intermediate detection model; constructing a second regression loss function based on the second disease area and the third disease area, and constructing a second classification loss function based on the second disease type and the third disease type; inputting multiple training samples into the intermediate detection model for iterative training, adjusting the parameters of the intermediate detection model based on the second regression loss function and the second classification loss function, and obtaining a disease detection model for roads, bridges and tunnels.

[0012] Optionally, obtaining multiple road, bridge, and tunnel defect images includes: obtaining inspection videos of roads, bridges, and tunnels, performing frame processing on the inspection videos to obtain multiple frame images; and screening out images of roads, bridges, or tunnels with defect features from the multiple frame images as road, bridge, and tunnel defect images.

[0013] Optionally, both the teacher model and the student model adopt the RetinaNet network model.

[0014] According to another aspect of an embodiment of the present application, a method for detecting road, bridge, or tunnel defects is provided, comprising: collecting images of roads, bridges, or tunnels; inputting the images into a pre-trained road, bridge, or tunnel defect detection model, wherein the road, bridge, or tunnel defect detection model is trained using the above-mentioned road, bridge, or tunnel defect detection model training method; and obtaining an output result of the road, bridge, or tunnel defect detection model, wherein the output result is used to reflect the defect area and defect type in the image.

[0015] According to another aspect of an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned training method of the road, bridge and tunnel disease detection model or the road, bridge and tunnel disease detection method through the computer program.

[0016] In an embodiment of the present application, a plurality of images of road, bridge and tunnel defects are first obtained, and the plurality of images are divided into a first image set and a second image set according to the number of preset objects in the images; image enhancement processing is performed on the first image set and the second image set respectively, and the image enhancement processing includes: image edge clipping or adding adversarial perturbations, and the first image set and the second image set after image enhancement processing are formed into a training sample set; a sample label corresponding to each training sample in the training sample set is determined, and the sample label is used to mark the defect area and the defect type; the plurality of training samples are input into the teacher model to obtain a first prediction result output by the teacher model; the student model is supervised and trained based on the plurality of training samples, the sample labels and the first prediction result to obtain a road, bridge and tunnel defect detection model. Among them, by performing enhancement processing on the training sample images, its recognition effect can be improved and the accuracy of the training model can be improved; by knowledge distillation, the size of the detection model can be further compressed, the amount of model calculation can be reduced while improving the applicability of the model, and the technical problem of large computational requirements and low applicability of the road surface defect detection model in the related art can be effectively solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a schematic diagram of the structure of an optional RetinaNet network according to an embodiment of the present application;

[0019] Figure 2 This is a flowchart of an optional method for training a road, bridge, and tunnel disease detection model according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of an optional training sample processing process according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of an optional knowledge distillation process according to an embodiment of the present application;

[0022] Figure 5 This is a flow chart of an optional method for detecting road, bridge, and tunnel defects according to an embodiment of the present application;

[0023] Figure 6 1 is a schematic structural diagram of an optional training device for a road, bridge, and tunnel disease detection model according to an embodiment of the present application;

[0024] Figure 7 It is a structural schematic diagram of an optional road, bridge and tunnel disease detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0026] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0027] In order to better understand the embodiments of the present application, some nouns or terms that appear in the description of the embodiments of the present application are first translated and explained as follows:

[0028] Knowledge distillation is a common method for model compression. Unlike pruning and quantization in traditional model compression, knowledge distillation builds a lightweight small model and trains it using supervisory information from a larger, higher-performing model to achieve better performance and accuracy. The large model is called the teacher, and the small model is called the student. The supervisory information output by the teacher is called knowledge, and the process of the student learning to transfer this supervisory information is called distillation.

[0029] YOLO (You Only Look Once) is an object detection algorithm. It doesn't actually remove candidate regions, but instead creatively combines candidate regions and object classification. This allows users to identify objects and their locations at a glance. This algorithm uses a single CNN model for end-to-end object detection. Its core concept is to use the entire image as input and directly regress the bounding box's location and category at the output layer.

[0030] Generative Adversarial Networks (GANs): A deep learning model that has emerged as one of the most promising methods for unsupervised learning on complex distributions in recent years. This model framework consists of two modules: a generative model (G) and a discriminative model (D). During training, the goal of the generative model (G) is to generate realistic images to deceive the discriminative model (D), while the goal of the discriminative model (D) is to distinguish images generated by the generative model (G) from real images. Through this interplay between the two modules, the GAN learns to generate a better output, often referred to as an adversarial perturbation.

[0031] RetinaNet: A classic first-order target detection model with a structure like Figure 1 As shown in the figure, the backbone is a residual network (ResNet), the neck is a feature pyramid network (FPN) structure, and the head includes two branches: classification and position detection. The two branches do not share parameters. This model introduces a new loss function, Focal Loss, based on the cross entropy (CE) loss function for binary classification problems. It is defined as follows:

[0032] FL(p t )=-∝(1-p t ) γ log(p t )

[0033] Among them, p t To predict the result, the parameter value is determined according to the distribution ratio of positive and negative samples, (1-p t ) γ It is an adjustment factor used to reduce the loss of high confidence samples.

[0034] Example 1

[0035] In order to solve the technical problems in related technologies that pavement defect detection models require large amounts of computation and have low applicability, the embodiment of the present application proposes a training method for a lightweight defect detection model for roads, bridges, and tunnels. This method is based on the idea of ​​knowledge distillation, and distills a deep learning model with large-scale parameters to obtain a relatively lightweight target detection model. While achieving satisfactory defect detection accuracy, it can effectively compress the model size, reduce the amount of computation, and improve applicability in various application scenarios.

[0036] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Figure 2 This is a flow chart of an optional training method for a road, bridge, and tunnel disease detection model according to an embodiment of the present application, such as Figure 2 As shown, the method includes at least steps S202-S210, wherein:

[0038] Step S202: Acquire a plurality of road, bridge, and tunnel disease images, and divide the plurality of road, bridge, and tunnel disease images into a first image set and a second image set, wherein the number of preset objects in the road, bridge, and tunnel disease images in the first image set is less than the number of preset objects in the road, bridge, and tunnel disease images in the second image set;

[0039] Step S204, performing image enhancement processing on the first image set and the second image set respectively, and forming the first image set and the second image set after the image enhancement processing into a training sample set, wherein the image enhancement processing includes at least one of the following: image edge clipping, adding adversarial perturbation;

[0040] Step S206, determining a sample label corresponding to each training sample in the training sample set, wherein the sample label is used to mark the disease area and disease type corresponding to the training sample;

[0041] Step S208: inputting multiple training samples into the teacher model to obtain a first prediction result output by the teacher model;

[0042] In step S210, supervised training is performed on the student model based on multiple training samples, sample labels, and the first prediction results to obtain a road, bridge, and tunnel disease detection model, wherein both the teacher model and the student model are used to predict the diseased area and disease type in the input image, and the parameter amount of the teacher model is greater than the parameter amount of the student model.

[0043] Among them, by enhancing the training sample images, its recognition effect can be improved and the accuracy of the training model can be improved; through knowledge distillation, the size of the detection model can be further compressed, the model calculation amount can be reduced while improving the model applicability, effectively solving the technical problems of high computational requirements and low applicability of pavement disease detection models in related technologies.

[0044] The following describes the training process of the road, bridge and tunnel disease detection model of the present application scheme in combination with specific implementation steps.

[0045] As an optional implementation, when obtaining multiple images of road, bridge and tunnel defects, the following method can be used: obtain an inspection video of the road, bridge and tunnel taken by an inspection vehicle, frame-process the inspection video to obtain multiple frame images; and filter out images of roads, bridges or tunnels with defect features from the multiple frame images as road, bridge and tunnel defect images.

[0046] In order to improve the accuracy of the model prediction results, the embodiment of the present application does not directly use the acquired road, bridge and tunnel disease images as training samples, but first classifies them and performs adaptive enhancement processing on different types of images to obtain training samples with better recognition effect.

[0047] Optionally, when dividing multiple road bridge and tunnel disease images into a first image set and a second image set, it can be done in the following way: for each road bridge and tunnel disease image, the number of preset objects in the road bridge and tunnel disease image is detected according to a preset object detection algorithm, wherein the object detection algorithm can adopt the YOLO target detection algorithm, and the types of preset objects include: shadow areas, cracks, passing vehicles, signs, etc.; when the number of preset objects does not exceed a preset threshold, the road bridge and tunnel disease image is determined to be a first type of image, and the road bridge and tunnel disease image is divided into the first image set; when the number of preset objects exceeds the preset threshold, the road bridge and tunnel disease image is determined to be a second type of image, and the road bridge and tunnel disease image is divided into the second image set.

[0048] The preset threshold is set by the user, such as 1 or 2, and is not specifically limited here. The first type of image is usually called a few-target image or a large-target image, and the second type of image is also called a multi-target image or a small-target image.

[0049] After the first image set and the second image set are divided, adaptive image enhancement processing operations can be performed based on the characteristics of the road, bridge and tunnel disease images in different image sets. The specific methods are as follows:

[0050] For the first image set, edge regions of the road, bridge, and tunnel defect images within a first preset ratio can be cropped to obtain a third image set. The first preset ratio is set by the user based on experience and is not specifically limited herein. Edge region cropping refers to randomly cropping edge regions of the road, bridge, and tunnel defect images that are not preset objects.

[0051] It can be understood that for images of large targets, the edge areas of non-preset objects in the original image are adaptively and randomly cropped at a certain ratio in order to focus on processing the key parts of the image, namely large areas of road, bridge and tunnel damage, such as large areas of road cracks, potholes or cracks in bridges and tunnels. From the perspective of image Fourier transform, this process focuses on processing the low-frequency part of the large target image, which can effectively improve the recognition effect of images with fewer targets.

[0052] For the second image set, multiple adversarial perturbations corresponding to the second image set can be generated based on the pre-trained adversarial generative network, and the adversarial perturbations of the second preset ratio are added to the second image set to obtain a fourth image set, wherein the second preset ratio is set by the user based on experience and is not specifically limited here.

[0053] It can be understood that for multi-target images with more complex backgrounds, adversarial perturbations are generated by the adversarial generative network and added to the original image in a certain proportion, which can enhance the high-frequency part of the multi-target image, thereby making the multi-target image have a better recognition effect.

[0054] Afterwards, the third image set and the fourth image set are combined to form a training sample set.

[0055] Figure 3 A schematic diagram of an optional training sample processing process is shown, which specifically includes the following steps:

[0056] S1, obtain inspection videos from inspection vehicles and pre-process them to obtain images of road, bridge and tunnel defects;

[0057] S2, for each road, bridge or tunnel disease image, use the target detection algorithm to detect the number of preset objects in the image;

[0058] S3, determine whether the number of preset objects exceeds 2, if so, execute step S4, if not, execute step S6;

[0059] S4, dividing the image into a first image set;

[0060] S5, performing image edge cropping on the images in the first image set according to a certain ratio to obtain a third image set;

[0061] S6, dividing the image into a second image set;

[0062] S7, adding adversarial perturbations to the second image set according to a certain ratio to obtain a fourth image set;

[0063] S8. Combine the third image set and the fourth image set into a training sample set.

[0064] As an optional implementation, the sample label corresponding to each training sample in the training sample set can be determined as follows: for each training sample, the first disease area and first disease type corresponding to the training sample are annotated using a pre-trained annotation tool to obtain the sample label corresponding to the training sample. The annotation tool can be a commonly used image annotation tool such as labelImg or ImgLab.

[0065] Currently, there are two main types of disease detection methods: one-stage object detection methods and two-stage object detection methods. One-stage object detection methods, such as the YOLO model and the SSD model, are end-to-end object detection methods. The idea behind two-stage object detection methods is to generate a sparse set of candidate proposals in the first stage, which includes all objects while filtering out most negative locations. In the second stage, the proposals are classified as foreground / background, that is, as objects or not. Generally speaking, one-stage object detection methods are faster but less accurate than two-stage object detection methods. Although two-stage object detection methods have high detection accuracy, their two-stage design results in relatively poor computational efficiency.

[0066] This application takes into account the limited computing and storage capabilities of the on-board intelligent inspection system and the high real-time requirements for the inspection results. Therefore, the teacher model and student model selected in the distillation process both adopt a one-stage RetinaNet network model. The two have the same structure, but the number of parameters of the student model is smaller than that of the teacher model.

[0067] As an optional implementation, during the entire distillation process, multiple training samples can be first input into the teacher model to obtain the first prediction result output by the teacher model, specifically including: for each training sample, the training sample is input into the teacher model to obtain the first prediction result output by the teacher model, the first prediction result includes the second disease area and the second disease type corresponding to the training sample predicted by the teacher model. The first prediction result is also called a soft result.

[0068] Afterwards, supervised training of the student model can be performed based on multiple training samples, sample labels, and the first prediction results in the following way:

[0069] For each training sample, the training sample is input into the student model, and the third disease area and the third disease type corresponding to the training sample output by the student model are obtained;

[0070] The student model can be first supervised and trained for the first time based on the real sample labels, specifically including: constructing a first regression loss function based on the first disease area and the third disease area, and constructing a first classification loss function based on the first disease type and the third disease type; inputting multiple training samples into the student model for iterative training, adjusting the parameters of the student model based on the first regression loss function and the first classification loss function, and obtaining an intermediate detection model.

[0071] Among them, the first classification loss function and the first regression loss function can both be calculated using the Kullback-Leible divergence. During the iterative training process, it is necessary to set the iteration rounds in advance and select a suitable optimizer. By minimizing the first regression loss function and the first classification loss function, the parameters of the student model are updated to obtain the intermediate detection model.

[0072] Afterwards, the intermediate detection model is supervised and trained again based on the soft results of the teacher model, specifically including: constructing a second regression loss function based on the second disease area and the third disease area, and constructing a second classification loss function based on the second disease type and the third disease type; multiple training samples are input into the intermediate detection model for iterative training, and the parameters of the intermediate detection model are adjusted according to the second regression loss function and the second classification loss function to obtain the disease detection model for roads, bridges and tunnels.

[0073] The second classification loss function and the second regression loss function are also calculated using the Kullback-Leible divergence. During the iterative training process, the number of iterations is set in advance, and a suitable optimizer is selected. The parameters of the intermediate detection model are updated by minimizing the second regression loss function and the second classification loss function until the final road, bridge and tunnel disease detection model is obtained.

[0074] Figure 4 A schematic diagram of an optional knowledge distillation process is shown, which specifically includes the following steps:

[0075] S1, determine the teacher model and student model;

[0076] S2, input multiple training samples into the teacher model to obtain the first prediction result;

[0077] S3, inputting multiple training samples into the student model to obtain a second prediction result;

[0078] S4, calculate the first regression loss and the first classification loss between the sample label and the second prediction result, and update the model parameters;

[0079] S5, calculating a second regression loss and a second classification loss between the first prediction result and the second prediction result, and continuing to update the model parameters;

[0080] S6, obtaining the distilled road, bridge and tunnel disease detection model.

[0081] In an embodiment of the present application, a plurality of images of road, bridge and tunnel defects are first obtained, and the plurality of images are divided into a first image set and a second image set according to the number of preset objects in the images; image enhancement processing is performed on the first image set and the second image set respectively, and the image enhancement processing includes: image edge clipping or adding adversarial perturbations, and the first image set and the second image set after image enhancement processing are formed into a training sample set; a sample label corresponding to each training sample in the training sample set is determined, and the sample label is used to mark the defect area and the defect type; the plurality of training samples are input into the teacher model to obtain a first prediction result output by the teacher model; the student model is supervised and trained based on the plurality of training samples, the sample labels and the first prediction result to obtain a road, bridge and tunnel defect detection model. Among them, by performing enhancement processing on the training sample images, its recognition effect can be improved and the accuracy of the training model can be improved; by knowledge distillation, the size of the detection model can be further compressed, the amount of model calculation can be reduced while improving the applicability of the model, and the technical problem of large computational requirements and low applicability of the road surface defect detection model in the related art can be effectively solved.

[0082] Example 2

[0083] Based on the training method of the road, bridge and tunnel defect detection model provided in Example 1, the embodiment of the present application provides a road, bridge and tunnel defect detection method that can be used in a vehicle-mounted inspection system.

[0084] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0085] Figure 5 FIG. 1 is a flow chart of an optional method for detecting road, bridge, and tunnel defects according to an embodiment of the present application. Figure 5 As shown, the method includes at least steps S502-S506, wherein:

[0086] Step S502: collecting images of roads, bridges or tunnels.

[0087] Specifically, while the inspection vehicle is driving, the on-board inspection system collects images of roads, bridges or tunnels through the on-board camera in real time.

[0088] Step S504: input the image into a pre-trained road, bridge, and tunnel defect detection model, wherein the road, bridge, and tunnel defect detection model is trained using the training method for the road, bridge, and tunnel defect detection model in Example 1.

[0089] Step S506: Obtain the output result of the road, bridge and tunnel disease detection model, wherein the output result is used to reflect the disease area and disease type in the image.

[0090] Among them, since the trained road, bridge and tunnel disease detection model is a lightweight detection model that has undergone knowledge distillation, it not only ensures the accuracy of disease detection but also does not require high system computing power. The on-board inspection system can use this model to perform real-time disease detection on roads, bridges and tunnels and detect problems in a timely manner.

[0091] Example 3

[0092] According to an embodiment of the present application, a training device for a road bridge and tunnel disease detection model for implementing the training method for a road bridge and tunnel disease detection model in Example 1 is also provided. Figure 6 As shown, the training device of the road, bridge and tunnel disease detection model includes at least: an acquisition module 61, a preprocessing module 62, a labeling module 63, a first training module 64 and a second training module 65, wherein:

[0093] The acquisition module 61 is used to acquire multiple road, bridge and tunnel disease images and divide the multiple road, bridge and tunnel disease images into a first image set and a second image set, wherein the number of preset objects in the road, bridge and tunnel disease images in the first image set is less than the number of preset objects in the road, bridge and tunnel disease images in the second image set.

[0094] As an optional implementation, the acquisition module can obtain inspection videos of roads, bridges and tunnels taken by inspection vehicles, frame the inspection videos, and obtain multiple frame images; and filter out images of roads, bridges or tunnels with disease features from the multiple frame images as road, bridge and tunnel disease images.

[0095] In order to improve the accuracy of the model prediction results, the embodiment of the present application does not directly use the acquired road, bridge and tunnel disease images as training samples, but first classifies them and performs adaptive enhancement processing on different types of images to obtain training samples with better recognition effect.

[0096] Optionally, for each road bridge and tunnel disease image, the acquisition module can detect the number of preset objects in the road bridge and tunnel disease image based on a preset object detection algorithm, wherein the object detection algorithm can adopt the YOLO target detection algorithm, and the types of preset objects include: shadow areas, cracks, passing vehicles, signs, etc.; when the number of preset objects does not exceed the preset threshold, the road bridge and tunnel disease image is determined to be a first type of image, and the road bridge and tunnel disease image is divided into the first image set; when the number of preset objects exceeds the preset threshold, the road bridge and tunnel disease image is determined to be a second type of image, and the road bridge and tunnel disease image is divided into the second image set.

[0097] The preset threshold is set by the user, such as 1 or 2, and is not specifically limited here. The first type of image is usually called a few-target image or a large-target image, and the second type of image is also called a multi-target image or a small-target image.

[0098] The preprocessing module 62 is used to perform image enhancement processing on the first image set and the second image set respectively, and form the first image set and the second image set after the image enhancement processing into a training sample set, wherein the image enhancement processing includes at least one of the following: image edge cropping, adding adversarial perturbation.

[0099] Optionally, after the first image set and the second image set are divided, the pre-processing module may perform adaptive image enhancement processing operations based on the characteristics of the road, bridge, and tunnel disease images in different image sets, specifically in the following manner:

[0100] For the first image set, the preprocessing module can perform edge cropping on a first preset ratio of the road, bridge, and tunnel defect images in the first image set to generate a third image set. The first preset ratio is user-defined based on experience and is not specifically limited herein. Edge cropping refers to randomly cropping edge regions of the road, bridge, and tunnel defect images that are not preset objects.

[0101] It can be understood that for images of large targets, the edge areas of non-preset objects in the original image are adaptively and randomly cropped at a certain ratio in order to focus on processing the key parts of the image, namely large areas of road, bridge and tunnel damage, such as large areas of road cracks, potholes or cracks in bridges and tunnels. From the perspective of image Fourier transform, this process focuses on processing the low-frequency part of the large target image, which can effectively improve the recognition effect of images with fewer targets.

[0102] For the second image set, the preprocessing module can generate multiple adversarial perturbations corresponding to the second image set based on the pre-trained adversarial generative network, and add the adversarial perturbations of a second preset ratio to the second image set to obtain a fourth image set, wherein the second preset ratio is set by the user based on experience and is not specifically limited here.

[0103] It can be understood that for multi-target images with more complex backgrounds, adversarial perturbations are generated by the adversarial generative network and added to the original image in a certain proportion, which can enhance the high-frequency part of the multi-target image, thereby making the multi-target image have a better recognition effect.

[0104] Afterwards, the preprocessing module combines the third image set and the fourth image set into a training sample set.

[0105] The labeling module 63 is used to determine a sample label corresponding to each training sample in the training sample set, wherein the sample label is used to mark the disease area and disease type corresponding to the training sample.

[0106] As an optional implementation, the sample label corresponding to each training sample in the training sample set can be determined as follows: for each training sample, the first disease area and first disease type corresponding to the training sample are annotated using a pre-trained annotation tool to obtain the sample label corresponding to the training sample. The annotation tool can be a commonly used image annotation tool such as labelImg or ImgLab.

[0107] The first training module 64 is used to input multiple training samples into the teacher model to obtain a first prediction result output by the teacher model.

[0108] The second training module 65 is used to supervise the training of the student model based on multiple training samples, sample labels and the first prediction results to obtain a road, bridge and tunnel disease detection model, wherein both the teacher model and the student model are used to predict the disease area and disease type in the input image, and the parameter amount of the teacher model is greater than the parameter amount of the student model.

[0109] This application takes into account the limited computing and storage capabilities of the on-board intelligent inspection system and the high real-time requirements for the inspection results. Therefore, the teacher model and student model selected in the distillation process both adopt a one-stage RetinaNet network model. The two have the same structure, but the number of parameters of the student model is smaller than that of the teacher model.

[0110] As an optional implementation, during the entire distillation process, the first training module can first input multiple training samples into the teacher model to obtain a first prediction result output by the teacher model, specifically including: for each training sample, input the training sample into the teacher model to obtain the first prediction result output by the teacher model, the first prediction result includes the second disease area and second disease type corresponding to the training sample predicted by the teacher model. The first prediction result is also called a soft result.

[0111] Afterwards, the second training module can supervise the student model based on the multiple training samples, sample labels, and the first prediction results in the following manner:

[0112] For each training sample, the training sample is input into the student model, and the third disease area and the third disease type corresponding to the training sample output by the student model are obtained;

[0113] The student model can be first supervised and trained for the first time based on the real sample labels, specifically including: constructing a first regression loss function based on the first disease area and the third disease area, and constructing a first classification loss function based on the first disease type and the third disease type; inputting multiple training samples into the student model for iterative training, adjusting the parameters of the student model based on the first regression loss function and the first classification loss function, and obtaining an intermediate detection model.

[0114] Among them, the first classification loss function and the first regression loss function can both be calculated using the Kullback-Leible divergence. During the iterative training process, it is necessary to set the iteration rounds in advance and select a suitable optimizer. By minimizing the first regression loss function and the first classification loss function, the parameters of the student model are updated to obtain the intermediate detection model.

[0115] Afterwards, the intermediate detection model is supervised and trained again based on the soft results of the teacher model, specifically including: constructing a second regression loss function based on the second disease area and the third disease area, and constructing a second classification loss function based on the second disease type and the third disease type; multiple training samples are input into the intermediate detection model for iterative training, and the parameters of the intermediate detection model are adjusted according to the second regression loss function and the second classification loss function to obtain the disease detection model for roads, bridges and tunnels.

[0116] The second classification loss function and the second regression loss function are also calculated using the Kullback-Leible divergence. During the iterative training process, the number of iterations is set in advance, and a suitable optimizer is selected. The parameters of the intermediate detection model are updated by minimizing the second regression loss function and the second classification loss function until the final road, bridge and tunnel disease detection model is obtained.

[0117] It should be noted that the modules in the training device for the road, bridge and tunnel disease detection model in the embodiment of the present application correspond one-to-one to the implementation steps of the training method for the road, bridge and tunnel disease detection model in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be elaborated here.

[0118] Example 4

[0119] According to an embodiment of the present application, a road bridge and tunnel disease detection device for implementing the road bridge and tunnel disease detection method in Example 2 is also provided. Figure 7 As shown, the road, bridge and tunnel disease detection device at least includes: a collection module 71, an input module 72 and an acquisition module 73, wherein:

[0120] The acquisition module 71 is used to acquire images of roads, bridges or tunnels.

[0121] Specifically, while the inspection vehicle is traveling, the acquisition module collects images of roads, bridges or tunnels in real time.

[0122] The input module 72 is used to input the image into the pre-trained road, bridge and tunnel disease detection model, wherein the road, bridge and tunnel disease detection model is trained using the road, bridge and tunnel disease detection model training method in Example 1.

[0123] The acquisition module 73 is used to obtain the output result of the road, bridge and tunnel disease detection model, wherein the output result is used to reflect the disease area and disease type in the image.

[0124] Among them, since the trained road, bridge and tunnel disease detection model is a lightweight detection model that has undergone knowledge distillation, while ensuring the accuracy of disease detection, the system computing power requirements are not high. The road, bridge and tunnel disease detection device of the embodiment of the present application can use this model to perform real-time disease detection on road, bridge and tunnels and discover problems in a timely manner.

[0125] It should be noted that the modules in the road, bridge and tunnel defect detection device in the embodiment of the present application correspond one-to-one to the implementation steps of the road, bridge and tunnel defect detection method in Example 2. Since a detailed description has been given in Example 2, some details not reflected in this embodiment can be referred to Example 2 and will not be elaborated here.

[0126] Example 5

[0127] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the training method of the road, bridge and tunnel disease detection model in Example 1 or the road, bridge and tunnel disease detection method in Example 2 by running the computer program.

[0128] According to an embodiment of the present application, a processor is also provided, which is used to run a computer program, wherein when the computer program is running, it executes the training method of the road, bridge and tunnel disease detection model in Example 1 or the road, bridge and tunnel disease detection method in Example 2.

[0129] According to an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the training method of the road, bridge and tunnel disease detection model in Example 1 or the road, bridge and tunnel disease detection method in Example 2 through the computer program.

[0130] Optionally, the processor is configured to execute the following steps through a computer program: acquiring a plurality of images of road, bridge and tunnel diseases, and dividing the plurality of images of road, bridge and tunnel diseases into a first image set and a second image set, wherein the number of preset objects in the images of road, bridge and tunnel diseases in the first image set is less than the number of preset objects in the images of road, bridge and tunnel diseases in the second image set; performing image enhancement processing on the first image set and the second image set respectively, and forming the first image set and the second image set after image enhancement processing into a training sample set, wherein the image enhancement processing includes at least one of the following: image edge cropping, adding adversarial perturbation; determining a sample label corresponding to each training sample in the training sample set, wherein the sample label is used to mark the disease area and disease type corresponding to the training sample; inputting the plurality of training samples into a teacher model to obtain a first prediction result output by the teacher model; performing supervised training on the student model based on the plurality of training samples, the sample labels and the first prediction result to obtain a road, bridge and tunnel disease detection model, wherein both the teacher model and the student model are used to predict the disease area and disease type in the input image, and the number of parameters of the teacher model is greater than the number of parameters of the student model.

[0131] Optionally, the processor is further configured to execute the following steps through a computer program: collecting images of roads, bridges or tunnels; inputting the images into a pre-trained road, bridge or tunnel disease detection model, wherein the road, bridge or tunnel disease detection model is trained using the training method of the road, bridge or tunnel disease detection model in Example 1; obtaining the output result of the road, bridge or tunnel disease detection model, wherein the output result is used to reflect the disease area and disease type in the image.

[0132] The serial numbers of the above embodiments are for description only and do not represent the advantages or disadvantages of the embodiments.

[0133] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0135] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0136] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0138] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A training method for a road, bridge, and tunnel disease detection model, characterized in that: include: Acquire a plurality of road, bridge, and tunnel disease images, and divide the plurality of road, bridge, and tunnel disease images into a first image set and a second image set, wherein the number of preset objects in the road, bridge, and tunnel disease images in the first image set is less than the number of preset objects in the road, bridge, and tunnel disease images in the second image set; Crop edge regions of the road, bridge, and tunnel disease images of a first preset ratio in the first image set to obtain a third image set; For the second image set, generating a plurality of adversarial perturbations corresponding to the second image set according to a pre-trained generative adversarial network, and adding a second preset proportion of the adversarial perturbations to the second image set to obtain a fourth image set; Combining the third image set and the fourth image set into a training sample set; Determining a sample label corresponding to each training sample in the training sample set, wherein the sample label is used to mark the disease area and disease type corresponding to the training sample; Inputting the plurality of training samples into a teacher model to obtain a first prediction result output by the teacher model; The student model is supervised and trained based on the multiple training samples, the sample labels and the first prediction results to obtain a road, bridge and tunnel disease detection model, wherein the teacher model and the student model are both used to predict the diseased area and disease type in the input image, and the parameter amount of the teacher model is greater than the parameter amount of the student model.

2. The method according to claim 1, characterized in that Dividing the plurality of road, bridge and tunnel disease images into a first image set and a second image set includes: For each of the road, bridge, and tunnel defect images, detecting the number of the preset objects in the road, bridge, and tunnel defect image according to a preset object detection algorithm, wherein the types of the preset objects include: shadow areas, cracks, passing vehicles, and signs; When the number of the preset objects does not exceed a preset threshold, determining that the road, bridge, and tunnel disease image is a first type of image, and classifying the road, bridge, and tunnel disease image into the first image set; When the number of the preset objects exceeds the preset threshold, the road, bridge and tunnel disease image is determined to be a second type of image, and the road, bridge and tunnel disease image is divided into the second image set.

3. The method according to claim 1, characterized in that Determining a sample label corresponding to each training sample in the training sample set includes: For each training sample in the training sample set, a pre-trained annotation tool is used to annotate the first disease area and the first disease type corresponding to the training sample to obtain the sample label corresponding to the training sample.

4. The method according to claim 3, characterized in that Inputting the plurality of training samples into a teacher model to obtain a first prediction result output by the teacher model includes: For each training sample, the training sample is input into the teacher model to obtain the first prediction result output by the teacher model, wherein the first prediction result includes the second disease area and the second disease type corresponding to the training sample.

5. The method according to claim 4, characterized in that Performing supervised training on the student model based on the plurality of training samples, the sample labels, and the first prediction results, comprising: For each training sample, inputting the training sample into the student model, and obtaining a third disease area and a third disease type corresponding to the training sample output by the student model; Constructing a first regression loss function based on the first disease area and the third disease area, and constructing a first classification loss function based on the first disease type and the third disease type; Inputting a plurality of training samples into the student model for iterative training, adjusting the parameters of the student model according to the first regression loss function and the first classification loss function, to obtain an intermediate detection model; Constructing a second regression loss function based on the second disease area and the third disease area, and constructing a second classification loss function based on the second disease type and the third disease type; Multiple training samples are input into the intermediate detection model for iterative training, and the parameters of the intermediate detection model are adjusted according to the second regression loss function and the second classification loss function to obtain the road, bridge and tunnel disease detection model.

6. The method according to claim 1, characterized in that Acquire multiple road, bridge and tunnel disease images, including: Obtaining an inspection video of a road, bridge, or tunnel, and performing frame processing on the inspection video to obtain multiple frame images; Images of roads, bridges or tunnels with damage features are screened out from the plurality of frame images as the road, bridge or tunnel damage images.

7. The method according to any one of claims 1 to 6, characterized in that The teacher model and the student model both adopt the RetinaNet network model.

8. A method for detecting road, bridge and tunnel defects, characterized in that: include: Collect images of roads, bridges, or tunnels; Inputting the image into a pre-trained road, bridge, and tunnel disease detection model, wherein the road, bridge, and tunnel disease detection model is trained using the road, bridge, and tunnel disease detection model training method according to any one of claims 1 to 6; An output result of the road, bridge and tunnel disease detection model is obtained, wherein the output result is used to reflect the disease area and disease type in the image.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the training method for a road, bridge, and tunnel disease detection model according to any one of claims 1 to 6 or the road, bridge, and tunnel disease detection method according to claim 8 through the computer program.

Images