Road surface crack detection method and device, electronic equipment and storage medium

Abstract

This application provides a method, apparatus, storage medium, and electronic device for detecting road surface cracks. The method includes: constructing a segmentation model using a preset first encoder and decoder, and constructing an evaluation model using a preset second encoder; alternately training the segmentation model and the evaluation model using a preset set of first crack images for training to obtain a trained segmentation model, wherein each first crack image is pre-labeled with an actual pixel-level crack marker; inputting a plurality of road surface images to be labeled into the trained segmentation model, and performing pixel-level image semantic segmentation on each road surface image to obtain a plurality of crack marker images containing pixel-level crack markers, and identifying the road surface cracks in the crack marker images.

Description

Technical Field

[0001] The embodiments of this application relate to the technical field of image recognition, and more particularly to a method, apparatus, electronic device, and storage medium for detecting road surface cracks. Background Technology

[0002] The existing methods for detecting road cracks are limited by factors such as labor and time costs. The crack images identified are mostly based on block-level markings, and the granularity of block-level markings is too coarse, which fundamentally limits the improvement of the accuracy of automated road crack detection.

[0003] On the other hand, road surface conditions are affected by climate, traffic flow, and other factors, resulting in a wide variety of crack types, such as shallow cracks, alligator cracks, and network cracks. Furthermore, there are many interfering factors, such as lane lines, stains, and uneven lighting, which greatly increases the difficulty of crack detection. Road surface crack detection models trained based on data from real-world application scenarios have poor recognition performance for fine cracks and a high rate of missed detection.

[0004] Therefore, a solution is needed that can achieve more granular pavement crack detection. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a method, device, electronic device and storage medium for detecting road surface cracks.

[0006] For the purposes described above, this application provides a method for detecting pavement cracks, including:

[0007] A segmentation model is constructed using a pre-set first encoder and decoder, and an evaluation model is constructed using a pre-set second encoder.

[0008] The segmentation model and the evaluation model are trained alternately using multiple first crack images for training to obtain a trained segmentation model. Each first crack image is pre-labeled with actual pixel-level crack markers.

[0009] Multiple road surface images to be labeled are input into the trained segmentation model, and pixel-level image semantic segmentation is performed on each road surface image to obtain multiple crack-labeled images containing pixel-level crack labels, and the road surface cracks in the crack-labeled images are identified.

[0010] Furthermore, before alternately training the segmentation model and the evaluation model using a pre-set set of first crack images for training, the method further includes:

[0011] A pre-trained densely connected convolutional neural network is used to predict cracks in multiple pre-set training images to obtain multiple initial crack images containing block-level crack labels.

[0012] Histogram equalization, median filtering, image grayscale logarithmic transformation, and morphological opening are performed on each initial crack image;

[0013] According to a preset dynamic threshold, each processed initial crack image is converted into a binary image and a pixel-level crack extraction algorithm based on edge detection is used to obtain the plurality of first crack images containing pixel-level crack markers.

[0014] Furthermore, a segmentation model is constructed using a pre-built first encoder and decoder, including:

[0015] The stride of the convolutional layer in the first encoder is set to 2, and upsampling is performed using transposed convolution in the decoder.

[0016] A guiding filter is added to both the first encoder and the decoder, and an attention mechanism is set in the decoder.

[0017] Furthermore, an evaluation model is constructed using a pre-built second encoder, including:

[0018] Set the stride of the convolutional layer in the second encoder to 2;

[0019] And set a loss function for evaluating the segmentation model and the evaluation model itself.

[0020] Furthermore, the segmentation model and the evaluation model are trained alternately, including:

[0021] The preset multiple first crack images are input into the segmentation model and the evaluation model;

[0022] The first model parameters of the segmentation model are kept unchanged, the evaluation model is trained, and the second target parameters of the evaluation model are determined. The second target parameters are used to update the second model parameters of the evaluation model.

[0023] The second objective parameter of the evaluation model is kept unchanged, the segmentation model is trained, and the first objective parameter of the segmentation model is determined. The first objective parameter is used to update the first model parameter.

[0024] Further, keeping the current first model parameters of the segmentation model unchanged, the evaluation model is trained, including:

[0025] Fix the current first model parameters in the segmentation model, and use the segmentation model to predict the plurality of first crack images to obtain a prediction result based on the current first model parameters;

[0026] The evaluation model is used to evaluate the feature loss between the predicted result and the actual pixel-level crack marker using the loss function shown below:

[0027]

[0028] Where L represents the feature loss, N represents the number of first crack images input to the segmentation model, and l mean x represents the mean absolute error of the prediction result of the segmentation model. n S(x) represents the pixel distribution of the nth first crack image. n ) represents the label distribution of the image predicted by the segmentation model, y n This represents the actual pixel-level marker distribution of the nth first crack image;

[0029] The feature loss is backpropagated, and the second model parameters are adjusted. The second model parameters at which the feature loss is maximized are determined as the second target parameters of the evaluation model.

[0030] Further, the segmentation model is trained while keeping the current second objective parameters of the evaluation model unchanged, including:

[0031] The second target parameter in the evaluation model is fixed, and the segmentation model is used to predict the plurality of first crack images to obtain a prediction result based on the first model parameter;

[0032] The evaluation model uses the loss function to evaluate the feature loss between the predicted result and the actual pixel-level crack marker:

[0033] The feature loss is backpropagated, and the first model parameters are adjusted. The first model parameters that minimize the feature loss are determined as the first target parameters of the segmentation model.

[0034] Based on the same inventive concept, this application also provides a road surface crack detection device, comprising:

[0035] Model building module, model training module, and model prediction module;

[0036] The model building module is configured to build a segmentation model using a preset first encoder and decoder, and to build an evaluation model using a preset second encoder.

[0037] The model training module is configured to alternately train the segmentation model and the evaluation model using a set of preset first crack images for training, so as to obtain a trained segmentation model. Each first crack image is pre-labeled with actual pixel-level crack markers.

[0038] The model prediction module is configured to input multiple road surface images to be labeled into the trained segmentation model, perform pixel-level image semantic segmentation on each road surface image to obtain multiple crack-labeled images containing pixel-level crack labels, and determine the road surface cracks in the crack-labeled images.

[0039] Based on the same inventive concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the road surface crack detection method as described in any of the above.

[0040] Based on the same inventive concept, this application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the road surface crack detection method described above.

[0041] As can be seen from the above, the road surface crack detection method, device, electronic device and storage medium provided in this application, based on the segmentation model constructed by the first encoder and decoder, and based on the evaluation model using the second encoder, can alternately train the segmentation model and the evaluation model to obtain an evaluation model that can accurately evaluate the performance of the segmentation model, and obtain a segmentation model that can effectively and accurately predict pixel-level crack markers. Based on the trained segmentation model, the road surface image to be marked can be semantically segmented to extract the final second crack image containing pixel-level crack markers. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a road surface crack detection method according to an embodiment of this application;

[0044] Figure 2 This is a flowchart illustrating the training data preprocessing process in an embodiment of this application.

[0045] Figure 3 This is a flowchart illustrating pixel-level marker extraction in an embodiment of this application.

[0046] Figure 4 This is a schematic diagram of the road surface crack detection device according to an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the electronic device structure according to an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0049] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0050] As described in the background section, existing methods for detecting pavement cracks are still insufficient to meet the needs of practical prediction.

[0051] In the process of developing this application, the applicant discovered that the main problem with the relevant pavement crack detection methods is that, on the one hand, due to the constraints of labor costs and time costs, the crack images identified are basically based on block-level markings, and the granularity of the block-level markings is too coarse, which fundamentally limits the improvement of the accuracy of automated pavement crack detection.

[0052] On the other hand, road surface conditions are affected by climate, traffic flow, and other factors, resulting in a wide variety of crack types, such as shallow cracks, alligator cracks, and network cracks. Furthermore, there are many interfering factors, such as lane lines, stains, and uneven lighting, which greatly increases the difficulty of crack detection. Road surface crack detection models trained based on data from real-world application scenarios have poor recognition performance for fine cracks and a high rate of missed detection.

[0053] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0054] In the embodiments of this application, a neural network is used to predict the acquired road surface image to obtain a crack image marked with pixel-level cracks.

[0055] Specifically, a pavement crack detection model that extracts pixel-level crack markers is constructed and an adversarial network is formed. The pavement crack detection model consists of a segmentation model and an evaluation model. The pavement crack detection model is trained using pre-set training images to obtain a trained segmentation model.

[0056] In this embodiment, before training the adversarial network, since training materials with pixel-level crack markings are not easy to obtain, the training images can be input into the pre-trained convolutional neural network to predict crack features and obtain crack images with block-level crack markings. The block-level crack-marked crack images are then converted into pixel-level crack-marked images by a pixel-level crack extraction algorithm based on edge detection before the adversarial network is trained.

[0057] Furthermore, based on the trained segmentation model, the road surface image to be labeled can be input into it, and the road surface image can be predicted to produce pixel-level labeled crack images.

[0058] refer to Figure 1 A method for detecting road surface cracks according to one embodiment of this application includes the following steps:

[0059] Step S101: Construct a segmentation model using a preset first encoder and decoder, and construct an evaluation model using a preset second encoder.

[0060] In the embodiments of this application, an adversarial network can be formed by constructing a segmentation model and an evaluation model, that is, a road surface crack detection model can be formed; wherein, the segmentation model can be constructed by setting the structure of an encoder and a decoder, and an encoder can be set in the evaluation model.

[0061] In this embodiment, the encoder in the segmentation model is used as the first encoder, and the encoder in the evaluation model is used as the second encoder.

[0062] Specifically, in the segmentation model, a U-shaped network is formed by a first encoder and a decoder. The first encoder has a convolutional layer with a stride of 2 to downsample the crack features of the image.

[0063] Furthermore, an attention mechanism is set in each convolutional layer of the decoder in the segmentation model, and transposed convolution is used to upsample the input image.

[0064] Furthermore, in the first encoder and decoder, a guide filter is set for each convolutional layer, and a corresponding guide image is set for it. Based on this, the feature information lost during the downsampling of the crack image can be compensated, ensuring that in a local area, when the guide image has an edge, the output image also keeps the edge unchanged, thereby improving the model's ability to identify fine cracks. At the same time, the non-edge areas and uneven parts in the input image are regarded as noise and the noise is minimized.

[0065] Furthermore, a second encoder is set in the evaluation model, and the stride of the convolutional layer in the second encoder is set to 2, so as to downsample the difference between the prediction results of the segmentation model and the actual labeled image.

[0066] Furthermore, based on the network structures of the segmentation model and the evaluation model determined above, a loss function for evaluating the segmentation model and the evaluation model itself can be set in the evaluation model.

[0067] Specifically, the loss function can be defined as the feature difference between the prediction result predicted by the segmentation model and the actual pixel-level crack markers pre-labeled in the training image. In this embodiment, the loss function can specifically represent multi-scale feature loss.

[0068] Furthermore, the segmentation model and evaluation model constructed above can be combined to form a road surface crack detection model for the following adversarial training.

[0069] Step S102: The segmentation model and the evaluation model are trained alternately using multiple first crack images for training to obtain a trained segmentation model. Each first crack image is pre-labeled with actual pixel-level crack markers.

[0070] In the embodiments of this application, based on the pavement crack detection model constructed with the segmentation model and the evaluation model described above, the segmentation model and the evaluation model can be trained alternately.

[0071] In this embodiment, before alternating training of the segmentation model and the evaluation model using a plurality of preset first crack images for training, preprocessing of the training images is required, specifically including the following steps:

[0072] Step S201: Use a pre-trained densely connected convolutional neural network to predict cracks in multiple pre-set training images to obtain multiple initial crack images containing block-level crack labels.

[0073] Step S202: Perform histogram equalization, median filtering, image grayscale logarithmic transformation, and morphological opening operation on each initial crack image.

[0074] Step S203: According to the preset dynamic threshold, each processed initial crack image is converted into a binary image and a pixel-level crack extraction algorithm based on edge detection is used to obtain the plurality of first crack images containing pixel-level crack markers.

[0075] In specific examples, based on Figure 3 The step S301 shown is to construct a road surface crack detection model. Step S302, inputting training images, can be further executed.

[0076] First, the initial images used for training are preprocessed. Specifically, since the acquired initial images are of different sizes and contain various types of interference, such as dirt interference, shadow interference, low-resolution interference, etc., the initial images can be preprocessed to remove interference in order to obtain training images.

[0077] Furthermore, a pre-trained DenseNet (densely connected convolutional neural network) can be used to predict cracks in the training images to obtain multiple initial crack images containing block-level crack labels.

[0078] That is, execution Figure 3 Step S303: Block-level crack prediction is performed using a pre-trained densely connected convolutional neural network.

[0079] Furthermore, based on the aforementioned crack prediction results, each initial crack image containing block-level crack markers can undergo further initial processing.

[0080] Specifically, histogram equalization can be performed sequentially to eliminate uneven illumination and enhance image contrast; further, median filtering can be applied to address the issue of the image becoming mottled after histogram equalization; further, logarithmic transformation of the image grayscale can be performed to make the feature changes in the low grayscale range more obvious; and further, morphological opening operations and other processing operations can be performed on the initial crack image.

[0081] Furthermore, a dynamic threshold can be designed for the initial crack image after the initial processing described above, so as to convert the initial crack image after initial processing into a binary image and obtain multiple initial sub-images.

[0082] Specifically, the dynamic threshold is designed according to the following formula, based on the average gray level and minimum gray level of each initial sub-image:

[0083]

[0084] Where thresh represents the dynamic threshold, patch_min represents the minimum gray level, patch_mean represents the average gray level, and dist represents the difference between the average gray level and the minimum gray level.

[0085] Based on the aforementioned dynamic threshold, each initial sub-image can be converted into a binary image.

[0086] Further, step S304 can be performed to extract pixel-level crack markers. Specifically, a contour extraction algorithm is used for each binary image to find crack contours and filter out contours with smaller areas, thereby obtaining multiple first crack images.

[0087] In some other embodiments, for each of the first crack images obtained above, a step can be added to determine whether it is qualified, that is, whether it meets the standard.

[0088] Specifically, step S305 is executed to determine whether the extraction effect meets the standard.

[0089] In a specific example, based on the judgment result of step S305 above, when the judgment result is yes, that is, when the extraction effect meets the standard, step S306 can be further executed to determine the first crack image as meeting the standard.

[0090] Furthermore, when the judgment result of step S305 is negative, that is, when the extraction effect is not up to standard, step S307 can be further executed for manual correction, and after correction, a prediction result that meets the standard is obtained, that is, step S306, the first crack image that is determined to meet the standard.

[0091] Based on this, multiple first crack images can be input into the aforementioned pavement crack detection model, that is, the segmentation model and the evaluation model can be trained alternately.

[0092] Furthermore, during the alternating training process, both the segmentation model and the evaluation model have their current model parameters set. The evaluation model can be trained by fixing the current model parameters of the segmentation model to obtain the model parameters of the trained evaluation model, which can then be used to replace the current model parameters of the evaluation model. Then, the segmentation model can be trained by fixing the trained model parameters of the evaluation model to obtain the model parameters of the trained segmentation model, which can then be used to replace the current model parameters of the segmentation model.

[0093] In this embodiment, the current model parameters of the segmentation model are used as the first model parameters, the current model parameters of the evaluation model are used as the second model parameters, the model parameters obtained in the segmentation model after training the segmentation model are used as the first target parameters, and the model parameters obtained in the evaluation model after training the evaluation model are used as the second target parameters.

[0094] In a specific example of this application, the current first model parameters of the segmentation model are fixed and unchanged, and the current segmentation model is used to predict multiple input first crack images.

[0095] Specifically, the first encoder is used to downsample each training image four times to extract crack features of road surface cracks in the training image.

[0096] Furthermore, the decoder performs four upsampling operations via transposed convolution to reconstruct the first crack image segmented by the first encoder.

[0097] Based on this, prediction results for each first crack image can be obtained from the segmentation model, which includes second crack images with pixel-level crack features.

[0098] Furthermore, the feature loss can be evaluated using the loss function set in the evaluation model.

[0099] Specifically, the feature difference between the prediction result of the segmentation model and the pre-marked actual pixel-level crack markers in the first crack image can be used as the feature loss, and the loss function can be set to the form shown below:

[0100]

[0101] Where L represents the feature loss, which in this embodiment can be a multi-scale feature loss, and N represents the number of first crack images input to the segmentation model. mean x represents the mean absolute error of the prediction results of the segmentation model. n S(x) represents the pixel distribution of the nth first crack image. n ) represents the label distribution predicted by the segmentation model, y n f represents the actual marker distribution of the nth first crack image. C This represents the multi-level features extracted by the evaluation model.

[0102] in,

[0103]

[0104] in, This represents the feature distribution extracted by the evaluation model from the i-th layer of the prediction results. This represents the pre-defined true feature distribution in the training image.

[0105] Furthermore, based on the calculated multi-scale feature loss, it is backpropagated, and the gradient of the evaluation model can be adjusted accordingly. In this embodiment, it can be gradient descent.

[0106] Furthermore, based on the adjustment of the gradient, the second model parameter for evaluating the model when the multi-scale feature loss is maximized can be determined, and the second model parameter for evaluating the model when the multi-scale feature loss is maximized can be used as the second objective parameter.

[0107] Furthermore, the second target parameters in the evaluation model are fixed, and the first crack image is input into the road surface crack detection model again. The segmentation model is then used to predict the training image again, and the prediction result is obtained, which is based on the current first model parameters of the segmentation model.

[0108] Furthermore, based on the prediction result, the multi-scale feature loss of this prediction is determined by using the second target parameter and loss function trained in the evaluation model, and backpropagation is performed on it to determine the first model parameter when the multi-scale feature loss is minimized. The first model parameter of the segmentation model when the multi-scale feature loss is minimized is taken as the first target parameter.

[0109] It can be seen that when training the segmentation model, in order to make the crack features segmented by the segmentation model more similar to the pre-set labels, it is necessary to minimize the multi-scale feature loss; while when training the evaluation model, in order to make the evaluation model better extract the difference between the predicted distribution results of the segmentation model and the true labels, it is necessary to maximize the multi-scale feature loss. By continuously alternating training, both the segmentation model and the evaluation model can achieve accurate results, resulting in a segmentation model with good segmentation performance.

[0110] Step S103: Input multiple road surface images to be labeled into the trained segmentation model, and perform pixel-level image semantic segmentation on each road surface image to obtain multiple crack-labeled images containing pixel-level crack labels, and determine the road surface cracks in the crack-labeled images.

[0111] In the embodiments of this application, the trained road surface crack detection model described above can be used to perform pixel-level detection of cracks in road surface images.

[0112]

[0113] In this embodiment, multiple road surface images to be marked can be input into the segmentation model trained in the aforementioned steps to perform pixel-level image semantic segmentation, so as to predict pixel-level crack markers from each road surface image, and thus obtain multiple crack marker images containing pixel-level crack markers.

[0114] Furthermore, based on the obtained crack marking images, pavement cracks can be identified to enable necessary pavement maintenance.

[0115] As can be seen, the road surface crack detection method of the present application, based on the segmentation model constructed by the first encoder and decoder, and based on the evaluation model using the second encoder, can alternately train the segmentation model and the evaluation model to obtain an evaluation model that can accurately evaluate the performance of the segmentation model, and obtain a segmentation model that can effectively and accurately predict pixel-level crack markers. Based on the trained segmentation model, the road surface image to be marked can be semantically segmented to extract the final second crack image containing pixel-level crack markers.

[0116] It should be noted that the method of the embodiments of this application can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of the embodiments of this application, and the multiple devices will interact with each other to complete the method described.

[0117] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the embodiments of this application also provide a road surface crack detection device.

[0119] refer to Figure 4 The road surface crack detection device includes: a model building module 401, a model training module 402, and a model prediction module 403;

[0120] The model building module 401 is configured to build a segmentation model using a preset first encoder and decoder, and to build an evaluation model using a preset second encoder.

[0121] The model training module 402 is configured to alternately train the segmentation model and the evaluation model using a plurality of preset first crack images for training, so as to obtain a trained segmentation model, wherein each first crack image is pre-labeled with actual pixel-level crack markers.

[0122] The model prediction module 403 is configured to input multiple road surface images to be labeled into the trained segmentation model, perform pixel-level image semantic segmentation on each road surface image to obtain multiple crack-labeled images containing pixel-level crack labels, and determine the road surface cracks in the crack-labeled images.

[0123] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0124] The apparatus described above is used to implement the corresponding pavement crack detection method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0125] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the road surface crack detection method as described in any of the above embodiments.

[0126] Figure 5 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0127] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0128] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this application are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0129] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0130] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0131] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0132] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this application, and not necessarily all the components shown in the figures.

[0133] The apparatus described above is used to implement the corresponding pavement crack detection method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0134] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the road surface crack detection method as described in any of the above embodiments.

[0135] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0136] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the pavement crack detection method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0137] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0138] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0139] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0140] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for detecting road surface cracks, characterized in that, include: A segmentation model is constructed using a pre-set first encoder and decoder, and an evaluation model is constructed using a pre-set second encoder. The segmentation model and the evaluation model are trained alternately using multiple first crack images for training to obtain a trained segmentation model. Each first crack image is pre-labeled with actual pixel-level crack markers. Multiple road surface images to be labeled are input into the trained segmentation model, and pixel-level image semantic segmentation is performed on each road surface image to obtain multiple crack-labeled images containing pixel-level crack labels, and the road surface cracks in the crack-labeled images are identified. The step of alternately training the segmentation model and the evaluation model using multiple pre-set first crack images for training to obtain a trained segmentation model specifically includes: The current first model parameters of the fixed segmentation model remain unchanged. The current segmentation model is used to predict multiple first crack images, and the prediction result for each first crack image is obtained. The prediction result includes a second crack image with pixel-level crack features. Set the loss function to the form shown below: ； Where L represents the feature loss, and N represents the number of first crack images input to the segmentation model. The mean absolute error represents the prediction result of the segmentation model. This represents the pixel distribution of the nth first crack image. This represents the label distribution predicted by the segmentation model. This represents the actual marker distribution of the nth first crack image. This represents the multi-level features extracted by the evaluation model; in, ； in, This represents the feature distribution extracted by the evaluation model from the i-th layer of the prediction results. This represents the pre-defined true feature distribution in the training images; Based on the calculated multi-scale feature loss, backpropagation is performed on it to adjust the gradient of the evaluation model; Based on the adjustment of the gradient, the second model parameter for evaluating the model when the multi-scale feature loss is maximized is determined, and the second model parameter for evaluating the model when the multi-scale feature loss is maximized is used as the second objective parameter. The second objective parameters in the fixed evaluation model are then input into the pavement crack detection model again, and the segmentation model is used again to predict the training image to obtain the prediction result, which is based on the current first model parameters of the segmentation model. Based on the prediction result, the multi-scale feature loss of this prediction is determined by using the second target parameter and loss function trained in the evaluation model, and backpropagation is performed on it to determine the first model parameter when the multi-scale feature loss is minimized. The first model parameter of the segmentation model when the multi-scale feature loss is minimized is taken as the first target parameter.

2. The method according to claim 1, characterized in that, Before alternatingly training the segmentation model and the evaluation model using a set of pre-defined first crack images for training, the method further includes: A pre-trained densely connected convolutional neural network is used to predict cracks in multiple pre-set training images to obtain multiple initial crack images containing block-level crack labels. Histogram equalization, median filtering, image grayscale logarithmic transformation, and morphological opening are performed on each initial crack image; According to a preset dynamic threshold, each processed initial crack image is converted into a binary image and a pixel-level crack extraction algorithm based on edge detection is used to obtain the plurality of first crack images containing pixel-level crack markers.

3. The method according to claim 1, characterized in that, The construction of the segmentation model using a pre-set first encoder and decoder includes: The stride of the convolutional layer in the first encoder is set to 2, and upsampling is performed using transposed convolution in the decoder. A guiding filter is set in both the first encoder and the decoder, and an attention mechanism is added to the decoder.

4. The method according to claim 1, characterized in that, The construction of the evaluation model using a pre-set second encoder includes: Set the stride of the convolutional layer in the second encoder to 2; And set a loss function for evaluating the segmentation model and the evaluation model itself.

5. The method according to claim 4, characterized in that, The alternating training of the segmentation model and the evaluation model includes: The preset multiple first crack images are input into the segmentation model and the evaluation model; The first model parameters of the segmentation model are kept unchanged, the evaluation model is trained, and the second target parameters of the evaluation model are determined. The second target parameters are used to update the second model parameters of the evaluation model. The second objective parameter of the evaluation model is kept unchanged, the segmentation model is trained, and the first objective parameter of the segmentation model is determined. The first objective parameter is used to update the first model parameter.

6. The method according to claim 5, characterized in that, The step of fixing the current first model parameters of the segmentation model and training the evaluation model includes: Fix the current first model parameters in the segmentation model, and use the segmentation model to predict the plurality of first crack images to obtain a prediction result based on the current first model parameters; The evaluation model is used to evaluate the feature loss between the predicted result and the actual pixel-level crack marker using the loss function shown below: Where L represents the feature loss, and N represents the number of first crack images input into the segmentation model. The mean absolute error represents the prediction result of the segmentation model. This represents the pixel distribution of the nth first crack image. This represents the label distribution of the image predicted by the segmentation model. This represents the actual pixel-level marker distribution of the nth first crack image; The feature loss is backpropagated, and the second model parameters are adjusted. The second model parameters at which the feature loss is maximized are determined as the second target parameters of the evaluation model.

7. The method according to claim 6, characterized in that, The step of fixing the current second objective parameters of the evaluation model and training the segmentation model includes: The second target parameter in the evaluation model is fixed, and the segmentation model is used to predict the plurality of first crack images to obtain a prediction result based on the first model parameter; The evaluation model uses the loss function to evaluate the feature loss between the predicted result and the actual pixel-level crack marker: The feature loss is backpropagated, and the first model parameters are adjusted. The first model parameters that minimize the feature loss are determined as the first target parameters of the segmentation model.

8. A road surface crack detection device, characterized in that, include: Model building module, model training module, and model prediction module; The model building module is configured to build a segmentation model using a preset first encoder and decoder, and to build an evaluation model using a preset second encoder. The model training module is configured to alternately train the segmentation model and the evaluation model using a set of preset first crack images for training, so as to obtain a trained segmentation model. Each first crack image is pre-labeled with actual pixel-level crack markers. The model prediction module is configured to input multiple road surface images to be labeled into the trained segmentation model, perform pixel-level image semantic segmentation on each road surface image to obtain multiple crack-labeled images containing pixel-level crack labels, and determine the road surface cracks in the crack-labeled images. The model training module specifically includes: The current first model parameters of the fixed segmentation model remain unchanged. The current segmentation model is used to predict multiple first crack images, and the prediction result for each first crack image is obtained. The prediction result includes a second crack image with pixel-level crack features. Set the loss function to the form shown below: ； Where L represents the feature loss, and N represents the number of first crack images input to the segmentation model. The mean absolute error represents the prediction result of the segmentation model. This represents the pixel distribution of the nth first crack image. This represents the label distribution predicted by the segmentation model. This represents the actual marker distribution of the nth first crack image. This represents the multi-level features extracted by the evaluation model; in, ； in, This represents the feature distribution extracted by the evaluation model from the i-th layer of the prediction results. This represents the pre-defined true feature distribution in the training images; Based on the calculated multi-scale feature loss, backpropagation is performed on it to adjust the gradient of the evaluation model; Based on the adjustment of the gradient, the second model parameter for evaluating the model when the multi-scale feature loss is maximized is determined, and the second model parameter for evaluating the model when the multi-scale feature loss is maximized is used as the second objective parameter. The second objective parameters in the fixed evaluation model are then input into the pavement crack detection model again, and the segmentation model is used again to predict the training image to obtain the prediction result, which is based on the current first model parameters of the segmentation model. Based on the prediction result, the multi-scale feature loss of this prediction is determined by using the second target parameter and loss function trained in the evaluation model, and backpropagation is performed on it to determine the first model parameter when the multi-scale feature loss is minimized. The first model parameter of the segmentation model when the multi-scale feature loss is minimized is taken as the first target parameter.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1 to 7.

Images