Concrete crack segmentation method, device and equipment based on data optimization and medium

Abstract

The invention relates to the technical field of computer vision and concrete crack recognition. The invention discloses a concrete crack segmentation method and device based on data optimization, equipment and a medium. The accuracy of identifying a crack segmentation result can be improved from the perspective of data optimization. The concrete crack segmentation method based on data optimization comprises the following steps: acquiring a concrete crack image; the concrete crack image is input into a crack semantic segmentation model for recognition processing, a crack segmentation result is obtained, and the crack semantic segmentation model is trained by a verification set and a target training set screened by a multi-expert cross evaluation voting model.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and concrete crack identification. More specifically, the present application relates to a method, device, equipment and medium for concrete crack segmentation based on data optimization. Background Art

[0002] In the development of damage detection for civil infrastructure, traditional crack identification methods rely on visual inspection by quality inspectors according to testing standards. However, the inherent labor-intensiveness, risk, subjectivity, and time-consuming nature of manual inspections have limited their effectiveness. With the rapid development of machine vision technology in recent years, it has been widely used to identify cracks in civil infrastructure. For example, semantic segmentation methods within machine vision are an effective approach for intelligent crack identification.

[0003] Current semantic segmentation methods typically use a classifier embedded at the end of a decoding structure to classify each pixel in the image to distinguish crack pixels from background pixels, thereby obtaining a crack segmentation result. The accuracy of the crack segmentation result depends on the detection performance of the classifier. Therefore, the semantic segmentation method improves the detection performance of the classifier by optimizing the algorithm and adjusting parameters. In this process of improving the detection performance of the classifier, it is assumed that the training samples used to train the classifier are completely reliable. However, the training samples usually have certain deviations. Specifically, the training samples may contain noise (for example, the training samples have incorrect labels), which will cause the training samples to deviate from the real samples. Regardless of the form of noise, the detection performance of the classifier will be limited by it, resulting in a serious reduction in classification accuracy. This serious reduction in classification accuracy will directly affect the accuracy of the crack segmentation results of the semantic segmentation method, thereby causing the semantic segmentation method to identify crack segmentation results that are not accurate enough. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, equipment and medium for concrete crack segmentation based on data optimization, which can improve the accuracy of crack segmentation results from the perspective of data optimization. The embodiments of the present application are mainly achieved through the following technical solutions:

[0005] A first aspect of an embodiment of the present application provides a method for segmenting concrete cracks based on data optimization, comprising:

[0006] Acquire concrete crack images;

[0007] The concrete crack image is input into a crack semantic segmentation model for recognition processing to obtain a crack segmentation result, wherein the crack semantic segmentation model is trained by a validation set and a target training set screened by a multi-expert cross-evaluation voting model.

[0008] According to one embodiment of the present application, the training step of the crack semantic segmentation model includes:

[0009] Obtaining the target training set, the validation set, and a first true label set, wherein the target training set is obtained by screening the multi-expert cross-evaluation voting model;

[0010] Inputting a first training sample into the crack segmentation network for recognition processing to obtain a first prediction result, wherein the first training sample is any element in the target training set and the validation set, and there is a one-to-one correspondence between the first training sample and one of the first true values ​​in the first true label set;

[0011] Calculating a first loss function based on the first prediction result and a first true value corresponding to the first training sample;

[0012] A first model parameter of the crack segmentation network is adjusted based on the first loss function to obtain the crack semantic segmentation model.

[0013] According to one embodiment of the present application, after the step of calculating a first loss function based on the first prediction result and a first true value corresponding to the first training sample, the method further includes:

[0014] Set the second loss function;

[0015] Calculate a third loss function based on the first loss function and the second loss function;

[0016] A second model parameter of the crack semantic segmentation model is adjusted based on the third loss function.

[0017] According to one embodiment of the present application, the step of training the crack semantic segmentation model further includes:

[0018] An evaluation method is used to evaluate the crack semantic segmentation model to obtain an evaluation result.

[0019] According to one embodiment of the present application, the step of obtaining the target training set includes:

[0020] Obtaining a first set of images to be processed;

[0021] Preprocessing the first to-be-processed image set to obtain an initial training set, the verification set, and a test set;

[0022] The initial training set and the validation set are screened using a multi-expert cross-evaluation voting model to obtain the target training set.

[0023] According to one embodiment of the present application, the step of preprocessing the first set of images to be processed to obtain an initial training set, the validation set, and the test set includes:

[0024] performing size normalization processing on each first image to be processed in the first set of images to be processed to obtain a second set of images to be processed;

[0025] performing data type adjustment processing on each second image to be processed in the second set of images to be processed to obtain a third set of images to be processed;

[0026] The third set of images to be processed is divided to obtain the initial training set, the verification set, and the test set.

[0027] According to one embodiment of the present application, the step of using a multi-expert cross-evaluation voting model to screen the initial training set and the validation set to obtain the target training set includes:

[0028] Dividing the initial training set and the validation set into multiple subsets;

[0029] Set up multiple expert models, each of which has a one-to-one correspondence with one of the subsets. However, each expert model is not used to evaluate its own corresponding subset, but is used to perform crack scoring on images of other subsets.

[0030] Using all target expert models to perform crack scoring processing on target images in a target subset, and obtaining a crack scoring result corresponding to each target expert model, wherein the target subset is any one of the multiple subsets, the target image is any one of the images in the target subset, and the target expert models are all expert models that do not have a one-to-one correspondence with the target subset;

[0031] Calculating the average of all crack scoring results to obtain a final crack assessment score corresponding to the target image;

[0032] Set preset thresholds;

[0033] When the final crack assessment score is greater than the preset threshold, the target image is added to the target training set.

[0034] A second aspect of the embodiments of the present application provides a concrete crack segmentation device based on data optimization, comprising:

[0035] A concrete crack image acquisition module is used to acquire concrete crack images;

[0036] The crack segmentation result acquisition module is used to input the concrete crack image into the crack semantic segmentation model for recognition processing to obtain the crack segmentation result, wherein the crack semantic segmentation model is trained by the validation set and the target training set screened by the multi-expert cross-evaluation voting model.

[0037] A third aspect of an embodiment of the present application provides a terminal device, including: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the steps of the data optimization-based concrete crack segmentation method provided in the first aspect of the embodiment of the present application.

[0038] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute the steps of the concrete crack segmentation method based on data optimization provided in the first aspect of the embodiment of the present application.

[0039] The beneficial effects of the embodiments of the present application include:

[0040] The embodiment of the present application improves the accuracy of crack semantic segmentation model recognition from the perspective of improving training data. Specifically, the embodiment of the present application will use the target training set screened by the multi-expert cross-evaluation voting model to train the crack semantic segmentation model, so that the crack semantic segmentation model can obtain accurate crack segmentation results when performing recognition processing on concrete crack images. Compared with the semantic segmentation method of the prior art that uses noisy training samples to train the model, the present application focuses on the quality of training data, thereby being able to improve the accuracy of crack segmentation results from the perspective of data optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A flowchart of a concrete crack segmentation method based on data optimization in some embodiments of the present application;

[0043] Figure 2 This is a reference image of concrete cracks in this application;

[0044] Figure 3Flowcharts of other embodiments of the concrete crack segmentation method based on data optimization in this application;

[0045] Figure 4 Flowchart for obtaining the target training set in this application;

[0046] Figure 5 A histogram of the performance indicators of the crack semantic segmentation model under different preset thresholds in this application;

[0047] Figure 6 These are two types of erroneous reference images that may occur in the process of obtaining the target training set in this application;

[0048] Figure 7 This is a reference diagram of the U-Net structure in some embodiments of this application;

[0049] Figure 8 Reference diagrams for the structures of U-Net, Unet++, and CrackU-net+ in some embodiments of this application;

[0050] Figure 9 Reference diagrams for hardware configurations and environmental parameters in some embodiments of this application;

[0051] Figure 10 This is a reference image of the segmentation results of various types of concrete crack images by the expert model in this application;

[0052] Figure 11 Reference graph of crack segmentation results of the initial model and crack semantic segmentation model (i.e., target model) in this application on the test set;

[0053] Figure 12 This is a reference diagram of the loss curve of the U-Net network with and without data filtering processing in this application;

[0054] Figure 13 This is a reference diagram of the IOU curve of the U-Net network with and without data filtering processing in this application;

[0055] Figure 14 This is a reference chart for the performance index values ​​of U-Net, Unet++, and CrackU-net+ on the test set in this application;

[0056] Figure 15 This is a principle block diagram of a concrete crack segmentation device based on data optimization in some embodiments of the present application;

[0057] Figure 16 This is a principle block diagram of the terminal device in some embodiments of the present application. DETAILED DESCRIPTION

[0058] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0059] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0060] The terms "exemplary" or "for example" are used to indicate an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0061] The terms "comprises," "comprising," or any other variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0062] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more of the relevant listed items.

[0063] In the development of damage detection for civil infrastructure, traditional crack identification methods rely on visual inspection by quality inspectors according to testing standards. However, the inherent labor-intensiveness, risk, subjectivity, and time-consuming nature of manual inspections have limited their effectiveness. With the rapid development of machine vision technology in recent years, it has been widely used to identify cracks in civil infrastructure. For example, semantic segmentation methods within machine vision are an effective approach for intelligent crack identification.

[0064] Current semantic segmentation methods typically use a classifier embedded at the end of a decoding structure to classify each pixel in the image to distinguish crack pixels from background pixels, thereby obtaining a crack segmentation result. The accuracy of the crack segmentation result depends on the detection performance of the classifier. Therefore, the semantic segmentation method improves the detection performance of the classifier by optimizing the algorithm and adjusting parameters. In this process of improving the detection performance of the classifier, it is assumed that the training samples used to train the classifier are completely reliable. However, the training samples usually have certain deviations. Specifically, the training samples may contain noise (for example, the training samples have incorrect labels), which will cause the training samples to deviate from the real samples. Regardless of the form of noise, the detection performance of the classifier will be limited by it, resulting in a serious reduction in classification accuracy. This serious reduction in classification accuracy will directly affect the accuracy of the crack segmentation results of the semantic segmentation method, thereby causing the semantic segmentation method to identify crack segmentation results that are not accurate enough.

[0065] Furthermore, noise can negatively impact the reliability of extracted data features, models built from the data, and judgments made based on the data. If the training set (which contains multiple training samples) used to train the model contains incorrectly labeled samples, the classifier's performance will significantly deteriorate, turning the semantic segmentation task into a weakly supervised learning problem.

[0066] Based on the above analysis, this application focuses on improving model performance from the perspective of improving the training set (i.e., training data). Specifically, multiple expert models are obtained through Unet++ network training. Data cleaning (which can be understood as data screening) is performed on poorly labeled images to generate high-quality training sets. This improves the performance of deep learning models from the perspective of dataset quality, thereby optimizing the accuracy of concrete crack segmentation.

[0067] The specific implementation of this application is further described below with reference to the accompanying drawings.

[0068] like Figure 1 As shown in FIG, it is a flow chart of a concrete crack segmentation method based on data optimization provided in the first aspect of the embodiment of the present application. Figure 1 In the embodiment, the concrete crack segmentation method based on data optimization includes:

[0069] S1. Obtain concrete crack images.

[0070] The concrete crack image can be referenced Figure 2 shown.

[0071] S2. Input the concrete crack image into a crack semantic segmentation model for recognition processing to obtain a crack segmentation result, wherein the crack semantic segmentation model is trained by a validation set and a target training set screened by a multi-expert cross-evaluation voting model.

[0072] The above technical solution improves the accuracy of crack semantic segmentation model recognition by improving training data. Therefore, compared with the existing semantic segmentation methods that use noisy training samples to train models, this application focuses on the quality of training data, thereby improving the accuracy of crack segmentation results from the perspective of data optimization.

[0073] In some embodiments, the step of training the crack semantic segmentation model includes:

[0074] S31. Obtain the target training set, the validation set, and the first true label set, wherein the target training set is obtained by screening the multi-expert cross-evaluation voting model.

[0075] Furthermore, the step of obtaining the target training set includes:

[0076] S311: Obtain a first set of images to be processed.

[0077] The first set of images to be processed may be a plurality of original crack images collected from a public network platform.

[0078] The original crack images on the public network platform are diverse in terms of crack morphology, background environment, and shooting distance, thereby improving the generalization ability of the crack semantic segmentation model and enhancing the robustness of the crack semantic segmentation model.

[0079] In the embodiment of the present application, the first set of images to be processed may include 2139 first images to be processed with different contents (ie, 2139 original torn text images with different contents). It should be understood that each of the first images to be processed has a corresponding manually annotated label.

[0080] S312, pre-processing the first set of images to be processed to obtain an initial training set, the verification set and the test set. The step S312 can be understood as Figure 3 The "Image Preprocessing" step in .

[0081] The pre-processing step can solve the problem that each image in the first set of images to be processed has different pixels and formats.

[0082] Further, refer to Figure 3 As shown, the step S312 includes:

[0083] S3121. Perform size normalization processing on each first image to be processed in the first set of images to be processed to obtain a second set of images to be processed.

[0084] Specifically, the step S3121 is to use the PIL (Python Imaging Library) library to standardize the image resolution of each of the first to-be-processed images to 448*448 dpi. The "*" symbol is a multiplication symbol in the field of mathematics.

[0085] In other implementations, the image resolution may take other values, which may be determined by those skilled in the art according to actual needs.

[0086] S3122: Perform data type adjustment processing on each second image to be processed in the second image set to be processed to obtain a third image set to be processed.

[0087] Specifically, the data type adjustment refers to converting the image format of each second image to be processed into PNG format.

[0088] In other embodiments, the image format is not limited to the aforementioned PNG format, and may be JPG or other formats. This application does not impose further restrictions on this, and it may be determined by those skilled in the art according to actual needs.

[0089] S3123, dividing the third set of images to be processed to obtain the initial training set, the verification set and the test set. The third set of images to be processed can be understood as Figure 3 The "Initial Benchmark Set" in .

[0090] For example, in this embodiment of the present application, the third image set to be processed has 2139 images. After step S3123, the total number of images in the initial training set and the validation set is 2067, of which the validation set accounts for 20% of the total number. The number of images in the test set is 72.

[0091] In other implementations, the specific numerical divisions of the division process are not limited to the aforementioned examples and can be set by those skilled in the art according to actual needs.

[0092] It should be understood that the purpose of dividing the validation set is to use the validation set to adjust hyperparameters.

[0093] After the test set is divided, the test set needs to be checked to ensure the accuracy of the annotation.

[0094] Furthermore, before the step S3123, the step S312 further includes:

[0095] S3124: Randomly shuffle all the third images to be processed in the third image set to be processed, and then perform batch renaming after the random shuffle.

[0096] The step S3124 makes the distribution of images in the third set of images to be processed random rather than regular.

[0097] S313, using a multi-expert cross-evaluation voting model to screen the initial training set and the validation set to obtain the target training set. Figure 3 The "Data Cleansing" step in

[0098] Further, refer to Figure 4 As shown, the step S313 includes:

[0099] S3131. Divide the initial training set and the validation set into multiple subsets.

[0100] Exemplary, reference Figure 4 As shown, the embodiment of the present application divides the initial training set and the verification set into N subsets, specifically subset 1, subset 2, subset 3, ..., subset N. It should be understood that the value of N cannot be too large or too small. If the N value is too large, the number of subsets is more, and each subset contains fewer images, which is likely to cause overfitting of the expert model. If the N value is too small and the number of subsets is insufficient, it is easy to produce a large deviation in the voting process of the cross-review, weakening the effect of the data selection mechanism. Thus, after experimental research, the embodiment of the present application defines the value of N as 4, that is, the embodiment of the present application divides the initial training set and the verification set into 4 subsets, and uses these 4 subsets for cross-review.

[0101] S3132. Set multiple expert models, each expert model has a one-to-one correspondence with one of the subsets, but each expert model is not used to evaluate its own corresponding subset, but is used to perform crack scoring processing on images of other subsets.

[0102] Exemplary, reference Figure 4 In this embodiment of the present application, the number of the multiple expert models is set to N, specifically expert model 1, expert model 2 ( Figure 4 ), expert model 3, . . . , expert model N, where expert model 1 has a one-to-one correspondence with subset 1, expert model 2 has a one-to-one correspondence with subset 2, expert model 3 has a one-to-one correspondence with subset 3, and so on, expert model N has a one-to-one correspondence with subset N.

[0103] S3133. Use all target expert models to perform crack scoring processing on the target images in the target subset to obtain a crack scoring result corresponding to each of the target expert models, wherein the target subset is any one of the multiple subsets, the target image is any image in the target subset, and the target expert model is all expert models that do not have a one-to-one correspondence with the target subset.

[0104] Exemplary, reference Figure 4 As shown, all expert models except expert model 2 (i.e., expert model 1, expert model 3, ..., expert model N) are used as the target expert models; subset 2 is used as the target subset; the jth image in subset 2 is used as the target image (i.e., Figure 4 Then, the target image is processed by the expert model 1 to score the cracks, and the crack scoring result S corresponding to the expert model 1 is obtained. 1j , use expert model 3 to perform crack scoring on the target image, and obtain the crack scoring result S corresponding to the expert model 3 3j , and so on, the target image is processed with the expert model N to perform crack scoring, and the crack scoring result S corresponding to the expert model N is obtained. Nj .

[0105] It is understood that the core of the cross-examination process of the embodiment of the present application is to evaluate the target image by using an expert model corresponding to another subset that does not include the target image. In addition, the multiple subsets should not contain repeated images.

[0106] Furthermore, the expert model can be expressed using the following calculation formula:

[0107]

[0108] Among them, Expert i is the i-th expert model; i = 1, 2, 3, ..., N; N is the total number of all target expert models; y j Subset is the true distribution of cracks in the target image j (i.e., the true label corresponding to the target image); i is the i-th subset; L is the fourth loss function; Expert i (y j ) is the crack distribution predicted by the i-th expert model for the target image j.

[0109] Furthermore, the calculation formula of the fourth loss function is:

[0110] L(y j ,Experti (y j ))=(y j -Expert i (y j )) 2 ;

[0111] Among them, y j is the true distribution of cracks in the target image (i.e., the jth image in the target subset), Expert i (y j ) is the crack distribution predicted by the i-th expert model for the target image.

[0112] In other implementations, the calculation formula of the fourth loss function may also be:

[0113] L(y j ,Expert i (y j ))=|y j -Expert i (y j )|;

[0114] Among them, y j is the true distribution of cracks in the target image, Expert i (y j ) is the crack distribution predicted by the i-th expert model for the target image.

[0115] In other implementations, the calculation formula of the fourth loss function may also be other formulas, which may be specifically set by those skilled in the art according to actual needs.

[0116] S3134: Calculate the average of all crack scoring results to obtain a final crack assessment score corresponding to the target image.

[0117] Specifically, the calculation formula of step S3134 is:

[0118]

[0119] Among them, S j is the final crack assessment score corresponding to the target image; j is the jth image in the target subset, i.e., the target image; N is the total number of all target expert models; Expert i (y j ) is the crack distribution predicted by the i-th expert model for the target image, and z is the index of the expert model corresponding to the target subset.

[0120] Furthermore, the scoring indicator used in the embodiment of the present application may be IOU (Intersection over Union) or Precision.

[0121] S3135. Set a preset threshold.

[0122] In this embodiment of the present application, the preset threshold is set to 0.4. Target images below the preset threshold will be identified as mislabeled samples. These samples will be filtered and not used to train the crack semantic segmentation model; target images above the preset threshold are defined as noise-free samples and are retained.

[0123] It should be understood that the preset threshold value cannot be set arbitrarily. Figure 5 As shown, Figure 5 For different preset thresholds (i.e. Figure 5 Histogram of the performance of the crack semantic segmentation model under the filtering threshold in . Figure 5 In the figure, as the preset threshold gradually increases, the amount of noise data removed continues to increase, and the performance of the model continues to improve. When the preset threshold reaches 0.4, that is, the samples whose IOUs mean is less than 0.4 in the expert model output prediction graph, the performance of the crack semantic segmentation model reaches the optimal and tends to be stable, and various indicators reach a peak. When the preset threshold reaches 0.5, the performance of the crack semantic segmentation model not only does not improve, but decreases. Because when the preset threshold exceeds a certain level, the probability of the data cleaning program making mistakes will increase, and the performance of the model will also decrease instead of increase. Therefore, after comprehensive consideration, the embodiment of the present application selects 0.4 as the preset threshold for voting to obtain the crack semantic segmentation model with the best performance and reduce the probability of expert model evaluation errors.

[0124] S3136: When the final crack assessment score is greater than the preset threshold, add the target image to the target training set.

[0125] Specifically, the step S3136 can be expressed as the following formula:

[0126]

[0127] Among them, Image j represents the target image, keep represents adding the target image to the target training set, S j is the final crack assessment score corresponding to the target image, θ represents the preset threshold, and delete represents deleting the target image.

[0128] It should be understood that when the final crack assessment score is less than or equal to the preset threshold, it indicates that the target image is noise data, which may have a degrading effect on the trained crack semantic segmentation model. Therefore, the target image should be deleted to optimize the target training set.

[0129] The S3134, S3135 and S3136 are the voting mechanism of the embodiment of the present application. This voting program traverses all images in the entire initial training set and the verification set, thereby obtaining a high-quality target training set by deleting low-quality noise images.

[0130] It should also be understood that in step S3136, two types of errors may occur. One is that a precisely labeled target image is mistakenly identified as an incorrectly labeled image and the target image is deleted. This is a first-type error (E1); the other is that a wrongly labeled target image is mistakenly identified as a precisely labeled image and the target image is added to the target training set. This is a second-type error (E2). Figure 6 As shown, Figure 6 Two cases of errors in the cross-voting filtering process are given, which reflect the fact that the expert model makes misjudgments due to the influence of interference factors. Figure 6 The middle (ab) lines belong to the first type of error (E1). Figure 6 The middle row (c) belongs to the second type of error (E2). Among them, the rows (ab) are all accurately labeled images. However, due to the relatively large proportion of noise samples in some subsets, the prediction performance of the expert model will have a certain deviation. The expert model is affected by the over-labeled samples and the missing labeled samples, and the output crack prediction map (i.e. Figure 6 The predicted image in the figure) and the actual crack distribution (reference Figure 6 (The provided labels in the image are inconsistent, with an IOU (intersection-over-union) value of only around 0.2. If such an image is ultimately filtered through voting, a Type I error will occur, resulting in the loss of accurate samples. Row (c) shows an overly wide mislabeled sample whose IOU (intersection-over-union) predicted value exceeds the preset threshold and is incorrectly determined to be eligible for inclusion in the target training set.

[0131] In order to avoid the two aforementioned errors, the multi-expert cross-voting scheme provided in the embodiment of the present application is related to the data partitioning. After considering the opinions of other expert models, the dependence of mislabeled data on data partitioning is reduced, thereby reducing the negative impact of mislabeled samples to a certain extent. However, there may still be some expert models with serious performance deviations, and after considering the opinions of other experts, the above two errors cannot be avoided. The potential limitations of this mechanism may include adding some penalty factors or performing multiple cross-examination filters (gradually increasing the preset threshold in each round of cross-voting, and making every effort to protect accurate samples) in the future upgrade process to reduce the probability of errors in the cross-voting process and optimize the performance of the crack semantic segmentation model.

[0132] It should also be understood that the steps S3131, S3132, S3133, S3134, S3135 and S3136 can be understood as Figure 3 The "Multi-expert cross-review to filter out low-quality noisy data" step in the

[15] section.

[0133] S32. Input a first training sample into a crack segmentation network for recognition processing to obtain a first prediction result, wherein the first training sample is any element in the target training set and the validation set, and the first training sample has a one-to-one correspondence with one of the first true values ​​in the first true label set.

[0134] The crack segmentation network is selected from one of U-Net, Unet++, or CrackU-net+. In other embodiments, those skilled in the art may also select other neural networks to replace the above examples, and this application does not impose any further limitations on this. The crack segmentation network is an encoder-decoder structure.

[0135] Further, the structure of U-Net can refer to Figure 7 As shown in Figure 2. For the traditional symmetric encoder-decoder structure, the downsampling layer is used to extract image features and reduce the resolution of the feature map, while the upsampling layer is used to output the final predicted image and restore the feature map to the size of the original image. Figure 7 In the figure, the dotted box on the left contains the downsampling layer, which is the basis of the encoder structure, while the dotted box on the right contains the upsampling layer, which is the main component of the decoder structure. Then, the matching feature maps of each layer of the encoder and decoder structures are fused together through the splicing operation to improve the segmentation performance of the network.

[0136] However, the limitation of U-Net is that features of different depths can only be fed back in a specific layer, and image feature fusion between different layers cannot be achieved. Unlike U-Net, Unet++ focuses on the information fusion of deep and shallow features. The feature maps of each layer are integrated together through skip connections to further obtain more dimensional and refined feature information. However, due to the increase in network parameters, the complexity of convergence in iterative calculations is aggravated to a certain extent, resulting in reduced recognition efficiency. The CrackU-net+ network removes the skip connections between the upsampled feature maps in the middle layer of Unet++, shortens the recognition cycle without reducing the recognition accuracy, and improves the recognition efficiency. The structure of the above segmentation network is as follows: Figure 8 shown.

[0137] Furthermore, to ensure the fairness of the experiments in this application, all experiments were conducted under the same hardware and software conditions. Figure 9 The figure shows the hardware configuration and environment parameters corresponding to the embodiment of the present application. Meanwhile, several hyperparameters set in this application are as follows: the initial learning rate is set to 0.001; Adam (an optimization algorithm) is used as the optimizer, which has an optimization speed that is better than other optimization algorithms; the decay coefficient is set to 0.0001; and the epoch of model training is equal to 1000.

[0138] S33. Calculate a first loss function based on the first prediction result and a first true value corresponding to the first training sample.

[0139] Furthermore, the calculation formula of the first loss function is:

[0140]

[0141] Among them, L BCE is the first loss function, namely the BCE loss function; N is the total number of pixels in the first prediction result; n∈1,2,…,N; y n is the real value of the n-th pixel, that is, the real value of the n-th pixel in the first real value corresponding to the first training sample; is the estimated value, that is, the first prediction result.

[0142] The first loss function is used to evaluate whether each pixel in the first training sample is accurately predicted.

[0143] In this embodiment of the present application, considering that crack segmentation is essentially a pixel-level binary classification task, the binary cross entropy loss function (BCE loss) is selected as the baseline loss function. In other embodiments, those skilled in the art may select other loss functions according to actual needs.

[0144] S34. Adjust first model parameters of the crack segmentation network based on the first loss function to obtain the crack semantic segmentation model.

[0145] In some embodiments, after step S33, the step of training the crack semantic segmentation model further includes:

[0146] S35. Set a second loss function.

[0147] Furthermore, the calculation formula of the second loss function is:

[0148]

[0149] Among them, L Dice is the second loss function, namely the Dice loss function; N is the total number of pixels in the first prediction result; n∈1,2,…,N; p n is the nth pixel in the predicted image, that is, the nth pixel in the first prediction result; q n The label of the nth pixel in the background or crack annotation data is q n When labeling the nth pixel in the background annotation data, q n is 0; in q n When labeling the nth pixel in the crack annotation data, q n is 1.

[0150] The second loss function is set because, in most sample images, crack pixels only account for a small fraction of all pixels. This imbalance in the distribution of positive and negative samples affects network performance. Therefore, the Dice loss function (also known as the second loss function) is used to address this imbalance.

[0151] S36. Calculate a third loss function based on the first loss function and the second loss function.

[0152] Furthermore, the calculation formula of the third loss function is:

[0153] L combine =1-I Dice +λL BCE ;

[0154] Among them, L combine is the third loss function; L Dice is the second loss function; λ is the penalty term coefficient, which is used to penalize the ratio of sample class imbalance; L BCE is the first loss function.

[0155] In the embodiment of the present application, the value of λ is 0.5. In other embodiments, the value of λ can be set by those skilled in the art according to actual needs.

[0156] This third loss function helps avoid the tendency to overemphasize background pixels during training when using only the BCE loss function. By combining the Dice and BCE loss functions, the network focuses on learning about foreground pixels while not completely ignoring their association with background pixels. In addition to addressing the issue of sample imbalance, using this combined loss function (i.e., the third loss function) for training makes the model more robust and tends to converge quickly.

[0157] S37. Adjust second model parameters of the crack semantic segmentation model based on the third loss function.

[0158] In some embodiments, the step of training the crack semantic segmentation model further includes:

[0159] S38. Using an evaluation method to evaluate the crack semantic segmentation model to obtain an evaluation result.

[0160] In the embodiments of the present application, the evaluation method is selected from one of IOU (Intersection over Union), Precision, Recall, and F1-Score. In other embodiments, other evaluation methods may also be used, and the specific method can be determined by those skilled in the art based on actual needs.

[0161] More specifically, the IOU calculation formula is:

[0162]

[0163] Among them, TP represents the number of concrete crack pixels in the image that are accurately predicted as cracks; FN represents the number of crack pixels that are incorrectly predicted as background; FP represents the number of background pixels that are incorrectly predicted as cracks.

[0164] The calculation formula of the Precision evaluation method is:

[0165]

[0166] Among them, TP represents the number of concrete crack pixels in the image that are accurately predicted as cracks; FP represents the number of background pixels that are incorrectly predicted as cracks.

[0167] The calculation formula for Recall is:

[0168]

[0169] Where TP represents the number of concrete crack pixels in the image that are accurately predicted as cracks; FN represents the number of crack pixels that are incorrectly predicted as background.

[0170] The calculation formula of F1-Score is:

[0171]

[0172] Precision is the percentage of crack pixels that are accurately identified, which represents the impact of false detection on the results. Recall is the probability that the model gives a correct crack prediction result, which indicates the impact of missed detection on the results. The overall ability to resist false detection and missed detection is evaluated using the F1-Score.

[0173] In some technical solutions, the embodiment of the present application uses the Unet++ network training expert model as a filter for the target training set (ie, training samples). The first set of images to be processed (ie, sample data) collected for the study comes from different experts on a public platform, and most of the elements in the first set of images to be processed are normally and accurately labeled, and the proportion of incorrectly labeled elements is relatively small. The expert model will learn the prior information of the proportion of samples in the first set of images to be processed, which may result in a class of samples (elements) with a small proportion having a lower weight at the loss function level, and accordingly, the model is more inclined to absorb feature information from a class of samples with a large proportion. In other words, in the cross-voting filtering process, the expert model is more likely to learn accurate features, and tends to give accurate segmentation results and distinguish noisy samples.

[0174] For example, let j be a data in the initial training set, Label j For the original label provided, the expert model's prediction for j is Pred j If Pred j With Label j If the overlap is high, the expert model determines that j is a noise-free label. On the contrary, j will be regarded as incorrectly labeled data. However, the voting method needs to integrate the opinions of multiple expert models, because each expert model may have different prediction results for j. In the embodiment of the present application, the initial training set and the validation set (which can also be expressed as a database) are divided into 4 subsets, and j is cross-reviewed by the remaining 3 expert models, which are trained from the remaining three subsets that do not contain j. Each image in the initial training set and the validation set is traversed and processed by this process. Next, the embodiment of the present application artificially sets a preset threshold (which can be understood as a filtering threshold), and the preset threshold is set to 0.4. When the average predicted IOU of the three expert models is lower than 0.4, j is regarded as mislabeled data and needs to be filtered out.

[0175] Furthermore, the embodiment of the present application combines the original image, the labeled image and the predicted image output by the expert model to provide a visual analysis of the cross-voting filtering process. For details, please refer to Figure 10 The analysis in row (a) shows that when processing precisely labeled data, the predicted positioning of the expert model basically coincides with the true labeling, and the IOU (intersection-over-union) value of these segmented images can reach above 0.8, which further confirms the fact that the filtering mechanism is more friendly to accurately labeled samples. Figure 10 The middle row (b) shows a typical over-wide labeled sample, which is a redundant error, that is, the background is mistakenly marked as the foreground. The segmentation and positioning of the expert model is more suitable for the crack distribution in the original image, so the IOU (intersection-over-union) value of this redundant labeled data is only 0.188. Figure 10 Rows (c) and (d) in the middle show the expert model's prediction results for samples with missing and rough labels, respectively. Row (c) is classified as an omission error, meaning the crack distribution in the original image is not fully labeled, mistakenly treating crack pixels as background. The expert model accurately identifies more detailed areas of crack distribution in the image. Therefore, these noisy samples, which degrade model performance, are distinguished and filtered because their IOU (intersection over union) value is only 0.399, below the preset threshold (also known as the filtering threshold). Row (d) is classified as a precision error, manifested by overly coarse labeling, resulting in confusion between background and foreground labels in some complex areas. Furthermore, the expert model's predictions better fit the original image, but its IOU (intersection over union) value remains above the preset threshold. Even as low as 0.518 in row (d) can prevent these samples from being properly filtered. In this case, it becomes increasingly important to consider the opinions of other expert models.

[0176] After cross-voting filtering, a new database (i.e., the target training set and the validation set) is obtained for training, and the target model (i.e., the crack semantic segmentation model) is further obtained. Figure 3 In the "Performance Change Evaluation" step, the performance of the new model (i.e., the crack semantic segmentation model) and the initial model is evaluated. The performance of the new model will be judged by Figure 11 Further elaborate. Figure 11 Correspondingly, the visualization results of the model performance evaluation on the test set are given. It is worth mentioning that the model evaluated in the embodiment of the present application is a model trained on a database that has been cross-voted filtered and a database that has not been cross-voted filtered. In addition, all images in the test set are accurately labeled, which is a key prerequisite for the usability of model performance evaluation. It should be understood that the database that has not been cross-voted filtered refers to the initial training set and the validation set, and the database that has been cross-voted filtered refers to the target training set and the validation set.

[0177] Specifically, Figure 11 (a) is a single ordinary concrete crack image, and Figure 11 The initial model (i.e., the model trained on the database without cross-voting filtering) is affected by redundant labels, resulting in a wider crack prediction range and an IOU (Intersection-over-Union) of only 0.333. However, after multi-expert cross-evaluation filtering, almost all redundant labels are removed. The target model (i.e., the model trained on the database with cross-voting filtering, i.e., the crack semantic segmentation model) learns more precise crack features and makes more accurate predictions, with an IOU (Intersection-over-Union) value of 0.863. This further demonstrates the superiority of the cross-voting filtering method. Figure 11 The middle row (b) is an example affected by rough annotation, where the initial model performs a very rough segmentation of the cracks in the branch area; filtering the training samples through cross voting greatly improves the segmentation performance of the model; this is most intuitively reflected in the increase in its IOU (i.e., intersection-over-union) value, which rises from 0.330 to 0.857. Figure 11 The image in (c) shows pavement cracks. Some fine cracks in the image are difficult to identify. The initial model does not recognize fine cracks well, which may be affected by the initial noise samples. In contrast, the target model achieves accurate segmentation of cracks of different sizes, and the output prediction map is very consistent with the actual distribution. Figure 11 The middle row (d) shows an example of the model's enhanced noise immunity. Crack detection is affected by scratches and paint marks distributed around the concrete. The initial model mistakenly identifies the paint stain as a crack, with an IOU (Intersection-over-Union) of only 0.583. However, after data cleaning, the model's noise immunity improves, accurately locating the crack, with an IOU (Intersection-over-Union) of 0.790. These experimental results demonstrate that the cross-voting-based data cleaning procedure removes poor-quality samples, improving the model's crack segmentation performance.

[0178] Regarding the effectiveness of the multi-expert cross-voting procedure, please refer to Figure 12 and Figure 13 As shown, Figure 12 The Loss curve of the U-Net network with and without data filtering is shown. Figure 13 The IOU (intersection-over-union) curves of the U-Net network with and without data filtering are shown. Figure 12 In , the model loss value of the target training set after voting filtering is 0.068, and the model loss value of the initial training set without voting filtering is 0.161; the loss value of the validation set corresponding to the target training set after voting filtering is 0.227, and the loss value of the validation set corresponding to the initial training set without voting filtering is 0.426. Figure 13The IOU (intersection-over-union) value of the target training set after voting filtering is 0.921, and the IOU (intersection-over-union) value of the initial training set without voting filtering is 0.821; the IOU (intersection-over-union) value of the validation set corresponding to the target training set after voting filtering is 0.753, and the IOU (intersection-over-union) value of the validation set corresponding to the initial training set without voting filtering is 0.567. Figure 12 and Figure 13 It shows that filtered datasets tend to train models with higher performance.

[0179] The performance data from the validation set and training set are not sufficient to fully measure the performance improvement of the model. The results of the test set cited in the embodiment of this application better reflect the effectiveness of the data cleaning procedure. Figure 14 As shown, Figure 14 The performance metrics of all crack segmentation networks on the test set are shown. After data filtering, the recognition performance of all three crack segmentation networks improved. While the Recall evaluation method (also known as the recall rate evaluation method) remained stable, the IoU evaluation method (also known as the Intersection over Union evaluation method) improved by approximately 4%, the Precision evaluation method (also known as the precision rate evaluation method) improved by 5%, and the F1-Score (also known as the F1 score) improved by approximately 2% to 3%.

[0180] In summary, this application takes concrete crack segmentation and identification as the background, and uses a multi-expert model cross-evaluation module to filter out (or screen) low-quality data samples, and optimizes the segmentation performance of the model by improving the quality of training samples. It can not only avoid the risks and subjectivity of manual inspection, but also provide reference value for building structure safety design and damage assessment.

[0181] like Figure 15 The figure shows a principle block diagram of a concrete crack segmentation device based on data optimization provided in the second aspect of the embodiment of the present application. Figure 15 In the embodiment, the concrete crack segmentation device 100 based on data optimization includes:

[0182] The concrete crack image acquisition module 101 is used to acquire concrete crack images.

[0183] The crack segmentation result obtaining module 102 is used to input the concrete crack image into the crack semantic segmentation model for recognition processing to obtain the crack segmentation result, wherein the crack semantic segmentation model is trained by the validation set and the target training set screened by the multi-expert cross-evaluation voting model.

[0184] In some implementations, the embodiments of the present application construct a high-performance crack segmentation detection platform based on the data optimization-based concrete crack segmentation device 100.

[0185] The third aspect of the embodiment of the present application provides a terminal device, the principle block diagram of the terminal device can be as follows: Figure 16 As shown. The terminal device includes a processor, a memory, a network interface, a display screen and a temperature sensor connected via a system bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a concrete crack segmentation method based on data optimization is implemented. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor is pre-set inside the terminal device to detect the operating temperature of the internal device.

[0186] Those skilled in the art will understand that Figure 16 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal device to which the solution of the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0187] In some embodiments, an embodiment of the present application provides a terminal device, which includes a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the steps of the data optimization-based concrete crack segmentation method provided in the first aspect of the above-mentioned embodiment of the present application.

[0188] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute the steps of the concrete crack segmentation method based on data optimization provided in the first aspect of the embodiment of the present application.

[0189] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0190] The technical features of the above embodiments can be combined without changing the basic principles of this application. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0191] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of patent protection for the present application shall be determined by the appended claims.

Claims

1. A concrete crack segmentation method based on data optimization, characterized in that: include: Acquire concrete crack images; The concrete crack image is input into a crack semantic segmentation model for recognition processing to obtain a crack segmentation result, wherein the crack semantic segmentation model is trained by a validation set and a target training set screened by a multi-expert cross-evaluation voting model.

2. The method for segmenting concrete cracks based on data optimization according to claim 1, characterized in that: The training steps of the crack semantic segmentation model include: Acquire the target training set, the validation set, and a first true label set, wherein the target training set is obtained by screening the multi-expert cross-evaluation voting model; Inputting a first training sample into a crack segmentation network for recognition processing to obtain a first prediction result, wherein the first training sample is any element in the target training set and the validation set, and the first training sample has a one-to-one correspondence with one of the first real values ​​in the first real label set; Calculate a first loss function based on the first prediction result and a first true value corresponding to the first training sample; A first model parameter of the crack segmentation network is adjusted based on the first loss function to obtain the crack semantic segmentation model.

3. The method for segmenting concrete cracks based on data optimization according to claim 2, characterized in that: After the step of calculating a first loss function based on the first prediction result and a first true value corresponding to the first training sample, the method further includes: Set the second loss function; Calculate a third loss function based on the first loss function and the second loss function; A second model parameter of the crack semantic segmentation model is adjusted based on the third loss function.

4. The method for segmenting concrete cracks based on data optimization according to claim 2 or 3, characterized in that: The training step of the crack semantic segmentation model also includes: An evaluation method is used to evaluate the crack semantic segmentation model to obtain an evaluation result.

5. The method for segmenting concrete cracks based on data optimization according to claim 2, characterized in that: The step of obtaining the target training set comprises: Obtaining a first set of images to be processed; Preprocessing the first to-be-processed image set to obtain an initial training set, the verification set, and a test set; The initial training set and the validation set are screened using a multi-expert cross-evaluation voting model to obtain the target training set.

6. The method for segmenting concrete cracks based on data optimization according to claim 5, characterized in that: The step of preprocessing the first to-be-processed image set to obtain an initial training set, the verification set and the test set comprises: Performing size normalization processing on each first image to be processed in the first set of images to be processed to obtain a second set of images to be processed; performing data type adjustment processing on each second image to be processed in the second image set to be processed to obtain a third image set to be processed; The third set of images to be processed is divided to obtain the initial training set, the verification set and the test set.

7. The method for segmenting concrete cracks based on data optimization according to claim 5, characterized in that: The step of using a multi-expert cross-evaluation voting model to screen the initial training set and the validation set to obtain the target training set includes: Dividing the initial training set and the validation set into a plurality of subsets; Multiple expert models are set, each of which has a one-to-one correspondence with one of the subsets. However, each expert model is not used to evaluate its own corresponding subset, but is used to process crack scores on images of other subsets. Using all target expert models to perform crack scoring processing on target images in a target subset, and obtaining a crack scoring result corresponding to each target expert model, wherein the target subset is any one of the multiple subsets, the target image is any one of the images in the target subset, and the target expert models are all expert models that do not have a one-to-one correspondence with the target subset; Calculating the average of all crack scoring results to obtain a final crack assessment score corresponding to the target image; Set preset thresholds; When the final crack assessment score is greater than the preset threshold, the target image is added to the target training set.

8. A concrete crack segmentation device based on data optimization, characterized in that: include: A concrete crack image acquisition module is used to acquire concrete crack images; The crack segmentation result acquisition module is used to input the concrete crack image into the crack semantic segmentation model for recognition processing to obtain the crack segmentation result, wherein the crack semantic segmentation model is trained by the validation set and the target training set screened by the multi-expert cross-evaluation voting model.

9. A terminal device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the steps of the concrete crack segmentation method based on data optimization as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the steps of the concrete crack segmentation method based on data optimization as described in any one of claims 1 to 7.

Images