Ballastless track slab crack measurement method based on multi-scale collaborative deep learning

Abstract

In the embodiment of the present invention, a method for measuring cracks in ballastless track slabs based on multi-scale collaborative deep learning is provided, which belongs to the field of image processing technology, and specifically includes: building a crack measurement framework of multi-scale collaborative deep learning; dividing multiple sample images into a training set and a validation set; adjusting the hyperparameters of the deep target detection network and outputting the optimal crack region extraction result corresponding to each sample image in the training set; performing boundary coordinate cropping on each crack region extraction result to obtain a crack image and input it into a deep semantic segmentation network to adjust the hyperparameters of the deep semantic segmentation network; obtaining a crack measurement model; inputting the collected target image corresponding to the target ballastless track slab into the crack detection model to obtain the continuous width value of the crack in the target image. Through the scheme of the present invention, the features of the three scales of image-pixel-width are collaboratively analyzed and transmitted, which reduces the pixel misjudgment caused by complex background and obtains a refined crack width measurement value.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a ballastless track slab crack measurement method based on multi-scale collaborative deep learning. Background Art

[0002] At present, with the increase in the operation time of high-speed railways, cracks gradually appear on the surface of ballastless track slabs and continue to grow due to the influence of climate, environment, service time, etc. The cracks not only reduce the strength of the concrete structure, but also shorten the service life of the track slabs. When the crack width expands to a certain degree of severity, it will cause the fasteners to loosen and the track to shift, which will lead to the failure of the entire ballastless track structure, seriously threatening the operation safety of the high-speed railway. Accurately measuring the width of cracks on the surface of ballastless track slabs and assessing their severity is an important part of the daily inspection work of high-speed railways, and is also a key basis for maintenance and repair decisions.

[0003] The detection accuracy of crack boundaries directly determines its severity, that is, the reliability of crack width measurement. Traditional measurement schemes based on image processing technology distinguish crack pixels from background pixels by processing shallow features in the inspection image (such as color, grayscale, contour, edge, and frequency). Although they are simple in structure and low in computational cost, the selection of optimal thresholds and seed pixels requires a lot of manual intervention and predefined formula design. If the crack characteristics or detection background change significantly, the calculation parameters need to be adjusted or even the algorithm needs to be redesigned. They have high specificity, low generalization and uncertainty, and are prone to produce blurred or discontinuous crack boundaries (missed detection), resulting in invalid width measurements.

[0004] Although deep learning-based solutions have greatly improved detection accuracy and efficiency compared to traditional image processing technologies, most algorithms are only applicable to monotonous and uniform scenes. Their commonality is that they only detect crack boundaries through pixel-scale analysis, making it difficult to distinguish between crack pixels and background pixels such as noise and contamination that are highly similar to the cracks. The poor adaptability to complex background conditions causes missed detections or false detections, resulting in crack width measurements that are seriously deviated from reality.

[0005] In summary, the refined measurement of crack width is the key basis for characterizing the crack status and making maintenance decisions for high-speed railway ballastless track slabs. However, the existing machine vision-based solutions are only applicable to monotonous and uniform scenes. Under complex background conditions, it is easy to miss detections or make false detections, causing the crack measurement results to deviate from the true value.

[0006] It can be seen that there is an urgent need for a ballastless track slab crack measurement method based on multi-scale collaborative deep learning with high adaptability and measurement accuracy. Summary of the invention

[0007] In view of this, an embodiment of the present invention provides a ballastless track slab crack measurement method based on multi-scale collaborative deep learning, which at least partially solves the problems of poor adaptability and measurement accuracy in the prior art.

[0008] In a first aspect, an embodiment of the present invention provides a ballastless track slab crack measurement method based on multi-scale collaborative deep learning, comprising:

[0009] Build a multi-scale collaborative deep learning crack measurement framework, wherein the crack measurement framework includes a deep object detection network, a deep semantic segmentation network and an improved orthogonal projection algorithm;

[0010] Dividing a plurality of sample images into a training set and a validation set, wherein the sample images are images corresponding to ballastless track slabs containing cracks;

[0011] Using the training set and the validation set to train the deep object detection network, adjusting the hyperparameters of the deep object detection network and outputting the optimal crack region extraction result corresponding to each sample image in the training set;

[0012] Perform boundary coordinate cropping on each of the crack region extraction results to obtain a crack image and input the crack image into the deep semantic segmentation network to adjust the hyper parameters of the deep semantic segmentation network;

[0013] A crack measurement model is obtained according to a deep object detection network with adjusted hyperparameters, a deep semantic segmentation network with adjusted hyperparameters, and the improved orthogonal projection algorithm;

[0014] The collected target image corresponding to the target ballastless track slab is input into the crack detection model to obtain continuous width values ​​of cracks in the target image.

[0015] According to a specific implementation of an embodiment of the present invention, the step of inputting the acquired target image corresponding to the target ballastless track slab into the crack detection model to obtain continuous width values ​​of cracks in the target image includes:

[0016] Inputting the target image into the crack detection model to obtain a crack region extraction result corresponding to the target image;

[0017] Perform boundary coordinate clipping on the crack region extraction result to obtain a clipped image;

[0018] A crack boundary pixel detection result is obtained according to the cropped image, and the continuous width value is calculated according to the improved orthogonal projection algorithm.

[0019] According to a specific implementation of the embodiment of the present invention, the step of obtaining the crack boundary pixel detection result according to the cropped image and calculating the continuous width value according to the improved orthogonal projection algorithm includes:

[0020] Input the crack boundary pixel detection result into the improved orthogonal projection algorithm to extract a single-pixel crack skeleton, wherein the direction of the crack skeleton is the tangent direction at each skeleton point derived from local adjacent skeleton points, and the center is the skeleton point;

[0021] Tracking and matching the crack boundary pixel detection results along the counterclockwise direction and the eight neighborhoods, and converting the binary boundary into a hierarchical sequence by matching all boundary pixels to obtain a complete crack outline;

[0022] An orthogonal projection ray is derived from each of the skeleton point pixels as a normal line of the crack skeleton, and the Euclidean distance between two intersection points of each of the orthogonal projection rays and two contour lines of the crack is used as the continuous width value.

[0023] According to a specific implementation of an embodiment of the present invention, the deep object detection network includes an input end, a feature extraction module and a coordinate prediction module.

[0024] According to a specific implementation manner of an embodiment of the present invention, the deep semantic segmentation network includes an encoder and a decoder.

[0025] According to a specific implementation of an embodiment of the present invention, before the step of training the deep object detection network with the training set and the validation set, adjusting the hyperparameters of the deep object detection network and outputting the optimal crack region extraction result corresponding to each sample image in the training set, the method further includes:

[0026] The cracks in each sample image are marked.

[0027] According to a specific implementation of an embodiment of the present invention, the step of training the deep object detection network with the training set and the validation set, adjusting the hyperparameters of the deep object detection network and outputting the optimal crack region extraction result corresponding to each sample image in the training set includes:

[0028] Inputting all of the training sets into the deep object detection network to obtain a predicted bounding box corresponding to each of the sample images;

[0029] A loss function is calculated based on the predicted bounding box and the annotated bounding box of each of the sample images, and the hyperparameters of the deep object detection network are adjusted according to the loss function until the average accuracy after verification with the validation set reaches a preset condition, the hyperparameters of the deep object detection network are fixed, and the optimal crack area extraction result corresponding to each of the sample images in the training set is output.

[0030] According to a specific implementation of an embodiment of the present invention, the step of performing boundary coordinate cropping on each of the crack region extraction results to obtain a crack image and inputting the image into the deep semantic segmentation network to adjust the hyperparameters of the deep semantic segmentation network includes:

[0031] Inputting all the crack images into the deep semantic segmentation network to obtain predicted pixels corresponding to each sample image;

[0032] The cross entropy loss is calculated according to the predicted pixels and the labeled pixels of each of the sample images, and the hyperparameters of the deep object detection network are adjusted according to the cross entropy loss until the average accuracy after verification with the verification set reaches a preset condition, and the hyperparameters of the deep object detection network are fixed.

[0033] The ballastless track slab crack measurement scheme based on multi-scale collaborative deep learning in the embodiment of the present invention includes: building a crack measurement framework of multi-scale collaborative deep learning, wherein the crack measurement framework includes a deep target detection network, a deep semantic segmentation network and an improved orthogonal projection algorithm; dividing a plurality of sample images into a training set and a validation set, wherein the sample images are images corresponding to ballastless track slabs containing cracks; training the deep target detection network with the training set and the validation set, adjusting the hyperparameters of the deep target detection network and outputting the optimal crack region extraction result corresponding to each sample image in the training set; performing boundary coordinate clipping on each crack region extraction result to obtain a crack image and inputting it into the deep semantic segmentation network to adjust the hyperparameters of the deep semantic segmentation network; obtaining a crack measurement model according to the deep target detection network with adjusted hyperparameters, the deep semantic segmentation network with adjusted hyperparameters and the improved orthogonal projection algorithm; inputting the collected target image corresponding to the target ballastless track slab into the crack detection model to obtain continuous width values ​​of the cracks in the target image.

[0034] The beneficial effects of the embodiments of the present invention are as follows: through the solution of the present invention, the features of the three scales of image-pixel-width are collaboratively analyzed and transmitted, the pixel misjudgment caused by complex background is reduced and a refined crack width measurement value is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic flow chart of a ballastless track slab crack measurement method based on multi-scale collaborative deep learning provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a crack measurement framework of multi-scale collaborative deep learning provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the overall framework of a deep object detection network provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the structure of a dilated convolution module provided in an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a ballastless track slab crack image data collection and annotation process provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of training results of a deep object detection network provided by an embodiment of the present invention;

[0042] Figure 7 A schematic diagram of training results of a deep semantic segmentation network provided in an embodiment of the present invention;

[0043] Figure 8 A schematic diagram of the overall framework of an improved crack width measurement method based on orthogonal projection provided by an embodiment of the present invention;

[0044] Fig. 9 A schematic diagram of crack skeleton extraction provided by an embodiment of the present invention;

[0045] Fig.10 A crack skeleton direction definition diagram provided by an embodiment of the present invention;

[0046] Fig.11 A schematic diagram of crack continuous width calculation based on orthogonal projection provided by an embodiment of the present invention;

[0047] Fig.12 A schematic diagram of continuous crack width measurement values ​​of a partial image provided by an embodiment of the present invention;

[0048] Fig.13A schematic diagram of crack boundary detection results of different deep learning algorithms on a test set provided by an embodiment of the present invention;

[0049] Fig.14 A schematic diagram of crack boundary detection results for four test images using different deep learning algorithms provided in an embodiment of the present invention;

[0050] Fig.15 A schematic diagram comparing crack width measurement results of a multi-scale collaborative algorithm and a traditional orthogonal projection method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0052] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0053] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present invention, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0054] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0055] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0056] An embodiment of the present invention provides a ballastless track slab crack measurement method based on multi-scale collaborative deep learning, which can be applied to the crack data measurement process in railway inspection scenarios.

[0057] See also Figure 1 , is a flow chart of a method for measuring cracks in ballastless track slabs based on multi-scale collaborative deep learning provided by an embodiment of the present invention. Figure 1 As shown, the method mainly comprises the following steps:

[0058] S101, building a multi-scale collaborative deep learning crack measurement framework, wherein the crack measurement framework includes a deep object detection network, a deep semantic segmentation network and an improved orthogonal projection algorithm;

[0059] Optionally, the deep object detection network includes an input end, a feature extraction module and a coordinate prediction module.

[0060] Optionally, the deep semantic segmentation network includes an encoder and a decoder.

[0061] In the specific implementation, considering that the refined measurement of crack width is the key basis for the characterization of crack status and maintenance decision-making of high-speed railway ballastless track slabs, the existing machine vision-based solutions are only applicable to monotonous and uniform scenes. Under complex background conditions, it is easy to miss or misdetect, resulting in crack measurement results deviating from the true value. In this way, a crack measurement framework based on multi-scale collaborative deep learning that integrates three scales of image, pixel and width can be established. Figure 2 As shown in the figure, it includes crack area extraction based on deep object detection network, crack boundary detection based on deep semantic segmentation network and crack width measurement based on improved orthogonal projection method. The input image is analyzed collaboratively with image scale and pixel scale, and features are transferred with width scale to improve adaptability to complex background conditions, forming a refined mapping from input inspection image to output continuous crack width measurement value, so as to achieve accurate characterization of crack severity.

[0062] For example, first, a deep object detection network for crack region extraction is built. The deep object detection network used in the present invention consists of an input end, a feature extraction module, and a coordinate prediction module. It encapsulates region classification and coordinate regression into one network. The confidence of the target in the bounding box and the probability of the target's category are captured directly from the complete image in one evaluation to achieve real-time and efficient end-to-end detection, such as Figure 3 shown.

[0063] Multi-scale crack images are fed into the deep object detection network from the input end and feature extracted in the form of a 3D tensor, where each dimension represents height, width, and channel (red, green, and blue). The input end divides the input image into an S×S grid. If the center of an object falls into a grid cell, then that grid cell is responsible for detecting the object.

[0064] The feature extraction module of the deep object detection network adopts the Darknet-53 structure (including layers 0 to 105). There are 53 independent convolutional layers from layer 0 to layer 74, and the rest are residual units. These convolutional layers use a large number of convolutional kernels (filters) of different sizes, such as 1, 3, and 5, to perform convolution operations with the neighborhood of the input image (the weight parameters are used to perform inner products with the data of the local window of the input image matrix), and then slide from the upper left to the lower right of the image with a certain step size, and output the convolution feature map of the entire input image. Each residual unit contains a 3×3 and a 1×1 convolution kernel, each of which contains three operation layers: convolution operation, BN operation, and Leaky ReLU activation operation. At the end of each residual unit, element-level addition is performed between the input and output vectors. Residual units simplify the training of deep networks and can effectively solve the problem of gradient disappearance. Furthermore, layers 75-105 are feature interaction layers of the YOLOv3 network, which acquire features from three scales (32, 16, and 8). At each scale, convolution kernels (3×3 and 1×1) implement local feature interaction between feature maps, and fully connected layers implement global feature interaction.

[0065] The coordinate prediction module of the deep object detection network constructs a multi-scale feature pyramid with three scale features (32, 16, and 8) using a single resolution input image. Each layer of the pyramid can be used to detect objects of different scales, where lower resolution feature maps with larger strides can produce a very coarse representation of the input image, while higher resolution feature maps with smaller strides have finer-grained features. Further, the multi-scale feature pyramid predicts three prediction boxes for each grid cell on the output feature maps of three different resolutions. Each prediction box consists of a confidence variable, four coordinate variables (bx, by, bh, bw), and a variable representing the class. These prediction variables are converted into object confidence, category probability, and location to generate prediction results for the crack area. The calculation process of the prediction box coordinates is as shown in Formula 1:

[0066]

[0067] Among them, c x and c y Indicates the grid cell coordinates to which the prediction box belongs, p w and p hRepresents the predefined height and weight of the prediction box, t x and t y represents the offset coordinates predicted by the deep object detection network, t w and t h Indicates the scale offset of the network prediction result.

[0068] Then, a deep semantic segmentation network for crack boundary detection is further established. The deep semantic segmentation network used in the present invention is an encoder-decoder architecture for semantic segmentation, such as Figure 3 As shown. In the encoder module (Encoder), the network uses the crack area extraction result output by the deep object detection network as the input of Inception-ResNet-v2 (network backbone) to calculate the deep features of the cracks, and uses the atrous convolution module (Atrous) to control the receptive field without changing the size of the feature map to extract multi-scale semantic information, and then enters the atrous spatial pyramid pooling module (ASPP) to filter and capture the most discriminative crack feature map. Finally, the extracted most discriminative feature map of the entire image is input into the decoder module (Decoder). The decoder module decodes the input feature map through an upsampling operation, and fuses it with the corresponding low-level features from the Inception-ResNet-v2 (network backbone) network and restores it to the spatial dimension consistent with the input image, thereby realizing the refined detection of boundary pixels in the crack area. The specific architecture of the encoder module is as follows:

[0069] The Inception-ResNet-v2 network composed of the Inception-resnet module and the Reduction module is used as the network backbone for crack pixel feature extraction. The Inception module combines convolutional layers in a parallel manner, which increases the width of the network and the nonlinearity of the structure, and reduces the number of parameters of the entire network. Compared with the direct serial combination of traditional models (AlexNet, VGGNet), it can speed up the calculation speed and mine deeper image features. The Resnet module directly connects the input layer and output layer of the Inception-resnet module, directly learns the difference between the input layer and the output layer, and adds the difference to the input layer for output. This can effectively solve the problem of gradient disappearance caused by too many network layers and improve the accuracy of pixel detection. The Reduction module aggregates the effective information in the feature information extracted from the crack image by halving the data space and deepening the channel.

[0070] The Atrous convolution module is a key component of the encoder module. It can control the receptive field without changing the size of the feature map, which is conducive to extracting multi-scale information. Figure 4As shown in Figure 2, the sampling rate controls the size of the receptive field. The larger the value, the larger the receptive field. The deep semantic segmentation network used in this paper further combines the dilated convolution module with the deep separable convolution. The advantage is that the deep separable convolution can greatly reduce the computational complexity while maintaining the same performance.

[0071] The encoder uses the Atrous Spatial Pyramid Pooling module (ASPP) to further extract multi-scale information, which uses dilated convolutions with different sampling rates to achieve this. The ASPP module mainly includes a 1×1 convolution layer, three 3x3 dilated convolutions, and a global average pooling layer to obtain image-level features, which are then sent to a 1x1 convolution layer and bilinearly interpolated to the original size. The features of different scales obtained by the above convolution layers are further combined and then sent to a 1x1 convolution for fusion to obtain the most discriminative crack feature map.

[0072] According to the detection results of crack boundary pixels obtained by the above two networks, an improved crack width measurement method based on orthogonal projection is proposed.

[0073] S102, dividing a plurality of sample images into a training set and a validation set, wherein the sample images are images corresponding to ballastless track slabs containing cracks;

[0074] For example, a high-resolution linear array camera installed on a high-speed integrated track inspection vehicle can be used to scan the surface of a CRTSⅢ ballastless track slab in a high-speed railway section to obtain the target data set required for training. Since the high-speed railway track structure is generally in a natural environment, it is affected by the superposition of natural light and camera light source, as well as the interference of complex environmental conditions such as the vibration and noise of the inspection vehicle itself. These collected original images have complex irregular backgrounds such as noise and contamination. These original images are processed by the following steps and then enter the crack measurement framework of multi-scale collaborative deep learning for analysis and calculation, such as Figure 5 As shown:

[0075] (1) The pixel size of the original inspection image collected is 4096 pixels × 4096 pixels, and the corresponding actual physical area is 235.5 mm × 235.5 mm.

[0076] (2) The images were reset to 400 pixels × 400 pixels, totaling more than 3,000 images.

[0077] (3) 500 images containing cracks are selected as sample data of the present invention. 10% of the 500 images are selected as validation sets each time, and the rest are selected as training sets, thus obtaining 10 sets of <training set, validation set>, and an additional 30 images are selected as test sets.

[0078] S103, training the deep object detection network with the training set and the validation set, adjusting the hyperparameters of the deep object detection network and outputting the optimal crack region extraction result corresponding to each sample image in the training set;

[0079] Optionally, in step S103, before using the training set and the validation set to train the deep object detection network, adjusting the hyperparameters of the deep object detection network and outputting the optimal crack region extraction result corresponding to each sample image in the training set, the method further includes:

[0080] The cracks in each sample image are marked.

[0081] Furthermore, in step S103, the deep object detection network is trained with the training set and the validation set, the hyper parameters of the deep object detection network are adjusted, and the optimal crack region extraction result corresponding to each sample image in the training set is output, including:

[0082] Inputting all of the training sets into the deep object detection network to obtain a predicted bounding box corresponding to each of the sample images;

[0083] A loss function is calculated based on the predicted bounding box and the annotated bounding box of each of the sample images, and the hyperparameters of the deep object detection network are adjusted according to the loss function until the average accuracy after verification with the validation set reaches a preset condition, the hyperparameters of the deep object detection network are fixed, and the optimal crack area extraction result corresponding to each of the sample images in the training set is output.

[0084] For example, labelme can be used to annotate the cracks in 10 sets of <training set, validation set> images (bounding boxes) and save them in .json file format to obtain a reference benchmark for the deep object detection network, and further annotate the crack boundaries within the box pixel by pixel to obtain a reference benchmark for the deep semantic segmentation network.

[0085] Then 10 groups of <training set, validation set> images and corresponding json files with crack area annotations are input into the deep target detection network for full learning and training, and the detection results of the crack target area are output. The present invention uses GPU as the computing core (CPU: AMD2990WX@3.0GHz, RAM=64GB; GPU: NVIDIA GeForce RTX 2080Ti), relying on Facebook's open source deep learning framework PyTorch1.2.0, and conducts hyperparameter adjustment experiments based on 10 groups of <training set, validation set>, and determines the optimal hyperparameter settings based on the training set loss function and performance evaluation indicators of the deep target detection network to output the optimal crack area extraction results. As shown in Table 1:

[0086] Table 1

[0087]

[0088] The loss function used by the deep object detection network is defined as the sum of the squared errors between the predicted bounding box and the labeled bounding box. The predicted bounding box coordinates and confidence values ​​can be corrected according to the numerical value of the loss function, as shown in Formula 2:

[0089]

[0090] Where: Represents the coordinates of the prediction box; (x i ,y i ,w i ,h i ,c i ,p i ) represents the coordinates of the annotation box; Indicates whether the object appears in the bounding box, the value is 1 if it exists, and 0 if it does not.

[0091] In step S103, the average precision (AP) of all categories of detection objects is used to evaluate the performance of the deep object detection network. The calculation steps of MAP are as follows:

[0092] (1) The first step is to calculate the Intersection over Union (IoU). IoU is defined as the degree of overlap between the predicted area and the labeled area. Its mathematical expression is shown in Formula 3.

[0093]

[0094] (2) The second step distinguishes positive samples from negative samples based on the IoU threshold, and sorts the confidence of all predictions in descending order. The IoU threshold is usually predefined (generally set to 0.5). When the IoU between the predicted bounding box and the annotated bounding box is greater than this threshold, the predicted bounding box is defined as a positive sample, otherwise it is a negative sample. In addition, the confidence threshold of the predicted bounding box is also used to distinguish positive predictions from negative predictions. When the IoU of the predicted result is greater than 0.5 and the prediction is correct, it indicates a true positive (TP). When the IoU of the predicted result is less than 0.5 or the prediction is wrong, it indicates a false positive (FP). When there is no IoU with the annotated bounding box, it indicates a false negative (FN), which indicates that the model cannot detect any object labels from the manual annotations.

[0095] (3) Calculate the precision and recall based on TP, FP, and FN. The precision is defined as the ratio of correctly detected targets to the total number of detected targets, and the recall is defined as the ratio of correctly detected targets to the total number of actual targets. The mathematical expressions are shown in formulas (4) and (5). Further calculate the precision and recall corresponding to different confidence thresholds and draw the PR curve. The value of MAP is obtained by integrating the PR curve, which represents the area enclosed by the PR curve and the coordinate axis.

[0096]

[0097]

[0098] The training result of the deep object detection network in step S103 is as follows: Figure 6 As shown in (a) of the figure, after 1000 rounds of sufficient training and learning of 450 images, the training set loss and MAP have reached convergence and stability, and the deep object detection network has reached a fitting state. Further calculation of the area under the PR curve drawn based on 50 validation set images shows that the MAP value of the deep object detection network for crack area extraction is 93.36%, as shown in Figure 6 (b) in .

[0099] S104, performing boundary coordinate cropping on each of the crack region extraction results to obtain a crack image and inputting the crack image into the deep semantic segmentation network to adjust the hyper parameters of the deep semantic segmentation network;

[0100] On the basis of the above embodiment, the step of performing boundary coordinate cropping on each of the crack region extraction results to obtain a crack image and inputting the crack image into the deep semantic segmentation network to adjust the hyper parameters of the deep semantic segmentation network as described in step S104 includes:

[0101] Inputting all the crack images into the deep semantic segmentation network to obtain predicted pixels corresponding to each sample image;

[0102] The cross entropy loss is calculated according to the predicted pixels and the labeled pixels of each of the sample images, and the hyperparameters of the deep object detection network are adjusted according to the cross entropy loss until the average accuracy after verification with the verification set reaches a preset condition, and the hyperparameters of the deep object detection network are fixed.

[0103] In specific implementation, after the boundary coordinates of each crack region extraction result are cropped to obtain the crack image, all the crack images can be used as the input of the deep semantic segmentation network. The encoder module of the deep semantic segmentation network extracts a feature map with rich semantic information from the input crack region, and the extracted feature map is further input into the decoder module and restored to the spatial dimension consistent with the output result of step S103, and the crack boundary pixel detection result at the pixel scale is output. The optimal hyperparameter settings of the deep semantic segmentation network used on the 450 training set images are shown in Table 2, and the cross entropy loss and MIoU distribution are used to evaluate the training effect and crack boundary detection performance of the deep semantic segmentation network.

[0104] Table 2

[0105]

[0106]

[0107] Then, the cross entropy loss can be used to define the error between the output predicted pixels and the labeled pixels to evaluate the training effect of the deep semantic segmentation network. Its mathematical expression is shown in formula (6):

[0108]

[0109] Where: loss is the total loss of DeepLabv3+; y is the label value; is the predicted value.

[0110] At the same time, MIoU can also be used to evaluate the detection performance of deep semantic segmentation networks such as DeepLabv3+. In the present invention, MIoU takes the average value of the IoU of objects of different categories and is used to measure the overlap between the predicted crack pixels and the marked crack pixels belonging to each category. Its mathematical expression is as shown in formula (7):

[0111]

[0112] Where n is the category of the detected object.

[0113] The training result of the deep semantic segmentation network in step S104 is as follows: Figure 7 As shown in Figure 1, (a) indicates that after 500 rounds of sufficient training and learning on 450 images, the training set loss and MIoU have converged and stabilized, and the deep semantic segmentation network has reached a fitting state. Further statistics on the average value of IoU on 50 validation set images show that the MIoU of the DeepLabv3+ network is 82.99%, as shown in Figure 1. Figure 7 as shown in (b).

[0114] S105, obtaining a crack measurement model according to a deep object detection network with adjusted hyperparameters, a deep semantic segmentation network with adjusted hyperparameters, and the improved orthogonal projection algorithm;

[0115] After adjusting the hyperparameters of the deep object detection network and the deep semantic segmentation network, the crack measurement model is obtained by combining the improved orthogonal projection algorithm. Subsequently, the collected crack image can be directly measured by the crack measurement model to output the continuous width value of the crack in the image.

[0116] S106, inputting the collected target image corresponding to the target ballastless track slab into the crack detection model to obtain continuous width values ​​of cracks in the target image.

[0117] On the basis of the above embodiment, as described in step S106, inputting the collected target image corresponding to the target ballastless track slab into the crack detection model to obtain the continuous width value of the crack in the target image includes:

[0118] Inputting the target image into the crack detection model to obtain a crack region extraction result corresponding to the target image;

[0119] Perform boundary coordinate clipping on the crack region extraction result to obtain a clipped image;

[0120] A crack boundary pixel detection result is obtained according to the cropped image, and the continuous width value is calculated according to the improved orthogonal projection algorithm.

[0121] Furthermore, the step of obtaining the crack boundary pixel detection result according to the cropped image and calculating the continuous width value according to the improved orthogonal projection algorithm includes:

[0122] Input the crack boundary pixel detection result into the improved orthogonal projection algorithm to extract a single-pixel crack skeleton, wherein the direction of the crack skeleton is the tangent direction at each skeleton point derived from local adjacent skeleton points, and the center is the skeleton point;

[0123] Tracking and matching the crack boundary pixel detection results along the counterclockwise direction and the eight neighborhoods, and converting the binary boundary into a hierarchical sequence by matching all boundary pixels to obtain a complete crack outline;

[0124] An orthogonal projection ray is derived from each of the skeleton point pixels as a normal line of the crack skeleton, and the Euclidean distance between two intersection points of each of the orthogonal projection rays and two contour lines of the crack is used as the continuous width value.

[0125] In the specific implementation, according to the detection results of crack boundary pixels obtained by the above two networks, an improved crack width measurement method based on orthogonal projection is proposed. The overall framework of the improved crack measurement method based on orthogonal projection is as follows: Figure 8 As shown, the specific steps include:

[0126] ① The first step is to define crack blocks and extract crack contours based on the crack boundary pixel detection results of the deep semantic segmentation network. Crack blocks are the basic unit for crack quantitative analysis and are defined as a group of pixels representing a single crack. Crack blocks can essentially represent the natural contour characteristics of cracks very well. However, since cracks present a non-planar, multi-branched complex extension pattern and the cracks intersect with each other, it is necessary to segment the interconnected cracks into separate crack blocks before performing crack geometric feature analysis. The present invention redefines crack blocks: a group of non-intersecting crack pixels with elongated features. According to this definition, each crack block should have two features: one is that it has two endpoints but no intersections; the other is that it exhibits an elongated pattern layout. The above two points are important features for distinguishing cracks in ballastless track slabs from peeling blocks or other diseases. According to the defined crack blocks, the edge pixels of the crack blocks are tracked and matched along the counterclockwise direction and the eight-neighborhood. By matching all edge pixels, the binary edges are converted into a hierarchical sequence, and then the complete crack contour is obtained.

[0127] ②The second step is to calculate the crack direction based on the crack skeleton information obtained. The crack skeleton is a very important geometric feature to characterize the crack. It is used to locate the basic structure of the crack in each crack block and is also an important part of the intelligent crack analysis. The crack block skeleton proposed by the skeleton extraction algorithm is the central white curve, and the extracted skeleton is pruned, such as Fig. 9 As shown, (a) is the crack block skeleton, and (b) is the image after the skeleton is pruned.

[0128] ③ The third step is to determine the direction of the crack skeleton and calculate the normal of the skeleton. The present invention defines the direction of the crack skeleton as the tangent direction at each skeleton point derived from the local adjacent skeleton points, with the center being the skeleton point, such as Fig.10As shown. It should be noted that the crack directions are different, and the crack skeleton determines its direction based on the entire crack block. There are usually two types of skeleton points in the crack skeleton, one is a non-terminal point and the other is a terminal point. For each non-terminal point, a 5×5 block matrix is ​​constructed on the skeleton graph centered on the skeleton point, and the overlap area between the matrix and the skeleton ( Fig.10 For the endpoint at the end of the skeleton, a 5×5 block matrix is ​​also constructed on the skeleton graph with the endpoint as the center, but an extension block needs to be constructed towards the inner skeleton point at the same time ( Fig.10 The matrix is ​​aligned with each point on the skeleton and moves the same pixel unit parallel and vertically to determine the direction of the crack skeleton. The pixel distance of the matrix movement can be adjusted according to user input. A smaller matrix is ​​needed to determine the change in the direction of the local crack block, so the present invention uses a 5×5 block matrix. The direction of the matrix θ is determined by extracting the connected components in the block matrix and calculating the second-order moment of each pixel matrix. Fig.10 As shown by the elliptical arrow in the figure, since the normal direction of each skeleton point is perpendicular to the orientation direction, the normal direction of each skeleton point is θ+π / 2.

[0129] ④ The fourth step is to extend the skeleton normal to the entire crack contour and then calculate the continuous width of the entire crack. The crack skeleton normal part marked between the two crack contours is defined as the crack width at that point, such as Fig.11 The thinner curve represents the extracted crack skeleton, and the thicker curve represents the orthogonal projection ray derived from each skeleton point pixel. Each orthogonal projection ray has two intersection points with the two contour lines of the crack, namely Fig.11 Therefore, the Euclidean distance between two intersection points can be used to determine the continuous width of a point in the crack.

[0130] Specifically, the Euclidean distance between the two intersection points is defined as the continuous width of a point inside the crack as shown in Formula 8. The continuous width value of the crack in the partial image output in step S5 is as follows: Fig.12 shown.

[0131] Further, the crack severity characterization index is calculated based on the distribution of the continuous crack width output in the above steps, including the minimum width, maximum width, average width (Formula 9), median width (Formula 10) and standard deviation (Formula 11) to comprehensively and systematically characterize the severity of the cracks.

[0132]

[0133]

[0134]

[0135]

[0136] Where: a ij is the continuous width of a point in the crack; x i , x j ∈C∩R; C is the crack contour point set; R is the orthogonal projection ray point set; n is the crack continuous width measurement point set; is the average width of the crack; is the median width of the crack; is the standard deviation.

[0137] The ballastless track slab crack measurement method based on multi-scale collaborative deep learning provided in this embodiment reduces pixel misjudgment caused by complex background and obtains refined crack width measurement values ​​by collaboratively analyzing and transmitting features at three scales: image, pixel, and width.

[0138] The present solution will be described below in conjunction with a specific embodiment. The output results of the crack measurement framework based on multi-scale collaborative deep learning constructed by the present invention are compared with the traditional solution to evaluate the crack measurement performance of the present invention, including the comparison with the deep learning algorithm at a single pixel scale and the comparison with the traditional orthogonal projection method, as follows:

[0139] (1) In the first step, 30 test set images are processed using the crack measurement framework based on multi-scale collaborative deep learning and U-Net and DeepLabv3+ networks built by the present invention to obtain the corresponding crack boundary detection results, and the performance of various solutions is compared from three aspects: MIOU, computational cost and detection results.

[0140] ① MIOU and computational cost of various solutions. Fig.13 The crack boundary detection results of the present invention and the existing two pixel-scale deep semantic segmentation networks for 30 test set images are given, among which the U-Net network has the lowest computational cost (single image processing time), but the MIoU on the test set is only 68.8%, while the MIoU of the DeepLabv3+ network reaches 77.59%, but it also brings higher computational costs. The total computational cost of the multi-scale collaborative algorithm in image-pixel scale collaborative analysis consists of two parts: crack area extraction (yellow area) and crack boundary detection (green area). Compared with the single-pixel-scale DeepLabv3+ network, although the total computational cost consumed by the multi-scale collaborative algorithm is slightly higher (single image processing time is 0.03s longer), its MIoU is improved by nearly 7%, and the image-scale crack area pre-extraction also makes the algorithm's computational cost lower than DeepLabv3+ when detecting crack boundaries at the pixel scale.

[0141] ② Crack boundary detection results of various solutions. Fig.14 The crack boundary detection results of the present invention and the existing deep semantic segmentation networks at two pixel scales for four test images are further compared. The U-Net network has a poor detection effect on crack details and is prone to produce discontinuous crack boundaries. In addition, both U-Net and DeepLabv3+ are prone to misjudge background pixels that are highly similar to cracks, such as noise and stains, as crack pixels ( Fig.13 Compared with U-Net and DeepLabv3+, the multi-scale collaborative algorithm effectively eliminates the interference of noise and contamination outside the target area when extracting the crack area at the image scale. The pixel-scale boundary segmentation of the crack area after coordinate clipping can obtain more refined detection results.

[0142] (2) In the second step, the crack measurement framework based on multi-scale collaborative deep learning constructed by the present invention and the traditional orthogonal projection method are used to process 30 test set images, respectively, to obtain the corresponding crack continuous width measurement results, and the crack severity characterization indicators output by the two solutions are compared to evaluate their detection performance. Fig.15 The calculation results of the quantitative indexes of crack severity of four test images (400 pixels × 400 pixels) were compared between the present invention and the traditional orthogonal projection method, including the minimum width, maximum width, average width, median width and standard deviation. Fig.15 In the figure, the histogram of (a) shows the calculation result of the traditional orthogonal projection method, and the dot-line graph shows the calculation result of the multi-scale collaborative algorithm. The maximum width values ​​of test image 1 and test image 2 calculated by the traditional orthogonal projection method are more than 8 times the calculation results of the multi-scale collaborative algorithm, and are much larger than their average width measurement values, indicating that the traditional orthogonal projection method is prone to misjudge complex backgrounds such as noise and contamination that are highly similar to cracks as crack pixels, causing skeleton deformation, which leads to a large deviation in the maximum width measurement value. Furthermore, the multi-scale collaborative algorithm can calculate the minimum width value of the cracks in the four test images, while the minimum width value of the cracks calculated by the traditional orthogonal projection method is approximately 0. This is because the Canny operator used in the traditional orthogonal projection method in the crack pixel contour extraction stage is prone to produce discontinuous crack boundaries, making it difficult to obtain the true minimum width value of the crack. In addition, the difference between the average width and median width of the cracks in the four test images calculated by the multi-scale collaborative algorithm is less than 0.1 mm, while for the traditional orthogonal projection method, this difference may reach 0.3 mm. Fig.15In (b), the standard deviations of the continuous widths of cracks measured by the two methods were further compared. They measured more than 350 continuous width values ​​along the crack skeleton line on four test images. The standard deviations of the continuous width measurement results of the multi-scale collaborative algorithm were all below 2, while the standard deviation of the traditional orthogonal projection method on test image 2 reached 18, showing that the multi-scale collaborative algorithm has better measurement stability and reliability. The above comparison results show that the multi-scale collaborative algorithm can achieve a more practical and stable crack severity characterization under the interference of complex background conditions such as noise and contamination.

[0143] The comparative analysis results show that the deep learning algorithm based on multi-scale collaboration proposed in the present invention can solve the shortcomings of existing machine vision solutions that are prone to missed detection or false detection under complex background conditions, resulting in measurement results deviating from the true value. It has high crack width measurement accuracy and low computational cost, and has excellent adaptability and robustness to complex backgrounds such as noise and contamination that are similar to the crack height.

[0144] It can be seen that the beneficial effects of the present invention include:

[0145] 1. The present invention proposes a deep learning algorithm based on multi-scale collaboration, which effectively solves the problem that the boundary detection of surface cracks on high-speed railway ballastless track slabs using the existing single-pixel-scale deep semantic segmentation network is easily disturbed by complex environmental conditions, resulting in serious deviations from actual detection results.

[0146] 2. The deep learning algorithm based on multi-scale collaboration proposed in the present invention can remove most of the pixel misjudgments caused by noise and contamination, and shows good adaptability to complex background conditions that are highly similar to cracks.

[0147] 3. The characterization index of the severity of surface cracks on ballastless track slabs calculated by the deep learning algorithm based on multi-scale collaboration proposed in the present invention is closer to the actual situation, and the fluctuation (standard deviation) is much lower than the traditional orthogonal projection method (amplitude up to 90%), showing that the crack width measurement performance of the algorithm has the characteristics of high precision, stability and good reliability.

[0148] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof.

[0149] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A ballastless track slab crack measurement method based on multi-scale collaborative deep learning, characterized in that: include: Build a multi-scale collaborative deep learning crack measurement framework, wherein the crack measurement framework includes a deep object detection network, a deep semantic segmentation network and an improved orthogonal projection algorithm; Dividing a plurality of sample images into a training set and a validation set, wherein the sample images are images corresponding to ballastless track slabs containing cracks; Using the training set and the validation set to train the deep object detection network, adjusting the hyperparameters of the deep object detection network and outputting the optimal crack region extraction result corresponding to each sample image in the training set; Perform boundary coordinate cropping on each of the crack region extraction results to obtain a crack image and input the crack image into the deep semantic segmentation network to adjust the hyper parameters of the deep semantic segmentation network; A crack detection model is obtained according to a deep object detection network with adjusted hyperparameters, a deep semantic segmentation network with adjusted hyperparameters, and the improved orthogonal projection algorithm; Inputting the collected target image corresponding to the target ballastless track slab into the crack detection model to obtain continuous width values ​​of cracks in the target image; The step of inputting the collected target image corresponding to the target ballastless track slab into the crack detection model to obtain continuous width values ​​of cracks in the target image comprises: Inputting the target image into the crack detection model to obtain a crack region extraction result corresponding to the target image; Perform boundary coordinate clipping on the crack region extraction result to obtain a clipped image; Obtaining crack boundary pixel detection results according to the cropped image, and calculating the continuous width value according to the improved orthogonal projection algorithm; The step of obtaining the crack boundary pixel detection result according to the cropped image and calculating the continuous width value according to the improved orthogonal projection algorithm comprises: Input the crack boundary pixel detection result into the improved orthogonal projection algorithm to extract a single-pixel crack skeleton, wherein the direction of the crack skeleton is the tangent direction at each skeleton point derived from local adjacent skeleton points, and the center is the skeleton point; Tracking and matching the crack boundary pixel detection results along the counterclockwise direction and the eight neighborhoods, and converting the binary boundary into a hierarchical sequence by matching all boundary pixels to obtain a complete crack outline; An orthogonal projection ray is derived from each of the skeleton point pixels as a normal line of the crack skeleton, and the Euclidean distance between two intersection points of each of the orthogonal projection rays and two contour lines of the crack is used as the continuous width value.

2. The method according to claim 1, characterized in that ,The deep object detection network includes an input end, a feature extraction module and a coordinate prediction module.

3. The method according to claim 1, characterized in that ,The deep semantic segmentation network includes an encoder and a decoder.

4. The method according to claim 1, characterized in that Before the step of training the deep object detection network with the training set and the validation set, adjusting the hyperparameters of the deep object detection network and outputting the optimal crack region extraction result corresponding to each sample image in the training set, the method further includes: The cracks in each sample image are marked.

5. The method according to claim 4, characterized in that The step of training the deep object detection network with the training set and the validation set, adjusting the hyperparameters of the deep object detection network and outputting the optimal crack region extraction result corresponding to each sample image in the training set comprises: Inputting all of the training sets into the deep object detection network to obtain a predicted bounding box corresponding to each of the sample images; A loss function is calculated based on the predicted bounding box and the annotated bounding box of each of the sample images, and the hyperparameters of the deep object detection network are adjusted according to the loss function until the average accuracy after verification with the validation set reaches a preset condition, the hyperparameters of the deep object detection network are fixed, and the optimal crack area extraction result corresponding to each of the sample images in the training set is output.

6. The method according to claim 4, characterized in that The step of performing boundary coordinate cropping on each crack region extraction result to obtain a crack image and inputting it into the deep semantic segmentation network to adjust the hyper parameters of the deep semantic segmentation network includes: Inputting all the crack images into the deep semantic segmentation network to obtain predicted pixels corresponding to each sample image; The cross entropy loss is calculated according to the predicted pixels and the labeled pixels of each of the sample images, and the hyperparameters of the deep object detection network are adjusted according to the cross entropy loss until the average accuracy after verification with the verification set reaches a preset condition, and the hyperparameters of the deep object detection network are fixed.

Images