Crack online detection method and system based on mobile device and super-resolution segmentation network

Abstract

The application discloses a crack online detection method and system based on a mobile device and a super-resolution reconstruction segmentation network, wherein a high-definition video acquisition instrument on a mobile device is used to scan the surface of a structure to be detected, and the collected video is compressed and then transmitted to a detection workstation online; a feature similarity module of a super-resolution reconstruction segmentation network model in the workstation is used to calculate the similarity between a crack feature image output by a single-image super-resolution module and a crack feature image output by a semantic segmentation super-resolution module, and the similarity is used to guide the semantic segmentation super-resolution module to learn a high-resolution representation; and then a trained single-image super-resolution module and semantic segmentation super-resolution module are used to perform super-resolution reconstruction and semantic segmentation on a crack image to be segmented. Compared with the prior art, the technical scheme can compress a high-definition video to achieve a real-time detection effect, and the crack shape segmented by the super-resolution reconstruction segmentation network model is more accurate.

Description

Technical Field

[0001] This invention relates to the field of intelligent detection of surface cracks in structures, and more particularly to an online crack detection method and system based on mobile devices and super-resolution reconstruction segmentation networks. Background Technology

[0002] Over the past decade, vision-based structural damage detection methods have received significant attention for monitoring civil infrastructure, including bridges, highways, railways, and tunnels. During periodic structural inspections, crack information provides crucial data for assessing the safety and durability of building projects; therefore, accurate crack detection and analysis are essential for proper building maintenance. An autonomous crack detection system can help reduce human intervention, lower costs, and improve the reliability and efficiency of the detection system. Computer vision-based crack detection technology offers advantages such as ease of operation, non-contact nature, and more intuitive interpretation of observation data, and has been widely applied in actual engineering sites. Digital cameras are combined with various types of drones and wall-climbing robots to detect cracks in target infrastructure. Crack detection capability largely depends on the quality and pixel resolution of the digital image. Digital image quality and resolution can vary depending on data acquisition conditions, such as working distance, shooting angle, compression factor, and operational vibration. Digital images acquired from structures using mobile devices may not guarantee accurate detection of minute cracks in terms of quality and resolution. Furthermore, if extremely high resolution is used for shooting, high-resolution video requires significant bandwidth to meet data transmission requirements during simultaneous detection. In reality, for safety reasons, mobile devices should maintain a certain working distance from the target structure. However, the pixel resolution corresponding to this working distance may be insufficient for visualizing microcracks. Therefore, noise, blurring, and insufficient resolution in digital images can lead to a decrease in the automatic crack detection capability, and the crack shape features extracted from digital images may not be accurate enough.

[0003] Therefore, existing automatic crack detection methods suffer from low pixel counts and losses during video compression and transmission, resulting in inaccurate crack shape features extracted from digital images. This has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention provides an online crack detection method and system based on mobile devices and super-resolution reconstruction segmentation networks, which solves the technical problem that existing automatic crack detection methods cannot accurately extract crack shape features from digital images due to low pixel counts.

[0005] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0006] An online crack detection method based on mobile devices and super-resolution reconstruction segmentation networks includes the following steps:

[0007] Acquisition of video data for structural appearance inspection: Install a high-definition video acquisition device and a rangefinder on a mobile device to scan the surface of the structure to be inspected. Compress the surface video of the structure to be inspected obtained by scanning and transmit it online to the inspection workstation.

[0008] A super-resolution reconstruction and segmentation network model is constructed and trained. The super-resolution reconstruction and segmentation network model includes a single-image super-resolution module, a semantic segmentation super-resolution module, and a feature similarity module. The feature similarity module calculates the similarity between the crack feature image output by the single-image super-resolution module and the crack feature image output by the semantic segmentation super-resolution module during training, and uses the similarity to guide the semantic segmentation super-resolution module in learning a high-resolution representation. The single-image super-resolution module is connected to the semantic segmentation super-resolution module, and both the single-image super-resolution module and the semantic segmentation super-resolution module are connected to the feature similarity module. The single-image super-resolution module takes a crack image as input and outputs a super-resolution image of the crack image to the semantic segmentation super-resolution module. The semantic segmentation super-resolution module receives the super-resolution image of the crack image and outputs a super-resolution semantic segmentation map of the super-resolution image.

[0009] The video frame images from the compressed video are obtained from the detection workstation, and the crack image to be segmented is input into the trained super-resolution reconstruction segmentation network model to obtain the super-resolution semantic segmentation map of the crack image to be segmented.

[0010] Preferably, the acquisition of structural appearance inspection video data includes the following steps:

[0011] A high-definition video acquisition device acquires video images of the surface of the structure to be inspected in real time, and a rangefinder records the distance between the acquisition device and the surface of the structure corresponding to each video frame. A computer center is set up, and workstations are configured in the computer center. Using the workstations as carriers, a super-resolution reconstruction and segmentation network model is constructed and trained. The high-definition video is compressed and transmitted to the workstations in real time for inspection.

[0012] Preferably, training a super-resolution reconstruction and segmentation network model includes the following steps:

[0013] Obtain the crack training images, their corresponding standard super-resolution images, and standard semantic segmentation maps from historical data;

[0014] The crack training image and its corresponding standard super-resolution image are input into the single image super-resolution module for training. The first loss of the single image super-resolution module is calculated based on the super-resolution image output by the single image super-resolution module and its corresponding standard super-resolution image.

[0015] The crack training image and its corresponding standard semantic segmentation image are input into the semantic segmentation super-resolution module for training. The second loss of the semantic segmentation super-resolution module is calculated based on the semantic segmentation image output by the semantic segmentation super-resolution module and its corresponding standard semantic segmentation image.

[0016] The feature similarity module calculates the feature similarity between the super-resolution image trained and output by the single-image super-resolution module and the corresponding semantic segmentation image trained and output by the semantic segmentation super-resolution module.

[0017] Then, the total loss of the super-resolution reconstruction segmentation network model is calculated based on the first loss, the second loss, and the feature similarity matrix. The training parameters of the super-resolution reconstruction segmentation network model are adjusted to minimize the total loss, thus obtaining the trained super-resolution reconstruction segmentation network model.

[0018] Preferably, the total loss of the super-resolution reconstruction segmentation network model is calculated based on the first loss, the second loss, and the feature similarity, using the following formula:

[0019] L total =L ce +w1L mse +w2L fa ;

[0020] Among them, L total For the total loss, L ce For the second loss, L mse As the first loss, L fa For feature similarity, w1 is the weight of the first loss and w2 is the weight of the second loss;

[0021] Preferably, the first loss L mse The calculation method is as follows:

[0022]

[0023] Among them, SISR(X) i Y is the i-th super-resolution image output by the single-image super-resolution module during training. i Let be the standard super-resolution image corresponding to the i-th super-resolution image; N is the total number of training samples for a single image super-resolution module.

[0024] The second loss Lce The calculation method is as follows:

[0025] L ce =w3L Bce +w4L Dice ;

[0026] L Bce L is the binary classification cross-entropy loss of the semantic segmentation super-resolution module; Dice w3 and w4 are the Dice loss of the semantic segmentation super-resolution module, and w3 and w4 are the weighting coefficients that balance the importance between the Bce loss and the Dice loss.

[0027] Preferably, the process of obtaining the crack training image, its corresponding standard super-resolution image, and standard semantic segmentation map from historical data includes the following steps:

[0028] Capture images of different types of cracks at different resolutions: For images of the same crack, use the low-resolution image as the training image and the high-resolution image as the corresponding standard super-resolution image. Mark the cracks in the standard super-resolution image at the pixel level, and then binarize the marked standard super-resolution image to obtain the standard semantic segmentation map of the training image.

[0029] Preferably, the super-resolution reconstruction segmentation network model is a U-shaped encoder-decoder network structure, and the backbone network of the U-shaped encoder-decoder network structure adopts ResNet152; the decoder module adopts ESPCN design, and the output size is consistent with the output size of the semantic segmentation super-resolution module; the single image super-resolution module and the semantic segmentation super-resolution module share the feature extraction module and encoder module of the super-resolution reconstruction segmentation network model, and the first loss function is used to optimize the decoder module of the U-shaped encoder-decoder network structure.

[0030] Preferably, the following steps are also included when constructing and training the super-resolution reconstruction and segmentation network model:

[0031] Multiple super-resolution reconstruction and segmentation network models with different hyperparameters were constructed and trained. The trained super-resolution reconstruction and segmentation network models were scored based on four indicators: peak signal-to-noise ratio, structural similarity, F1-score, and intersection-over-union ratio. The super-resolution reconstruction and segmentation network model with the highest score was selected as the optimal model, and the optimal model was used to perform super-resolution reconstruction and semantic segmentation on the crack image to be segmented.

[0032] Preferably, multiple trained super-resolution reconstruction and segmentation network models are evaluated using four metrics: peak signal-to-noise ratio, structural similarity, F1-score, and intersection-over-union ratio (IoU). This is achieved through the following formula:

[0033] S=λ1PSNR+λ2SSIM+λ3(F1-score)+λ4IoU

[0034] Where S represents the score; PSNR represents the peak signal-to-noise ratio of a single image super-resolution module; SSIM represents the structural similarity of a single image super-resolution module; F1-score represents the balanced F-score of the semantic segmentation super-resolution module, which is the harmonic mean of precision and recall; IoU represents the intersection-union ratio of the semantic segmentation super-resolution module, and λ1, λ2, λ3, and λ4 represent the weights of PSNR, SSIM, F1-score, and IoU, respectively.

[0035] Preferably, after obtaining the super-resolution semantic segmentation map of the crack image to be segmented, the method further includes the following steps:

[0036] The mid-axis transformation algorithm is used to process the super-resolution semantic segmentation map of the crack image to be segmented, so as to obtain the skeleton of the crack and the crack feature value in pixels in the crack image to be segmented.

[0037] Based on the image distance recorded in the image acquisition instrument of the crack image to be segmented and the known camera parameters, the true size of the crack feature value is calculated;

[0038] The degree of danger of the crack is assessed based on the application scenario of the structure in which the crack is located and the actual size of the crack's characteristic values.

[0039] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a method.

[0040] The present invention has the following beneficial effects:

[0041] 1. The online crack detection method and system based on mobile devices and super-resolution reconstruction and segmentation networks of this invention involves installing a high-definition video acquisition device and a rangefinder on a mobile device to scan the surface of the structure to be detected. The acquired video is compressed and transmitted online to a detection workstation. A super-resolution reconstruction and segmentation network model is constructed. During training, the similarity between the crack feature image output by the single-image super-resolution module and the crack feature image output by the semantic segmentation super-resolution module is calculated through the feature similarity module of the super-resolution reconstruction and segmentation network model. This similarity is used to guide the semantic segmentation super-resolution module to learn a high-resolution representation. Then, the trained single-image super-resolution module and semantic segmentation super-resolution module are used to perform super-resolution reconstruction and semantic segmentation on the crack image to be segmented, resulting in a super-resolution semantic segmentation map of the crack image to be segmented. Compared with existing technologies, this technical solution can compress high-definition video to achieve real-time detection. The crack shape segmented by its super-resolution reconstruction and segmentation network model is more accurate, achieving higher precision and faster mobile online crack detection.

[0042] 2. In the preferred embodiment, this technical solution constructs and trains multiple super-resolution reconstruction and segmentation network models with different hyperparameters. The trained super-resolution reconstruction and segmentation network models are scored based on four indicators: peak signal-to-noise ratio, structural similarity, F1-score, and intersection-over-union ratio. The super-resolution reconstruction and segmentation network model with the highest score is selected as the optimal model. The optimal model is then used to perform super-resolution reconstruction and semantic segmentation on the crack image to be segmented, which can further improve the accuracy of the crack shape output by the super-resolution reconstruction and segmentation network model.

[0043] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0044] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0045] Figure 1 This is a flowchart of an online crack detection method based on a mobile device and a super-resolution reconstruction segmentation network according to a preferred embodiment of the present invention.

[0046] Figure 2 This is a diagram of the super-resolution reconstruction segmentation network structure in a preferred embodiment of the present invention.

[0047] Figure 3 This is a partial semantic segmentation dataset illustration from a preferred embodiment of the present invention.

[0048] Figure 4 This is a comparison diagram of the semantic segmentation results of super-resolution images at different magnifications in a preferred embodiment of the present invention.

[0049] Figure 5 This is a comparison diagram of the semantic segmentation results of the online crack detection method based on mobile devices and super-resolution reconstruction segmentation network in a preferred embodiment of the present invention with other methods. Detailed Implementation

[0050] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0051] Example 1:

[0052] This embodiment discloses an online crack detection method based on mobile devices and super-resolution reconstruction segmentation networks, including the following steps:

[0053] Acquisition of video data for structural appearance inspection: Install a high-definition video acquisition device and a rangefinder on a mobile device to scan the surface of the structure to be inspected. Compress the surface video of the structure to be inspected obtained by scanning and transmit it online to the inspection workstation.

[0054] A super-resolution reconstruction and segmentation network model is constructed and trained. The super-resolution reconstruction and segmentation network model includes a single-image super-resolution module, a semantic segmentation super-resolution module, and a feature similarity module. The feature similarity module calculates the similarity between the crack feature image output by the single-image super-resolution module and the crack feature image output by the semantic segmentation super-resolution module during training, and uses the similarity to guide the semantic segmentation super-resolution module in learning a high-resolution representation. The single-image super-resolution module is connected to the semantic segmentation super-resolution module, and both the single-image super-resolution module and the semantic segmentation super-resolution module are connected to the feature similarity module. The single-image super-resolution module takes a crack image as input and outputs a super-resolution image of the crack image to the semantic segmentation super-resolution module. The semantic segmentation super-resolution module receives the super-resolution image of the crack image and outputs a super-resolution semantic segmentation map of the super-resolution image.

[0055] The video frame images from the compressed video are obtained from the detection workstation, and the crack image to be segmented is input into the trained super-resolution reconstruction segmentation network model to obtain the super-resolution semantic segmentation map of the crack image to be segmented.

[0056] Furthermore, this embodiment also discloses a computer system including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method.

[0057] By installing a high-definition video acquisition device and a rangefinder on a mobile device, the surface of the structure to be inspected is scanned. The acquired video is compressed and transmitted online to the inspection workstation. A super-resolution reconstruction and segmentation network model is constructed. During training, the feature similarity module of the super-resolution reconstruction and segmentation network model calculates the similarity between the crack feature images output by the single-image super-resolution module and the crack feature images output by the semantic segmentation super-resolution module. This similarity is used to guide the semantic segmentation super-resolution module to learn a high-resolution representation. Then, the trained single-image super-resolution module and semantic segmentation super-resolution module are used to perform super-resolution reconstruction and semantic segmentation on the crack image to be segmented, resulting in a super-resolution semantic segmentation map of the crack image to be segmented. Compared with existing technologies, this solution can compress high-definition video to achieve real-time detection. The crack shape segmented by its super-resolution reconstruction and segmentation network model is more accurate, achieving higher precision and faster mobile online crack detection.

[0058] Example 2:

[0059] Example 2 is a preferred embodiment of Example 1, differing from Example 1 in that the specific steps of the online crack detection method based on mobile devices and super-resolution reconstruction segmentation networks have been optimized:

[0060] In this embodiment, as Figure 1 As shown, an online crack detection method based on mobile devices and a super-resolution reconstruction segmentation network is disclosed. The method uses a high-definition video acquisition device to acquire video images of the surface of the structure to be detected in real time, and a rangefinder records the distance between the acquisition device and the surface of the structure corresponding to each video frame image. A computer center is built, and workstations are configured in the computer center. Using the workstations as carriers, a super-resolution reconstruction segmentation network model is constructed and trained. The high-definition video is compressed and transmitted to the workstations in real time for detection.

[0061] Specifically, the following steps are included:

[0062] Step 1: Constructing a dataset for super-resolution reconstruction

[0063] A high-definition video acquisition device is used to acquire video images of the surface of the structure to be inspected in real time as raw sample data. A rangefinder records the distance between the acquisition device and the surface of the structure corresponding to each video frame. A computer center is built and workstations are configured in the computer center. The high-definition video is compressed and transmitted to the workstations in real time for inspection. Data augmentation is performed on the acquired raw sample data, and a one-to-one corresponding high and low resolution dataset is established for training, verification and testing.

[0064] A dataset for super-resolution reconstruction was constructed, containing images of the same crack at the same location at different resolutions. Therefore, images were acquired by fixing the position of a digital camera and adjusting its resolution, using three resolutions: 320×240, 640×480, and 1280×960. The 320×240 crack images were used as training images in the low-resolution dataset, the 640×480 crack images as standard super-resolution images in the 2x high-resolution dataset, and the 1280×960 crack images as standard super-resolution images in the 4x high-resolution dataset. A total of 500 cracks were captured during the acquisition process.

[0065] A dataset for semantic segmentation was constructed, containing semantically segmented images of cracks from 2x and 4x higher resolution datasets. The cracks in the images were pixel-level labeled using the labeling tool Lalabelme. The labeled images were then binarized, resulting in crack locations labeled in white with a pixel value of 255, while other background areas were labeled in black with a pixel value of 0, yielding a standard semantic segmentation map. The constructed dataset is shown below. Figure 3 As shown.

[0066] When creating the deep learning dataset, 10% of the total collected data was selected as the test set, and 80% of the remaining data was used as the training set and 20% as the validation set. The test set did not require data augmentation. Data augmentation of the remaining data primarily involved various geometric operations such as flipping, rotating, cropping, deforming, and scaling.

[0067] Step 2: Using the workstation as a platform, construct and train a super-resolution reconstruction and segmentation network model.

[0068] (1) Constructing a super-resolution reconstruction segmentation network model:

[0069] like Figure 2As shown, the super-resolution reconstruction and segmentation network model is a U-shaped encoder-decoder network structure, including a feature extraction module, an encoder module, a single-image super-resolution module (SISR), a semantic segmentation super-resolution module (SSSR), a decoder module, and a feature similarity module (FA). The feature extraction module is connected to the encoder module, which is also connected to the single-image super-resolution module and the semantic segmentation super-resolution module. The single-image super-resolution module is also connected to the semantic segmentation super-resolution module, which is further connected to the decoder module. The feature similarity module is also connected to both the single-image super-resolution module and the semantic segmentation super-resolution module. The backbone network of the U-shaped encoder-decoder network structure uses ResNet152. The decoder module uses an ESPCN design, and its output size is the same as that of the semantic segmentation super-resolution module. The single-image super-resolution module and the semantic segmentation super-resolution module share the feature extraction module and the encoder module of the super-resolution reconstruction and segmentation network model.

[0070] The semantic segmentation super-resolution module adds an extra upsampling layer at the end of the traditional semantic segmentation model, resulting in a semantic segmentation map that is several times larger than the original image. For example, an input image of size 512×1024 will output a 1024×2048 image, twice the size of the input image. Other semantic segmentation methods typically use 512×1024 images for training and testing, then enlarge them to 1024×2048 in post-processing. However, our method fully utilizes ground truth, avoiding the loss of effective label information caused by preprocessing.

[0071] The decoder module alone is insufficient to recover a similar high-resolution semantic feature representation obtained using the original image as input. Since the decoder is a simple bilinear upsampling layer or sub-network, it doesn't contain any additional information due to the low resolution of the input image. The single-image super-resolution module aims to reconstruct a high-resolution image from a low-resolution input, effectively reconstructing fine-grained structural information, which is always helpful for semantic segmentation. High-resolution features extracted from the single-image super-resolution module guide the learning of the high-resolution representation in the semantic segmentation super-resolution module. These details can be modeled through correlations or relationships between internal pixels; relationship learning compensates for the simple design of the decoder. Furthermore, it only assists in training; the entire module is automatically removed during inference. The encoder part of this module is shared with the semantic segmentation super-resolution module, while the decoder part adopts an ESPCN design. The final output has the same size as the output of the semantic segmentation super-resolution module.

[0072] Since the single-image super-resolution module contains more complete structural information than the semantic segmentation super-resolution module, feature similarity learning is introduced to guide the semantic segmentation super-resolution module in learning high-resolution representations. The feature similarity module aims to learn the distance between the similarity matrices of the single-image super-resolution module and the semantic segmentation super-resolution module, where the similarity matrix primarily describes the pairwise relationships between pixels.

[0073] The concept of super-resolution is integrated into existing semantic segmentation to formulate a semantic segmentation super-resolution module. Then, a fine-grained structural representation of the single-image super-resolution module is performed using a feature similarity module, further enhancing the high-resolution capability of the semantic segmentation super-resolution module. Furthermore, both parts share the same feature extractor, and the single-image super-resolution module is optimized using reconstruction supervision during training, allowing it to be freely removed from the network during inference, thus saving overhead.

[0074] (2) The training steps for the super-resolution reconstruction segmentation network model are as follows:

[0075] The crack training image and its corresponding standard super-resolution image are input into the single image super-resolution module for training. The first loss of the single image super-resolution module is calculated based on the super-resolution image output by the single image super-resolution module and its corresponding standard super-resolution image.

[0076] The crack training image and its corresponding standard semantic segmentation image are input into the semantic segmentation super-resolution module for training. The second loss of the semantic segmentation super-resolution module is calculated based on the semantic segmentation image output by the semantic segmentation super-resolution module and its corresponding standard semantic segmentation image.

[0077] The feature similarity module calculates the feature similarity between the super-resolution image trained and output by the single-image super-resolution module and the corresponding semantic segmentation image trained and output by the semantic segmentation super-resolution module.

[0078] Then, the total loss of the super-resolution reconstruction segmentation network model is calculated based on the first loss, the second loss, and the feature similarity matrix. The training parameters of the super-resolution reconstruction segmentation network model are adjusted to minimize the total loss, thus obtaining the trained super-resolution reconstruction segmentation network model.

[0079] During training, a dataset of a certain multiple is selected. After inputting low-resolution crack images into the network, the super-resolution reconstruction branch outputs super-resolution crack images. The error between the super-resolution images and the high-resolution crack images at the corresponding multiple is calculated using a loss function. The network parameters of the decoder branch (such as the specific kernel parameters in the convolution kernel) are optimized based on the error. The semantic segmentation super-resolution module outputs high-resolution semantic segmentation maps at the corresponding multiple. The error between the semantic segmentation maps and the crack labels is calculated using a loss function. The network parameters are then optimized and trained based on the error.

[0080] Specifically, the formula for calculating the total loss of the super-resolution reconstruction segmentation network model is as follows:

[0081] L total =L ce +w1L mse +w2L fa ;

[0082] Among them, L total For the total loss, L ce For the second loss, L mse As the first loss, L fa For feature similarity, w1 is the weight of the first loss and w2 is the weight of the second loss;

[0083] Specifically, the first loss L mse The calculation method is as follows:

[0084]

[0085] Among them, SISR(X) i Y is the i-th super-resolution image output by the single-image super-resolution module during training. i Let be the standard super-resolution image corresponding to the i-th super-resolution image; N is the total number of training samples for a single image super-resolution module.

[0086] The second loss L ce The calculation method is as follows:

[0087] L ce =w3L Bce +w4L Dice ;

[0088] L Bce L is the binary classification cross-entropy loss of the semantic segmentation super-resolution module; Dice w3 and w4 are the Dice loss of the semantic segmentation super-resolution module, and w3 and w4 are the weighting coefficients that balance the importance between the Bce loss and the Dice loss.

[0089] In the semantic segmentation branch of the network, a combined loss function combining cross-entropy loss and Dice loss is designed to address the issue of imbalanced pixel samples during the training of the crack segmentation network. Combining the Bce and Dice loss functions is well-suited for crack segmentation models. The combined loss function focuses on both pixel-level and image-level saliency, resulting in stable model training and effectively handling the problem of imbalanced positive and negative samples in crack pixels.

[0090] Step 3: Select the optimal hyperparameters for the super-resolution reconstruction segmentation network model to obtain the optimal model.

[0091] To select the optimal combination of hyperparameters (learning rate, batch size, input size, optimizer, kernel size and number, etc.) and the best model under this combination, model validation is performed on a validation set after training. Multiple super-resolution reconstruction and segmentation network models with different hyperparameter combinations are scored, and the model with the best performance score is selected as the optimal model. The scoring evaluates model performance based on super-resolution reconstruction results of crack images and semantic segmentation accuracy. The evaluation metrics for super-resolution reconstruction results are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), while the evaluation metrics for semantic segmentation accuracy are F1-score and Intersection over Union (IoU).

[0092] PSNR provides an objective metric for measuring the similarity between two images based on the differences between their pixels. A higher value indicates less distortion in the image. SSIM is an image quality evaluation standard that is more in line with human vision. It assesses the similarity between two images by combining the brightness, contrast, and structure of the images. The closer its value is to 1, the better the quality of the generated image.

[0093] The formula for calculating PSNR is as follows:

[0094] The SSIM calculation formula is as follows:

[0095] Where W, H, and C represent the width, length, and number of channels of the image, respectively; X represents the SR image; and Y represents the original image. MAX I It is the maximum possible pixel value in the image. If each sample point has 8 bits, MAX I =255. μ x μ y Let σ represent the mean of image X and Y, respectively. x σ y Let σ represent the variances of images X and Y, respectively. xyThe covariance of images X and Y is represented by C1 and C2, which are constants used to avoid the denominator being zero.

[0096] The formula for calculating the F1 score is as follows:

[0097] The formula for calculating IoU is as follows:

[0098] Wherein, True positive (TP) is the number of pixels that correctly detected cracks, False positive (FP) is the number of pixels that were falsely detected as cracks in non-crack areas, False negative (FN) is the number of pixels that did not correctly detect cracks, and N is the number of samples in the test set.

[0099] To more objectively evaluate model performance using four evaluation metrics, a weighted average method is used to calculate the model performance score. The calculation formula is as follows:

[0100] S=λ1PSNR+λ2SSIM+λ3(F1-score)+λ4IoU

[0101] Wherein, λ1, λ2, λ3, and λ4 represent the weights of PSNR, SSIM, F1-score, and IoU, respectively, and the optimal model is finally selected based on the highest score.

[0102] Step 4: Use drones or wall-climbing robots to collect images of cracks on the structural surface, and use the trained optimal model to perform super-resolution reconstruction and semantic segmentation on the newly collected crack images.

[0103] Drones or wall-climbing robots are used to collect images of cracks on the structural surface. Simultaneously, the distance from the imaging point of the acquisition instrument to the surface of the structure is measured using LiDAR. A reconstruction magnification is selected, and the optimal training model at that magnification is chosen. The acquired images are preprocessed and then input into the optimal training model to obtain a super-resolution reconstructed image of the newly acquired crack image and a semantic segmentation image at the corresponding magnification resolution.

[0104] like Figures 4-5 As shown, the crack map segmented by the optimal model in this technical solution can accurately perform semantic segmentation of microcracks in low-resolution and blurry crack images.

[0105] Step 5: Based on the image distance obtained from the image acquisition instrument, extract the skeleton from the semantic segmentation results obtained above and quantify its feature values ​​(length, width, etc.). Finally, assess the hazard level of the crack feature values ​​obtained above according to the relevant specifications.

[0106] The skeleton of cracks in an image is extracted by improving the Median Transformation (MAT) algorithm. The extracted information includes a set of points and the minimum distance l corresponding to the set of points. d MAT is a commonly used target skeleton extraction technique. Based on the extracted skeleton and contour information, morphological features such as crack length, width and area can be obtained in pixels.

[0107] Crack skeleton extraction yields a single-pixel-wide crack skeleton, where the length of the crack skeleton is the same as the original crack length. Due to the complex and irregular shape of the crack, the extracted crack skeleton line is not a simple straight line; therefore, a piecewise summation method is used to calculate the crack length. Adaptive segmentation of the crack skeleton divides each curve into a series of crack segments, the length of which can be defined as the Euclidean distance between its two endpoints. By summing the lengths of all crack segments, the total length of the entire crack curve can be obtained. The definition is as follows: Where n represents the number of segments in the crack skeleton, (x i1 y i1 ), (x i2 y i2 ) represent the starting coordinates and ending coordinates of the i-th crack segment, respectively.

[0108] The extracted information includes the minimum distance l from each point on the skeleton to the boundary point. d The formula for calculating the maximum crack width, max_width, is: max_width = 2 × Max(l) d ).

[0109] The area of ​​the crack is determined by the number of crack pixels in the segmentation image. The formula for calculating the average width is as follows:

[0110] A three-point laser rangefinder is installed along the camera's imaging axis. The laser rangefinder measures the object distance L, i.e., the distance between the rangefinder and the target, synchronously with the camera shutter. Based on the lens imaging principle:

[0111] In the formula, L' is the image distance; f is the lens focal length. Let A' be the actual size of the target, i.e., the actual physical width of the crack; A' is the imaging size, then we have Thus obtain Imaging size A' is In the formula, A” represents the number of pixels in the image; d represents the physical size of the long side of the image sensor; D represents the number of pixels on the long side of the image sensor, and the pixel resolution is: It represents the actual physical size represented by a unit pixel and serves as a conversion factor between the actual physical size and the number of pixels. By performing a series of processing steps on the digital image, the number of pixels occupied by the target object in the entire image is obtained, from which the actual physical size of the target object (crack) can be calculated.

[0112] Based on the image distance recorded by the image acquisition instrument and the known camera parameters, the true size of its characteristic value is calculated, and finally, the degree of danger of the crack characteristic value obtained above is assessed according to the relevant specifications.

[0113] In summary, the method provided by this invention can perform semantic segmentation, feature quantization, and evaluation of microcracks in low-resolution and blurry crack images. It has the advantages of being safe and effective, easy to operate, more accurate, and more intelligent, providing effective reference for structural maintenance managers in decision-making.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A crack online detection method based on mobile devices and super-resolution reconstruction segmentation networks, characterized in that, Includes the following steps: Acquisition of video data for structural appearance inspection: Install a high-definition video acquisition device and a rangefinder on a mobile device to scan the surface of the structure to be inspected. Compress the surface video of the structure to be inspected obtained by scanning and transmit it online to the inspection workstation. A super-resolution reconstruction and segmentation network model is constructed and trained. The super-resolution reconstruction and segmentation network model includes a single-image super-resolution module, a semantic segmentation super-resolution module, and a feature similarity module. The feature similarity module calculates the similarity between the crack feature image output by the single-image super-resolution module and the crack feature image output by the semantic segmentation super-resolution module during training, and uses the similarity to guide the semantic segmentation super-resolution module in learning a high-resolution representation. The single-image super-resolution module is connected to the semantic segmentation super-resolution module, and both the single-image super-resolution module and the semantic segmentation super-resolution module are connected to the feature similarity module. The single-image super-resolution module takes a crack image as input and outputs a super-resolution image of the crack image to the semantic segmentation super-resolution module. The semantic segmentation super-resolution module receives the super-resolution image of the crack image and outputs a super-resolution semantic segmentation map of the super-resolution image. The video frame images in the compressed video are obtained from the detection workstation, and the crack image to be segmented is input into the trained super-resolution reconstruction segmentation network model to obtain the super-resolution semantic segmentation map of the crack image to be segmented. Training a super-resolution reconstruction and segmentation network model includes the following steps: Obtain the crack training images, their corresponding standard super-resolution images, and standard semantic segmentation maps from historical data; The crack training image and its corresponding standard super-resolution image are input into the single image super-resolution module for training. The first loss of the single image super-resolution module is calculated based on the super-resolution image output by the single image super-resolution module and its corresponding standard super-resolution image. The crack training image and its corresponding standard semantic segmentation image are input into the semantic segmentation super-resolution module for training. The second loss of the semantic segmentation super-resolution module is calculated based on the semantic segmentation image output by the semantic segmentation super-resolution module and its corresponding standard semantic segmentation image. The feature similarity module calculates the feature similarity between the super-resolution image trained and output by the single-image super-resolution module and the corresponding semantic segmentation image trained and output by the semantic segmentation super-resolution module. Then, the total loss of the super-resolution reconstruction segmentation network model is calculated based on the first loss, the second loss, and the feature similarity matrix. The training parameters of the super-resolution reconstruction segmentation network model are adjusted to minimize the total loss, thus obtaining the trained super-resolution reconstruction segmentation network model.

2. The online crack detection method based on mobile device and super-resolution reconstruction segmentation network according to claim 1, characterized in that, The acquisition of structural appearance inspection video data includes the following steps: A high-definition video acquisition device acquires video images of the surface of the structure to be inspected in real time, and a rangefinder records the distance between the acquisition device and the surface of the structure corresponding to each video frame. A computer center is set up, and workstations are configured in the computer center. Using the workstations as carriers, a super-resolution reconstruction and segmentation network model is constructed and trained. The high-definition video is compressed and transmitted to the workstations in real time for inspection.

3. The online crack detection method based on mobile device and super-resolution reconstruction segmentation network according to claim 1, characterized in that, The total loss of the super-resolution reconstruction segmentation network model is calculated based on the first loss, the second loss, and the feature similarity, using the following formula: ； in, For the total loss, This is the second loss. The first loss, For feature similarity, The weight of the first loss, This is the weight for the second loss.

4. The online crack detection method based on mobile device and super-resolution reconstruction segmentation network according to claim 3, characterized in that, The first loss The calculation method is as follows: ； in, This refers to the i-th super-resolution image output by the single-image super-resolution module during training. The standard super-resolution image corresponding to the i-th super-resolution image; This represents the total number of training samples for a single image super-resolution module; Second loss The calculation method is as follows: ； L Bce The binary classification cross-entropy loss of the semantic segmentation super-resolution module; L Dice The Dice loss of the semantic segmentation super-resolution module is given. and Weighting coefficients to balance the importance of Bce loss and Dice loss.

5. The online crack detection method based on mobile device and super-resolution reconstruction segmentation network according to claim 4, characterized in that, Obtain the crack training image, its corresponding standard super-resolution image, and standard semantic segmentation map from historical data, including the following steps: Capture images of different types of cracks at different resolutions: For images of the same crack, use the low-resolution image as the training image and the high-resolution image as the corresponding standard super-resolution image. Mark the cracks in the standard super-resolution image at the pixel level, and then binarize the marked standard super-resolution image to obtain the standard semantic segmentation map of the training image.

6. The online crack detection method based on mobile device and super-resolution reconstruction segmentation network according to claim 4, characterized in that, The super-resolution reconstruction segmentation network model is a U-shaped encoder-decoder network structure, with the backbone network of the U-shaped encoder-decoder network structure using ResNet152; the decoder module adopts the ESPCN design, and its output size is consistent with the output size of the semantic segmentation super-resolution module; the single-image super-resolution module and the semantic segmentation super-resolution module share the feature extraction module and encoder module of the super-resolution reconstruction segmentation network model, and the first loss function is used to optimize the decoder module of the U-shaped encoder-decoder network structure.

7. The online crack detection method based on mobile device and super-resolution reconstruction segmentation network according to claim 6, characterized in that, The following steps are also included when building and training the super-resolution reconstruction and segmentation network model: Multiple super-resolution reconstruction and segmentation network models with different hyperparameters were constructed and trained. The trained super-resolution reconstruction and segmentation network models were scored based on four indicators: peak signal-to-noise ratio, structural similarity, F1-score, and intersection-over-union ratio. The super-resolution reconstruction and segmentation network model with the highest score was selected as the optimal model, and the optimal model was used to perform super-resolution reconstruction and semantic segmentation on the crack image to be segmented.

8. The online crack detection method based on mobile device and super-resolution reconstruction segmentation network according to claim 7, characterized in that, The trained super-resolution reconstruction and segmentation network models are evaluated using four metrics: peak signal-to-noise ratio, structural similarity, F1 score, and intersection-over-union ratio (IoU). This evaluation is achieved through the following formula: in, The score is represented by: PSNR (Peak Signal-to-Noise Ratio) for a single-image super-resolution module; SSIM (Structural Similarity Score) for a single-image super-resolution module; F1-score (Harmonic Mean of Precision and Recall) for a semantic segmentation super-resolution module; and IoU (Intersection over Union) for a semantic segmentation super-resolution module. , , , These represent the weights of PSNR, SSIM, F1-score, and IoU, respectively.

9. The online crack detection method based on mobile device and super-resolution reconstruction segmentation network according to claim 1, characterized in that, After obtaining the super-resolution semantic segmentation map of the crack image to be segmented, the following steps are also included: The mid-axis transformation algorithm is used to process the super-resolution semantic segmentation map of the crack image to be segmented, so as to obtain the skeleton of the crack and the crack feature value in pixels in the crack image to be segmented. Based on the image distance recorded in the image acquisition instrument of the crack image to be segmented and the known camera parameters, the true size of the crack feature value is calculated; The degree of danger of the crack is assessed based on the application scenario of the structure in which the crack is located and the actual size of the crack's characteristic values.

10. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the steps of any one of claims 1 to 9 when executing a computer program.

Images