Submarine tunnel pavement crack extraction method and system

Abstract

The invention discloses a subsea tunnel pavement crack extraction method and system, and relates to the technical field of image recognition processing, and the method comprises the steps: obtaining a to-be-segmented real crack image; inputting a real crack image to be segmented into the CycleGAN network pre-training model to obtain a migration image; the CycleGAN network pre-training model is a trained model obtained by training based on a virtual crack image data set and a real crack image data set, and comprises two pairs of generators and discriminators for alternate adversarial training. Obtaining a migration image similar to crack characteristics in the virtual crack image; and inputting the migration image into the semantic segmentation pre-training model, and extracting to obtain a corresponding crack region in the real crack image to be segmented. According to the invention, the inter-domain difference between the virtual crack image data set and the real crack image data set can be reduced, and the segmentation precision of the model on the real crack image is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image recognition and processing, and in particular to a method and system for extracting cracks from a submarine tunnel pavement. Background Art

[0002] Pavement cracks in undersea tunnels are one of the common diseases in road structures. Accurate detection of pavement cracks in undersea tunnels is crucial for the maintenance and safety management of undersea tunnels. With the rapid development of computer vision and deep learning technologies, semantic segmentation models have been widely used in pavement crack detection scenarios. However, existing technologies still face many challenges in practical applications, especially in data set acquisition and processing. Traditional real crack image datasets are mainly obtained by manually annotating real-world pavement images. However, this method's annotation process is time-consuming and labor-intensive, with high production costs and long production cycles. In addition, errors may occur during the manual annotation process, which directly affects the training effect of the model and the accuracy of its detection results.

[0003] In order to solve the above problems of real crack image datasets, researchers began to explore the use of computer graphics technology to generate virtual crack image datasets. Virtual crack image datasets have obvious advantages such as low production cost, short cycle, and accurate annotation. However, due to the domain differences between virtual crack images and real crack images in terms of illumination, texture, noise, etc., the model directly trained on the virtual crack image dataset is difficult to adapt to real application scenarios, which often results in low segmentation accuracy of the model on real crack images. In addition, although computer graphics technology can generate high-quality virtual crack images, the crack features and morphology in these virtual crack images cannot fully cover the diversity of real crack images, which will also reduce the segmentation accuracy of the model on real crack images.

[0004] Based on this, how to reduce the inter-domain difference between the virtual crack image dataset and the real crack image dataset and improve the segmentation accuracy of the model for real crack images has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0005] The purpose of this application is to provide a method and system for extracting cracks from an undersea tunnel pavement, which can reduce the inter-domain difference between a virtual crack image dataset and a real crack image dataset, and improve the segmentation accuracy of the model for real crack images.

[0006] To achieve the above objectives, this application provides the following solutions.

[0007] In a first aspect, the present application provides a method for extracting cracks from an undersea tunnel pavement, the method comprising the following steps.

[0008] Acquire a real crack image to be segmented; the real crack image to be segmented is a real crack image to be segmented and processed obtained by collecting cracks on the pavement of an undersea tunnel.

[0009] The real crack image to be segmented is input into the CycleGAN network pre-training model to obtain a migration image; the CycleGAN network pre-training model is a trained model obtained by training based on a virtual crack image dataset and a real crack image dataset, and the CycleGAN network pre-training model is used to perform style migration and image conversion on the real crack image to obtain a migration image similar to the crack features in the virtual crack image, and the crack features include morphological features, position features, curvature features and texture features of the cracks; the virtual crack image dataset includes a number of virtual crack images, and the real crack image dataset includes a number of real crack images, each of the virtual crack images and each of the real crack images is correspondingly annotated with a semantic segmentation label, and the semantic segmentation label is used to annotate the road surface background and the crack area; the CycleGAN network pre-training model includes two pairs of generators and discriminators, and the two pairs of generators and discriminators are alternately trained in adversarial manner, the generator is used to learn to convert the real crack image into the virtual crack image and generate the migration image, and the discriminator is used to distinguish the migration image generated by the generator from the virtual crack image.

[0010] The migration image is input into a semantic segmentation pre-training model to extract the corresponding crack area in the real crack image to be segmented; the semantic segmentation pre-training model is a trained model trained based on the virtual crack image dataset and its semantic segmentation label, and the semantic segmentation pre-training model is used to perform crack detection and segmentation on the migration image and extract the crack area.

[0011] Optionally, before the step of acquiring the real crack image to be segmented, the method for extracting cracks from the undersea tunnel pavement further includes the following steps.

[0012] Construct a virtual crack image dataset.

[0013] Based on the virtual crack image dataset and the real crack image dataset, the CycleGAN network model is trained to obtain a CycleGAN network pre-trained model; the CycleGAN network model is an untrained model corresponding to the CycleGAN network pre-trained model.

[0014] Optionally, constructing a virtual crack image dataset specifically includes the following steps.

[0015] Create a road surface background entity; the road surface background entity refers to a plane entity in three-dimensional space as a road surface background.

[0016] A coordinate point is randomly selected on the road surface background entity as the starting point for crack generation, and a modifier is used to simulate crack characteristics to generate a crack entity; the modifier includes a subdivision surface modifier, a simple deformation modifier, a displacement modifier, a wave modifier and a solidification modifier.

[0017] The crack entity and the road surface background entity are combined to obtain a combined entity including the crack and the road surface background.

[0018] A Boolean operation is performed on the combined entity to obtain a pavement crack entity; the pavement crack entity is an entity obtained by subtracting the overlapping portion of the crack entity and the pavement background entity from the combined entity and hiding the crack entity.

[0019] According to the pavement crack entity, rendering parameters are adjusted to generate a virtual crack image; the rendering parameters include the material texture of the pavement, the material texture of the crack, the light intensity and the light direction.

[0020] According to the virtual crack image, a material channel is adjusted or a label layer is used to generate a semantic segmentation label corresponding to the virtual crack image.

[0021] Repeat steps “creating a pavement background entity” to “adjusting the material channel or using the label layer according to the virtual crack image to generate a semantic segmentation label corresponding to the virtual crack image” to obtain a virtual crack image dataset.

[0022] Optionally, after the step of training the CycleGAN network model based on the virtual crack image dataset and the real crack image dataset to obtain the CycleGAN network pre-training model, the method for extracting cracks from the undersea tunnel pavement further includes the following steps.

[0023] The CycleGAN network pre-training model is used to perform style transfer and image conversion on each of the real crack images in the real crack image dataset to obtain a plurality of the transferred images.

[0024] A migration image dataset is established based on the plurality of migration images.

[0025] The migration image dataset, the virtual crack image dataset and the real crack image dataset are compared in terms of crack features, intra-domain distribution and inter-domain distribution to evaluate the performance of the CycleGAN network pre-training model.

[0026] Optionally, the virtual crack image dataset is a SyCrack dataset, the real crack image dataset is a CRACK500 dataset, and the migration image dataset is a T-CRACK500 dataset.

[0027] Optionally, the loss function of the CycleGAN network pre-training model includes generating adversarial loss and cycle consistency loss.

[0028] Optionally, before the step of inputting the migration image into a semantic segmentation pre-training model to extract the corresponding crack area in the real crack image to be segmented, the method for extracting cracks from the undersea tunnel pavement further includes the following steps.

[0029] Based on the virtual crack image dataset and its semantic segmentation labels, a semantic segmentation model is trained with the goal of minimizing cross entropy loss, and a back propagation algorithm is used to optimize the model parameters of the semantic segmentation model. After the training is completed, a semantic segmentation pre-trained model is obtained; the semantic segmentation model is an untrained model corresponding to the semantic segmentation pre-trained model.

[0030] Optionally, the semantic segmentation pre-training model is FCN, U-net, Erfnet, DeepLabV3, PSPNet or SegFormer.

[0031] In a second aspect, the present application provides a system for extracting cracks from an undersea tunnel pavement, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for extracting cracks from an undersea tunnel pavement described in the first aspect.

[0032] According to the specific embodiments provided in this application, this application discloses the following technical effects: The present application provides a method and system for extracting cracks from the pavement of an undersea tunnel, which uses a CycleGAN network pre-training model. The CycleGAN network pre-training model uses the adversarial training principle between a generator and a discriminator based on a virtual crack image dataset and a real crack image dataset, and uses the generator to continuously learn to convert the real crack image into a virtual crack image and generate a migration image. At the same time, the discriminator is used to distinguish the migration image and the virtual crack image generated by the generator as much as possible. After the real crack image is input into the CycleGAN network pre-training model, the real crack image can be extracted through the CycleGAN network pre-training model. By performing style transfer and image conversion, a migration image with morphological features, position features, curvature features and texture features similar to those in the virtual crack image is output, and the real crack image is converted into a pseudo-real crack image with a style similar to that of the virtual crack image. This can effectively reduce the inter-domain difference between the virtual crack image dataset and the real crack image dataset, and then the migration image output by the CycleGAN network pre-training model is input into the semantic segmentation pre-training model to accurately segment and extract the crack area, thereby improving the segmentation accuracy of the model for the real crack image, and providing solid technical support for practical applications in fields such as road maintenance and intelligent transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A flow chart of a method for extracting cracks from an undersea tunnel pavement provided in one embodiment of the present application.

[0035] Figure 2 A schematic diagram of the overall architecture of a method for extracting cracks from a submarine tunnel pavement provided in one embodiment of the present application.

[0036] Figure 3 A schematic diagram of the structure of a CycleGAN network provided in one embodiment of the present application.

[0037] Figure 4 A schematic diagram of the structure of a generator provided in one embodiment of the present application.

[0038] Figure 5 A schematic diagram of the structure of a discriminator provided in one embodiment of the present application.

[0039] Figure 6A pixel distribution histogram of the CRACK500 dataset provided in an embodiment of the present application.

[0040] Figure 7 This is a pixel distribution histogram of the T-CRACK500 dataset provided in an embodiment of the present application.

[0041] Figure 8 This is a domain distribution diagram of the CRACK500, T-CRACK500 and SyCrack data sets provided in an embodiment of the present application on a two-dimensional plane.

[0042] Fig. 9 This is a domain distribution diagram of the CRACK500, T-CRACK500 and SyCrack data sets provided in an embodiment of the present application on a two-dimensional plane.

[0043] Fig.10 A schematic diagram of the mPrecision indicator segmentation results provided in an embodiment of the present application.

[0044] Fig.11 A schematic diagram of the mRecall indicator segmentation result provided in an embodiment of the present application.

[0045] Fig.12 A schematic diagram of the mFscore indicator segmentation results provided in an embodiment of the present application.

[0046] Fig.13 A schematic diagram of the mIoU indicator segmentation result provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] like Figure 1 As shown, this embodiment proposes a method for extracting cracks from an undersea tunnel pavement. The method is mainly used to detect, segment and extract crack areas from an undersea tunnel pavement, and specifically includes the following steps.

[0049] Step S1, obtaining a real crack image to be segmented.

[0050] In this embodiment, the real crack image to be segmented is a real crack image to be segmented and processed by collecting cracks on the road surface of an undersea tunnel. The segmentation processing here refers to crack detection and segmentation of the real crack image to be segmented, and the real crack image to be segmented is segmented into a road surface background and a crack area, thereby extracting the crack area.

[0051] Step S2: input the real crack image to be segmented into a CycleGAN (Cycle-Consistent Generative Adversarial Networks) pre-trained model to obtain a migration image.

[0052] In this embodiment, the CycleGAN network pre-training model is a trained CycleGAN network model obtained by training based on a virtual crack image dataset and a real crack image dataset. The CycleGAN network pre-training model is used to perform style transfer and image conversion on the real crack image to obtain a migration image similar to the crack features in the virtual crack image, wherein the crack features include morphological features, position features, curvature features and texture features of the cracks; the virtual crack image dataset includes a plurality of virtual crack images, and the real crack image dataset includes a plurality of real crack images, each of the virtual crack images and each of the real crack images is correspondingly annotated with a semantic segmentation label, and the semantic segmentation label is used to annotate the road surface background and the crack area; the CycleGAN network pre-training model includes two pairs of generators and discriminators, and the two pairs of generators and discriminators are alternately subjected to adversarial training, the generator is used to learn to convert the real crack image into the virtual crack image and generate the migration image, and the discriminator is used to distinguish the migration image generated by the generator from the virtual crack image.

[0053] Step S3: input the migration image into the semantic segmentation pre-training model to extract the corresponding crack area in the real crack image to be segmented.

[0054] In this embodiment, the semantic segmentation pre-trained model is a trained model obtained by training based on a virtual crack image dataset and its semantic segmentation labels. The semantic segmentation pre-trained model is used to detect and segment cracks in the migration image and extract crack areas.

[0055] In this embodiment, before the step of obtaining the real crack image to be segmented in step S1, the method for extracting cracks from the pavement of an undersea tunnel further includes the following steps.

[0056] (1) Construct a virtual crack image dataset. The specific steps include the following steps.

[0057] (1-1) Creating a pavement background entity; the pavement background entity refers to a plane entity in a three-dimensional space as a pavement background.

[0058] (1-2) Randomly select a coordinate point on the road surface background entity as the starting point for crack generation, and use a modifier to simulate crack characteristics to generate a crack entity; the modifier includes a subdivision surface modifier, a simple deformation modifier, a displacement modifier, a wave modifier, and a solidify modifier.

[0059] (1-3) Combining the crack entity and the pavement background entity to obtain a combined entity including the crack and the pavement background.

[0060] (1-4) Performing a Boolean operation on the combined entity to obtain a pavement crack entity; the pavement crack entity is an entity obtained by subtracting the overlapping portion of the crack entity and the pavement background entity from the combined entity and hiding the crack entity.

[0061] (1-5) According to the pavement crack entity, adjusting rendering parameters to generate a virtual crack image; the rendering parameters include the material texture of the pavement, the material texture of the crack, the light intensity and the light direction.

[0062] (1-6) According to the virtual crack image, adjust the material channel or use the label layer to generate a semantic segmentation label corresponding to the virtual crack image.

[0063] (1-7) Repeat steps (1-1) to (1-6) to obtain a virtual crack image dataset.

[0064] Compared with the traditional manual labeling method, this embodiment automatically generates a virtual crack image dataset with high-precision semantic segmentation labels. The automated process significantly reduces the cost and time of virtual crack image dataset production, and greatly improves the efficiency of virtual crack image dataset production. By automatically and efficiently generating a large number of high-quality virtual crack image datasets for training high-precision semantic segmentation models, it helps road management departments to more effectively monitor and maintain roads and extend the service life of roads. Combined with intelligent transportation systems, real-time monitoring and early warning of road cracks can be achieved, improving road safety and service life. It also provides a high-quality data generation and training platform for researchers and students in the fields of road engineering, computer vision, etc., promoting the development of scientific research and teaching in related fields, and promoting the development of road crack detection and maintenance technology.

[0065] (2) Based on the virtual crack image dataset and the real crack image dataset, the CycleGAN network model is trained to obtain a CycleGAN network pre-trained model. The CycleGAN network model is an untrained model corresponding to the CycleGAN network pre-trained model.

[0066] In this embodiment, after step (2), the method for extracting cracks from the undersea tunnel pavement further includes the following steps.

[0067] (3) Using the CycleGAN network pre-training model, perform style transfer and image conversion on each of the real crack images in the real crack image dataset to obtain a plurality of the transferred images.

[0068] (4) A migration image dataset is established based on the plurality of migration images.

[0069] (5) performing crack feature comparison, intra-domain distribution comparison, and inter-domain distribution comparison on the migration image dataset, the virtual crack image dataset, and the real crack image dataset to evaluate the performance of the CycleGAN network pre-training model.

[0070] In this embodiment, the virtual crack image dataset is SyCrack (Synthetic Crack Dataset), the real crack image dataset is CRACK500 (Crack 500) dataset, and the migration image dataset is T-CRACK500 (T-Crack 500) dataset.

[0071] In this embodiment, the loss function of the CycleGAN network pre-training model includes generative adversarial loss and cycle consistency loss.

[0072] This embodiment uses the CycleGAN network pre-training model to convert real crack images into pseudo-real crack images with a similar style to virtual crack images, thereby effectively reducing the domain differences between virtual crack image datasets and real crack image datasets, and improving the detection accuracy and generalization ability of the model in real scenes. The application of this technology in the field of crack detection is still novel, and it provides a new solution for dealing with domain differences. In addition, this embodiment realizes image style transfer without paired data by combining generative adversarial loss with cycle consistency loss, which can significantly improve the applicability and accuracy of subsequent semantic segmentation models on real crack images and enhance the performance of the model in complex real scenes.

[0073] In this embodiment, before the step of inputting the migration image into the semantic segmentation pre-training model in step S3 to extract the corresponding crack area in the real crack image to be segmented, the method for extracting cracks from the undersea tunnel pavement further includes the following steps.

[0074] (6) Based on the virtual crack image dataset and its semantic segmentation labels, a semantic segmentation model is trained with the goal of minimizing the cross entropy loss, and a back propagation algorithm is used to optimize the model parameters of the semantic segmentation model. After the training is completed, a semantic segmentation pre-trained model is obtained. The semantic segmentation model is an untrained model corresponding to the semantic segmentation pre-trained model.

[0075] In this embodiment, the semantic segmentation pre-training model is FCN (Fully Convolutional Networks), U-net (Deep Learning Convolutional Neural Networks), Erfnet (Efficient residual factorized networks), DeepLabV3 (Deep Learning Network Based on Semantic Segmentation), PSPNet (Pyramid Scene Parsing Network) or SegFormer (Simple and efficient design For semantic segmentation with transformers).

[0076] In order to make the technical solution of this embodiment clearer, the model training process and specific application process of this embodiment are described in detail below in the form of examples.

[0077] like Figure 2 As shown, this embodiment builds a complete semantic segmentation architecture based on a virtual crack image dataset and unsupervised domain adaptation (UDA) technology. The semantic segmentation architecture consists of three core modules: a virtual pavement crack dataset generation module, a migration network module, and a segmentation network module. Among them, the virtual pavement crack dataset generation module is used to construct a virtual pavement crack dataset; the migration network module is used to perform migration conversion of real crack images based on the CycleGAN network model to obtain a migration image similar to the virtual crack image; the segmentation network module is used to perform image segmentation on the migration image based on the semantic segmentation model to extract the crack area.

[0078] In the first stage, a virtual pavement crack dataset with semantic segmentation labels was created based on the virtual pavement crack dataset generation module. In the second stage, the migration network and segmentation network were trained based on the migration network module. The migration network was trained adversarially between its generator and discriminator, with virtual crack images as the Y domain and real crack images as the X domain. This adversarial training gradually improved the style transfer ability of the CycleGAN network model. Then, six types of semantic segmentation models including FCN, U-net, Erfnet, DeepLabV3, PSPNet or SegFormer were trained using virtual crack images and semantic segmentation labels in the virtual crack image dataset. The third stage is the testing stage based on the segmentation network module. After training, the CycleGAN network model converts real crack images into images that are closer to the style of virtual crack images, called migration images. Then, the converted migration images are input into the semantic segmentation network for crack detection and segmentation. By combining the generation of virtual crack image datasets with unsupervised domain adaptation technology, the applicability and accuracy of virtual crack image datasets in real road crack images are significantly improved without manual pixel-level labeling.

[0079] This embodiment uses the open source 3D computer graphics software Blender to achieve highly automated generation of virtual crack image data sets, which not only reduces the production cost of virtual crack image data sets, but also improves the efficiency and quality of virtual crack image data set generation, providing a solid data foundation for model training. By combining the geometric modeling, rendering and material processing capabilities of Blender software, random parameter adjustment is used to generate diversified virtual crack images, ensuring that the generated virtual crack image data sets are highly similar to real crack images in terms of crack morphology, position and texture, etc., providing rich and high-quality data support for the training of semantic segmentation models, and can help train more robust and adaptable semantic segmentation models. The generation of virtual crack image data sets specifically includes the following steps.

[0080] Step 1: Create a road surface background entity.

[0081] In the 3D space of Blender software, first create a plane entity as the pavement background and determine its dimensions (such as length, width, and height). These dimensions will serve as the basic reference framework for subsequent crack generation to ensure that crack generation does not exceed the boundaries of the pavement entity.

[0082] Step 2: Generate crack entities.

[0083] Randomly select a starting point: Randomly select a coordinate point on the pavement background entity as the starting point for crack generation to increase the randomness and diversity of the crack location.

[0084] This embodiment uses a modifier to simulate crack characteristics, as follows.

[0085] Subdivision Surface: Add a subdivision surface modifier to the crack entity. By increasing the subdivision level of the mesh, the crack surface becomes smoother and more delicate, simulating the microstructure of a real crack.

[0086] Simple Deformation: Apply a simple deformation modifier, such as the Stretch method, to deform the cracks along a specific direction (such as the Z axis), with the deformation factor set to a random value to ensure diversity in the crack lengths.

[0087] Displacement: Use the displacement modifier and noise texture to generate irregular undulations on the crack surface, simulating the rough texture of natural cracks. Control the complexity and naturalness of the texture by adjusting the depth and turbulence parameters of the noise texture.

[0088] Wave: Through the wave modifier, the height, width and narrowness parameters are randomly adjusted to simulate the wear and tear or weathering effects in the natural environment, making the cracks more realistic.

[0089] Solidify: Add thickness to the two-dimensional crack plane to make it a three-dimensional object, making the crack more three-dimensional and realistic visually. The thickness value can be set randomly.

[0090] Through the combined use of the above modifiers, not only the shape, texture and curvature of the cracks are closer to the real situation, but also the introduction of random parameters ensures that each generated crack is unique.

[0091] Step 3: Combine the crack entity with the pavement background entity. In this step, the modified crack entity is combined with the pavement background entity to form a combined entity containing the crack and the pavement background. At this point, the crack entity and the pavement background entity are combined together, but no Boolean operation has been performed.

[0092] Step 4: Perform Boolean operations on the combined entities to generate pavement crack entities. Next, a realistic crack effect is generated by performing Boolean operations on the combined entities, specifically subtracting the overlapped portion of the crack entity and the pavement background and hiding the crack entity. In this way, the crack is naturally integrated into the pavement background, forming the final pavement crack entity.

[0093] Step 5: Generate a virtual crack image.

[0094] In this embodiment, adjusting rendering parameters to generate a virtual crack image specifically includes the following contents.

[0095] (1) Setting the global environment and camera position: Adjust the lighting conditions and camera position of the global environment to ensure that the visual effect of the generated image is close to the real scene.

[0096] (2) Generate virtual crack images: Apply material textures and lighting conditions, and apply different material textures to the pavement background entity and the crack entity according to the material characteristics of the pavement and the crack. At the same time, adjust the lighting intensity and direction to generate realistic virtual crack images.

[0097] Step 6: Generate labels for virtual crack images.

[0098] When generating labels, this embodiment generates semantic segmentation labels corresponding to the virtual crack image by adjusting the material channel or using a dedicated label layer under the same camera position and rendering settings. These labels accurately mark the crack area and road surface background, providing supervision information for subsequent semantic segmentation model training.

[0099] Through the above steps, a large number of high-quality virtual crack images and their corresponding semantic segmentation labels can be quickly generated. These images are highly similar to real crack images in terms of crack morphology, location, and texture features, providing sufficient and diverse data sets for training deep learning models. In addition, since the entire generation process is highly automated, the cost and time of data set production are significantly reduced. A virtual crack image of pavement cracks can be produced every 0.5 seconds, and the corresponding semantic segmentation labels are automatically generated, which improves the availability and practicality of the data set.

[0100] In order to ensure the diversity of the dataset, the above steps are repeated to generate a large number of virtual crack image datasets with high-quality pixel-level annotations, namely the SyCrack dataset. The SyCrack dataset contains 2000 virtual crack images and their corresponding semantic segmentation labels. The resolution of each image is 448×448 pixels.

[0101] By adjusting rendering parameters and modifier settings, diverse crack morphologies, locations, and texture features are generated, ensuring a high degree of diversity and realism in the dataset.

[0102] In this embodiment, the CRACK500 dataset is a real crack image dataset, and SyCrack is a virtual crack image dataset. Compared with the CRACK500 dataset, the virtual crack images in the SyCrack dataset have a high degree of similarity with the real crack images in the CRACK500 dataset in terms of crack shape, curvature, position, and texture features of the interface with the road surface.

[0103] Through the above method, a large number of high-quality virtual crack images and their corresponding semantic segmentation labels can be quickly generated, providing sufficient data support for training the segmentation model. It has the advantages of low cost, high efficiency and customizability, and effectively solves the problems of high production cost, small scale and low accuracy of existing semantic segmentation datasets.

[0104] In this embodiment, in order to effectively reduce the domain difference between the virtual crack image dataset and the real dataset, an unsupervised domain adaptation technique is used, and the CycleGAN network model is used to perform style transfer and image conversion on the real crack image, so that the converted migration image is closer to the virtual crack image in style, thereby improving the performance of the subsequent semantic segmentation model. The following is a detailed refinement of the unsupervised domain adaptation method. The style transfer based on the CycleGAN network model specifically includes the following steps.

[0105] Step 1: Train the CycleGAN network model to obtain the CycleGAN network pre-training model.

[0106] Data preparation: The source domain data is the CRACK500 dataset. These images contain diverse characteristics of cracks in real scenes, but the annotation cost is high and the number is limited. The target domain data is the SyCrack dataset, which contains a large number of high-quality, pixel-level annotated virtual crack images. Network construction: Figure 3 As shown, this embodiment adopts the CycleGAN network model, which consists of two pairs of generators and discriminators ( , , , ), which are responsible for image conversion from source domain to target domain and from target domain to source domain respectively. Loss function: Define the generation adversarial loss ( ) and cycle consistency loss ( ), ensuring that the image style is transferred while retaining key semantic information.

[0107] Figure 4 Shows the generator The network structure of the generator Learn how to convert a CRACK500 to SyCrack, creating a T-CRACK500. Figure 5 The discriminator is shown The network structure, Try to distinguish SyCrack from the generator Generated T-CRACK500.

[0108] Adversarial training: By alternately training the generator and the discriminator, the network parameters are continuously optimized until the Nash equilibrium is reached. Training results: After sufficient training, the CycleGAN network can convert the real crack images in CRACK500 into migration images that are similar in style to the SyCrack dataset. The corresponding migration image dataset is T-CRACK500.

[0109] It should be noted that the CycleGAN network pre-trained model in this embodiment refers to a CycleGAN network model that has been pre-trained, and the CycleGAN network model refers to a CycleGAN network model that has not completed training. Both the CycleGAN network model and the CycleGAN network pre-trained model have the function of converting real crack images into migration images. The two are CycleGAN network models at different stages before and after training.

[0110] Step 2: Generate the migration image dataset T-CRACK500.

[0111] Using the CycleGAN network pre-training model, we transfer the style of each real crack image in the CRACK500 dataset and generate the corresponding migration image, ensuring that the converted migration image retains the semantic information of the real crack (such as crack location, shape, etc.) while being similar in style to the SyCrack dataset. The migration image not only captures the pixel-level features of the real crack image, but also retains the defect features of the real crack image.

[0112] Figure 6 and Figure 7 The pixel distribution histograms of the CRACK500 dataset and the T-CRACK500 dataset are respectively. Compared with the CRACK500 dataset, the T-CRACK500 dataset shows a more obvious separation between cracks and pavement background features. In the T-CRACK500 dataset, the background pixel distribution becomes more concentrated, forming a sharp peak, which reduces the background noise and enhances the uniformity of the background area. This improvement helps the crack detection algorithm to better distinguish the background information. In addition, the pixel distribution of the crack area is more concentrated in the lower value range, making the crack area in the image more consistent and prominent. This improvement improves the algorithm's ability to accurately capture and locate cracks. The difference between the pixel distribution of the background and crack areas has increased significantly, enhancing the contrast between the cracks and the background, thereby improving the detection accuracy.

[0113] In the T-CRACK500 dataset, the position and morphology of the cracks remain unchanged, and the morphology and geometric features of the cracks in the CRACK500 dataset are retained. In order to study the relationship between different image domains during domain use, this embodiment uses t-SNE (t-distributed Stochastic Neighbor Embedding) to project the high-dimensional features of the CRACK500 dataset, T-CRACK500 dataset, and SyCrack dataset into a two-dimensional feature space for visualization. Figure 8 and Fig. 9The inter-domain distribution and intra-domain distribution of the three datasets CRACK500, T-CRACK500 and SyCrack on the two-dimensional plane are shown respectively. It can be seen intuitively that there are significant differences in the intra-domain distribution and inter-domain distribution between the CRACK500 dataset and the SyCrack dataset. In contrast, the cross-domain distribution of the T-CRACK500 dataset is between the SyCrack dataset and the CRACK500 dataset, and is closer to the SyCrack dataset. In addition, compared with the SyCrack dataset, the intra-domain distribution of the CRACK500 dataset is more concentrated, while the intra-domain distribution of the T-CRACK500 dataset is closer to the SyCrack dataset. This shows that through domain adaptation, the T-CRACK500 dataset is closer to the SyCrack dataset in terms of intra-domain distribution and inter-domain distribution.

[0114] Step 3: Use the virtual crack image dataset to train the semantic segmentation model to obtain a semantic segmentation pre-trained model.

[0115] Model selection: Select a variety of mainstream semantic segmentation models for training, including U-Net, Erfnet, DeepLabV3, FCN, PSPNet and Segformer, which have shown excellent performance in the field of image segmentation.

[0116] Training process: Use the virtual crack image dataset SyCrack and its corresponding semantic segmentation labels as training data. Use the back propagation algorithm to optimize the model parameters, minimize the loss function (such as cross entropy loss), and improve the model's ability to identify crack areas.

[0117] It should be noted that the semantic segmentation pre-trained model in this embodiment refers to a semantic segmentation model that has been pre-trained, while the semantic segmentation model refers to a semantic segmentation model that has not completed training. Both the semantic segmentation model and the semantic segmentation pre-trained model have the functions of semantic segmentation and extraction of crack areas. The two are semantic segmentation models at different stages before and after training respectively.

[0118] Step 4: Test the model.

[0119] The semantic segmentation pre-trained model is applied to the migration image dataset T-CRACK500 to evaluate its performance in segmenting the migration images. Based on the segmentation results on the T-CRACK500 dataset, the reasons for the poor performance of the model (such as mis-segmentation, missed segmentation, etc.) are analyzed. Multiple evaluation indicators such as Precision, Recall, Fscore, and IoU are used to measure the segmentation effect of the model.

[0120] Through the above-mentioned unsupervised domain adaptation method, this embodiment can effectively reduce the inter-domain differences and intra-domain differences between the SyCrack dataset and the CRACK500 dataset, so that the semantic segmentation model trained on the virtual crack image dataset can be more effectively applied to the segmentation task of real crack images.

[0121] The datasets used in this embodiment include the CRACK500 dataset, the T-CRACK500 dataset, and the SyCrack dataset. The CRACK500 dataset includes 1896 training images and 348 test images, and the resolution of each image is 448×448 pixels. The significant variations in crack morphology and width in the CRACK500 dataset, as well as the complex background textures in most images, make crack segmentation in the CRACK500 dataset particularly challenging. The T-CRACK500 dataset contains 348 images, which are migrated from the CRACK500 dataset through domain adaptation to improve the segmentation performance of the model on real crack images. The SyCrack dataset includes 2000 automatically generated virtual pavement crack images, each of which is labeled with high-precision pixel-level segmentation labels.

[0122] As shown in Table 1, in order to evaluate the cross-domain performance of the semantic segmentation model trained on the virtual crack image dataset, this embodiment conducted experimental tests on three sets of datasets: Source only, Adaptive, and Oracle. By selecting different training and test datasets, the differences between domains are gradually reduced. In Source only, the SyCrack dataset is used for training, and the real CRACK500 dataset is used for testing. In Adaptive, the SyCrack dataset is used for training, and the T-CRACK500 dataset is used for testing. In Oracle, the CRACK500 dataset is used for training and testing.

[0123] Table 1 Experimental test

[0124] In this embodiment, the hardware environment includes an Intel(R) i7-13620H CPU, an RTX 4060 GPU with 8GB VRAM and 32GB RAM. The software environment includes PyTorch 1.7.1, Python 3.8, CUDA 11.0, and the operating system is Windows. The network batch size of CycleGAN is 1, the initial learning rate is 0.0002, and 3 epochs are trained using the Adam optimizer.

[0125] This embodiment uses mPrecision (mean Precision), mRecall (meanRecall), mFscore (mean Fscore) and mIoU (mean IoU) as evaluation indicators. mPrecision, mRecall, mFscore and mIoU are the average values ​​of Precision, Recall, Fscore and IoU on different segmentation categories. Precision, Recall, Fscore and IoU are defined as shown in formulas (1) to (4).

[0126] (1).

[0127] (2).

[0128] (3).

[0129] (4).

[0130] Among them, TP is a true positive example, which indicates the number of pixels accurately identified as cracks, FP is a false positive example, which indicates the number of pixels incorrectly identified as cracks, FN is a false negative example, which indicates the number of pixels incorrectly identified as non-cracks, and TN is a true negative example, which indicates the number of pixels accurately identified as non-cracks.

[0131] Figures 10 to 13 The segmentation results of six models, FCN, U-net, Erfnet, DeepLabV3, PSPNet and SegFormer, on three datasets, Source only, Adaptive and Oracle, are shown respectively. Fig.10 This is a schematic diagram of the segmentation results of the mPrecision indicator. Fig.11 This is a schematic diagram of the mRecall indicator segmentation results. Fig.12 This is a schematic diagram of the mFscore indicator segmentation results. Fig.13Schematic diagram of the segmentation results of the mIoU indicator. In the Oracle dataset, there is no domain difference between the training set and the test set, and all six models show high accuracy. Among them, the mPrecision of five models exceeds 86.0%, while the mPrecision of DeepLabV3 and SegFormer exceeds 90.0%. In contrast, the mPrecision of FCN is slightly lower, at 83.88%. The mRecall of the six models ranges from 84% to 88%, and the mFscore ranges from 85% to 89%. Except for SegFormer with an mIoU of 81.18%, the mIoU of the remaining models is between 76% and 80%. On the Source only dataset, due to significant domain differences, the accuracy of the model drops significantly. The mFscore of DeepLabV3 and PSPNet is about 50%, and the mIoU is about 48%. The mFscore of FCN is 55.5%, and the mIoU is 51.01%. The remaining three models have an mFscore of about 60% and an mIoU of about 54%. Although DeepLabV3, FCN, and PSPNet have an mPrecision of over 93%, their mRecall is low, only about 50%. This shows that although these models have high precision when predicting, they fail to correctly identify a large number of true positive examples. On the Adaptive dataset, the segmentation accuracy is significantly improved. The mFscore and mIoU of the six models increased by 16.55% and 12.62%, respectively. Specifically, U-net's mFscore increased from 61.31% to 79.63%, an increase of 18.32%, and its mIoU increased from 54.39% to 69.77%, an increase of 15.38%. Similarly, SegFormer's mFscore increased from 61.07% to 76.24%, an increase of 15.17%, and its mIoU increased from 54.06% to 66.27%, an increase of 12.21%. Although the improvements of DeepLabV3 and PSPNet are relatively small, their mFscore and mIoU are still improved by about 14% and 9%, respectively.

[0132] Taking U-net as an example, on the Adaptive dataset, domain adaptation significantly improved the robustness and detection sensitivity of the model, thereby enhancing the segmentation performance of the model. On the Source only dataset, due to the high interference of the road background, the model misclassified the road background as cracks. However, under the Adaptive condition, this missegmentation was significantly reduced. In addition, for images where cracks are difficult to distinguish from the road background, under the Source only condition, the model tends to misclassify cracks as part of the road background. Under the Adaptive condition, the possibility of such misclassification in the transferred image is reduced. This further proves that the segmentation ability of the model is significantly improved after domain adaptation.

[0133] This embodiment uses Blender software to develop a virtual crack image dataset, which can generate a virtual crack image every 0.5 seconds and automatically generate corresponding semantic segmentation labels. The virtual crack image dataset SyCrack contains 2,000 high-quality pixel-level annotated virtual crack images, which are highly similar to real crack images in terms of crack morphology, position, curvature and other features, providing effective virtual data support for semantic segmentation model training. In addition, this embodiment uses the unsupervised domain adaptation network CycleGAN to effectively reduce the domain difference between real crack images and virtual crack images, and improve the segmentation performance of multiple semantic segmentation models on real crack images. The experimental results show that the performance of this method on real data sets has been significantly improved, with the advantages of low cost, high efficiency and customizability, and significantly improves the segmentation accuracy of the model on real crack images, and has broad application prospects and market value.

[0134] In view of the problems faced by the current pavement crack segmentation field, such as high data annotation cost, scarcity of real annotation datasets, and significant domain differences between virtual crack image datasets and real datasets, this embodiment performs pavement crack segmentation based on virtual crack image datasets and unsupervised domain adaptation, effectively reducing the domain differences between virtual crack image datasets and real crack image datasets, improving the segmentation accuracy of the model on real crack images, and providing solid technical support for practical applications in the fields of road maintenance and intelligent transportation. It can be refined into the following aspects.

[0135] (1) Reduce data annotation costs: Traditional methods rely on a large number of manual annotations of real crack images, which is not only time-consuming and labor-intensive, but also the annotation quality is easily affected by human factors. This embodiment overcomes this bottleneck by automatically generating a high-quality virtual crack image dataset with semantic segmentation labels, significantly reducing the cost of data annotation and improving the accuracy of annotation.

[0136] (2) Improving the quality and diversity of virtual crack image datasets: By using advanced computer graphics technology (such as Blender), this embodiment can generate high-quality virtual crack images containing rich and diverse crack features. These images are highly similar to real crack images in terms of shape, position, curvature, texture, etc., providing sufficient and diverse training samples for training high-performance semantic segmentation models.

[0137] (3) Reducing the inter-domain difference between the virtual crack image dataset and the real crack image dataset: By introducing unsupervised domain adaptation technology (such as CycleGAN), the style of the real crack image can be transferred to the domain of the virtual crack image dataset, generating pseudo-real crack images that are similar in style to the virtual crack images but retain the semantic information of the real cracks, effectively reducing the inter-domain difference between the virtual crack image dataset and the real crack image dataset, so that the model trained on the virtual crack image dataset can better adapt to real-world application scenarios.

[0138] (4) Improving the segmentation accuracy of the model for real crack images: This embodiment significantly improves the segmentation accuracy of the model in segmenting real crack images, which is not only reflected in the accurate identification of crack areas, but also in the fine processing of crack boundaries, providing road maintenance departments with more reliable and accurate crack detection results.

[0139] In an exemplary embodiment, a system for extracting cracks from an undersea tunnel pavement is also provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for extracting cracks from an undersea tunnel pavement.

[0140] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for extracting cracks from a submarine tunnel pavement, characterized in that: The method for extracting cracks from the pavement of a submarine tunnel comprises: Acquire a real crack image to be segmented; the real crack image to be segmented is a real crack image to be segmented and processed obtained by collecting cracks on the pavement of an undersea tunnel; The real crack image to be segmented is input into the CycleGAN network pre-training model to obtain a migration image; the CycleGAN network pre-training model is a trained model obtained by training based on a virtual crack image dataset and a real crack image dataset, and the CycleGAN network pre-training model is used to perform style migration and image conversion on the real crack image to obtain a migration image similar to the crack features in the virtual crack image, and the crack features include morphological features, position features, curvature features and texture features of the cracks; the virtual crack image dataset includes a number of virtual crack images, and the real crack image dataset includes a number of real crack images, each of the virtual crack images and each of the real crack images is correspondingly annotated with a semantic segmentation label, and the semantic segmentation label is used to annotate the road surface background and the crack area; the CycleGAN network pre-training model includes two pairs of generators and discriminators, and the two pairs of generators and discriminators are alternately trained in adversarial manner, the generator is used to learn to convert the real crack image into the virtual crack image and generate the migration image, and the discriminator is used to distinguish the migration image generated by the generator from the virtual crack image; The migration image is input into a semantic segmentation pre-training model to extract the corresponding crack area in the real crack image to be segmented; the semantic segmentation pre-training model is a trained model trained based on the virtual crack image dataset and its semantic segmentation label, and the semantic segmentation pre-training model is used to perform crack detection and segmentation on the migration image and extract the crack area.

2. The method for extracting cracks in a submarine tunnel pavement according to claim 1, characterized in that: Before the step of obtaining the real crack image to be segmented, the method for extracting cracks from the pavement of an undersea tunnel further comprises: Construct virtual crack image dataset; Based on the virtual crack image dataset and the real crack image dataset, the CycleGAN network model is trained to obtain a CycleGAN network pre-trained model; the CycleGAN network model is an untrained model corresponding to the CycleGAN network pre-trained model.

3. The method for extracting cracks from a submarine tunnel pavement according to claim 2, characterized in that: Construct a virtual crack image dataset, including: Creating a road surface background entity; the road surface background entity refers to a plane entity in a three-dimensional space as a road surface background; A coordinate point is randomly selected on the road surface background entity as a starting point for crack generation, and a modifier is used to simulate crack characteristics to generate a crack entity; the modifier includes a subdivision surface modifier, a simple deformation modifier, a displacement modifier, a wave modifier and a solidification modifier; Combining the crack entity and the pavement background entity to obtain a combined entity including the crack and the pavement background; Performing a Boolean operation on the combined entity to obtain a pavement crack entity; the pavement crack entity is an entity obtained by subtracting the overlapping portion of the crack entity and the pavement background entity from the combined entity and hiding the crack entity; According to the pavement crack entity, adjusting rendering parameters to generate a virtual crack image; the rendering parameters include the material texture of the pavement, the material texture of the crack, the light intensity and the light direction; According to the virtual crack image, adjusting a material channel or using a label layer to generate a semantic segmentation label corresponding to the virtual crack image; Repeat steps "creating a pavement background entity" to "adjusting the material channel or using the label layer according to the virtual crack image to generate a semantic segmentation label corresponding to the virtual crack image" to obtain a virtual crack image dataset.

4. The method for extracting cracks from a submarine tunnel pavement according to claim 2, characterized in that: After the step of training the CycleGAN network model based on the virtual crack image dataset and the real crack image dataset to obtain the CycleGAN network pre-training model, the subsea tunnel pavement crack extraction method further includes: Using the CycleGAN network pre-training model, performing style migration and image conversion on each of the real crack images in the real crack image dataset to obtain a plurality of migration images; According to the plurality of migration images, a migration image dataset is established; The migration image dataset, the virtual crack image dataset and the real crack image dataset are compared in terms of crack features, intra-domain distribution and inter-domain distribution to evaluate the performance of the CycleGAN network pre-training model.

5. The method for extracting cracks in a submarine tunnel pavement according to claim 4, characterized in that: The virtual crack image dataset is the SyCrack dataset, the real crack image dataset is the CRACK500 dataset, and the migration image dataset is the T-CRACK500 dataset.

6. The method for extracting cracks in a submarine tunnel pavement according to claim 1, characterized in that: The loss function of the CycleGAN network pre-training model includes generation adversarial loss and cycle consistency loss.

7. The method for extracting cracks in a submarine tunnel pavement according to claim 1, characterized in that: Before the step of inputting the migration image into the semantic segmentation pre-training model to extract the corresponding crack area in the real crack image to be segmented, the method for extracting cracks from the undersea tunnel pavement further includes: Based on the virtual crack image dataset and its semantic segmentation labels, a semantic segmentation model is trained with the goal of minimizing cross entropy loss, and a back propagation algorithm is used to optimize the model parameters of the semantic segmentation model. After the training is completed, a semantic segmentation pre-trained model is obtained; the semantic segmentation model is an untrained model corresponding to the semantic segmentation pre-trained model.

8. The method for extracting cracks in a submarine tunnel pavement according to claim 1, characterized in that: The semantic segmentation pre-training model is FCN, U-net, Erfnet, DeepLabV3, PSPNet or SegFormer.

9. A system for extracting cracks from a submarine tunnel pavement, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for extracting cracks from the pavement of an undersea tunnel according to any one of claims 1 to 8.

Images