A method for characterizing typical bridge defects

Abstract

This invention discloses a method for characterizing typical bridge defects, which relates to the field of bridge inspection technology. First, a sample library of deterioration defects is constructed, and a surrounding environment model and a bridge BIM model are established. Second, a virtual bridge scene is constructed using the UE5 engine, and virtual defect samples are obtained. Finally, combining real and virtual samples, an improved dual-branch generative adversarial network is used for data enhancement to generate true-like deterioration samples and expand real sample data. An improved Unet network trained with real sample data is then used to identify and extract defect areas from the inspection image. Finally, the identification results are mapped to the BIM model for three-dimensional visualization. This method addresses the problem of insufficient defect samples through a generative adversarial network, improves the accuracy of fine-grained defect identification using an improved Unet, and implements digital characterization of bridge defects throughout the entire process, from data acquisition and intelligent identification to three-dimensional dynamic display, significantly improving the accuracy and visualization of defect detection.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge detection, and more particularly to a method for characterizing typical bridge defects. Background Art

[0002] With the rapid development of railway transportation networks, railway bridges, as crucial transportation hubs, bear enormous traffic pressure. The rapid development of high-speed and heavy-haul railways, in particular, has placed higher demands on bridge structures. However, long-term use leads to the gradual deterioration of bridge structural materials, resulting in frequent problems such as cracks, rust, and falling blocks. These defects directly threaten bridge safety, potentially reducing structural strength and even causing accidents. In the field of railway bridge health monitoring, intelligent characterization of typical defects (such as cracks, rust, and spalling) is a key technology for ensuring safe bridge operation. Traditional methods for characterizing typical railway bridge defects (such as cracks, spalling, and rust) rely primarily on manual data collection and image analysis, which suffer from limitations in sample acquisition, high costs, insufficient model generalization, and limited visualization. In recent years, with the advancement of artificial intelligence (AI), deep learning-based methods have become a research hotspot, with key technologies including image segmentation, object detection, and classification. For example, convolutional neural networks (CNNs) are used to extract image features, combined with models such as regional convolutional neural networks (R-CNN) or YOLO to locate and identify defects. Simultaneously, research is also using transfer learning and data augmentation techniques to improve the model's adaptability to different environments and lighting conditions. Furthermore, representation learning-based methods can better distinguish different types of defects, such as cracks, rust, and spalling, by extracting high-level features of defects.

[0003] However, existing technologies still have some limitations: first, traditional image processing methods perform poorly in complex backgrounds or with small sample data; second, deep learning models are highly dependent on labeled data and have difficulty effectively solving problems such as overlapping diseased areas and blurred boundaries; finally, due to the particularity of the railway bridge environment (such as changes in perspective and uneven lighting conditions), the generalization ability and robustness of existing models still need to be further improved.

[0004] Therefore, how to improve the accuracy of bridge disease identification and achieve accurate characterization is an urgent problem that technical personnel in this field need to solve. Summary of the Invention

[0005] In light of this, the present invention provides a method for characterizing typical bridge defects. This method constructs a degradation defect library containing typical defects (cracks, corrosion, and spalling). Using an improved Generative Adversarial Network (GAN), the method generates high-quality, true-to-life degradation defect samples and simultaneously generates high-precision annotated masks to expand the deterioration defect dataset, effectively alleviating the issues of data uniformity, scarcity, and high annotation costs. Furthermore, by introducing an image recognition method based on representation learning, the method enables more precise extraction of defect features and high-precision identification. This approach not only improves the model's generalization and adaptability but also provides a new technical path for intelligent bridge monitoring.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for characterizing typical bridge defects includes the following steps:

[0008] Step 1: Collect bridge operation and maintenance data, build a parameterized model library of typical bridge defects, and establish a sample library of bridge deterioration defects;

[0009] Step 2: Based on the design drawings and relevant operation and maintenance information of the target bridge, the family files and mileage information of each subdivided component of the bridge are sorted out. The bridge BIM model is quickly constructed using the rapid modeling plug-in combined with the bridge deterioration disease sample library. Based on the geographical location of the target bridge, the corresponding image maps and elevation data are downloaded to establish a surrounding environment model.

[0010] Step 3: Based on the UE5 engine, the bridge BIM model and the surrounding environment model are integrated to obtain a virtual bridge scene, and the camera in the virtual bridge scene is initialized to obtain virtual disease samples;

[0011] Step 4: Collect real deterioration damage images of existing bridges and target bridges to build a real deterioration damage sample library. Use the real deterioration damage sample library and virtual damage samples to train an improved two-branch generative adversarial network. Use the trained improved two-branch generative adversarial network to obtain real deterioration damage samples to expand the real deterioration damage sample library.

[0012] Step 5: Train the improved Unet network based on the expanded real deterioration disease sample library. Input the collected images of the target bridge to be inspected into the trained improved Unet network to identify deterioration diseases and extract the deterioration disease area mask.

[0013] Step 6: Map the deterioration and disease area mask onto the bridge BIM model for characterization.

[0014] Preferably, typical bridge defects include cracks, corrosion and spalling; cracks include cross cracks, oblique cracks, horizontal cracks and vertical cracks; the crack information is described in detail through four indicators: shape, length, width and depth, and a corresponding crack family model is created to realize data entry and visualization; for corrosion and spalling, area is used as the main descriptive indicator to complete the entry and visualization of relevant information.

[0015] Preferably, the bridge deterioration disease sample library includes apparent disease types such as cracks (classified by direction), rust, and spalling, and deterioration disease models that subdivide each type based on plane morphology, severity and size parameters. A deterioration disease model corresponding to each disease is created based on the BIM platform according to the typical disease data recorded in the parametric model library; when there are no specific requirements for the deterioration disease model, splines are used to define the direction of cracks, the area of ​​rust and spalling, and the length, width and depth can be parametrically adjusted; when it is necessary to construct a deterioration disease model based on actual measurement data, the specific contour coordinates of cracks, rust and spalling are imported into point coordinate data through Dynamo to generate corresponding line contours to form the corresponding deterioration disease model.

[0016] Preferably, a rapid modeling plug-in is used to generate a bridge BIM model based on the family files and mileage information used for each subdivided component of the target bridge in the bridge design drawings and related operation and maintenance information, and the degradation disease model in the bridge degradation disease sample library is loaded into the bridge BIM model; the image map and elevation data of the surrounding environment of the target bridge are downloaded, and the image map is exported as raster data in JPG format through Global Mapper software, and the elevation data is exported as DEM data in TIF format through Global Mapper software, and the DEM data is then exported as a grayscale image through QGIS. The raster data and grayscale image constitute the surrounding environment model; Twinmotion software is used to perform beautification operations such as greening and modifying materials to build a greening model.

[0017] Preferably, the bridge BIM model, greening model and surrounding environment model are imported into the UE5 engine, and the oblique photography model is integrated to form a virtual bridge scene.

[0018] Preferably, the method further includes updating the virtual bridge scene in real time based on the real-time detection image of the service status of the target bridge, including eliminating the deterioration defects that have been repaired and adding new deterioration defects.

[0019] Preferably, the improved dual-branch generative adversarial network includes a dual-channel generator and a dual-modal discriminator; the dual-channel generator adopts a dual-branch Unet network structure, including a shared encoder and a decoder; the decoder includes an image generation branch decoder and a mask generation branch decoder, which are respectively integrated with a spatial adaptive normalization (SPADE) processing layer; the dual-modal discriminator includes a multi-scale image discriminator and a mask discriminator.

[0020] Preferably, the process of training the improved two-branch generative adversarial network is:

[0021] Step 41: Collect real deterioration disease images to build a real deterioration disease sample library, and perform data association and structured processing on the real deterioration disease images and virtual disease samples to obtain processed real disease masks and virtual disease masks;

[0022] Step 411: Collect and annotate a real deterioration and disease image. Manually annotate the deterioration and disease targets in the real deterioration and disease image or pre-annotate using an existing segmentation model to obtain a real mask image.

[0023] Step 412: performing mask preprocessing: converting the virtual mask image and the real mask image in the virtual disease sample from RGB encoding into a multi-channel binary mask, and obtaining processed virtual disease masks and real disease masks respectively;

[0024] Step 42: Input the virtual defect image and the corresponding processed virtual defect mask into a dual-channel generator to obtain a true image and a true mask of the virtual degraded defect corresponding to the virtual degraded defect image;

[0025] Step 421: The input layer transmits the virtual disease image to the shared encoder;

[0026] Step 422: The shared encoder uses convolution operations to extract underlying common features from the virtual disease image to reduce redundant calculations;

[0027] Step 423: The decoder's image generation branch decoder uses the virtual defect mask as a spatial condition and fuses it with the underlying common features to obtain a true-to-image image. The decoder's mask generation branch decoder uses the virtual defect mask as a spatial condition and fuses it with the underlying common features to obtain a true-to-image mask.

[0028] Step 43: Input the true image and the true mask, the real degraded disease image and the real disease mask into the bimodal discriminator for image discrimination and semantic consistency discrimination, and obtain the true and false discrimination space matrices of the image and mask respectively;

[0029] Step 431: The input layer of the bimodal discriminator inputs the true image and the real degraded disease image to the multi-scale image discriminator, and inputs the true mask and the real disease mask to the mask discriminator;

[0030] Step 432: The multi-scale image discriminator performs multi-level downsampling on the true image and the real degraded disease image, and applies the PatchGAN discriminator to perform discrimination at different scales to improve the discrimination ability of local textures and obtain the true and false discrimination space matrix for each area of ​​the image;

[0031] The mask discriminator uses a convolutional network to determine whether the topological structures of the true-like mask and the real disease mask are consistent, and generates a true and false discrimination space matrix for the mask;

[0032] Step 44: Backpropagation is performed using a comprehensive loss function based on the true / false discrimination space matrix of the image and mask to adjust the parameters of the multi-scale image discriminator and mask discriminator of the dual-channel generator and the dual-modal discriminator;

[0033] Step 45: The finally obtained true image and true mask are combined into a true degradation disease sample, which is then merged into the true degradation disease sample library.

[0034] Preferably, the comprehensive loss function of the dual-channel generator includes adversarial loss, pixel reconstruction loss, and feature matching loss calculated based on feature extraction of the VGG network; the comprehensive loss function of the bimodal discriminator includes real image discrimination loss, real mask discrimination loss, true-like image discrimination loss, and true-like mask discrimination loss.

[0035] Preferably, the improved Unet network includes an encoder, a decoder and a segmentation head; wherein the encoder uses a standard Resnet encoder structure to extract five layers of feature maps of different scales for subsequent jump connection operations with the decoder; the decoder adds an attention mechanism to fuse the feature maps extracted by the corresponding encoder; the segmentation head maps the tensor output by the decoder to the target category number through convolution and outputs a degraded disease area mask; and the weighted cross entropy loss function is used to train the improved Unet network.

[0036] Preferably, the specific process of performing virtual-real mapping on the extracted deterioration disease area mask in step 6 is:

[0037] S61: Importing the extracted image of the deterioration disease area mask into the UE5 engine to generate the corresponding deterioration material;

[0038] S62: Create a deterioration defect plane in the UE5 engine and attach a deterioration material to it. Combined with the latitude and longitude information of the drone's specific location at the time of shooting, which is carried in the image to be inspected, and the distance and angle between the drone and the target bridge at the time, after coordinate conversion, accurately obtain the deterioration defect location in the bridge BIM model corresponding to the deterioration defect area mask in the image to be inspected. Place the deterioration defect plane at the deterioration defect location.

[0039] S63: Rotate and fine-tune the deterioration and disease plane to complete the placement of the deterioration and disease plane.

[0040] Through the above technical solutions, it can be seen that compared with the existing technology, the present invention provides a typical bridge disease characterization method. First, a parameterized model library of typical railway bridge diseases such as cracks (cross cracks, horizontal cracks and vertical cracks), rust and spalling is constructed to form a bridge deterioration disease sample library; secondly, a BIM model of the railway bridge (including the steel beam section) is constructed, and a surrounding environment model is established. The model is integrated in UE5 and UE5 (Unreal Engine) is used to simulate the surrounding environment. 5) uses powerful functions and multi-level modeling technology to generate highly realistic virtual bridge scenes, initialize the camera to obtain multi-angle and multi-distance images of the virtual scene and corresponding semantic annotation information (mask image), and extract high-quality virtual disease samples; thirdly, based on the real deterioration disease images of the actual bridge, combined with the virtual disease samples, the improved two-branch generative adversarial network (GAN) is used to expand the typical disease sample set of railway bridges, realize the domain adaptation of virtual samples to real images, and simultaneously output high-precision refined masks, effectively utilize the existing small number of high-quality real and virtual disease samples, generate more diverse real disease samples and semantic annotations, make up for the problems of small number of real collected data and single type, and reduce the generation The data annotation cost is reduced, further improving the generalization performance of the defect detection model. Next, representation learning is performed based on the improved Unet network's representation learning capability. By extracting multi-dimensional high-level features from the image, the accurate identification and classification of typical railway bridge defects are achieved, the recognition ability of small samples and clear edge targets is improved, and intelligent detection of typical railway bridge defects is achieved. Finally, based on the detection results, a deterioration defect area mask is extracted. The extracted deterioration defect area mask is imported into the Unreal Engine (UE), and the image is used in the engine to create a dedicated deterioration defect material. Based on the latitude and longitude information contained in the original image, it is accurately mapped to the corresponding coordinate position of the bridge modeling system, generating a high-fidelity deterioration defect representation in the corresponding area. The method of the present invention not only significantly enriches the defect sample resources and reduces the cost and time investment of manual collection, but also greatly improves the accuracy and operating efficiency of the detection system, providing strong technical support for ensuring the safe operation and maintenance of railway bridges. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0042] Figure 1 A schematic flow chart of a typical bridge disease characterization method provided by the present invention;

[0043] Figure 2 This is a schematic diagram of the modeling process of the degradation disease model provided by the present invention;

[0044] Figure 3 A schematic diagram of the process of constructing a virtual bridge scene provided by the present invention;

[0045] Figure 4 Schematic diagram of the process of extracting virtual disease samples from the virtual bridge scene provided by the present invention;

[0046] Figure 5 A schematic diagram of the data transmission relationship of the improved dual-branch generative adversarial network provided by the present invention;

[0047] Figure 6 A schematic diagram of the dual-channel generator network structure of the improved dual-branch generative adversarial network provided by the present invention;

[0048] Figure 7 A schematic diagram of the multi-scale image discriminator network structure of the improved dual-branch generative adversarial network provided by the present invention;

[0049] Figure 8 Schematic diagram of the mask discriminator network structure of the improved dual-branch generative adversarial network provided by the present invention;

[0050] Figure 9 A schematic diagram of the training process of the improved Unet network provided by the present invention;

[0051] Figure 10 Schematic diagram of the improved Unet network structure provided by the present invention;

[0052] Figure 11 This is a schematic diagram of extracting deterioration and diseased areas provided by the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] The embodiment of the present invention discloses a method for characterizing typical bridge defects, such as Figure 1 As shown, the following steps are included:

[0055] S1: Collect railway bridge operation and maintenance data, build a parameterized model library of typical railway bridge diseases, and establish a bridge deterioration disease sample library;

[0056] S2: Based on the bridge design drawings and related operation and maintenance information, the family files and mileage information of each subdivided component of the bridge are sorted out. The bridge BIM model is quickly constructed by combining the rapid modeling plug-in with the bridge deterioration disease sample library. Based on the geographical location of the target bridge, the corresponding image map and elevation data are downloaded to establish the surrounding environment model.

[0057] S3: Based on the UE5 engine, the bridge BIM model and the surrounding environment model are integrated to obtain a virtual bridge scene. The camera in the virtual bridge scene is initialized to obtain virtual disease samples.

[0058] S4: Collect real deterioration disease images of bridges to build a real deterioration disease sample library, and use the real deterioration disease sample library and virtual disease samples to train an improved two-branch generative adversarial network. Use the trained improved two-branch generative adversarial network to obtain real deterioration disease samples to expand the real deterioration disease sample library;

[0059] S5: The improved Unet network is trained based on the expanded real deterioration disease sample library. The collected images of the target bridge to be inspected are input into the trained improved Unet network to identify deterioration diseases and extract deterioration disease area masks.

[0060] S6: Map the deterioration and disease area mask onto the bridge BIM model for representation.

[0061] Furthermore, the virtual disease samples include images of the virtual scene at multiple angles and distances and corresponding mask images (semantic annotation information).

[0062] Furthermore, in order to systematically study the characteristics and development patterns of these diseases and provide data support for subsequent diagnosis and repair, a standardized bridge deterioration disease sample library was created; typical diseases of railway bridges include cracks, rust and spalling (chipping); cracks include cross cracks, oblique cracks, horizontal cracks and vertical cracks. The crack information is described in detail through four indicators: shape, length, width and depth, and a corresponding crack family model is created to realize data entry and visualization display; for rust and spalling, area is used as the main descriptive indicator to complete the entry and visualization of relevant information.

[0063] Furthermore, based on relevant data on typical defects, a deterioration defect model was constructed, covering surface defect types such as cracks (classified by direction), corrosion, and spalling. Each type was further subdivided based on planar morphology, severity, and size parameters. The collection of all deterioration defect models constituted a bridge deterioration defect sample library. Deterioration defect models were created using BIM (Building Information Modeling) technology, and the creation of these models can be divided into two scenarios: the first, when there are no specific requirements for deterioration defects, uses splines to define the direction of cracks, the area of ​​corrosion, and the area of ​​chipping, and can be parametrically adjusted for length, width, and depth; the second, based on the specific contour coordinates of cracks, corrosion, and spalling obtained by measurement, point coordinate data is imported through Dynamo, corresponding line contours are generated, and ultimately a corresponding family model is formed. This approach not only improves the efficiency of railway bridge deterioration defect management but also provides accurate data support for subsequent maintenance and repair.

[0064] Furthermore, the specific modeling process of the degradation disease model is as follows: Figure 2 As shown, the specific steps include:

[0065] S11: Select a suitable family template;

[0066] S12: Input relevant data in the parameterized model library, set disease parameters, and create a benchmark;

[0067] When there is no measured data, create a four-way reference plane to form the basic positioning, establish a parameterized system based on the crack type (horizontal / vertical / cross), and set core parameters such as length (L), width (W), and depth (D). Set the area (S) parameter for rust and spalling diseases;

[0068] When there is measured data, Dynamo is used to parse the coordinate point file (CSV / Excel), automatically generate discrete point data, and establish a data-driven association mechanism;

[0069] S13: Set the material and perform material visualization;

[0070] Set the crack filling material to a semi-transparent crack material (50% transparency is recommended), and add a normal map to enhance the three-dimensional effect;

[0071] Set up the rust material, set up a layered material system, including a base layer and an oxide layer, and use PBR material to simulate the metal oxidation effect;

[0072] Set the material of the peeling area, call the exposed aggregate material library, and use the edge damage parameter to control the gradient effect.

[0073] Furthermore, based on the family files and mileage information used for each subdivided component of the target bridge in the bridge design drawings and related operation and maintenance information, a rapid modeling plug-in was used to quickly generate a bridge BIM model. The generated deterioration and disease model was loaded into the bridge project, and the deterioration and disease model was added to the corresponding position of the bridge BIM model, and the .udatasmith format file was exported. The image map and elevation data of the surrounding environment of the target bridge were downloaded from the existing database. The image map and elevation data were exported as raster data in JPG format and dem data in tif format respectively through Global Mapper software. The dem data was then exported as a grayscale image through QGIS. The raster data and grayscale image constituted the surrounding environment model. Twinmotion software was used to perform beautification operations such as greening and material modification, and the .udatasmith format file was exported to build the greening model.

[0074] Furthermore, the process of S2-S3 is as follows Figure 3 As shown in the figure, the specific implementation process of creating a virtual bridge scene in S3 is as follows:

[0075] S311: Open the UE5 engine to create a new level and add a dynamic sky sphere;

[0076] S312: Use the Datasmith plug-in in the UE5 engine to import the BIM model of the bridge with deterioration and the surrounding environment model;

[0077] S313: Export the oblique photography model of the railway bridge section collected in OSGB format into FBX format using OSGBLab software;

[0078] S314: Use the ObliquePhotography plug-in in the UE5 engine to import the oblique photography model of the FBX grid to make the virtual scene more realistic;

[0079] S315: The imported bridge BIM model, greening model, surrounding environment model and oblique photography model are integrated to form a virtual bridge scene (digital twin base).

[0080] Furthermore, the bridge BIM model in the virtual bridge scenario is updated in real time based on the target bridge's service status. Based on real-time inspection images of the target bridge's service status, repaired deterioration defects in the virtual bridge scenario are promptly eliminated. For newly added deterioration defects (such as cracks, spalling, and rust), a new deterioration defect model is created in real time in the virtual bridge scenario through S5-S6, and the relevant information is recorded based on the corresponding IoT sensor data and on-site inspection information, as well as information extracted using image recognition technology. This dynamic update and synchronization mechanism ensures a high degree of consistency and accuracy between the virtual and physical worlds, providing reliable data support for the full lifecycle management and condition assessment of bridges.

[0081] Furthermore, the specific steps for updating the virtual bridge scene in real time include:

[0082] S316: Create cracked, rusted, and peeling blueprint actors in the UE5 engine;

[0083] S317: Use the GeoReferencing plug-in. Drag the GeoReferencingSystem into the project from the Place Actor control panel. Set the Planet Shape, Projected CRS, Geographic CRS, and the latitude and longitude coordinates of the zero point to complete the registration point coordinate settings.

[0084] S318: Adding new defects or removing deterioration defects on the virtual bridge scene by inputting the latitude and longitude coordinates of the registration points;

[0085] S319: After creating a deterioration defect, you need to input some IoT sensor data and on-site inspection information, as well as defect information extracted using image recognition technology. Create a widget to record the input defect information. Taking cracks as an example, the defect information includes the length, width, height of the crack, the inspection personnel and inspection time, or the collected defect image.

[0086] S3110: Save the input damage information and the created deterioration damage. When the created virtual bridge scene project is running, all the related deterioration damage models added or deleted last time and the related information corresponding to the models are loaded when the scene is opened for viewing.

[0087] Furthermore, in S3, three types of deterioration disease models, namely cracks, rust, and spalling, are rendered in layers in the virtual bridge scene and distinguished by different colors, which is helpful for the segmentation and expansion of virtual disease samples in the later stage. The process of extracting virtual disease samples in the virtual bridge scene is as follows: Figure 4 As shown, specifically:

[0088] S321: Search for motion blur in the UE5 engine's project settings and turn it off; search for Custom and change the custom depth template to "Enable Template";

[0089] S322: Drag the "Post Process Volume" component into the scene and set its range to "Infinite";

[0090] S323: Modify the CustomStencil material in the UE5 engine; use the mask function to mask the R / G / B channels respectively, define the custom channel template value, use if to make a judgment, and realize layered rendering;

[0091] S324: Select all deterioration disease models corresponding to cracks, rust, and spalling, set the custom depth template values ​​to 1, 2, and 3 respectively, and set the custom depth template value of the beam to 0. This way, the diseases on both sides of the beam will not interfere with each other during layered rendering.

[0092] The custom depth template value for cracks is 1, representing red; the custom depth template value for rust is 2, representing green; the custom depth template value for spalling is 3, representing blue; and the custom depth template value for beams is 0, representing black.

[0093] S325: Drag the movie camera actor in the UE5 engine into the virtual bridge scene, adjust its height and rotation angle, aim the camera at the location of the deterioration defect, and adjust the camera's aperture, focal length, and focus distance so that the deterioration defect is clearly displayed in the camera;

[0094] S326: Create a level sequence, drag the initialized camera into it, set the camera's aperture, focal length, focus distance, and position information. Each time the camera moves, add a keyframe at an appropriate interval in the level sequence.

[0095] S327: Output the level sequence and set related parameters; output the image in the Custom Render Passes format and select two rendering channels: the custom template and the final image;

[0096] S328: Customize the template image and final image, and output the virtual disease sample.

[0097] Furthermore, the improved dual-branch generative adversarial network (improved GAN) includes a dual-channel generator and a dual-modal discriminator; the dual-channel generator uses a dual-branch Unet network as the basic network structure, and is composed of a shared encoder and decoder. The decoder includes an image generation branch decoder and a mask generation branch decoder that are integrated with the spatial adaptive normalization (SPADE) processing layer; the dual-modal discriminator includes a multi-scale image discriminator and a mask discriminator. The specific structure of the multi-scale image discriminator is as follows: Figure 7 As shown in the figure, the specific structure of the mask discriminator is as follows Figure 8 As shown. The improved GAN data transmission relationship is as follows Figure 5 As shown in the figure, a dual-channel generator is designed to simultaneously generate degraded images and their semantically annotated masks, and a dual-modal discriminator is designed to simultaneously distinguish the authenticity of images and the rationality of masks. Domain adaptation of virtual data and real data is achieved through cross-domain alignment of feature spaces, and high-precision masks are generated synchronously to form true-to-life degraded disease samples, thereby expanding the existing real-to-life degraded disease sample library data.

[0098] Furthermore, the process of improving GAN training is as follows:

[0099] S41: Collecting real deterioration disease images, and performing data association (matching real data with virtual data) and structural processing on the real deterioration disease images and virtual disease samples;

[0100] S411: Collecting and labeling a real deterioration and disease image, manually labeling the deterioration and disease targets in the real deterioration and disease image or pre-labeling using an existing segmentation model to obtain a real mask image;

[0101] S412: performing mask preprocessing: converting the virtual mask image and the real mask image corresponding to the virtual disease sample from RGB encoding into a multi-channel binary mask (such as background, cracks, rust, and peeling) to obtain processed virtual disease masks and real disease masks;

[0102] S42: Inputting the virtual disease image and the processed virtual disease mask into a dual-channel generator to obtain a true image and a true mask corresponding to the virtual degraded disease image;

[0103] S421: The input layer transmits the virtual disease image to the shared encoder;

[0104] S422: The shared encoder uses convolution operations to extract underlying common features from the virtual disease image to reduce redundant calculations;

[0105] S423: The decoder's image generation branch decoder uses the virtual disease mask as a spatial condition and fuses it with the underlying common features to obtain a true-like image; the decoder's mask generation branch decoder uses the virtual disease mask as a spatial condition and fuses it with the underlying common features to obtain a true-like mask;

[0106] S43: Inputting the true-like image and true-like mask obtained by the virtual degraded disease image and mask through the dual-channel generator, and the collected real degraded disease image and real disease mask into the dual-modal discriminator for image discrimination and semantic consistency discrimination, and obtaining the true and false discrimination space matrices of the image and mask respectively;

[0107] S431: The input layer of the bimodal discriminator inputs the true image and the real degraded disease image into the multi-scale image discriminator, and inputs the true mask and the real disease mask into the mask discriminator;

[0108] S432: The multi-scale image discriminator performs multi-level downsampling on the true-looking image and the real degraded disease image. The PatchGAN discriminator is applied to distinguish at different scales to improve the ability to distinguish local textures and obtain the true and false discrimination space matrix for each area of ​​the image.

[0109] The mask discriminator uses a convolutional network to determine whether the topological structures of the true-like mask and the real disease mask are consistent, and generates a true and false discrimination space matrix for the mask;

[0110] S44: Based on the true and false discrimination space matrices of the multi-scale image discriminator and the mask discriminator, a comprehensive loss function is used for back propagation to adjust the parameters of the multi-scale image discriminator and the mask discriminator of the dual-channel generator and the bimodal discriminator;

[0111] S45: The finally obtained true-like image and true-like mask are combined into a true-like degradation disease sample, which is then merged into a true degradation disease sample library.

[0112] The constructed virtual damage images and masks are combined with the real deterioration damage images and the corresponding real masks. Through a dual-branch generative adversarial network plus a multi-task loss function, domain adaptation from virtual damage images to real deterioration damage images is achieved, and high-precision true-like masks are output simultaneously, realizing the expansion and automatic labeling of samples in the bridge deterioration damage sample library of typical railway bridge diseases.

[0113] Further, such as Figure 6 As shown in the figure, the shared encoder performs feature extraction, including four groups of feature extraction modules and two groups of convolutional layers Conv connected in sequence. Each group of feature extraction modules includes a convolutional layer Conv, an activation function Relu, a convolutional layer Conv, an activation function Relu and a maximum pooling layer MaxPool connected in sequence. The maximum pooling layer of each group of feature extraction modules is connected to the convolutional layer of the next group of feature extraction modules; the second activation function of each group of feature extraction modules and the second convolutional layer of the shared encoder are connected to the decoder.

[0114] Further, such as Figure 6As shown in the figure, the image generation branch decoder in the decoder includes four groups of image generation modules and a group of convolution layers Conv connected in sequence, and outputs the generated image as a true image; each group of image generation modules includes a deconvolution layer ConvTranspose2d, a spatial adaptive normalization processing layer SPADE, a fusion layer Concat, a convolution layer Conv, an activation function Relu, a convolution layer Conv, and an activation function Relu connected in sequence. The activation function Relu of each group of image generation modules is connected to the deconvolution layer of the next group of image generation modules; the second activation function of each group of feature extraction modules is connected to the activation function of a corresponding image generation module. The fusion layer and the second convolution layer of the shared encoder are connected to the deconvolution layer of the first image generation module; the virtual degraded disease mask is input to the spatial adaptive normalization processing layer; the mask generation branch decoder in the decoder includes four groups of mask generation modules and a group of convolution layers connected in sequence, and outputs the generated mask as a true mask; the structure of each group of mask generation modules is the same as that of the image generation module, and the connection relationship of the four groups of mask generation modules is the same as that of the image generation module; the second activation function of each group of feature extraction modules is connected to the fusion layer of a corresponding mask generation module; the virtual degraded disease mask is input to the spatial adaptive normalization processing layer of the mask generation module.

[0115] Furthermore, the comprehensive loss function of the dual-channel generator includes adversarial loss (image adversarial loss, mask adversarial loss), pixel reconstruction loss, and feature matching loss calculated based on feature extraction from the VGG network. Hybrid training is used during training to balance the contributions of virtual and real data. The comprehensive loss function of the bimodal discriminator includes real image discrimination loss, real mask discrimination loss, true-like image discrimination loss, and true-like mask discrimination loss. The discriminator parameters are adjusted through backpropagation by calculating the loss value between the virtual diseased samples and the collected real degraded diseased images. The specific loss function is as follows:

[0116] Comprehensive loss function for dual-channel generator Expressed as:

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123] in, represents the image adversarial loss; represents the mask adversarial loss; represents pixel reconstruction loss; represents the mask Dice loss; represents the perceptual loss based on vgg; 、 、 and Respectively represent weight coefficients; G represents a dual-channel generator; Represents a multi-scale image discriminator; True represents the true value in the true and false discrimination space matrix; represents the mask discriminator; x represents the real degraded disease image; m represents the real disease mask; represents a virtual disease image in a virtual disease sample; Indicates the virtual disease mask corresponding to the virtual disease image; represents mse loss; Indicates the The weight coefficient of the layer; Indicates the The VGG network of the layer is used to extract the features of the real-like image and the real degraded disease image; the image adversarial loss is calculated using the image's true and false discrimination space matrix, and the mask adversarial loss is calculated using the mask's true and false discrimination space matrix;

[0124] Comprehensive loss function of bimodal discriminator Expressed as:

[0125]

[0126]

[0127]

[0128]

[0129]

[0130] in, represents the real image discrimination loss; represents the real mask discrimination loss; represents the real image discrimination loss; Represents the class-true mask discrimination loss; False represents the false value in the true and false discrimination space matrix; the real image discrimination loss and the class-true image discrimination loss are calculated using the image’s true and false discrimination space matrix, and the real mask discrimination loss and the class-true mask discrimination loss are calculated using the mask’s true and false discrimination space matrix.

[0131] Furthermore, a two-channel generator is used to generate a realistic mask of the degraded diseased image, which enables automatic refinement of semantic information. At the same time, the minimum enclosing rectangle of the mask is calculated to automatically generate the degraded bounding box to complete the target labeling.

[0132] Furthermore, in order to adapt to the extraction of disease features of different sizes, the traditional Unet network is improved by combining multi-scale and attention mechanisms, and an improved Unet network is constructed, such as Figure 10 As shown in the figure, the improved Unet network consists of an encoder, a decoder, and a segmentation head. The encoder uses the standard Resnet encoder structure to extract five layers of feature maps of different scales for subsequent jump connection operations with the decoder; the decoder adds an attention mechanism to fuse the feature maps extracted by the corresponding encoder, focus on important areas and features, and suppress irrelevant background noise; the segmentation head maps the tensor output by the decoder to the number of target categories through convolution operations, and outputs a degraded disease mask; the weighted cross entropy loss function is further used to train the improved Unet network. The training process of the improved Unet network is as follows: Figure 9 shown.

[0133] Furthermore, the training process of the improved Unet network is as follows:

[0134] S51: Data preprocessing and annotation conversion;

[0135] S511: Multi-source data fusion, expanded real degradation disease sample library (including real degradation disease images and real mask images collected during real inspections, and the real-like images and masks generated by the improved GAN in S4 as expanded real degradation disease samples);

[0136] S512: Data enhancement: Perform online enhancement on sample data (degraded images and masks) through random rotation, translation, Gaussian noise, and illumination simulation to ensure synchronization of annotations.

[0137] S513: Color coding conversion, converting the colors of the true mask image in S411 and the true-like mask generated by the dual-channel generator in S423 into category labels, for example: red cracks correspond to category 1, green rust corresponds to category 2, blue peeling corresponds to category 3, and other background areas correspond to category 0;

[0138] S52: Model training;

[0139] S521: The real deterioration disease images from real inspections and the deterioration disease images (real-like images) generated by the improved GAN are used as the input data of the model. The masks corresponding to the deterioration disease images are used as the output data of the model to train the improved Unet network.

[0140] S522: The improved Unet network uses the standard Resnet encoder structure to extract 5 different scale feature maps to achieve multi-scale feature extraction operations; the decoder fuses the features extracted by the corresponding encoder through upsampling, and simultaneously adds an attention mechanism to focus on important areas and features; the segmentation head maps the decoder output to the target category number through convolution operations to achieve segmentation function.

[0141] Furthermore, combined with the sample category distribution, a weighted cross entropy loss function is constructed to assign higher weights to different degradations and relatively small weights to the background; weighted cross entropy loss function as follows:

[0142]

[0143] Where N represents the number of pixels; C represents the number of categories; Indicates a label; represents the predicted class probability; Represents the weight of category c, which is inversely proportional to the category frequency.

[0144] Furthermore, after the training of the improved Unet network is completed, the degradation detection is performed on the collected image to be detected, and the degradation disease mask of the image to be detected, that is, the segmentation image, is obtained. The degradation disease area in the image to be detected is extracted according to the color value of the degradation disease area in the mask, and the extracted image of the degradation disease area is obtained as shown in FIG. Figure 11 shown.

[0145] Furthermore, the specific process of performing virtual-real mapping on the extracted deterioration and diseased area in S6 is as follows:

[0146] S61: Importing the extracted images of the deteriorated diseased area into the UE5 engine, and subsequently using these images to generate exclusive deteriorated materials;

[0147] S62: Create a plane in the UE5 engine and attach a deterioration material to it. Combined with the latitude and longitude information of the drone's specific location at the time of inspection, carried in the image taken by the drone, and the distance and angle between the drone and the bridge at the time, after coordinate conversion, the deterioration damage location is accurately obtained, and the deterioration damage plane is placed. The plane serves as the carrier of the deterioration material.

[0148] S63: Perform necessary and reasonable rotation and fine-tuning operations on the deterioration and disease plane to ensure high-precision mapping between the physical world and the digital virtual world and maintain a high degree of consistency.

[0149] This paper aims at typical defects of railway bridges, such as cracks, falling blocks, and rust, and constructs a parameterized model library for degradation characterization through geometric characteristic parameters such as length, width, and area. Based on the UE (Unreal Engine) platform, by integrating the geographic coordinate data of the defects detected in reality and converting the coordinates, the corresponding defect models are dynamically generated in the corresponding areas, realizing the intelligent mapping and visualization display between the physical deterioration entity of the railway bridge and the digital twin. A dual-branch generative adversarial network is proposed to construct a multi-task loss function to achieve domain adaptation from virtual samples to real images, and simultaneously generate high-precision mask images, automatically refine the semantic masks of the labeled samples, and form a realistic degradation defect sample library to make up for the problems of insufficient real collected image data and high cost of manual labeling. An improved method combining multi-scale and attention mechanism is proposed. The Unet network is used to extract small samples and deterioration targets with clear edges, improve detection accuracy, and realize intelligent and accurate detection of typical bridge defects. It further realizes automatic extraction of deterioration areas based on semantic annotation of deterioration defect areas. It can realize two intelligent representations of typical bridge defects: one is to automatically add parameterized defects and manually fine-tune them to achieve similarity in appearance; the second is to automatically extract deterioration defects (location, shape, type, etc.). By making deterioration defect materials and converting the defect coordinates based on the longitude and latitude of the original detection image, it is automatically mapped to the corresponding area of ​​the twin, achieving both similarity in appearance and spirit.

[0150] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0151] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for characterizing typical bridge defects, characterized in that: The following steps are involved: Step 1: Collect bridge operation and maintenance data, build a parameterized model library of typical bridge defects, and establish a sample library of bridge deterioration defects; Step 2: Based on the design drawings and relevant operation and maintenance information of the target bridge, a BIM model of the target bridge is constructed using the rapid modeling plug-in combined with the bridge deterioration disease sample library. Based on the geographical location of the target bridge, corresponding imagery and elevation data are downloaded to construct a surrounding environment model. Step 3: Based on the UE5 engine, the bridge BIM model and the surrounding environment model are integrated to obtain a virtual bridge scene, and the camera in the virtual bridge scene is initialized to obtain virtual disease samples; Step 4: Collect real degradation disease images to build a real degradation disease sample library, and use the real degradation disease sample library and virtual disease samples to train an improved two-branch generative adversarial network. Use the trained improved two-branch generative adversarial network to obtain real degradation disease samples and expand the real degradation disease sample library; The improved dual-branch generative adversarial network includes a dual-channel generator and a bimodal discriminator; The dual-channel generator adopts a dual-branch Unet network structure, including a shared encoder and decoder; the decoder includes an image generation branch decoder and a mask generation branch decoder, each of which incorporates a spatially adaptive normalization processing layer; the dual-modal discriminator includes a multi-scale image discriminator and a mask discriminator; Step 5: Train the improved Unet network based on the expanded real deterioration disease sample library. Input the collected images of the target bridge to be inspected into the trained improved Unet network to identify deterioration diseases and extract the deterioration disease area mask. Improved Unet network including encoder, decoder and segmentation head; The encoder uses the Resnet encoder structure to extract five layers of feature maps of different scales for skip connection operations with the decoder; The decoder adds an attention mechanism to fuse the feature maps extracted by the corresponding encoder; The segmentation head maps the tensor output by the decoder to the number of target categories through convolution operations and outputs the mask of the degraded diseased area; Step 6: Map the deterioration and disease area mask onto the bridge BIM model for characterization.

2. A typical bridge disease characterization method according to claim 1, characterized in that: Typical bridge defects include cracks, corrosion, and spalling. The bridge deterioration defect sample library includes deterioration defect models for cracks, corrosion, and spalling with different plane shapes, severity, and size parameters. When there are no specific requirements for the deterioration defect model, splines are used to define the direction of cracks, the area of ​​corrosion and spalling, and the length, width, and depth are parametrically adjusted. When constructing a deterioration defect model based on actual measurement data, the specific contour coordinates of cracks, corrosion, and spalling are imported into point coordinate data through Dynamo to generate the corresponding line contours, forming the corresponding deterioration defect model.

3. A typical bridge disease characterization method according to claim 2, characterized in that: Based on the family files and mileage information used for each subdivided component of the target bridge in the design drawings and related operation and maintenance information, a rapid modeling plug-in is used to generate a bridge BIM model. The degradation disease model in the bridge degradation disease sample library is loaded into the bridge BIM model. Download the image map and elevation data of the target bridge's surrounding environment, export the image map into raster data in JPG format using GlobalMapper software, and export the elevation data into DEM data in TIF format using GlobalMapper software. The DEM data is then exported into a grayscale image using QGIS. The raster data and grayscale image constitute the surrounding environment model. Use Twinmotion software to perform landscaping and modify material beautification operations to build a greening model.

4. A typical bridge disease characterization method according to claim 3, characterized in that: Import the bridge BIM model, greening model and surrounding environment model into the UE5 engine, and integrate the oblique photography model to form a virtual bridge scene.

5. A typical bridge disease characterization method according to claim 1, characterized in that: Based on the real-time detection images of the service status of the target bridge, the virtual bridge scene is updated in real time, including eliminating the deterioration defects that have been repaired and adding new deterioration defects.

6. A typical bridge disease characterization method according to claim 1, characterized in that: The training process of the improved two-branch generative adversarial network is as follows: Step 41: Collect real deterioration disease images to build a real deterioration disease sample library, and perform data association and structured processing on the real deterioration disease images and virtual disease samples to obtain processed real disease masks and virtual disease masks; Step 411: labeling the real deterioration disease image to obtain a real mask image; Step 412: converting the virtual mask image and the real mask image in the virtual disease sample from RGB encoding into a multi-channel binary mask to obtain a processed virtual disease mask and a real disease mask respectively; Step 42: Input the processed virtual disease mask and the virtual disease image in the corresponding virtual disease sample into a dual-channel generator to obtain a true image and a true mask of the virtual degraded disease; Step 421: The input layer transmits the virtual disease image to the shared encoder; Step 422: The shared encoder extracts underlying common features from the virtual disease image based on a convolution operation; Step 423: The decoder's image generation branch uses the virtual disease mask as a spatial condition and fuses it with the underlying common features to obtain a true-like image. The decoder's mask generation branch decoder uses the virtual disease mask as a spatial condition and fuses it with the underlying common features to obtain a true-like mask; Step 43: Input the true image, the true mask, the true degraded disease image, and the true disease mask into the bimodal discriminator for image discrimination and semantic consistency discrimination, and obtain the true and false discrimination space matrices of the image and mask respectively; Step 431: The input layer of the bimodal discriminator inputs the true image and the real degraded disease image to the multi-scale image discriminator, and inputs the true mask and the real disease mask to the mask discriminator; Step 432: The multi-scale image discriminator performs multi-level downsampling on the true image and the real degraded disease image, and applies the PatchGAN discriminator to perform discrimination at different scales to obtain the true and false discrimination space matrix for each region of the image; The mask discriminator uses a convolutional network to determine whether the topological structures of the true-like mask and the real disease mask are consistent, and generates a true and false discrimination space matrix for the mask; Step 44: Backpropagate using a comprehensive loss function based on the true / false discrimination space matrix of the image and mask to adjust the parameters of the dual-channel generator and the bimodal discriminator. Step 45: The finally obtained true image and true mask are combined into a true degradation disease sample, which is then merged into the true degradation disease sample library.

7. A typical bridge disease characterization method according to claim 6, characterized in that: The comprehensive loss function of the dual-channel generator includes adversarial loss, pixel reconstruction loss, and feature matching loss calculated based on the features extracted by the VGG network; the comprehensive loss function of the bimodal discriminator includes real image discrimination loss, real mask discrimination loss, class-real image discrimination loss, and class-real mask discrimination loss.

8. A typical bridge disease characterization method according to claim 1, characterized in that: In step 5, the weighted cross entropy loss function is used to train the improved Unet network.

9. A typical bridge disease characterization method according to claim 1, characterized in that: The specific process of performing virtual-real mapping on the extracted deterioration disease area mask in step 6 is as follows: S61: Importing the extracted image of the deterioration disease area mask into the UE5 engine to generate the corresponding deterioration material; S62: Create a deterioration defect plane in the UE5 engine and attach a deterioration material to it. Combined with the latitude and longitude information of the drone's specific location at the time of shooting, carried in the image to be inspected, and using the distance and angle between the drone and the target bridge at the time, after coordinate conversion, the corresponding deterioration defect location in the bridge BIM model is obtained. The deterioration defect plane is then placed at the deterioration defect location. S63: Rotate and fine-tune the deterioration and disease plane.

Images