Typical bridge disease characterization method

Abstract

The invention discloses a typical bridge disease characterization method, and relates to the technical field of bridge detection, and the method comprises the steps: firstly constructing a degraded disease sample library, and building a surrounding environment model and a bridge BIM model; secondly, constructing a virtual bridge scene by using a UE5 engine, and obtaining a virtual disease sample; performing data enhancement by adopting an improved double-branch generative adversarial network in combination with real and virtual samples, generating a similar real degradation sample, and expanding real sample data; carrying out disease area identification and extraction on the detection image by using an improved Unet network trained by using real sample data; and finally, mapping an identification result to a BIM model to realize three-dimensional visual representation. According to the method, the problem of insufficient disease samples is solved through the generative adversarial network, the fine-grained disease recognition precision is improved by utilizing the improved Unet, full-process digital representation of bridge diseases from data acquisition, intelligent recognition to three-dimensional dynamic display is realized, and the accuracy and visualization effect of disease detection are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge detection, and more specifically, to a method for characterizing typical diseases of bridges. Background Art

[0002] With the rapid development of the railway transportation network, railway bridges, as important transportation hub facilities, bear huge traffic pressure. Especially with the rapid development of high-speed railways and heavy-haul railways, higher requirements are put forward for bridge structures. However, long-term use has led to the gradual deterioration of bridge structure materials, and problems such as cracks, rust, and spalling occur frequently. These diseases will directly threaten the safety of bridges, possibly leading to a decline in structural strength and even triggering safety accidents. In the field of railway bridge health monitoring, the intelligent characterization technology of typical diseases (such as cracks, rust, and spalling) is one of the key technologies to ensure the safe operation of bridges. For the characterization of typical diseases of railway bridges (such as cracks, spalling, rust, etc.), traditional methods mainly rely on manual collection and image analysis, and there are problems such as limited sample collection, high cost, insufficient model generalization ability, and unclear visualization effect. In recent years, with the development of artificial intelligence technology, methods based on deep learning have gradually become a research hotspot, and the main technologies include image segmentation, object detection, and classification, etc. For example, convolutional neural networks (CNNs) are used for image feature extraction, combined with models such as Region-based Convolutional Neural Networks (R-CNNs) or YOLO to achieve disease location and recognition; at the same time, some studies have also adopted transfer learning and data augmentation techniques to improve the adaptability of the model to different environments and lighting conditions. In addition, methods based on representation learning can better distinguish different types of diseases such as cracks, rust, and spalling by extracting high-level features of diseases.

[0003] However, there are still some limitations in the existing technologies: First, traditional image processing methods perform poorly in complex backgrounds or small-sample data; second, deep learning models are highly dependent on labeled data and are difficult to effectively solve problems such as overlapping disease regions and blurred boundaries; finally, due to the particularity of the railway bridge environment (such as changing perspectives, uneven lighting conditions, etc.), the generalization ability and robustness of existing models still need to be further improved.

[0004] Therefore, how to improve the accuracy of bridge disease recognition and achieve accurate characterization is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method for characterizing typical diseases of bridges, constructs a deterioration disease library containing typical diseases (cracks, rust, spalling), and uses an improved Generative Adversarial Network (GAN) to generate high-quality realistic deterioration disease samples, synchronously generates high-precision annotation masks, expands the deterioration diseases, and effectively alleviates the problems of data singularity, scarcity, and high annotation cost. In addition, by introducing an image recognition method based on representation learning, disease features are more accurately extracted and high-precision recognition is achieved. This method not only improves the generalization ability and adaptability of the model, but also provides a new technical path for the intelligent monitoring of bridges.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for characterizing typical diseases of bridges, comprising the following steps:

[0008] Step 1: Collect bridge operation and maintenance data, construct a parameterized model library of typical diseases of bridges, and establish a bridge deterioration disease sample library;

[0009] Step 2: According to the design drawings of the target bridge and relevant operation and maintenance information, sort out the family files and mileage information of each subdivision component of the bridge, quickly construct a bridge BIM model through a rapid modeling plug-in in combination with the bridge deterioration disease sample library, and download the corresponding image maps and elevation data according to the geographical location of the target bridge to establish a surrounding environment model;

[0010] Step 3: Based on the UE5 engine, fuse the bridge BIM model and the surrounding environment model to obtain a virtual bridge scene, and initialize the camera in the virtual bridge scene to obtain virtual disease samples;

[0011] Step 4: Collect real deterioration disease images of existing bridges and the target bridge to construct a real deterioration disease sample library, and use the real deterioration disease sample library and virtual disease samples to train an improved dual-branch generative adversarial network, and use the trained improved dual-branch generative adversarial network to obtain realistic deterioration disease samples to expand the real deterioration disease sample library;

[0012] Step 5: Train the improved Unet network according to the expanded real deterioration disease sample library, input the to-be-detected image of the target bridge collected into the trained improved Unet network for deterioration disease recognition, and extract the deterioration disease area mask;

[0013] Step 6: Map the deterioration disease area mask on the bridge BIM model for characterization.

[0014] Preferably, the typical diseases of bridges include cracks, corrosion and spalling; cracks include cross cracks, diagonal cracks, horizontal cracks and vertical cracks; the crack information is described in detail through four indicators: shape, length, width and depth, and the corresponding crack family models are created to realize data entry and visual display; for corrosion and spalling, the area is used as the main description index to complete the entry and visualization of relevant information.

[0015] Preferably, the bridge deterioration disease sample library includes deterioration disease models covering apparent disease types such as cracks (classified by orientation), corrosion, and spalling, and each type is subdivided based on planar morphology, severity, and dimensional parameters. According to the typical disease data recorded in the parametric model library, the deterioration disease model corresponding to each disease is created based on the BIM platform; when there are no specific requirements for the deterioration disease model, spline curves are used to define the orientation of cracks, and the areas of corrosion and spalling, and the length, width, and depth can be parametrically adjusted; when it is necessary to construct a deterioration disease model based on actual measurement data, the specific contour coordinates of cracks, corrosion, and spalling are imported into point coordinate data through Dynamo to generate the corresponding line contours, forming the corresponding deterioration disease model.

[0016] Preferably, according to the family files and mileage information of each subdivision component of the target bridge in the bridge design drawings and relevant operation and maintenance information, a bridge BIM model is generated using a rapid modeling plug-in, and the deterioration disease models in the bridge deterioration disease sample library are loaded into the bridge BIM model; the image map and elevation data of the surrounding environment of the target bridge are downloaded, the image map is exported as raster data in JPG format through Global Mapper software, the elevation data is exported as dem data in tif format through Global Mapper software, and the dem data is exported as a grayscale map through QGIS. The raster data and the grayscale map constitute the surrounding environment model; Twinmotion software is used for beautification operations such as greening and modifying materials to construct a greening model.

[0017] Preferably, the bridge BIM model, the greening model and the surrounding environment model are imported into the UE5 engine and fused with the oblique photography model to form a virtual bridge scene.

[0018] Preferably, it also includes real-time updating of the virtual bridge scene according to the real-time detection images of the service status of the target bridge, including eliminating the deteriorated diseases that have been renovated and adding newly emerged deteriorated diseases.

[0019] Preferably, the improved dual-branch generative adversarial network includes a dual-channel generator and a dual-modal discriminator; the dual-channel generator adopts a dual-branch Unet network structure, including a shared encoder and a decoder; the decoder includes an image generation branch decoder and a mask generation branch decoder, which respectively incorporate spatial adaptive normalization (SPADE) processing layers; the dual-modal discriminator includes a multi-scale image discriminator and a mask discriminator.

[0020] Preferably, the process of training the improved dual-branch generative adversarial network is as follows:

[0021] Step 41: Collect real deteriorated disease images to construct a real deteriorated disease sample library, and perform data association and structuring processing on the real deteriorated disease images and virtual disease samples to obtain processed real disease masks and virtual disease masks;

[0022] Step 411: Collect real deteriorated disease images and perform annotation. Manually annotate the deteriorated disease targets in the real deteriorated disease images or use an existing segmentation model for pre-annotation to obtain real mask images;

[0023] Step 412: Perform mask preprocessing: Convert the virtual mask images and real mask images in the virtual disease samples from RGB encoding to multi-channel binary masks to obtain processed virtual disease masks and real disease masks respectively;

[0024] Step 42: Input the virtual disease images and the corresponding processed virtual disease masks into the dual-channel generator to obtain a pseudo-real image and a pseudo-real mask corresponding to the virtual deteriorated disease image;

[0025] Step 421: The input layer transmits the virtual disease images to the shared encoder;

[0026] Step 422: The shared encoder extracts underlying common features from the virtual disease images using convolution operations to reduce redundant calculations;

[0027] Step 423: The image generation branch decoder of the decoder uses the virtual disease mask as a spatial condition and fuses it with the underlying common features to obtain a pseudo-real image; the mask generation branch decoder of the decoder uses the virtual disease mask as a spatial condition and fuses it with the underlying common features to obtain a pseudo-real mask;

[0028] Step 43: Input the pseudo-real image, the pseudo-real mask, the real deteriorated disease image, and the real disease mask into the dual-modal discriminator for image discrimination and semantic consistency discrimination to obtain true / false discrimination space matrices for the images and masks respectively;

[0029] Step 431: The input layer of the bimodal discriminator inputs the class-true image and the real deteriorated disease image into the multi-scale image discriminator, and inputs the class-true mask and the real disease mask into the mask discriminator;

[0030] Step 432: The multi-scale image discriminator performs multi-level downsampling on the class-true image and the real deteriorated disease image, applies the PatchGAN discriminator for discrimination at different scales to enhance the discrimination ability for local textures, and obtains the true / false discrimination space matrix for each region of the image;

[0031] The mask discriminator judges whether the topological structures of the class-true mask and the real disease mask are consistent through a convolutional network, and generates the true / false discrimination space matrix of the mask;

[0032] Step 44: According to the true / false discrimination space matrices of the image and the mask, perform backpropagation using the comprehensive loss function, and adjust the parameters of the multi-scale image discriminator and the mask discriminator of the dual-channel generator and the bimodal discriminator;

[0033] Step 45: Combine the finally obtained class-true image and class-true mask to form a class-true deteriorated disease sample, and merge it into the real deteriorated disease sample library.

[0034] Preferably, the comprehensive loss function of the dual-channel generator includes adversarial loss, pixel reconstruction loss, and feature matching loss calculated by extracting features based on the vgg network; the comprehensive loss function of the bimodal discriminator includes real image discrimination loss, real mask discrimination loss, class-true image discrimination loss, and class-true mask discrimination loss.

[0035] Preferably, the improved Unet network includes an encoder, a decoder, and a segmentation head; among them, the encoder uses the standard Resnet encoder structure to extract feature maps of five different scales for subsequent skip connection operations with the decoder; the decoder adds an attention mechanism to fuse the feature maps extracted by the corresponding encoder; the segmentation head maps the tensor output by the decoder to the number of target categories through convolution, and outputs the deteriorated disease area mask; the improved Unet network is trained using the weighted cross-entropy loss function.

[0036] Preferably, the specific process of performing virtual-real mapping on the extracted deteriorated disease area mask in step 6 is as follows:

[0037] S61: Import the image of the extracted deteriorated disease area mask into the UE5 engine to generate the corresponding deteriorated material;

[0038] S62: Create a deterioration disease plane in the UE5 engine, attach a deterioration material to the deterioration disease plane, and combine the longitude and latitude information of the specific position of the drone at the time of shooting carried in the image to be detected taken by the drone. Through the distance and angle between the drone and the target bridge at that time, after coordinate conversion, accurately obtain the position of the deterioration disease in the BIM model of the bridge corresponding to the mask of the deterioration disease area in the image to be detected, and place the deterioration disease plane at the position of the deterioration disease;

[0039] S63: Perform rotation and fine-tuning operations on the deterioration disease plane to complete the placement of the deterioration disease plane.

[0040] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a method for characterizing typical diseases of bridges. First, a parametric model library of typical diseases of railway bridges such as cracks (cross cracks, horizontal cracks, and vertical cracks), rust, and spalling is constructed to form a bridge deterioration disease sample library; secondly, a BIM model of a railway bridge (including a steel beam section) is constructed, and a surrounding environment model is established, and the models are fused in UE5. Using the powerful functions and multi-level modeling technology of UE5 (Unreal Engine 5), a highly realistic virtual bridge scene is generated, and the camera is initialized to obtain images and corresponding semantic annotation information (mask images) at multiple angles and distances in the virtual scene, and high-quality virtual disease samples are extracted; thirdly, according to the real deterioration disease images of the actual bridge, combined with the virtual disease samples, an improved dual-branch generative adversarial network (GAN) is used to expand the typical disease sample set of railway bridges, realize the domain adaptation from virtual samples to real images, and synchronously output a high-precision refined mask, effectively utilize a small amount of existing high-quality real and virtual disease samples, generate more diverse realistic disease samples and semantic annotations, make up for the problems of small quantity and single type of real collected data, and at the same time reduce the cost of generating data annotation, and further improve the generalization performance of the disease detection model; then, based on the representation learning ability of the improved Unet network, perform representation learning, and realize the accurate recognition and classification of typical diseases of railway bridges by extracting multi-dimensional high-level features in the image, improve the recognition ability of small samples and clear-edge targets, and realize the intelligent detection of typical diseases of railway bridges; finally, extract the mask of the deterioration disease area according to the detection result, import the extracted mask of the deterioration disease area into the Unreal Engine (UE), apply an image in the engine to create a dedicated deterioration disease material, and according to the longitude and latitude information contained in the original image, accurately map it to the corresponding coordinate position of the bridge modeling system, and generate a high-fidelity deterioration disease performance in the corresponding area. The method of the present invention not only significantly enriches the disease sample resources, reduces the cost and time investment of manual collection, but also greatly improves the accuracy and operation efficiency of the detection system, providing strong technical support for ensuring the safe operation and maintenance of railway bridges. Description of the Drawings

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0042] Figure 1 Schematic diagram of the process of a typical bridge disease characterization method provided by the present invention;

[0043] Figure 2 Schematic diagram of the process of building a deterioration disease model provided by the present invention;

[0044] Figure 3 Schematic diagram of the process of constructing a virtual bridge scene provided by the present invention;

[0045] Figure 4 Schematic diagram of the process of extracting virtual disease samples from a virtual bridge scene provided by the present invention;

[0046] Figure 5 Schematic diagram of the data transmission relationship of the improved dual-branch generative adversarial network provided by the present invention;

[0047] Figure 6 Schematic diagram of the network structure of the dual-channel generator of the improved dual-branch generative adversarial network provided by the present invention;

[0048] Figure 7 Schematic diagram of the network structure of the multi-scale image discriminator of the improved dual-branch generative adversarial network provided by the present invention;

[0049] Figure 8 Schematic diagram of the network structure of the mask discriminator of the improved dual-branch generative adversarial network provided by the present invention;

[0050] Figure 9 Schematic diagram of the training process of the improved Unet network provided by the present invention;

[0051] Figure 10 Schematic diagram of the network structure of the improved Unet network provided by the present invention;

[0052] Figure 11 Schematic diagram of the extraction of the deterioration disease area provided by the present invention. Detailed implementation manners

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0054] An embodiment of the present invention discloses a method for characterizing typical diseases of bridges, as Figure 1 shown, including the following steps:

[0055] S1: Collect the operation and maintenance data of railway bridges, construct a parameterized model library of typical diseases of railway bridges, and establish a sample library of bridge deterioration diseases;

[0056] S2: According to the bridge design drawings and relevant operation and maintenance information, sort out the family files and mileage information of each subdivision component of the bridge, and quickly construct a bridge BIM model through a rapid modeling plug-in in combination with the bridge deterioration disease sample library; according to the geographical location of the target bridge, download the corresponding image map and elevation data to establish a surrounding environment model;

[0057] S3: Based on the UE5 engine, fuse the bridge BIM model and the surrounding environment model to obtain a virtual bridge scene, and initialize the camera in the virtual bridge scene to obtain virtual disease samples;

[0058] S4: Collect real deterioration disease images of the bridge to construct a real deterioration disease sample library, and use the real deterioration disease sample library and virtual disease samples to train an improved dual-branch generative adversarial network. Use the trained improved dual-branch generative adversarial network to obtain quasi-real deterioration disease samples and expand the real deterioration disease sample library;

[0059] S5: Train the improved Unet network according to the expanded real deterioration disease sample library, input the to-be-detected image of the target bridge collected into the trained improved Unet network for deterioration disease identification, and extract the deterioration disease area mask;

[0060] S6: Map the deterioration disease area mask onto the bridge BIM model for characterization.

[0061] Furthermore, the virtual disease samples include images of multiple angles and multiple distances in the virtual scene and the corresponding mask images (semantic annotation information).

[0062] Furthermore, to systematically study the characteristics and development laws of these diseases and provide data support for subsequent diagnosis and repair, a standardized sample library of bridge deterioration diseases is created; typical diseases of railway bridges include cracks, rust, and spalling (chipping); cracks include cross cracks, inclined cracks, horizontal cracks, and vertical cracks, and the crack information is described in detail through four indicators: shape, length, width, and depth, and the corresponding crack family model is created to realize data entry and visual display; for rust and spalling, the area is used as the main description index to complete the entry and visualization of relevant information.

[0063] Furthermore, based on the relevant data of typical diseases, a deterioration disease model is constructed, including apparent disease types such as cracks (classified by orientation), rust, and spalling, and each type is further subdivided based on planar morphology, severity, and size parameters. The set of all deterioration disease models constitutes the bridge deterioration disease sample library; the deterioration disease model is created based on BIM (Building Information Modeling) technology, and there are two cases: the first case is when there are no specific requirements for the deterioration disease, the spline is used to define the orientation of the crack, the area of rust and chipping, and the length, width, and depth can be parametrically adjusted; the second case is based on the specific contour coordinates of cracks, rust, and spalling obtained by measurement, and the point coordinate data is imported through Dynamo to generate the corresponding line contour, and finally the corresponding family model is formed. This method not only improves the efficiency of railway bridge deterioration disease management but also provides accurate data support for subsequent maintenance and repair.

[0064] Furthermore, the specific modeling process of the deterioration disease model is as Figure 2 shown, specifically including the following steps:

[0065] S11: Select a suitable family template;

[0066] S12: Input relevant data in the parametric model library, set disease parameters, and create a benchmark;

[0067] When there is no measured data, create a four-way reference plane to form the basic positioning, establish a parametric system according to the crack type (horizontal / vertical / cross), and set core parameters such as length (L), width (W), and depth (D). Set the area (S) parameter for rust and spalling diseases;

[0068] When there is measured data, parse the coordinate point file (CSV / Excel) through Dynamo to automatically generate discrete point data and establish a data-driven association mechanism;

[0069] S13: Set the material and perform material visualization processing;

[0070] Set the crack filling material, set it to a semi-transparent crack material (recommended 50% transparency), and add a normal map to enhance the three-dimensional effect;

[0071] Set the rust material and the layered material system, including the base layer and the oxidation layer, and use PBR materials to simulate the metal oxidation effect;

[0072] Set the material for the spalling area, call the exposed aggregate material library, and use the edge breakage parameter to control the gradient effect.

[0073] Furthermore, according to the family files and mileage information of each subdivision component of the target bridge in the bridge design drawings and relevant operation and maintenance information, use the rapid modeling plug-in to quickly generate the bridge BIM model, load the generated deterioration disease model into the bridge project, add the deterioration disease model to the corresponding position of the bridge BIM model, and export the.udatasmith format file; download the image map and elevation data of the surrounding environment of the target bridge from the existing database. The image map and elevation data are respectively exported as raster data in JPG format and dem data in tif format through Global Mapper software. The dem data is then exported as a grayscale map through QGIS. The raster data and the grayscale map constitute the surrounding environment model; use Twinmotion software to perform beautification operations such as greening and modifying materials, and export the.udatasmith format file to construct the greening model.

[0074] Furthermore, the process of S2 - S3 is as Figure 3 shown. The specific implementation process of creating the virtual bridge scene in S3 is as follows:

[0075] S311: Open the UE5 engine to create a new level and add a dynamic sky sphere;

[0076] S312: Use the datasmith plug-in in the UE5 engine to import the bridge BIM model with deterioration diseases and the surrounding environment model;

[0077] S313: Export the oblique photography model of the section where the railway bridge is located in OSGB format collected through the OSGBLab software as FBX format;

[0078] S314: Use the ObliquePhotography plug-in in the UE5 engine to import the FBX-format oblique photography model to make the virtual scene more realistic;

[0079] S315: Integrate the imported bridge BIM model, greening model, surrounding environment model, and oblique photography model to form a virtual bridge scene (digital twin base).

[0080] Furthermore, the BIM model of the bridge in the virtual bridge scene is updated in real time according to the service status of the target bridge; for the deteriorated diseases that have been renovated in the virtual bridge scene based on the real-time detection images of the service status of the target bridge, elimination operations are performed in a timely manner; for newly added deteriorated diseases (such as cracks, spalling, corrosion, etc.), based on the IoT sensor data and on-site inspection information corresponding to the newly added deteriorated diseases, as well as the information extracted by using image recognition technology, a newly added deteriorated disease model is created in real time in the virtual bridge scene through S5-S6 and relevant information is recorded. This dynamic update and synchronization mechanism can ensure that the virtual world and the physical world always maintain a high degree of consistency and accuracy, thus providing reliable data support for the whole life cycle management and condition assessment of the bridge.

[0081] Furthermore, the specific operation steps for real-time updating of the virtual bridge scene include:

[0082] S316: Create blueprint Actors for cracks, corrosion, and spalling in the UE5 engine;

[0083] S317: Use the GeoReferencing plugin to drag "GeoReferencingSystem" from the "Place Actor" control panel into the project, and set the "Planet Shape", "Projected CRS", "Geographic CRS", and the longitude and latitude coordinates of the zero point to complete the registration point coordinate settings;

[0084] S318: Add new diseases or destroy deteriorated diseases on the virtual bridge scene by inputting the longitude and latitude coordinates of the registration points;

[0085] S319: After creating a deteriorated disease, some IoT sensor data, on-site inspection information, and disease information extracted by using image recognition technology need to be input, and a Widget is created to record the input disease information; taking cracks as an example, the disease information includes the length, width, and height of the cracks, the inspectors and inspection time, or the disease pictures collected;

[0086] S3110: Save the input disease information and the created deteriorated diseases. When the created virtual bridge scene project runs, when opening the scene for viewing, all the relevant deteriorated disease models added or deleted last time and the relevant information corresponding to the models are loaded.

[0087] Furthermore, in S3, the three deteriorated disease models of cracks, corrosion, and spalling are rendered in layers in the virtual bridge scene and distinguished by different colors, which helps with the segmentation and expansion of virtual disease samples in the later stage; the process of extracting virtual disease samples from the virtual bridge scene is as Figure 4 shown, specifically:

[0088] S321: Search for "Motion Blur" in the project settings of the UE5 engine and turn it off; search for "Custom" and change the custom depth stencil to "Enable Stencil".

[0089] S322: Drag the "Post Process Volume" component into the scene and set its range to "Infinity".

[0090] S323: Modify the CustomStencil material in the UE5 engine; use a masking function to mask the R / G / B channels respectively, define the custom channel template values, and use if for judgment to achieve layered rendering.

[0091] S324: Select all the deterioration disease models corresponding to cracks, rust, and spalling, and set the custom depth stencil values to 1, 2, and 3 respectively. Set the custom depth stencil value of the beam body to 0, so that the diseases on both sides of the beam body will not interfere with each other during layered rendering.

[0092] Among them, the custom depth stencil value of the crack is 1, representing red; the custom depth stencil value of the rust is 2, representing green; the custom depth stencil value of the spalling is 3, representing blue; the custom depth stencil value of the beam body is 0, representing black.

[0093] S325: Drag the Cine Camera Actor in the UE5 engine into the virtual bridge scene, adjust its height and rotation angle, aim the camera at the deterioration disease location, and adjust the aperture, focal length, and focus distance of the camera so that the deterioration disease is clearly presented in the camera.

[0094] S326: Create a level sequence, drag the initialized camera into it, set the aperture, focal length, focus distance, and position information of the camera, and add a keyframe at an appropriate time interval in the level sequence every time the camera moves.

[0095] S327: Output the level sequence and set the relevant parameters; output pictures in the Custom Render Passes format and select two rendering channels: the custom template and the final image.

[0096] S328: Customize the template picture and the final image to output virtual disease samples.

[0097] Furthermore, the improved dual-branch generative adversarial network (Improved GAN) includes a dual-channel generator and a dual-modal discriminator; the dual-channel generator is based on the dual-branch Unet network structure and consists of a shared encoder and a decoder. The decoder includes an image generation branch decoder and a mask generation branch decoder with a fused spatial adaptive normalization (SPADE) processing layer; the dual-modal discriminator includes a multi-scale image discriminator and a mask discriminator. The specific structure of the multi-scale image discriminator is asFigure 7 As shown in Figure 8 shown, the specific structure of the mask discriminator is as Figure 5 shown. The data transmission relationship of the improved GAN is as

[0098] shown. A dual-channel generator that simultaneously generates degraded images and their semantic annotation masks, and a dual-modal discriminator that simultaneously discriminates the authenticity of images and the rationality of masks are designed. Domain adaptation between virtual data and real data is achieved through cross-domain alignment in the feature space, and high-precision masks are generated synchronously to form pseudo-real degraded disease samples, expanding the data in the existing real degraded disease sample library.

[0099] Further, the process of training the improved GAN is as follows:

[0100] S41: Collect real degraded disease images, and perform data association (corresponding real data with virtual data) and structured processing on the real degraded disease images and virtual disease samples;

[0101] S411: Collect real degraded disease images and perform annotation. Manually annotate the degraded disease targets in the real degraded disease images or use existing segmentation models for pre-annotation to obtain real mask images;

[0102] S412: Perform mask preprocessing: Convert the virtual mask images corresponding to the virtual disease samples and the real mask images from RGB encoding to multi-channel binary masks (such as background, cracks, rust, peeling), obtaining the processed virtual disease masks and real disease masks;

[0103] S42: Input the virtual disease images and the processed virtual disease masks into the dual-channel generator to obtain pseudo-real images and pseudo-real masks corresponding to the virtual degraded disease images;

[0104] S421: The input layer transmits the virtual disease images to the shared encoder;

[0105] S422: The shared encoder extracts underlying common features from the virtual disease images using convolutional operations, reducing redundant calculations;

[0106] S423: The image generation branch decoder of the decoder takes the virtual disease mask as a spatial condition and fuses it with the underlying common features to obtain pseudo-real images; the mask generation branch decoder of the decoder takes the virtual disease mask as a spatial condition and fuses it with the underlying common features to obtain pseudo-real masks;

[0107] S431: The input layer of the dual-modal discriminator inputs the class-real images and real deteriorated disease images into the multi-scale image discriminator, and inputs the class-real masks and real disease masks into the mask discriminator;

[0108] S432: The multi-scale image discriminator performs multi-level downsampling on the class-real images and real deteriorated disease images, applies the PatchGAN discriminator for discrimination at different scales to enhance the discrimination ability for local textures, and obtains the true-false discrimination space matrix for each region of the image;

[0109] The mask discriminator determines whether the topological structures of the class-real mask and the real disease mask are consistent through a convolutional network, and generates the true-false discrimination space matrix of the mask;

[0110] S44: According to the true-false discrimination space matrices of the multi-scale image discriminator and the mask discriminator, use a comprehensive loss function for backpropagation to adjust the parameters of the multi-scale image discriminator and the mask discriminator of the dual-channel generator and the dual-modal discriminator;

[0111] S45: Combine the finally obtained class-real images and class-real masks to form class-real deteriorated disease samples, and merge them into the real deteriorated disease sample library.

[0112] Combine the constructed virtual disease images and masks in the real deteriorated disease images and the corresponding real masks. Through the dual-branch generative adversarial network plus the multi-task loss function, realize the domain adaptation from the virtual disease images to the real deteriorated disease images, and synchronously output high-precision class-real masks to realize the expansion and automatic annotation functions of the samples in the bridge deteriorated disease sample library of typical diseases of railway bridges.

[0113] Furthermore, as Figure 6 shown, the shared encoder performs feature extraction, including four groups of feature extraction modules and two convolutional layers Conv connected in sequence. Each group of feature extraction modules includes a convolutional layer Conv, an activation function Relu, a convolutional layer Conv, an activation function Relu, and a max pooling layer MaxPool connected in sequence. The max pooling layer of each group of feature extraction modules is connected to the convolutional layer of the next group of feature extraction modules; the second activation function of each group of feature extraction modules and the second convolutional layer of the shared encoder are connected to the decoder.

[0114] Furthermore, as Figure 6As shown in the figure, the image generation branch decoder in the decoder includes four groups of image generation modules and a convolutional layer Conv connected in sequence, and outputs the generated image as a pseudo-real image; each group of image generation modules includes a transposed convolutional layer ConvTranspose2d, a spatially adaptive normalization processing layer SPADE, a fusion layer Concat, a convolutional layer Conv, an activation function Relu, a convolutional layer Conv, and an activation function Relu connected in sequence. The activation function Relu of each group of image generation modules is connected to the transposed convolutional layer of the next group of image generation modules; the second activation function of each group of feature extraction modules is connected to the fusion layer of a corresponding image generation module, and the second convolutional layer of the shared encoder is connected to the transposed convolutional layer of the first image generation module; the virtual deterioration disease mask is input to the spatially adaptive normalization processing layer; the mask generation branch decoder in the decoder includes four groups of mask generation modules and a convolutional layer connected in sequence, and outputs the generated mask as a pseudo-real mask; the structure of each group of mask generation modules is the same as that of the image generation module, and the connection relationship of the four groups of mask generation modules is the same as that of the image generation module; the second activation function of each group of feature extraction modules is connected to the fusion layer of a corresponding mask generation module; the virtual deterioration disease mask is input to the spatially adaptive normalization processing layer of the mask generation module.

[0115] Furthermore, the comprehensive loss function of the dual-channel generator includes adversarial losses (image adversarial loss, mask adversarial loss), pixel reconstruction loss, and feature matching loss calculated based on the features extracted by the vgg network. During the training process, mixed training is adopted to balance the contributions of virtual and real data; the comprehensive loss function of the dual-modal discriminator includes real image discrimination loss, real mask discrimination loss, pseudo-real image discrimination loss, and pseudo-real mask discrimination loss. The discriminator parameters are adjusted by backpropagation by calculating the loss value between the virtual disease samples and the collected real deteriorated disease images. The specific loss functions are as follows:

[0116] The comprehensive loss function of the dual-channel generator is expressed as:

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123] Among them, represents the image adversarial loss; Denotes the masked adversarial loss; Denotes the pixel reconstruction loss; Denotes the masked Dice loss; Denotes the vgg-based perceptual loss; , , and Denote the weight coefficients respectively; G denotes the dual-channel generator; Denotes the multi-scale image discriminator; True denotes the true value in the true / false discrimination space matrix; Denotes the masked discriminator; x denotes the real deteriorated disease image; m denotes the real disease mask; Denotes the virtual disease image in the virtual disease sample; Denotes the virtual disease mask corresponding to the virtual disease image; Denotes the mse loss; Denotes the weight coefficient of the Denotes the vgg network of the

[0124] Composite loss function of the dual-modal discriminator Is expressed as:

[0125]

[0126]

[0127]

[0128]

[0129]

[0130] Among them, Denotes the real image discrimination loss; Denotes the real mask discrimination loss; Denotes the pseudo-real image discrimination loss; Denotes the pseudo-real mask discrimination loss; False denotes the false value in the true / false discrimination space matrix; The real image discrimination loss and the pseudo-real image discrimination loss are calculated using the true / false discrimination space matrix of the image, and the real mask discrimination loss and the pseudo-real mask discrimination loss are calculated using the true / false discrimination space matrix of the mask.

[0131] Furthermore, a pseudo mask of deteriorated disease images is generated by a dual-channel generator to achieve automatic refinement annotation of semantic information. Meanwhile, the minimum bounding rectangle of the mask can be calculated to automatically generate a deteriorated bounding box to complete the target annotation.

[0132] Furthermore, to adapt to the extraction of disease features of different sizes, the traditional Unet network is improved by combining multi-scale and attention mechanisms to construct an improved Unet network. As Figure 10 shown, the improved Unet network consists of an encoder, a decoder, and a segmentation head. Among them, the encoder uses a standard Resnet encoder structure to extract five layers of feature maps of different scales for subsequent skip connection operations with the decoder; the decoder adds an attention mechanism to fuse the feature maps extracted by the corresponding encoder, focusing on important regions and features and suppressing irrelevant background noise; the segmentation head maps the tensor output by the decoder to the number of target categories through convolution operations to output a deteriorated disease mask; furthermore, a weighted cross-entropy loss function is used to train the improved Unet network. The training process of the improved Unet network is as Figure 9 shown.

[0133] Furthermore, the training process of the improved Unet network is as follows:

[0134] S51: Data preprocessing and annotation conversion;

[0135] S511: Multi-source data fusion, an expanded real deteriorated disease sample library (including real deteriorated disease images and real mask images collected from real inspections, and the pseudo images and pseudo masks generated by the improved GAN in S4 as expanded real deteriorated disease samples);

[0136] S512: Data augmentation, online augmentation of sample data (deteriorated images and masks) through random rotation, translation, Gaussian noise, light simulation, etc., to ensure synchronous annotation;

[0137] S513: Color coding conversion, converting the colors of the real mask images in S411 and the pseudo masks generated by the dual-channel generator in S423 into category labels. For example: red cracks correspond to category 1, green rust corresponds to category 2, blue spalling corresponds to category 3, and other background regions correspond to category 0;

[0138] S52: Model training;

[0139] S521: The real deteriorated disease images from real inspections and the deteriorated disease images (pseudo images) generated by the improved GAN are used as the input data of the model, and the masks corresponding to each deteriorated disease image are used as the output data of the model to train the improved Unet network;

[0140] S522: The improved Unet network uses a standard Resnet encoder structure to extract five different-scale feature maps, achieving multi-scale feature extraction operations; the decoder fuses the features extracted by the corresponding encoder through upsampling and synchronously adds an attention mechanism to focus on important regions and features; the segmentation head maps the decoder output to the number of target categories through convolution operations to achieve the segmentation function.

[0141] Furthermore, in combination with the sample category distribution, a weighted cross-entropy loss function is constructed, assigning higher weights to different deteriorations and relatively small weights to the background; the weighted cross-entropy loss function is as follows:

[0142]

[0143] where N represents the number of pixels; C represents the number of categories; represents the label; represents the predicted category probability; represents the weight of category c, which is inversely proportional to the category frequency.

[0144] Furthermore, after the training of the improved Unet network is completed, the collected images to be detected are subjected to deterioration detection to obtain the deterioration disease mask of the images to be detected, that is, the segmentation image. According to the color values marked in the deterioration disease area in the mask, the deterioration disease areas in the images to be detected are extracted to obtain the extracted images of the deterioration disease areas as Figure 11 shown.

[0145] Furthermore, the specific process of performing virtual-real mapping on the extracted deterioration disease areas in S6 is as follows:

[0146] S61: Import the images of the extracted deterioration disease areas into the UE5 engine, and then use these images to generate exclusive deterioration materials;

[0147] S62: Create a plane in the UE5 engine and attach the deterioration material to the plane. Combining the longitude and latitude information of the specific position of the drone carried by the images to be detected taken by the drone, through the distance and angle between the drone and the bridge at that time, after coordinate conversion, the deterioration disease position is accurately obtained to complete the placement of the deterioration disease plane; the plane is the carrier of the deterioration material;

[0148] S63: Perform necessary and reasonable rotation and fine-tuning operations on the deterioration disease plane to ensure high-precision mapping between the physical world and the digital virtual world and maintain its high consistency.

[0149] In view of typical diseases of railway bridges, such as cracks, spalling, and corrosion, a parametric model library of deterioration characterization parameters is constructed through geometric feature parameters such as length, width, and area. Based on the UE (Unreal Engine) platform, by integrating the geographical coordinate data of diseases detected in reality and through coordinate conversion, corresponding disease models are dynamically generated in the corresponding areas, realizing the intelligent mapping and visual display between the physical deterioration entity and the digital twin of railway bridges. A dual-branch generative adversarial network is proposed to construct a multi-task loss function, realizing domain adaptation from virtual samples to real images, and synchronously generating high-precision mask images to automatically refine the semantic mask of the labeled samples, forming a realistic deterioration disease sample library to make up for problems such as insufficient real collected image data and high manual annotation costs. An improved Unet network combining multi-scale and attention mechanisms is proposed to realize the extraction of small-sample and clearly edged deterioration targets, improve the detection accuracy, and realize the intelligent and accurate detection of typical diseases of bridges. Further, the automatic extraction of the deteriorated area is realized according to the semantic annotation of the deteriorated disease area. It can realize the intelligent characterization of two typical diseases of bridges: one is to automatically add parametric diseases and manually fine-tune them to achieve a similar appearance; the second is to automatically extract deteriorated diseases (position, shape, type, etc.), map them automatically to the corresponding areas of the digital twin by making the materials of the deteriorated diseases and according to the disease coordinates converted from the longitude and latitude of the original detection image, achieving both similar appearance and spirit.

[0150] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0151] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for characterizing typical diseases of bridges, characterized in that, It includes the following steps: Step 1: Collect bridge operation and maintenance data, construct a parametric model library of typical bridge diseases, and establish a bridge deterioration disease sample library; Step 2: According to the design drawings and relevant operation and maintenance information of the target bridge, use a rapid modeling plug-in to combine with the bridge deterioration disease sample library to construct the BIM model of the target bridge, and download the corresponding image map and elevation data according to the geographical location of the target bridge to construct the surrounding environment model; Step 3: Based on the UE5 engine, fuse the bridge BIM model and the surrounding environment model to obtain a virtual bridge scene, and initialize the camera in the virtual bridge scene to obtain virtual disease samples; Step 4: Collect real deterioration disease images to construct a real deterioration disease sample library, and use the real deterioration disease sample library and virtual disease samples to train an improved dual-branch generative adversarial network. Use the trained improved dual-branch generative adversarial network to obtain realistic deterioration disease samples and expand the real deterioration disease sample library; Step 5: Train the improved Unet network according to the expanded real deterioration disease sample library, input the collected test images of the target bridge into the trained improved Unet network for deterioration disease recognition, and extract the deterioration disease area mask; Step 6: Map the deterioration disease area mask onto the bridge BIM model for characterization.

2. The method for characterizing typical diseases of a bridge according to claim 1, wherein The typical diseases of the bridge include cracks, rust, and spalling; the bridge deterioration disease sample library includes deterioration disease models of cracks, rust, and spalling with different planar forms, severity levels, and size parameters. When there are no specific requirements for the deterioration disease model, use splines to define the orientation of the cracks, and the areas of rust and spalling, and parametrically adjust the length, width, and depth; when constructing the deterioration disease model based on actual measurement data, import the specific contour coordinates of cracks, rust, and spalling through Dynamo to import point coordinate data, generate corresponding line contours, and form the corresponding deterioration disease model.

3. A method for characterizing typical diseases of bridges according to claim 2, characterized in that, According to the family files and mileage information used by each subdivision component of the target bridge in the design drawings and relevant operation and maintenance information, use a rapid modeling plug-in to generate the BIM model of the bridge, and load the deterioration disease models in the bridge deterioration disease sample library into the BIM model of the bridge; Download the image map and elevation data of the surrounding environment of the target bridge, export the image map as raster data in JPG format through Global Mapper software, export the elevation data as dem data in tif format through Global Mapper software, and then export the dem data as a grayscale image through QGIS. The raster data and the grayscale image constitute the surrounding environment model; use Twinmotion software to perform beautification operations such as greening and modifying materials to construct a greening model.

4. A method for characterizing typical diseases of bridges according to claim 3, characterized in that, Import the bridge BIM model, greening model, and surrounding environment model into the UE5 engine, and fuse the oblique photography model to form a virtual bridge scene.

5. A method for characterizing typical diseases of bridges according to claim 1, characterized in that, According to the real-time detection images of the service status of the target bridge, update the virtual bridge scene in real time, including eliminating the deteriorated diseases that have been renovated and adding newly added deteriorated diseases.

6. The method for characterizing typical diseases of bridges according to claim 1, wherein, The improved dual-branch generative adversarial network includes a two-channel generator and a dual-modal discriminator; The dual-channel generator adopts a dual-branch Unet network structure, including a shared encoder and a decoder; the decoder includes an image generation branch decoder and a mask generation branch decoder, which respectively integrate spatial adaptive normalization processing layers; the dual-modal discriminator includes a multi-scale image discriminator and a mask discriminator.

7. A method for characterizing typical diseases of bridges according to claim 6, characterized in that, The process of training the improved dual-branch generative adversarial network is as follows: Step 41: Collect real deteriorated disease images to construct a real deteriorated disease sample library, and perform data association and structuring processing on the real deteriorated disease images and virtual disease samples to obtain processed real disease masks and virtual disease masks; Step 411: Annotate the real deteriorated disease images to obtain real mask images; Step 412: Convert the virtual mask images and real mask images in the virtual disease samples from RGB encoding to multi-channel binary masks to obtain processed virtual disease masks and real disease masks respectively; Step 42: Input the processed virtual disease masks and the virtual disease images in the corresponding virtual disease samples into the dual-channel generator to obtain pseudo-real images and pseudo-real masks of virtual deteriorated diseases; Step 421: The input layer transmits the virtual disease images to the shared encoder; Step 422: The shared encoder extracts underlying common features from the virtual disease images based on convolutional operations; Step 423: The image generation branch decoder of the decoder uses the virtual disease mask as a spatial condition and fuses it with the underlying common features to obtain a pseudo-real image; The mask generation branch decoder of the decoder uses the virtual disease mask as a spatial condition and fuses it with the underlying common features to obtain a pseudo-real mask; Step 43: Input the pseudo-real images, pseudo-real masks, real deteriorated disease images, and real disease masks into the dual-modal discriminator for image discrimination and semantic consistency discrimination to obtain true / false discrimination spatial matrices for images and masks respectively; Step 431: The input layer of the dual-modal discriminator inputs the pseudo-real images and real deteriorated disease images into the multi-scale image discriminator, and inputs the pseudo-real masks and real disease masks into the mask discriminator; Step 432: The multi-scale image discriminator performs multi-level downsampling on the pseudo-real images and real deteriorated disease images, and applies a PatchGAN discriminator for discrimination at different scales to obtain a true / false discrimination spatial matrix for each region of the image; The mask discriminator determines whether the topological structures of the pseudo-real mask and the real disease mask are consistent through a convolutional network, and generates a true / false discrimination spatial matrix for the mask; Step 44: Perform backpropagation using a comprehensive loss function according to the true / false discrimination spatial matrices of the images and masks, and adjust the parameters of the dual-channel generator and the dual-modal discriminator; Step 45: Combine the finally obtained pseudo-real images and pseudo-real masks to form pseudo-real deteriorated disease samples, and merge them into the real deteriorated disease sample library.

8. A method for characterizing typical diseases of bridges according to claim 7, characterized in that, The comprehensive loss function of the dual-channel generator includes adversarial loss, pixel reconstruction loss, and feature matching loss calculated based on features extracted by the vgg network; the comprehensive loss function of the dual-modal discriminator includes real image discrimination loss, real mask discrimination loss, pseudo-real image discrimination loss, and pseudo-real mask discrimination loss.

9. A method for characterizing typical diseases of bridges according to claim 1, characterized in that, In step 5, the improved Unet network includes an encoder, a decoder, and a segmentation head; the encoder uses a Resnet encoder structure to extract five layers of feature maps with different scales for skip connection operations with the decoder; The decoder adds an attention mechanism to fuse the feature maps extracted by the corresponding encoder; The segmentation head maps the tensor output by the decoder to the number of target categories through convolution operations to output a mask of the deteriorated disease area; The weighted cross-entropy loss function is used to train the improved Unet network.

10. A method for characterizing typical diseases of bridges according to claim 1, characterized in that, In step 6, the specific process of performing virtual-real mapping on the extracted deteriorated disease area mask is as follows: S61: Import the image of the extracted deteriorated disease area mask into the UE5 engine to generate a corresponding deteriorated material; S62: Create a deteriorated disease plane in the UE5 engine, attach the deteriorated material to the deteriorated disease plane, and combine the longitude and latitude information of the specific position of the drone during shooting carried by the image to be detected taken by the drone. Through the distance and angle of the drone from the target bridge at that time, after coordinate conversion, obtain the position of the deteriorated disease in the corresponding bridge BIM model, and place the deteriorated disease plane at the position of the deteriorated disease; S63: Perform rotation and fine-tuning operations on the deteriorated disease plane.

Images