Abnormality recognition method for long-span bridge structure health monitoring based on image classification

Abstract

The present invention discloses a method for identifying abnormalities in health monitoring of large-span bridge structures based on image classification. The present invention relates to the technical field of health monitoring of bridge structures. The following steps are as follows: Step 1: Generate abnormal samples using a generative adversarial network (GAN), wherein the abnormal samples include cracks, spalling, and rust. The method for identifying abnormalities in health monitoring of large-span bridge structures based on image classification generates and optimizes abnormal samples through a generative adversarial network, thereby alleviating the data scarcity situation; the dynamic generation-learning-optimization closed-loop system and distributed collaborative platform reduce the annotation cost; the annotation quality stability is improved by means of weakly supervised learning, knowledge graphs, and automated quality inspection tools; the utilization rate of unlabeled data is improved when multimodal data is fused using self-supervised learning technology, which solves the problems of data annotation and quality in an all-round way, and provides strong support for bridge structure health monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge structure health monitoring, and in particular to an abnormality recognition method for long-span bridge structure health monitoring based on image classification. Background Art

[0002] In the health monitoring of long-span bridge structures, anomaly recognition methods based on image classification have gradually become an important technical means for bridge safety management; however, current technologies have significant deficiencies in data annotation and quality, which directly affect the training effect of image classification models and the accuracy of anomaly recognition; on the one hand, the data collection of bridge anomalies (such as cracks, spalling, rust, etc.) is highly professional, and abnormal samples are usually scarce, and obtaining high-quality annotated data requires a lot of manpower and time; on the other hand, due to the complex structure of bridges and long-term exposure to different environments (such as strong winds, rain and snow, direct sunlight, etc.), the quality of the collected images varies, and some images may be blurred, noisy or occluded, which further aggravates the difficulty of annotation; in addition, existing data annotation mainly relies on manual operation, and the annotation process may result in inaccurate annotation due to the operator's lack of experience or misjudgment. Accurate or inconsistent; for example, there may be subjective bias in the annotation of specific parameters such as crack width and length, which in turn affects the model's ability to identify subtle anomalies; at the same time, the annotation efficiency is low and it is difficult to meet the processing needs of large-scale bridge monitoring data; and in the health monitoring of large-span bridges, it is often necessary to process a large amount of unlabeled data under different bridge structure types and different climatic conditions. These unlabeled data have not been fully utilized, resulting in a waste of resources; in summary, the large-span bridge structure health monitoring method based on image classification still faces major technical bottlenecks in data annotation and quality, including data scarcity, high annotation cost, unstable annotation quality and low utilization of unlabeled data; these deficiencies have become key obstacles to further improving the recognition accuracy and promoting the application of monitoring methods, and an innovative solution is urgently needed to break through the limitations of existing technologies. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention provides a large-span bridge structure health monitoring anomaly identification method based on image classification, which solves the problems of data scarcity, high annotation cost, unstable annotation quality and low utilization of unlabeled data in the health monitoring method of large-span bridge structures.

[0004] To achieve the above objectives, the present invention is implemented by the following technical scheme: a method for identifying abnormalities in health monitoring of a long-span bridge structure based on image classification, the method comprising the following steps:

[0005] Step 1: Generate abnormal samples using a generative adversarial network (GAN), including cracks, spalling, and rust; dynamically adjust the distribution of the generated model based on active learning feedback signals to make the generated data consistent with the actual bridge abnormality distribution; use image enhancement technology to denoise, super-resolution reconstruct, and optimize clarity of low-quality images to improve data quality;

[0006] Step 2: Collect multimodal monitoring data including image data, vibration data, strain data and environmental condition data; use the cross-modal feature alignment model to extract and fuse multimodal features; generate comprehensive abnormal samples containing time series, environmental context and spatial features to improve the perception of long-term evolution trends of bridges;

[0007] Step 3: Establish a dynamic generation-learning-optimization closed-loop system, including: generating preliminary annotations using a generative adversarial network; annotating high-uncertainty areas based on weakly supervised learning and active learning; using automated quality inspection tools to detect the boundaries and regional consistency of the annotated data, and optimizing the annotation priority and range through the knowledge graph; feeding back the newly annotated samples to the generation model and the annotation model through closed-loop feedback to dynamically improve the model performance;

[0008] Step 4: Decompose the large-scale labeling task into fine-grained subtasks based on the uncertainty analysis of the model, including anomaly type classification, dimension labeling, and area division; use the distributed collaboration platform to assign subtasks to the cloud labeling model or manual labeling team; automatically merge and align the distributed labeling results, and use quality verification tools to ensure labeling consistency and reliability;

[0009] Step 5: Institutions share model updates through the federated learning framework without sharing original data. The federated learning model evaluates and optimizes the quality of labeled data from each institution. It combines the data distribution of multiple institutions to generate universal anomaly samples suitable for multiple scenarios and improve the generalization ability of the model.

[0010] Preferably, the process of dynamically optimizing the abnormal samples generated by the generative adversarial network (GAN) through active learning feedback signals comprises the following steps:

[0011] Step a1: Generate preliminary abnormal sample data using the generator and discriminator of the Generative Adversarial Network (GAN), where the generator generates image samples simulating bridge abnormalities based on random noise vectors. The abnormality types include but are not limited to cracks, spalling, rust, and structural deformation;

[0012] Step a2: Perform a preliminary quality assessment on the generated abnormal samples, verify their effectiveness through model training, and select high-confidence samples for model training; the samples must meet the following standards:

[0013] The feature similarity with the actual bridge abnormality image distribution exceeds the set similarity threshold;

[0014] The probability of being correctly classified as anomaly in the pre-trained image classification model is over 90%;

[0015] Step a3: Train the classification model for the selected samples, and find high uncertainty samples through model uncertainty assessment, and optimize the annotations in the following ways:

[0016] Using entropy-based active learning algorithms, we can identify samples where the model predicts low confidence in classification tasks.

[0017] Submit high-uncertainty samples to experts for manual annotation to obtain more accurate anomaly classification and detailed descriptions, such as crack width and spalling area;

[0018] Step a4: Re-input the manually annotated high-uncertainty samples into the generative model, and optimize the generated samples by adjusting the training target distribution of the generator so that the newly generated samples are more consistent with the actual bridge anomaly distribution. The training target distribution includes the proportion of anomaly types and the expression of feature details.

[0019] Step a5: After each round of sample generation and labeling is completed, the effectiveness of the sample generation strategy is evaluated based on the changes in the classification accuracy and recall rate in the model performance indicators, and the training parameters of the GAN generator and discriminator are adjusted to form a dynamic optimization loop;

[0020] Through the above process, the realism and effectiveness of the generated samples are gradually improved in each round of optimization, thereby effectively supplementing high-quality abnormal data under conditions of data scarcity.

[0021] Preferably, the multimodal data fusion process comprises the following steps:

[0022] Step b1: Acquire multimodal data related to bridge health monitoring, which includes image data, vibration data, strain data, and environmental condition data; the image data includes high-resolution images of cracks and spalling, the vibration data includes bridge vibration characteristics recorded by acceleration sensors, the strain data includes strain changes collected by fiber Bragg grating sensors, and the environmental condition data includes monitoring data of temperature, humidity, wind speed, and corrosion factors;

[0023] Step b2: For image data, convolutional neural network (CNN) is used to extract spatial texture and structural features, including crack width, shape and direction; for vibration data, fast Fourier transform (FFT) or wavelet analysis is used to extract frequency domain features, which include vibration amplitude and natural frequency; for strain data, time series modeling is used to extract change trends and abnormal fluctuation characteristics; for environmental condition data, time series features and context information are extracted through statistical analysis;

[0024] Step b3: Use the Transformer model to align the features of data from different modalities, and capture the association between image features and non-image features through the attention mechanism; unify the cross-modal features and generate a fused high-dimensional feature vector as a comprehensive anomaly description;

[0025] Step b4: Combine the results of cross-modal feature alignment to generate abnormal samples containing multi-modal information, with detailed annotations of corresponding time, environment and space features; and use self-supervised learning technology to optimize cross-modal feature representation in combination with unlabeled data;

[0026] Step b5: Verify the effectiveness of the fusion features through the multimodal anomaly detection model, and compare the improvement of single-modal data in classification accuracy and anomaly recognition ability;

[0027] Through the above multimodal data fusion process, the problem of insufficient information of single-modal data is overcome, and a comprehensive description of complex bridge anomalies is achieved.

[0028] Preferably, the data annotation optimization and closed-loop learning system optimizes the annotation process by the following steps:

[0029] Step c1: Generate preliminary annotation data using a generative adversarial network (GAN), including category annotations and bounding box annotations for abnormal areas; apply weakly supervised learning techniques to optimize preliminary annotation results based on a small amount of high-quality manually annotated data;

[0030] Step c2: Construct a knowledge graph containing bridge structure features, anomaly types and environmental influencing factors, where bridge structure features include material and node type, anomaly types include cracks, spalling and rust, and environmental influencing factors include humidity, temperature and corrosion factors; and use knowledge graph reasoning to guide the priority allocation of the annotation range, such as giving priority to annotating anomaly features in high-risk areas under environmental conditions; for anomaly samples after annotation, analyze the causes of anomalies and possible evolution trends through knowledge graphs;

[0031] Step c3: Use automated quality inspection tools to detect problems in the labeled data, including boundary overlap, inconsistency, and missing annotations; automatically correct annotation errors based on rule engines or deep learning models, such as adjusting crack boundaries or demarcation of spalling areas;

[0032] Step c4: After each round of annotation optimization, use new high-quality annotation data to retrain the generation model and the annotation model; compare and analyze the optimized annotation results with the preliminary annotation results to gradually improve the performance of the annotation model;

[0033] Through the above-mentioned annotation optimization process, the annotation efficiency and quality are effectively improved, and the knowledge graph is used to enhance the scientificity and rationality of annotation.

[0034] Preferably, the distributed collaboration platform completes the labeling task decomposition and collaboration through the following steps:

[0035] Step d1: According to the complexity of the anomalies in the bridge image data and the uncertainty of the model, the large-scale labeling task is automatically decomposed into multiple fine-grained subtasks; the fine-grained subtasks include: anomaly type classification, dimension labeling, region segmentation and time series association. Combined with the task characteristics, the best labeling method is selected through the task allocation algorithm, including machine labeling or manual assisted labeling;

[0036] Among them, the anomaly type classification includes cracks or spalling, the dimensioning includes crack width and length, the region segmentation includes the boundary of the spalling region, and the time series association includes the abnormal changes in environmental conditions;

[0037] Step d2: Build a distributed collaboration platform based on the cloud annotation system. The platform supports real-time synchronization of task status and allows multiple annotation teams to complete different subtasks in parallel. The platform schedules resources through task priority queues and assigns high-priority tasks to experienced annotators or models with the best performance.

[0038] Step d3: Use the automated alignment tool to integrate the annotation results generated by different subtasks into a complete anomaly sample annotation; use the deep learning model to perform consistency verification and conflict detection on the integration results to ensure the logical consistency and data reliability between the annotation modules;

[0039] Step d4: For low-confidence or conflicting areas, automatically generate task feedback and reallocate them to appropriate annotation teams for secondary annotation or adjustment; regularly evaluate the overall annotation quality of the platform through the quality control module, and adjust the task allocation strategy for frequently occurring annotation deviations;

[0040] Step d5: Combine the historical data of the annotation results in the platform to train the task allocation optimization model to improve the accuracy and efficiency of task scheduling; in a distributed collaborative environment, dynamically adjust the task priority through the task progress prediction mechanism to ensure that key tasks are completed on time;

[0041] Through the above-mentioned distributed collaboration platform and task decomposition method, efficient processing of large-scale bridge monitoring data labeling tasks is achieved, while ensuring the consistency and high quality of the labeling results.

[0042] Preferably, the cross-institutional data sharing and optimization driven by federated learning includes the following steps:

[0043] Step e1: Each participating organization stores its bridge monitoring data, including images, vibration, and environmental information, locally, ensuring that the data is not transmitted to protect privacy. Within each organization, an initial anomaly recognition model is trained based on local data to generate model weights and feature representations.

[0044] Step e2: Build a federated learning framework that supports distributed transmission and updating of model parameters to avoid sharing of original data; use the Federated Averaging algorithm to aggregate model updates uploaded by multiple institutions to form a global optimization model;

[0045] Step e3: The federated learning model evaluates the quality of the labeled data provided by each institution and identifies areas with inconsistent or large deviations in the labels. It uses the aggregated knowledge of the global model to correct the labeling deviations and generates customized labeling optimization strategies for each institution.

[0046] Step e4: Based on the data distribution characteristics of multiple institutions, a globally optimized generative adversarial network (GAN) is used to generate general abnormal samples applicable to multiple scenarios. Based on the general abnormal samples, a locally optimized sample set is generated for each institution to improve the adaptability of the model.

[0047] Step e5: In each round of federated learning iteration, monitor the model performance indicators in real time, including classification accuracy, anomaly recognition accuracy, and annotation consistency across institutions; dynamically adjust the aggregation strategy of federated learning based on performance feedback to optimize the adaptability of the global model to diverse data distribution;

[0048] Through the above-mentioned federated learning method, various institutions have achieved efficient data sharing and collaborative modeling under privacy protection, which not only improved the generalization ability of the model, but also significantly enhanced the quality and consistency of labeled data.

[0049] Preferably, the specific steps of strengthening the tiny features of the image data by contrastive learning technology in the closed-loop learning process include:

[0050] Step f1: Perform data augmentation operations on the original image data, including random cropping, rotation, and noise addition, to generate multi-view samples; use the self-supervised learning model SimCLR to extract the feature embedding vector of the enhanced samples and construct feature pairs of similar samples;

[0051] Step f2: Design a loss function for contrastive learning to maximize the similarity between embedding vectors of samples of the same category and minimize the similarity between embedding vectors of samples of different categories; in the feature space, optimize the sensitivity of the model to small feature differences by adjusting the temperature parameter of the loss function;

[0052] Step f3: In the embedding space, the separation distance between abnormal features and background features is used to evaluate the feature differentiation ability of the model; feature contrast technology is used to enhance the representation ability of small abnormal features and reduce the probability of misclassification due to background interference, where small abnormal features include cracks and spalling;

[0053] Step f4: Feedback the enhanced abnormal features to the annotation model and the generation model to optimize the automatic annotation accuracy of the abnormal area; after each round of contrast learning iteration, the performance of the model is improved by contrast measurement, including the abnormal recognition rate and the accuracy of detail annotation;

[0054] Through the application of the above-mentioned contrastive learning technology, the model's ability to perceive subtle abnormal features has been effectively improved, which is particularly suitable for detection scenarios of subtle anomalies such as cracks and rust.

[0055] Preferably, the method is applicable to structural health monitoring of long-span bridges of various bridge types and under complex environmental conditions, and its specific implementation includes the following steps:

[0056] Step g1: Collect and classify data of different types of long-span bridges, including suspension bridges, cable-stayed bridges, arch bridges, and continuous beam bridges; extract abnormal features of key parts according to the structural characteristics of the bridge type, including rust on the main cable of the suspension bridge, fatigue cracks on the cables of the cable-stayed bridge, and peeling areas of the arch ring of the arch bridge;

[0057] Step g2: monitor the key parameters of the environment in which the bridge is located, including meteorological data and special conditions. Through the environmental context association technology, analyze the impact of environmental conditions on bridge structural abnormalities and generate corresponding environmental factor weights. Meteorological data include temperature, humidity, rainfall, and wind speed. Special conditions include salt spray corrosion in the marine environment and the impact of extreme weather.

[0058] Step g3: Optimize the quality of image data collected in complex environments, especially noise removal and occlusion completion; use super-resolution reconstruction technology to improve the clarity of blurred images to capture detailed abnormal features;

[0059] Step g4: During the labeling and training process, the data distribution of the generative adversarial network (GAN) is dynamically adjusted according to different bridge types and environmental conditions to generate abnormal samples that are more in line with the characteristics of the scene; the active learning algorithm is used to dynamically detect high-uncertainty samples under different environments and bridge structures to optimize the model's adaptability to specific scenarios;

[0060] Step g5: Establish a specific abnormal feature library for key structures of different bridge types, including main beams, cables, and arch rings; combine knowledge graphs to analyze the causes and evolution patterns of abnormal features, and provide health monitoring solutions for different bridge types;

[0061] Through the above steps, the method can effectively adapt to various bridge types and complex environmental conditions, and provide accurate and scenario-based anomaly recognition capabilities for structural health monitoring of long-span bridges.

[0062] Preferably, the automated quality inspection tool combines a rule engine and a deep learning model to complete the quality inspection and optimization of the labeled data through the following steps:

[0063] Step h1: define a set of annotation rules for abnormal areas, including boundary consistency, area integrity and category accuracy; use the rule engine to check the annotation data one by one; the content of the rule engine includes: whether the boundary of the abnormal area is closed, whether the size of the anomaly is beyond the reasonable range, and whether the annotation categories in the same image are consistent;

[0064] Step h2: Use deep learning models to detect the labeled data, including regional comparative analysis based on convolutional neural networks; identify potential labeling errors or deviations, including areas with unclear boundaries and areas that are missing labels; automatically generate correction suggestions or directly adjust the labeled data, including: optimizing the pixel-level accuracy of crack boundaries and adjusting the polygonal labeling range of spalling areas;

[0065] Step h3: Perform consistency comparison analysis between multimodal annotated data, where the multimodal annotated data includes image and vibration data.

[0066] The consistency comparison analysis process includes: whether the cracks marked in the image are consistent with the location of the vibration feature changes, whether the rusted area is highly correlated with the humidity in the environmental conditions;

[0067] Step h4: grade the quality of the annotation data and optimize the annotations of high-priority areas first; use the corrected high-quality annotation data to retrain the model and verify the effectiveness of the annotation optimization;

[0068] Through the automated application of the above-mentioned quality inspection tools, the subjective errors in manual annotation are significantly reduced, and the overall reliability and consistency of the annotation data are improved.

[0069] Preferably, the dual-circulation technology framework includes two stages: preliminary circulation and deep circulation, which are specifically implemented as follows:

[0070] Step i1: Grade the quality of the original image based on image clarity, noise level, and occlusion ratio; apply denoising, super-resolution reconstruction, and texture enhancement techniques to low-quality images to improve the reliability of basic annotations; use a pre-trained generative adversarial network (GAN) to generate preliminary annotation results, including the category and location of abnormal areas; apply weakly supervised learning methods combined with a small amount of manually labeled data to optimize the accuracy of generated annotations; use automated quality inspection tools to detect errors in preliminary annotations, including unclear region boundaries or mislabeled categories; after correcting the labeled data, form a basic annotation result set to provide input for the deep loop;

[0071] Step i2: Generate high-dimensional feature vectors through self-supervised learning to separate abnormal features from background features;

[0072] Use contrastive learning methods to enhance the perception of minor anomalies, optimize the boundaries and classification of the initial annotation area based on deep features; combine multimodal auxiliary annotation (such as vibration and image) to optimize the accurate description of abnormal features; feed back the features mined by deep loops to the generative adversarial network (GAN) and annotation model to dynamically optimize data generation and annotation quality; adjust the optimization strategy of the next stage loop according to the performance indicators of the deep loop;

[0073] Through the dual-loop technical framework, the method achieves dynamic improvement of the entire process from rough labeling to fine optimization, improving labeling efficiency and recognition accuracy.

[0074] The present invention provides a method for identifying abnormalities in long-span bridge structure health monitoring based on image classification. It has the following beneficial effects:

[0075] (I) This large-span bridge structure health monitoring anomaly recognition method based on image classification generates and optimizes abnormal samples through generative adversarial networks, alleviating the data scarcity situation; the dynamic generation-learning-optimization closed-loop system and distributed collaboration platform reduce the labeling cost; the annotation quality stability is improved with the help of weakly supervised learning, knowledge graphs and automated quality inspection tools; the self-supervised learning technology is used to improve the utilization rate of unlabeled data in multimodal data fusion, comprehensively solving the problems of data labeling and quality, and providing strong support for bridge structure health monitoring.

[0076] (II) This large-span bridge structure health monitoring anomaly recognition method based on image classification uses a generative adversarial network (GAN) to generate abnormal samples, including common bridge anomaly types such as cracks, spalling, and rust; dynamically adjusts the distribution of the generated model through active learning feedback signals to make it conform to the actual bridge anomaly conditions; combines multi-institutional data and shares model updates through a federated learning framework. Without sharing the original data, it integrates the data distribution of multiple institutions to generate general abnormal samples suitable for multiple scenarios, which expands the data volume and diversity, provides richer materials for model training, and improves the problem of data scarcity. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a flowchart of an abnormality recognition method for long-span bridge structure health monitoring based on image classification;

[0078] Figure 2 Schematic diagram of the process of anomaly identification in long-span bridge structure health monitoring based on GAN and multimodal data fusion. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] Example 1, please refer to Figure 1 and Figure 2 The present invention provides a technical solution: a method for identifying abnormalities in the health monitoring of a large-span bridge structure based on image classification, the method comprising: using a high-resolution digital camera, an unmanned aerial vehicle or a fixed camera to photograph key parts of the bridge, especially areas where cracks, spalling and corrosion may occur; these devices can be installed on the bridge deck, piers, hangers and other parts of the bridge for regular or real-time monitoring; each image should contain sufficient contextual information, that is, when shooting, it is necessary to ensure the clarity of the target area and capture the relevant features of the overall structure of the bridge; in addition to image data, bridge vibration, deformation and stress data collected by strain sensors, accelerometers and temperature sensors will be used as supplementary information for multimodal data analysis;

[0081] In order to reduce the interference caused by image quality problems, the median filter and mean filter in the denoising algorithm are used to denoise the collected images; through denoising, the noise caused by weather and light interference is removed; the image is enhanced by histogram equalization, contrast stretching, and data enhancement methods to improve the contrast and clarity of the image, ensuring that the damage features: cracks and corrosion are more prominent, thereby improving the accuracy of subsequent image classification;

[0082] Construct a deep convolutional neural network (CNN), which contains multiple convolutional layers, pooling layers, fully connected layers and output layers; the convolutional layer is used to extract local features in the image: cracks, corrosion marks, the pooling layer is used to reduce the dimension and reduce the computational complexity, and the fully connected layer is used to combine and classify high-level features; finally, the network outputs the classification results, including no damage, cracks, corrosion, and peeling;

[0083] Use manual annotation methods to classify image data and annotate the specific damage types in each image. To improve classification accuracy, use a large number of damage type images to ensure that the training set is representative and covers various typical situations. Considering the large amount of data and computing resources required to train deep neural networks, transfer learning methods can be used to fine-tune deep learning models that have been pre-trained in the field of natural image classification. Through transfer learning, model training can be accelerated and its accuracy can be improved.

[0084] The trained CNN model is applied to the actual monitoring process to automatically detect and identify abnormal phenomena in bridge images. By reasoning about the image data, the model can identify cracks, corrosion, and deformation abnormalities on the bridge surface, and mark the location and degree of the abnormality.

[0085] Based on the location information of abnormalities such as cracks or corrosion in the image, combined with strain and vibration sensor data, the impact of these abnormalities on the safety of the bridge structure is analyzed; for example, the expansion of cracks and the depth of corrosion may lead to a reduction in the bearing capacity of the bridge structure;

[0086] Build a decision support system based on image recognition results; based on the identified abnormalities, the system provides maintenance personnel with real-time reports on the specific damage type, damage degree, and possible impact range to assist maintenance personnel in making decisions; for example, for identified cracks, the system will give recommendations on the length and width of the cracks and whether they need to be repaired immediately;

[0087] In addition to image data, strain and vibration in sensor data are also important bases for judging the health of bridge structures. Through data fusion technology, image data and sensor data are combined to form comprehensive bridge health monitoring data. For example, if the crack position identified by the image is also detected by the strain sensor as a strain change, it can be determined whether the crack poses a threat to the structural safety.

[0088] Deep learning fusion model: A multi-input neural network model is used to process image and sensor data separately, and finally fuse information from different modes to output a comprehensive judgment result; this model can further improve the accuracy of bridge health judgment based on the complementarity of image and sensor data;

[0089] The image acquisition system, sensor system and data processing unit are integrated into a unified platform and installed at key locations on the bridge. The cameras and sensors should be able to cover every important part of the bridge and monitor the health of the structure regularly or in real time. All collected data, including image data, sensor data and recognition results, will be stored, processed and analyzed through the cloud platform. The cloud platform can provide powerful computing power to support large-scale image data processing and real-time monitoring. Finally, a user-friendly interface is developed for real-time monitoring and data query by bridge managers. The interface can intuitively display the health status of each monitoring point and provide abnormal alarm information.

[0090] Efficiency: The efficiency of bridge structure health monitoring has been greatly improved through the automated monitoring system based on image classification. Compared with the traditional manual inspection method, automated monitoring can not only reduce labor costs, but also ensure the real-time and comprehensiveness of monitoring.

[0091] Accuracy: After being trained on a large-scale dataset, the deep learning model can accurately identify various types of damage on the bridge surface, especially in low-light or complex environments;

[0092] Decision support: The system can provide maintenance personnel with detailed abnormality reports, helping them to accurately identify the parts of the bridge that need attention and give repair suggestions, which enhances the scientificity and timeliness of decision-making;

[0093] Accuracy evaluation: The accuracy of the image classification model is evaluated by comparing the manually labeled data with the model output results; the accuracy of the model is above 80%, and through continuous optimization, the recognition accuracy is further improved;

[0094] Model optimization: As the amount of data increases, the model is regularly updated and retrained to ensure that the model can adapt to the structural characteristics of different bridges and environmental changes and improve generalization capabilities;

[0095] The anomaly recognition method for health monitoring of large-span bridge structures based on image classification, combined with advanced technologies such as deep learning, image processing, and sensor data fusion, provides an efficient and accurate solution for bridge health monitoring. Through automated monitoring and anomaly recognition, the safety and service life of bridge structures are significantly improved, and the workload of manual inspections is reduced. This method is not only innovative in theory, but also shows good results in practical applications. It can be widely used in the health monitoring of large-span bridges, tunnels and other infrastructure.

[0096] Example 2: Abnormal identification of health monitoring of large-span bridge structures based on GAN and multimodal data fusion. Abnormal samples are generated based on the generative adversarial network (GAN), and image, vibration, strain and environmental data are combined to realize abnormal identification of large-span bridge structures through cross-modal data fusion and automatic annotation optimization technology. The steps are as follows:

[0097] Step 1: First, use high-resolution cameras to capture images of key parts of long-span bridges to obtain image data of abnormalities such as cracks, peeling, and rust on the bridges. At the same time, install sensors to collect vibration, strain, and environmental data. For low-quality images collected, apply image denoising, super-resolution reconstruction, and clarity enhancement techniques to improve the usability of images and the ability to express abnormal features.

[0098] Step 2: Generate image samples of simulated bridge anomalies, including cracks, spalling, and rust, using a generative adversarial network (GAN). The generator generates high-quality anomaly images by learning the characteristics of bridge anomalies. Active learning methods are used to select samples with high uncertainty in the model generation process. These samples are submitted to human experts for annotation and fed back to the GAN generator to optimize its training process, so that the generated anomaly samples are more consistent with the anomaly distribution of actual bridges.

[0099] Step 3: The collected multimodal data include images, vibration, strain and environmental data. First, a convolutional neural network (CNN) is used to extract abnormal features such as cracks and spalling in the image. The frequency domain features of the vibration data are extracted through Fourier transform, and the change trend of the strain data is analyzed using a time series model. At the same time, the impact of temperature, humidity and wind speed in the environmental data on bridge abnormalities is statistically analyzed. The Transformer model is used to align image features and non-image features (such as vibration, strain, and environmental data), and the features of different modalities are fused through the attention mechanism to generate a comprehensive abnormality description.

[0100] Step 4: Use the abnormal samples generated by GAN and the automated annotation tool to perform preliminary annotation, including crack width and spalling area; the annotation results will be transmitted to the annotation model and optimized through weakly supervised learning and quality detection tools; based on the performance of each round of annotation results and the model, combined with the feedback data of manual annotation, the abnormal samples generated by GAN are further optimized through closed-loop learning, and the accuracy of annotation is improved;

[0101] Step 5: Based on the generated and optimized labeled data, train an anomaly detection model to perform real-time detection of the health status of the bridge structure; the model can identify anomalies such as cracks, peeling, rust, etc. of the bridge, and generate a detailed anomaly report; combined with the actual condition of the bridge, generate a decision support system to provide maintenance personnel with suggestions for bridge maintenance and repair, and respond to abnormal conditions in a timely manner.

[0102] Example 3: Abnormal identification of bridge structure health monitoring based on federated learning and distributed collaboration, using a federated learning framework, sharing the training results of the abnormal identification model through cross-institutional collaboration, while ensuring data privacy; combining distributed labeling tasks and labeling optimization to complete abnormal identification in long-span bridge structure health monitoring; including the following steps:

[0103] Step 1: Collect images, vibration, strain and environmental condition data from multiple bridges; bridge monitoring units in different regions independently collect and store data to ensure data privacy; apply denoising and enhancement techniques to the collected image data to remove noise in the image and improve image quality; apply fast Fourier transform to vibration data to extract key frequency domain features, and analyze abnormal fluctuations in strain data through time series modeling;

[0104] Step 2: Each participating institution trains a preliminary anomaly recognition model locally using local data and generates model weights; all institutions use the same deep learning model structure to ensure that model parameters can be shared; the model parameters uploaded by each institution are aggregated through a federated averaging algorithm to form a global optimization model; the global model improves the generalization and adaptability of the model by combining monitoring data from multiple institutions; through a federated learning framework, ensure that the original data of each institution is not shared, only model updates are shared, and data privacy is protected;

[0105] Step 3: Based on the model’s prediction results and uncertainty analysis, decompose the large-scale annotation task into fine-grained subtasks, including crack classification and spalling area segmentation; assign these tasks to different annotation teams for collaboration, which include a mix of machine and manual annotation; use automated quality inspection tools to check the consistency of the annotation results, check for overlapping areas, missing annotations, and annotation errors; optimize the annotation results through the knowledge graph, adjust the annotation priority, and further improve the model training through feedback;

[0106] Step 4: Use the globally optimized deep learning model to analyze the multimodal data of the bridge and identify abnormal features such as cracks, spalling, and rust. Generate a detailed abnormality report through the abnormal labels and location annotations output by the model.

[0107] Feedback mechanism: The detected anomaly information is provided to relevant maintenance personnel through the feedback mechanism, and maintenance suggestions are provided based on the health status of the bridge. The results of anomaly detection will be used as input for model iteration to further optimize the recognition model.

[0108] Step 5: Share model updates and training results between different institutions to jointly promote the advancement of anomaly recognition technology without leaking data; each institution optimizes the local model according to its specific bridge monitoring needs and shares updates through the federated learning framework; combine monitoring data from different regions and bridge types to generate a general anomaly recognition model suitable for multiple scenarios; this model can adapt to bridge health monitoring needs in climate and corrosive environments.

[0109] Through the abnormal recognition method of long-span bridge structure health monitoring based on image classification, generative adversarial network (GAN) and active learning are used to optimize abnormal samples to ensure the authenticity of abnormal samples; through the fusion of multimodal data and cross-modal feature alignment, the model's ability to recognize abnormalities of complex bridge structures is improved; distributed annotation is combined with automatic optimization to ensure the consistency and high quality of data annotation; cross-institutional collaboration and privacy protection are achieved through the federated learning framework to improve the generalization ability of the model; and the complete feedback mechanism and decision support system ensure real-time response and accurate diagnosis of bridge health monitoring.

[0110] It should be further explained that, in the specific implementation process, the generative adversarial network (GAN) is used to generate abnormal samples, including common bridge abnormalities such as cracks, spalling, and rust; the distribution of the generated model is dynamically adjusted through active learning feedback signals to make it conform to the actual bridge abnormalities; combined with multi-institutional data, the model update is shared through the federated learning framework, and the multi-institutional data distribution is integrated to generate general abnormal samples suitable for multiple scenarios without sharing the original data, which expands the data volume and diversity, provides richer materials for model training, and improves the problem of data scarcity;

[0111] A dynamic generative learning optimization closed-loop system is established, in which a generative adversarial network generates preliminary annotations, reducing the workload of manual initial annotations; weakly supervised learning and active learning are used to annotate high uncertainty areas, improving the pertinence and efficiency of annotations; and a distributed collaborative platform is used to automatically decompose large-scale annotation tasks into fine-grained subtasks based on model uncertainty and abnormal complexity, and reasonably allocate them to cloud-based annotation models or manual annotation teams, achieving efficient use of resources and improving annotation efficiency, thereby reducing annotation costs and solving the problem of high annotation costs;

[0112] In the data annotation optimization and closed-loop learning system, weakly supervised learning technology is used to optimize the initial annotation results based on a small amount of high-quality manually annotated data, thereby improving the accuracy of annotation. A knowledge graph containing bridge structural characteristics, anomaly types, and environmental influencing factors is constructed to guide the priority allocation of annotation ranges, and analyze the causes and evolution trends of anomalies, thereby enhancing the scientificity and rationality of annotation and overcoming the problem of unstable annotation quality.

[0113] The automated quality inspection tool combines the rule engine and deep learning model to detect boundary overlap, inconsistency, and missing annotations in the annotation data, and automatically corrects the annotation errors. This includes adjusting the crack boundary or spalling area division, and performing consistency comparison analysis between multi-modal annotation data to ensure the consistency of image and vibration data annotation, which significantly improves the stability of annotation quality.

[0114] In the process of multimodal data fusion, self-supervised learning technology is used in combination with unlabeled data to optimize cross-modal feature representation; this includes extracting features from each modal data separately and fusing them through the Transformer model after acquiring multimodal data such as images, vibrations, strains and environmental conditions. In the fusion process, unlabeled data is used to further optimize the features, fully tapping the value of unlabeled data, improving the utilization rate of unlabeled data in model training, and solving the problem of low utilization rate of unlabeled data.

[0115] The anomaly recognition method for long-span bridge structure health monitoring based on image classification has achieved remarkable results in solving the problems of insufficient data labeling and quality. It generates and optimizes abnormal samples through generative adversarial networks, alleviating the data scarcity situation; the dynamic generation-learning-optimization closed-loop system and distributed collaboration platform reduce the labeling cost; the stability of labeling quality is improved with the help of weakly supervised learning, knowledge graphs and automated quality inspection tools; the self-supervised learning technology is used to improve the utilization rate of unlabeled data when multimodal data is fused, which comprehensively solves the problems of data labeling and quality and provides strong support for bridge structure health monitoring.

[0116] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0117] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An abnormality recognition method for long-span bridge structure health monitoring based on image classification, characterized in that: The method comprises the following steps: Step 1: Generate abnormal samples using a generative adversarial network (GAN), including cracks, spalling, and rust; dynamically adjust the distribution of the generated model based on active learning feedback signals to make the generated data consistent with the actual bridge abnormality distribution; use image enhancement technology to denoise, super-resolution reconstruct, and optimize clarity of low-quality images to improve data quality; Step 2: Collect multimodal monitoring data including image data, vibration data, strain data and environmental condition data; extract and fuse multimodal features using a cross-modal feature alignment model; generate comprehensive anomaly samples including time series, environmental context and spatial features; Step 3: Establish a dynamic generation-learning-optimization closed-loop system, including: generating preliminary annotations using a generative adversarial network; annotating high-uncertainty areas based on weakly supervised learning and active learning; using automated quality inspection tools to detect the boundaries and regional consistency of the annotated data, and optimizing the annotation priority and range through the knowledge graph; feeding back the newly annotated samples to the generation model and annotation model through closed-loop feedback; Step 4: Decompose the large-scale labeling task into fine-grained subtasks based on the uncertainty analysis of the model, including anomaly type classification, dimension labeling, and area division; use the distributed collaboration platform to assign subtasks to the cloud labeling model or manual labeling team; automatically merge and align the distributed labeling results, and use quality verification tools to ensure labeling consistency and reliability; Step 5: Institutions share model updates through the federated learning framework without sharing original data. The federated learning model evaluates and optimizes the quality of labeled data from each institution. Combined with the data distribution of multiple institutions, general anomaly samples suitable for multiple scenarios are generated.

2. The method for identifying abnormalities in long-span bridge structure health monitoring based on image classification according to claim 1 is characterized in that: The abnormal samples generated by the generative adversarial network (GAN) are dynamically optimized through active learning feedback signals, including: screening high-confidence samples for model training; after manually annotating high-uncertainty samples, the newly annotated data is used to update the generation model.

3. The method for identifying abnormalities in long-span bridge structure health monitoring based on image classification according to claim 2 is characterized in that: The fused multimodal features during the extraction and fusion of multimodal features using the cross-modal feature alignment model, i.e., multimodal data fusion, include: extracting features from image, vibration and strain data based on self-supervised learning technology; and contextualizing feature information of abnormal samples by integrating environmental conditions using a time series model.

4. The method for identifying abnormalities in long-span bridge structure health monitoring based on image classification according to claim 3 is characterized in that: The step three also includes the following process: Step c1: Generate preliminary annotation data using a generative adversarial network (GAN), including category annotations and bounding box annotations for abnormal areas; apply weakly supervised learning techniques to optimize preliminary annotation results based on a small amount of high-quality manually annotated data; Step c2: Construct a knowledge graph that includes bridge structure characteristics, anomaly types, and environmental influencing factors, and use knowledge graph reasoning to guide the priority allocation of the annotation range; for the anomaly samples after annotation, analyze the anomaly causes and possible evolution trends through the knowledge graph; Step c3: Use automated quality inspection tools to detect problems in the labeled data, including boundary overlap, inconsistency, and missing annotations; automatically correct annotation errors based on rule engines or deep learning models; Step c4: After each round of annotation optimization, use new high-quality annotation data to retrain the generation model and the annotation model; compare and analyze the optimized annotation results with the preliminary annotation results to gradually improve the performance of the annotation model.

5. The method for identifying abnormalities in long-span bridge structure health monitoring based on image classification according to claim 4 is characterized in that: The distributed collaboration platform completes the labeling task decomposition and collaboration through the following steps: Step d1: According to the complexity of the anomalies in the bridge image data and the uncertainty of the model, the large-scale labeling task is automatically decomposed into multiple fine-grained subtasks; the fine-grained subtasks include: anomaly type classification, dimension labeling, region segmentation and time series association. Combined with the task characteristics, the best labeling method is selected through the task allocation algorithm, including machine labeling or manual assisted labeling; Step d2: Build a distributed collaboration platform based on the cloud annotation system. The platform supports real-time synchronization of task status and allows multiple annotation teams to complete different subtasks in parallel. The platform schedules resources through task priority queues and assigns high-priority tasks to experienced annotators or models with the best performance. Step d3: Use the automated alignment tool to integrate the annotation results generated by different subtasks into abnormal sample annotations; use the deep learning model to perform consistency verification and conflict detection on the integration results to ensure the logical consistency and data reliability between the annotation modules; Step d4: For low-confidence or conflicting areas, automatically generate task feedback and reallocate them to appropriate annotation teams for secondary annotation or adjustment; regularly evaluate the overall annotation quality of the platform through the quality control module, and adjust the task allocation strategy for frequently occurring annotation deviations; Step d5: Combine the historical data of the annotation results in the platform to train the task allocation optimization model; in a distributed collaborative environment, dynamically adjust the task priority through the task progress prediction mechanism.

6. The method for identifying abnormalities in long-span bridge structure health monitoring based on image classification according to claim 5 is characterized in that: The cross-institutional data sharing and optimization driven by federated learning includes the following steps: Each institution retains local monitoring data and shares model parameter updates through federated learning; The federated learning model dynamically evaluates the quality of labeled data and optimizes the performance of the generation model and the labeling model.

7. The method for identifying abnormalities in long-span bridge structure health monitoring based on image classification according to claim 6 is characterized in that: In the step three, the closed-loop feedback feeds the newly labeled samples back to the generation model and the labeling model, including a closed-loop learning process, and the closed-loop learning process strengthens the tiny features of the image data through contrastive learning technology, including: generating feature embedding vectors of anomalies and background through self-supervised learning; applying contrastive learning methods to enhance the perception of tiny anomalies.

8. The method for identifying abnormalities in long-span bridge structure health monitoring based on image classification according to claim 7 is characterized in that: The method is applicable to a variety of bridge types and environmental conditions, including health monitoring of long-span bridges under strong winds, rain, snow, and direct sunlight.

9. The method for identifying abnormalities in long-span bridge structure health monitoring based on image classification according to claim 8 is characterized in that: The automated quality inspection tool combines a rules engine and a deep learning model to detect problems in the labeled data, such as inaccurate annotation area boundaries, inconsistent annotation of anomaly types, and deviations between annotation dimensions and actual features.

10. The method for identifying abnormalities in long-span bridge structure health monitoring based on image classification according to claim 9 is characterized in that: The method is implemented through a dual-loop technical framework, including a preliminary loop and a deep loop. The preliminary loop is used to complete basic annotation and optimize low-quality images; the deep loop further improves the annotation quality and model performance through deep feature mining and multi-stage optimization.

Images