Bridge defect automatic identification and inspection method based on deep learning

Abstract

The invention relates to the technical field of bridge detection, in particular to a bridge defect automatic identification and inspection method based on deep learning, which comprises the following steps: S1, multi-modal data acquisition: acquiring multi-modal data of the surface and internal structure of a bridge; s2, multi-modal data preprocessing: generating a space-time aligned enhanced data set; s3, feature fusion and extraction: generating a fusion feature spectrum; s4, adaptively optimizing the characteristic spectrum: adjusting the connection weight between network layers according to the real-time environment parameters; s5, defect positioning and type identification: positioning space coordinates of defects and synchronously outputting probability distribution of defect types; and S6, defect analysis and report generation: generating an inspection report including defect evolution trend analysis based on the result of defect positioning and type identification. According to the invention, the bridge inspection efficiency is improved, the labor cost is reduced, and the intelligent upgrading of the bridge management field is further promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge detection, and particularly to an automatic bridge defect recognition and inspection method based on deep learning. Background Art

[0002] With the rapid development of the transportation industry, bridges, as a key part of transportation infrastructure, bear a huge traffic flow and ensure people's travel safety. The health status of bridges directly affects traffic safety and the service life of bridges. Therefore, regular bridge inspection and maintenance are particularly important. Traditional bridge inspection methods mainly rely on manual inspection. Workers conduct close observation and detection by climbing, scaling, etc. This method not only has a high work intensity and low efficiency, but also has certain safety hazards and high personnel costs, and it is difficult to comprehensively cover all parts of the bridge structure, especially some inaccessible parts.

[0003] Currently, with the continuous development of unmanned aerial vehicle (UAV) technology and sensor devices, UAV-based bridge inspection technology has gradually been applied. However, there are still some deficiencies in the existing UAV bridge inspection technology. First, in terms of data collection, although UAVs can carry a variety of sensors, the data collected by different sensors often have problems of spatio-temporal asynchrony, which leads to great difficulties in data fusion and subsequent analysis. Second, most of the existing defect recognition methods rely on manual or simple rule algorithms, lacking intelligence and automation, and unable to make full use of the advantages of technologies such as deep learning, resulting in the accuracy and efficiency of defect recognition being unable to meet actual needs. Finally, the existing technologies generally lack defect evolution trend analysis and intelligent decision support, making it difficult to predict the development trend of defects in advance and unable to provide effective decision support for bridge maintenance and repair.

[0004] The purpose of the present invention is to overcome the deficiencies existing in the prior art, and propose an automatic bridge defect recognition and inspection method based on deep learning, which improves the automation and intelligence level of bridge inspection, reduces labor costs, and extends the service life of bridges to ensure traffic safety. Summary of the Invention

[0005] The present invention provides an automatic bridge defect recognition and inspection method based on deep learning.

[0006] The automatic bridge defect recognition and inspection method based on deep learning includes the following steps:

[0007] S1, multi-modal data collection: Obtain multi-modal data of the bridge surface and internal structure through multi-dimensional image acquisition devices carried by UAVs, including visible light images, infrared thermal imaging data, and three-dimensional laser point cloud data;

[0008] S2, Multimodal Data Preprocessing: Use the spatio-temporal synchronization enhancement algorithm to preprocess the multimodal data and generate a spatio-temporally aligned enhanced dataset;

[0009] S3, Feature Fusion and Extraction: Input the spatio-temporally aligned enhanced dataset into the multimodal feature fusion network, and extract defect features of different dimensions through a cross-scale convolutional kernel group to generate a fused feature map;

[0010] S4, Adaptive Optimization of the Feature Map: Based on the dynamic weight allocation mechanism, adaptively optimize the fused feature map and adjust the connection weights between network layers according to real-time environmental parameters;

[0011] S5, Defect Location and Type Recognition: Input the optimized fused feature map into the bidirectional attention localization mechanism to locate the spatial coordinates of the defect and simultaneously output the probability distribution of the defect type;

[0012] S6, Defect Analysis and Report Generation: Based on the results of defect location and type recognition, generate an inspection report including the analysis of the defect evolution trend.

[0013] Optionally, the multimodal data acquisition in S1 includes:

[0014] S11, Selection and Configuration of UAV-Borne Equipment: Select a UAV platform and carry multi-dimensional image acquisition equipment, including a visible light image acquisition unit, an infrared thermal imaging unit, and a three-dimensional laser point cloud scanner;

[0015] S12, Data Acquisition of Bridge Surface and Internal Structure: By controlling the UAV to fly to different positions and angles of the bridge, acquire multimodal data of the bridge surface and internal structure, and obtain a dataset including visible light images, infrared thermal imaging data, and three-dimensional laser point cloud data.

[0016] Optionally, the multimodal data preprocessing in S2 includes:

[0017] S21, Spatio-Temporal Data Synchronization Analysis: Analyze the time and space dimensions in the multimodal data, and identify the synchronization deviation between different data sources by calculating the timestamps and spatial coordinates of each modality data;

[0018] S22, Application of Spatio-Temporal Enhancement Algorithm: Enhance the multimodal data through the spatio-temporal synchronization enhancement algorithm to improve the spatio-temporal alignment accuracy of each modality data and generate an enhanced dataset D aug ;

[0019] S23, Generation of Spatio-Temporally Aligned Dataset: According to the generated enhanced dataset D aug , generate a spatio-temporally aligned enhanced dataset D aligned ;

[0020] S24, Data quality assessment: Evaluate the spatio-temporal alignment accuracy of the dataset by calculating the Mean Alignment Error (MAE).

[0021] Optionally, the feature fusion and extraction in S3 includes:

[0022] S31, Input the enhanced dataset with spatio-temporal alignment: Input the enhanced dataset with spatio-temporal alignment into the multi-modal feature fusion network;

[0023] S32, Feature extraction using cross-scale convolutional kernel groups: Extract defect feature maps F of different scales and dimensions through the cross-scale convolutional kernel groups in the multi-modal feature fusion network i ;

[0024] S33, Fuse features from different modalities: Fuse the feature maps F1, F2, …, F from different modalities n to generate a multi-modal fusion feature map F fusion .

[0025] Optionally, the adaptive optimization of the feature map in S4 includes:

[0026] S41, Real-time acquisition of environmental parameters: Real-time acquire the current environmental parameters E through environmental sensors, including temperature, humidity, and light intensity;

[0027] S42, Design of dynamic weight allocation mechanism: Adjust the weights of each layer of the network through an adaptive mechanism according to the real-time environmental parameters E;

[0028] S43, Optimize the weight connections of the fusion feature map: Dynamically optimize the connection weights of each network layer according to the real-time environmental parameters and output the adaptively optimized fusion feature map F opt .

[0029] Optionally, the defect location and type identification in S5 includes:

[0030] S51, Defect spatial coordinate location: Input the adaptively optimized fusion feature map into the bidirectional attention location mechanism to locate the spatial coordinates of the defect;

[0031] S52, Defect type identification and probability distribution output: Based on the spatial coordinates of the located defect, the bidirectional attention mechanism identifies the type of each defect and outputs the probability distribution of the defect type.

[0032] Optionally, the defect spatial coordinate location in S51 includes:

[0033] S511, Bidirectional attention mechanism calculation: In the bidirectional attention mechanism, through forward propagation, for the adaptively optimized fusion feature map F optProcess it to calculate the importance weight A at each position i,j and adjust and strengthen the weights of the defect areas through backpropagation;

[0034] S512, Defect space coordinate calculation: Based on the calculated attention weight A i,j , select the area with the highest weight as the space coordinate C of the defect.

[0035] Optionally, the defect type recognition and probability distribution output in S52 include:

[0036] S521, Defect area extraction based on space coordinates: Based on the obtained space coordinate C=(x c , y c ) of the defect, extract the feature map F opt corresponding to the defect area from the adaptively optimized fused feature map F region ;

[0037] S522, Defect type recognition: Input the extracted defect area feature map F region into the bidirectional attention mechanism, perform type recognition according to the features of the defect area, and calculate the probability P t of each defect type through forward propagation;

[0038] S523, Output defect type probability distribution: According to the calculated defect type probabilities P1, P2,..., P T , output the type and its corresponding probability distribution of each defect area.

[0039] Optionally, the defect analysis and report generation in S6 include:

[0040] S61, Defect evolution trend analysis: Based on historical multi-modal data and the space coordinates and type recognition results of current defects, analyze the evolution trend of defects, and predict the future development trend of defects through a regression analysis model;

[0041] S62, Generate inspection report: Combine the results of defect location and type recognition and defect evolution trend analysis to generate an inspection report including the defect evolution trend.

[0042] Optionally, the generation of the inspection report in S62 includes:

[0043] S621, Defect location and type: List the position coordinates and probability distribution of type recognition of each defect;

[0044] S622, Defect evolution trend: Show the evolution trend of defects at different time points and predict the state change of defects in the future;

[0045] S623, Maintenance suggestions: According to the type and evolution trend of defects, corresponding repair or reinforcement suggestions are put forward, including crack repair, corrosion prevention and control, local reinforcement and replacement, bridge deck or bearing replacement, extended monitoring and tracking, and structural reconstruction and renovation.

[0046] Advantages of the present invention:

[0047] In the present invention, by comprehensively applying drones, multi-modal data acquisition technology and spatio-temporal synchronization enhancement algorithms, multi-modal data on the surface and internal structure of bridges can be efficiently and accurately collected, and spatio-temporal alignment and enhancement processing can be carried out, providing high-quality input data for subsequent defect identification and analysis. Through the fusion and precise alignment of multiple modal data, the integrity and complementarity of the data are significantly improved, thereby effectively enhancing the defect recognition ability of the deep learning model and ensuring stable and efficient automated inspection in various environments.

[0048] In the present invention, an adaptive optimization mechanism is introduced to adjust the weights of the feature maps based on real-time environmental parameters, improving the adaptability and accuracy of the model in complex environments. Combined with the bidirectional attention mechanism, the spatial coordinates of defects can be accurately located, and the types of defects can be identified and the probability distribution output. Through this adaptive optimization and precise positioning, efficient detection and classification of bridge defects are achieved, not only improving the recognition accuracy, but also providing a reliable basis for subsequent maintenance decisions, further promoting the development of bridge inspection intelligence and automation.

[0049] In the present invention, through defect type identification and evolution trend analysis, an inspection report including the evolution trend of defects can be automatically generated, providing comprehensive and accurate defect assessment and maintenance suggestions for bridge managers. By combining the type and evolution trend of defects, the report can early warn of potential structural risks, help managers make timely repair or reinforcement decisions, optimize resource allocation, extend the service life of bridges, and ensure the safety and reliability of bridges. This intelligent inspection and report generation mode greatly improves the efficiency of bridge inspection, reduces labor costs, and further promotes the intelligent upgrade in the field of bridge management. Description of the drawings

[0050] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are only those of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0051] Figure 1 It is a schematic flow diagram of the identification and inspection method for the embodiments of the present invention;

[0052] Figure 2Schematic diagram of the adaptive optimization feature map according to an embodiment of the present invention. Detailed implementation manners

[0053] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. At the same time, it should be noted here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments. For some well-known technologies, those skilled in the art can also adopt other alternative methods for implementation; moreover, the accompanying drawings are only for more specifically describing the embodiments, and are not intended to specifically limit the present invention.

[0054] It should be noted that in the specification, references to "an embodiment", "embodiments", "exemplary embodiments", "some embodiments", etc. indicate that the described embodiments may include specific features, structures, or characteristics, but not necessarily every embodiment includes such specific features, structures, or characteristics. Additionally, when combining embodiments to describe specific features, structures, or characteristics, implementing such features, structures, or characteristics in combination with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the relevant art.

[0055] Generally, terms can be understood at least in part from their use in context. For example, at least in part depending on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or can be used to describe a combination of features, structures, or characteristics in a plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey a set of exclusive factors, but rather can alternatively, at least in part depending on the context, allow for the existence of other factors that may not be explicitly described.

[0056] As Figure 1 - Figure 2 shown, the automatic bridge defect identification and inspection method based on deep learning includes the following steps:

[0057] S1, Multi-modal data acquisition: Obtain multi-modal data of the bridge surface and internal structure through a multi-dimensional image acquisition device carried by a drone, including visible light images, infrared thermal imaging data, and three-dimensional laser point cloud data;

[0058] S2, Multi-modal data preprocessing: Use a spatio-temporal synchronization enhancement algorithm to preprocess the multi-modal data to generate a spatio-temporally aligned enhanced data set;

[0059] S3, Feature fusion and extraction: Input the spatio-temporally aligned enhanced data set into a multi-modal feature fusion network, and extract defect features of different dimensions through a cross-scale convolutional kernel group to generate a fused feature map;

[0060] S4, Adaptive optimization of the feature map: Based on a dynamic weight allocation mechanism, adaptively optimize the fused feature map, and adjust the connection weights between network layers according to real-time environmental parameters to improve the model recognition accuracy;

[0061] S5, Defect Location and Type Identification: Input the optimized fused feature map into the bidirectional attention location mechanism to locate the spatial coordinates of the defect and simultaneously output the probability distribution of the defect type.

[0062] S6, Defect Analysis and Report Generation: Generate an inspection report including the analysis of the defect evolution trend based on the results of defect location and type identification.

[0063] Through the above content, efficient and accurate bridge defect identification and location are achieved. It can comprehensively utilize the multi-dimensional image data collected by the unmanned aerial vehicle, automatically analyze and evaluate the health status of the bridge structure, provide real-time and accurate defect detection results, and generate an inspection report with the analysis of the evolution trend, improving the automation and intelligence level of bridge inspection, reducing the labor cost, and at the same time enhancing the efficiency and safety of bridge management and maintenance.

[0064] The multi-modal data collection in S1 includes:

[0065] S11, Selection and Configuration of Equipment Carried by the Unmanned Aerial Vehicle: Select the unmanned aerial vehicle platform and carry multi-dimensional image collection equipment, including a visible light image collection unit, an infrared thermal imaging unit, and a three-dimensional laser point cloud scanner, to ensure the compatibility and data synchronization of each device.

[0066] S12, Data Collection of the Bridge Surface and Internal Structure: Control the unmanned aerial vehicle to fly to different positions and angles of the bridge to collect multi-modal data of the bridge surface and internal structure, and obtain a data set including visible light images, infrared thermal imaging data, and three-dimensional laser point cloud data.

[0067] Through the above content, detailed information about the bridge surface and internal structure can be comprehensively and efficiently obtained. By reasonably configuring and optimizing the equipment, the data synchronization and high-precision collection are ensured, enabling the accurate capture of key defect information under different environments and complex structure conditions. This not only improves the coverage and accuracy of data collection but also greatly reduces the risks and costs of manual inspection, enhancing the automation and intelligence level of bridge health monitoring.

[0068] The multi-modal data preprocessing in S2 includes:

[0069] S21, Spatiotemporal Data Synchronization Analysis: Analyze the time and space dimensions in the multi-modal data. By calculating the timestamps and spatial coordinates of each modal data, identify the synchronization deviation between different data sources to ensure the consistency of all data in time and space, expressed as:

[0070] T aligned = T visible + ΔT;

[0071] Among them, T aligned is the aligned timestamp, T visible is the timestamp of the visible light image data, and ΔT is the adjustment deviation time;

[0072] S22, Application of spatio-temporal enhancement algorithm: Enhance multi-modal data through the spatio-temporal synchronization enhancement algorithm to improve the spatio-temporal alignment accuracy of each modal data. Let the input multi-modal data be D = [D1, D2,..., D n , i = [1, 2,..., n], where each D i represents a type of modal data (visible light, infrared, laser point cloud). The spatio-temporal enhancement algorithm generates the enhanced data set D aug by adjusting the temporal and spatial relationships between modalities, which is expressed as:

[0073] D aug = f(D, T aligned , S aligned );;

[0074] Among them, D aug is the enhanced multi-modal data set, f(·) is the spatio-temporal enhancement function, T aligned is the aligned timestamp, and S aligned is the aligned spatial coordinate;

[0075] The spatio-temporal enhancement function f(·) is expressed as:

[0076]

[0077] Among them, D aug is the multi-modal data set after spatio-temporal enhancement, w i is the weight coefficient of the modal data D i , is the processing function for spatio-temporal alignment of the modal data D i , including temporal and spatial synchronization operations, and are the aligned timestamp and spatial coordinate of the modality D i respectively, and n is the number of modalities (visible light image, infrared thermal imaging data, 3D laser point cloud data);

[0078]

[0079] Among them, is an interpolation function for spatio-temporal compensation and interpolation of data;

[0080]

[0081] Among them, D i is the modal data D iThe value at the current moment, T current Is the current timestamp, T aligned Is the alignment timestamp of the modal data, ΔT is the time period length, D i ' Is the modal data D i The value at the interpolated moment;

[0082] S23, Generate a spatio-temporally aligned dataset: According to the generated enhanced dataset D aug , Generate a spatio-temporally aligned enhanced dataset D allgned , Expressed as:

[0083]

[0084] Among them, Represents each modal data and its corresponding timestamp and spatial coordinates, i = [1, 2,..., n];

[0085] S24, Data quality assessment: By calculating the mean alignment error (MAE), evaluate the spatio-temporal alignment accuracy of the dataset, expressed as:

[0086]

[0087] Among them, MAE is the mean alignment error, Is the aligned timestamp, Is the reference timestamp, N is the number of data points;

[0088] Through the above content, the consistency of various types of data in time and space is ensured. Through weighted interpolation and spatio-temporal compensation, the accuracy and reliability of data fusion are improved, enabling different modal data to work accurately together at the same time and space points. This not only improves the integrity and information complementarity of the data but also provides higher-quality input data for subsequent deep learning models, thereby enhancing the accuracy and robustness of defect recognition and further promoting the intelligent and automated development of bridge health monitoring.

[0089] The feature fusion and extraction in S3 include:

[0090] S31, Input the spatio-temporally aligned enhanced dataset: Input the enhanced dataset that has undergone spatio-temporal alignment into the multi-modal feature fusion network;

[0091] S32, Feature extraction by cross-scale convolutional kernel groups: Through the cross-scale convolutional kernel groups in the multi-modal feature fusion network, extract defect feature maps F of different scales and dimensions i , Each modal data D i Performs convolution operations through multiple convolutional kernels of different sizes. The goal of the cross-scale convolutional kernel group is to extract multi-level features from each modal data and capture the multi-scale information of the defect, expressed as:

[0092]

[0093] Among them, F i is the feature map extracted from modality D i and W k is the k-th convolutional kernel with a size of k×k, and D i is the i-th modality data of the input, and K is the total number of convolutional kernels (the number of cross-scale convolutional kernel groups);

[0094] S33, fusing features from different modalities: Fusing the feature maps F1, F2, …, F n from different modalities to generate a multi-modal fusion feature map F fusion , which is expressed as:

[0095]

[0096] Among them, F fusion is the fused feature map, w i ' is the weight coefficient of modality i, indicating the contribution degree of this modality to the fused feature map, and F i is the feature map of the i-th modality;

[0097] Through the above content, the advantageous information of each modality can be effectively integrated, the expression ability of defect features can be improved, multi-level features are extracted through the cross-scale convolutional kernel group, and combined with the weighted average method, it ensures that different modality features are reasonably weighted according to their importance during the fusion process, not only enhancing the complementarity of each modality data, but also improving the accuracy and robustness of the fused feature map.

[0098] The adaptive optimization feature map in S4 includes:

[0099] S41, real-time environmental parameter acquisition: The current environmental parameters E, including temperature, humidity, and light intensity, are collected in real time through environmental sensors, which is expressed as:

[0100] E = [E1, E2,..., E m ;

[0101] Among them, E i is the i-th environmental parameter, and i = [1, 2,..., m];

[0102] S42, design of dynamic weight allocation mechanism: According to the real-time environmental parameters E, the weights of each layer of the network are adjusted through an adaptive mechanism, which is expressed as:

[0103]

[0104] Among them, w i,t' is the dynamically adjusted weight of the i-th layer at time t, α i is the basic weight coefficient of this layer, β i is the environmental parameter E i the influence coefficient on the weight of this layer, E i is the i-th real-time environmental parameter, and max(E) is the maximum value among all environmental parameters for normalization;

[0105] S43. Optimize the weight connection of the fused feature map: Dynamically optimize the connection weights of each network layer according to the real-time environmental parameters to ensure that the output of each level adjusts its importance according to the current environment, and output the adaptively optimized fused feature map F opt , expressed as:

[0106]

[0107] where F opt is the adaptively optimized fused feature map, F i is the feature map of the i-th layer;

[0108] Through the above content, the connection weights between network layers are adaptively adjusted according to the real-time environmental parameters, enabling the model to flexibly cope with the impact of different environmental conditions on data quality and feature performance. By optimizing the weights of the fused feature map in real time, the adaptability of the model to various environmental changes can be effectively improved, the recognition accuracy and robustness can be enhanced. It not only improves the performance of the model in complex environments but also ensures fine-tuning during the feature extraction and fusion process, making defect recognition more accurate.

[0109] The defect location and type recognition in S5 include:

[0110] S51. Defect spatial coordinate location: Input the adaptively optimized fused feature map into the bidirectional attention location mechanism to locate the spatial coordinates of the defect. Through the attention mechanism, the model can focus on the defect area, suppress irrelevant information, and thus effectively identify and calibrate the position of the defect;

[0111] S52. Defect type recognition and probability distribution output: Based on the located spatial coordinates of the defect, the bidirectional attention mechanism performs type recognition on each defect and outputs the probability distribution of the defect type;

[0112] Through the above content, the spatial coordinates of the defect are accurately located, and the defect type is classified and recognized. Through the attention mechanism, the model can focus on the defect area, effectively exclude interference information, thereby improving the location accuracy and classification accuracy. It not only enhances the reliability of defect detection but also provides the probability distribution for different types of defects, providing strong support for subsequent bridge maintenance and repair decisions, and enhancing the practicality and accuracy of the intelligent inspection system.

[0113] The defect space coordinate positioning in S51 includes:

[0114] S511, calculating with the bidirectional attention mechanism: In the bidirectional attention mechanism, through forward propagation, the adaptively optimized fused feature map F opt is processed to calculate the importance weight A i,j for each position, and the weight of the defect area is adjusted and strengthened through backpropagation. The bidirectional attention mechanism is expressed as:

[0115]

[0116] where A i,j is the attention weight of the i-th row and j-th column in the feature map F opt , representing the degree of attention of this position to the defect. F opt (i,j) is the feature value of the fused feature map F opt at the i-th row and j-th column. H is the height or the number of rows of the feature map. F opt (i,k) is the feature value of the fused feature map F opt at the i-th row and k-th column;

[0117] S512, calculating the defect space coordinates: According to the calculated attention weight A i,j , the area with the highest weight is selected as the defect space coordinate C, which is expressed as:

[0118] C = argmax i,j A i,j ;

[0119] where C is the defect space coordinate, representing the image position with the highest attention weight;

[0120] Through the above content, it is possible to accurately focus on the defect area, suppress irrelevant information, thereby significantly improving the accuracy of defect positioning. Through the adaptively optimized fused feature map, the data advantages of different modalities can be fully utilized for accurate spatial coordinate positioning, which not only improves the recognition accuracy of the defect area but also can be dynamically adjusted according to the attention weight of the feature map to ensure stable and efficient detection and positioning of defects in different environments.

[0121] The defect type recognition and probability distribution output in S52 include:

[0122] S521, extracting the defect area based on the spatial coordinates: Based on the obtained defect space coordinate C = (x c , y c ), the feature map F opt corresponding to the defect area is extracted from the adaptively optimized fused feature map F region , which is expressed as:

[0123] F region = Extract(F opt , C);

[0124] where F region is the extracted defect region feature map, C = (x c , y c ) is the spatial coordinate of the defect, and Extract represents the operation of extracting the defect region from the feature map;

[0125] S522, Defect type identification: Input the extracted defect region feature map F region into the bidirectional attention mechanism, and perform type identification according to the features of the defect region. Through forward propagation, calculate the probability P t of each defect type, expressed as:

[0126]

[0127] where P t is the probability of the t-th defect type, W t is the classification weight of the t-th defect type, F region is the extracted defect region feature map, T is the total number of defect types, and W k is the classification weight of other defect types;

[0128] S523, Output the defect type probability distribution: According to the calculated defect type probabilities P1, P2,..., P T , output the type and its corresponding probability distribution of each defect region, expressed as:

[0129] P type = [P1, P2,..., P T ;

[0130] where P type is the defect type probability distribution, indicating the possibility of each type of defect;

[0131] Through the above content, it is possible to accurately identify the type based on the located defect region and output the probability distribution of each defect type. By extracting the features of the defect region and combining the attention mechanism, it is possible to focus on important features, suppress noise, and improve the classification accuracy. It can not only accurately identify the defect type but also provide a reliable probability distribution for each type, helping decision-makers evaluate the severity of the defect and the repair priority, thereby optimizing the bridge maintenance strategy.

[0132] The defect analysis and report generation in S6 include:

[0133] S61, Defect Evolution Trend Analysis: Based on historical multimodal data and the spatial coordinates and type recognition results of current defects, analyze the evolution trend of defects, and predict the future development trend of defects through a regression analysis model, expressed as:

[0134] e i+1 = α·e i + β·Δe i ;

[0135] where e i+1 is the predicted value of the defect state at the next time point, α and β are the regression coefficients of the model, and Δe i is the change in the current defect state, representing the change of the defect within the time interval, and e i is the state of the defect at the i-th time point;

[0136] S62, Generate Inspection Report: Combine the results of defect location and type recognition and defect evolution trend analysis to generate an inspection report including the defect evolution trend;

[0137] Through the above content, it is possible to provide more comprehensive and accurate data support for bridge maintenance. Through the prediction of the evolution trend, potential risks and defect expansion situations can be identified in advance, helping decision-makers to take more effective preventive and maintenance measures. At the same time, the detailed defect types and evolution trend analysis in the report can provide continuous reference for the long-term health monitoring of the bridge, improve the accuracy and efficiency of inspection, and ultimately optimize the allocation of maintenance resources.

[0138] The inspection report generated in S62 includes:

[0139] S621, Defect Location and Type: List the location coordinates of each defect and the probability distribution of type recognition;

[0140] S622, Defect Evolution Trend: Show the evolution trend of the defect at different time points and predict the state change of the defect in the future;

[0141] S623, Maintenance Suggestions: According to the type and evolution trend of the defect, put forward corresponding repair or reinforcement suggestions, including crack repair, corrosion prevention and control, local reinforcement and replacement, bridge deck or bearing replacement, extended monitoring and tracking, structural reconstruction and renovation;

[0142] Through the above content, the defect locations, the probability distributions of type recognition, the evolution trends and future predictions are listed in detail. Combined with the maintenance suggestions, it provides comprehensive and accurate data support for the maintenance decision-making of the bridge. By showing the evolution trends of the defects, the report can early warn of potential structural risks, help the managers make timely repair or reinforcement decisions. The maintenance suggestions cover different types of defects and repair schemes, ensuring that the most appropriate maintenance measures can be taken under different circumstances. This report not only improves the intelligent and refined level of bridge management, but also optimizes the resource allocation, extends the service life of the bridge, and ensures the safety and reliability of the bridge.

[0143] This invention covers any alternatives, modifications, equivalent methods and solutions made on the essence and scope of this invention. For the public to have a thorough understanding of this invention, specific details are elaborated in the following preferred embodiments of this invention. However, those skilled in the art can fully understand this invention even without the description of these details. Additionally, well-known methods, processes, procedures, components and circuits are not described in detail to avoid unnecessary confusion to the essence of this invention.

[0144] The above are only the preferred embodiments of this invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of this invention, several improvements and refinements can still be made, and these improvements and refinements should also be regarded as the protection scope of this invention.

Claims

1. An automatic bridge defect recognition and inspection method based on deep learning, characterized in that It includes the following steps: S1, Multimodal data acquisition: Obtain multimodal data of the bridge surface and internal structure through a multi-dimensional image acquisition device carried by a drone, including visible light images, infrared thermal imaging data, and three-dimensional laser point cloud data; S2, Multimodal data preprocessing: Use a spatio-temporal synchronization enhancement algorithm to preprocess the multimodal data and generate a spatio-temporally aligned enhanced data set; S3, Feature fusion and extraction: Input the spatio-temporally aligned enhanced data set into a multimodal feature fusion network, and extract defect features of different dimensions through a cross-scale convolutional kernel group to generate a fused feature map; S4, Adaptive optimization of the feature map: Based on a dynamic weight allocation mechanism, adaptively optimize the fused feature map and adjust the connection weights between network layers according to real-time environmental parameters; S5, Defect location and type recognition: Input the optimized fused feature map into a bidirectional attention location mechanism to locate the spatial coordinates of the defect and simultaneously output the probability distribution of the defect type; S6, Defect analysis and report generation: Based on the results of defect location and type recognition, generate an inspection report including an analysis of the defect evolution trend.

2. The automatic bridge defect recognition and inspection method based on deep learning according to claim 1, characterized in that The multimodal data acquisition in S1 includes: S11, Selection and configuration of the drone-borne equipment: Select a drone platform and carry a multi-dimensional image acquisition device, including a visible light image acquisition unit, an infrared thermal imaging unit, and a three-dimensional laser point cloud scanner; S12, Data acquisition of the bridge surface and internal structure: By controlling the drone to fly to different positions and angles of the bridge, acquire multimodal data of the bridge surface and internal structure, and obtain a data set including visible light images, infrared thermal imaging data, and three-dimensional laser point cloud data.

3. The automatic bridge defect identification and inspection method based on deep learning according to claim 1, characterized in that The multimodal data preprocessing in S2 includes: S21, Spatio-temporal data synchronization analysis: Analyze the time and space dimensions in the multimodal data, and identify the synchronization deviation between different data sources by calculating the timestamps and spatial coordinates of each modality data; S22, Application of spatio-temporal enhancement algorithm: enhancing multi-modal data through the spatio-temporal synchronization enhancement algorithm to improve the spatio-temporal alignment accuracy of each modal data and generate the enhanced dataset D aug ; S23, Generate a spatio-temporally aligned dataset: Based on the generated enhanced dataset D aug , generate an enhanced dataset D that is spatio-temporally aligned aligned ; S24, Data quality assessment: Evaluate the spatio-temporal alignment accuracy of the data set by calculating the average alignment error.

4. The automatic recognition and inspection method for bridge defects based on deep learning according to claim 1, characterized in that, The feature fusion and extraction in S3 includes: S31, Input the spatio-temporally aligned enhanced data set: Input the enhanced data set that has been spatio-temporally aligned into a multimodal feature fusion network; S32, Feature extraction using cross-scale convolutional kernel groups: Through the cross-scale convolutional kernel groups in the multi-modal feature fusion network, defect feature maps F of different scales and dimensions are extracted i ; S33, Fusing different modality features: Fusing the feature maps F1, F2, …, F from different modalities n to generate a multi-modal fusion feature map F fusion .

5. The automatic bridge defect recognition and inspection method based on deep learning according to claim 3, characterized in that, The adaptive optimization of the feature map in S4 includes: S41, Real-time environmental parameter acquisition: Real-time acquire the current environmental parameters E through environmental sensors, including temperature, humidity, and light intensity; S42, Design of the dynamic weight allocation mechanism: According to the real-time environmental parameters E, adjust the weights of each layer of the network through an adaptive mechanism; S43. Optimize the weight connection of the fused feature map: Dynamically optimize the connection weights of each network layer according to real-time environmental parameters, and output the fused feature map F after adaptive optimization opt .

6. The automatic bridge defect recognition and inspection method based on deep learning according to claim 5, characterized in that, The defect location and type recognition in S5 includes: S51, Defect spatial coordinate location: Input the adaptively optimized fused feature map into a bidirectional attention location mechanism to locate the spatial coordinates of the defect; S52, Defect type recognition and probability distribution output: Based on the spatial coordinates of the located defect, the bidirectional attention mechanism performs type recognition on each defect and outputs the probability distribution of the defect type.

7. The automatic bridge defect identification and inspection method based on deep learning according to claim 6, characterized in that, The defect spatial coordinate location in S51 includes: S511, Bidirectional Attention Mechanism Calculation: In the bidirectional attention mechanism, the adaptively optimized fused feature map F is processed through forward propagation opt to calculate the importance weight A at each position i,j , and the weights of the defect regions are adjusted and strengthened through backpropagation; S512, Defect space coordinate calculation: Based on the calculated attention weight A i,j , select the region with the highest weight as the spatial coordinate C of the defect.

8. The method for automatic identification and inspection of bridge defects based on deep learning according to claim 7, characterized in that, The defect type recognition and probability distribution output in S52 includes: S521, Defect area extraction based on spatial coordinates: Based on the obtained spatial coordinates C = (x c , y c ), extract the feature map F opt corresponding to the defect area from the adaptively optimized fused feature map F region ; S522, Defect type recognition: Input the extracted defect region feature map F region into the bidirectional attention mechanism, perform type recognition based on the features of the defect region, and calculate the probability P of each defect type through forward propagation t ; S523, Output the probability distribution of defect types: Based on the calculated defect type probabilities P1, P2,..., P T , output the type of each defect area and its corresponding probability distribution.

9. The automatic bridge defect recognition and inspection method based on deep learning according to claim 8, characterized in that The defect analysis and report generation in S6 includes: S61, Defect evolution trend analysis: Based on historical multi-modal data and the spatial coordinates and type recognition results of current defects, analyze the evolution trend of defects, and predict the future development trend of defects through a regression analysis model; S62, Generate inspection report: Combine the results of defect location and type recognition and defect evolution trend analysis to generate an inspection report including the defect evolution trend.

10. The automatic bridge defect recognition and inspection method based on deep learning according to claim 9, characterized in that The generation of the inspection report in S62 includes: S621, Defect location and type: List the location coordinates of each defect and the probability distribution of type recognition; S622, Defect evolution trend: Show the evolution trend of defects at different time points and predict the state changes of defects in the future; S623, Maintenance suggestions: According to the type and evolution trend of defects, put forward corresponding repair or reinforcement suggestions, including crack repair, corrosion prevention and reinforcement, local reinforcement and replacement, bridge deck or bearing replacement, extended monitoring and tracking, structure reconstruction and renovation.

Images