An infrastructure crack diagnosis method based on a multi-modal dynamic evolution model

Abstract

This invention relates to the field of infrastructure safety detection technology, specifically to a method for diagnosing infrastructure cracks based on a multimodal dynamic evolution model. The steps include: constructing a multimodal evolution simulator capable of internalizing the dynamic evolution laws of cracks during an offline training phase; using this simulator as a constraint on physical laws to drive a conditional generative adversarial network to generate augmented data conforming to the evolution laws, forming a virtual-real data fusion closed loop to train a final model with extremely strong generalization ability; and using this solidified model in the online inference phase to dynamically extrapolate real-time multimodal data, achieving early warning and accurate location of crack activity states by calculating the feature change rate. This invention solves problems such as static detection, single-modal analysis, sample scarcity, and data physical distortion, realizing a shift from passive identification to proactive prognosis.

Description

Technical Field

[0001] This invention relates to the field of infrastructure safety inspection technology, and in particular to a method for diagnosing infrastructure cracks based on a multimodal dynamic evolution model. Background Technology

[0002] Crack detection in critical infrastructure is a crucial step in ensuring structural safety and long-term stable operation, and is widely used in major infrastructure scenarios such as hydropower station water conveyance tunnels, dam pipe corridors, and thermal power boiler furnaces. With the continuous improvement of monitoring and intelligent capabilities, higher demands are being placed on the accuracy, real-time performance, and prognostic capabilities of crack identification and early warning.

[0003] Existing crack detection technologies for critical infrastructure suffer from the following limitations: First, most methods rely on single-modal information analysis, such as using only visible light or infrared images, making it difficult to comprehensively capture the integrated characteristics and evolutionary patterns of defects under multi-physics conditions. Second, crack samples in infrastructure scenarios, especially time-series data covering the entire lifecycle, are extremely scarce. Although data augmentation techniques such as generative adversarial networks have been used to alleviate the sample shortage problem, the generated data often lacks physical plausibility and fails to conform to the inherent dynamic laws of crack evolution, easily leading to a decline in the model's generalization ability in practical industrial applications. Furthermore, existing crack detection systems are mostly passive response modes, issuing alarms only after cracks have become significant, lacking the ability to predict and forecast dynamic evolution trends such as crack initiation and propagation, thus failing to achieve early intervention and precise location.

[0004] Although deep learning and data augmentation technologies have made significant progress in recent years, research on combining multimodal information fusion with crack dynamic evolution modeling to form an efficient and intelligent crack prognosis and diagnosis system for major infrastructure is still in its early stages. Most existing systems can only process static or single-modal data when detecting cracks, making it difficult to cope with the dynamic changes of defects and the multiphysics coupling characteristics in complex industrial environments, and failing to fully leverage their advantages in multimodal information fusion and predictive maintenance.

[0005] In summary, the limitations of existing technologies in multimodal fusion, dynamic modeling, and data generation make it difficult to achieve true predictive maintenance. Summary of the Invention

[0006] The purpose of this invention is to overcome the limitations of existing major infrastructure crack detection technologies, such as sample scarcity and physical distortion of generated data, and to provide a method for the prognosis and diagnosis of major infrastructure cracks based on a multimodal dynamic evolution model.

[0007] The objective of this invention can be achieved through the following technical solutions: A method for diagnosing infrastructure cracks based on a multimodal dynamic evolution model, the method comprising the following steps: Offline training phase: Generate multimodal evolution sequences covering the complete life cycle of defects based on physical law models, and construct the initial training dataset; A multimodal evolution simulator is initially trained using an initial training dataset. The multimodal evolution simulator includes a multimodal feature extractor, a recurrent evolution prediction head, and a segmentation decoder. Using a pre-trained multimodal evolution simulator as a constraint of physical laws, a conditional generative adversarial network is driven to generate augmented data that conforms to the laws of evolution. The generated augmented data was used to enhance the training of the multimodal evolution simulator to obtain the final diagnostic model; Online inference phase: Using a multimodal evolution simulator that has undergone reinforcement training, forward inference and backward tracing are performed on real-time collected industrial multimodal data to obtain future state features and historical state features; Calculate the rate of change of features between the current state features and the historical and future state features, and diagnose and locate the active state of the defect based on the rate of change of features.

[0008] As a preferred technical solution, the multimodal evolution sequence is specifically generated as follows: Collect and construct a real multimodal crack image dataset covering various infrastructure materials and crack morphologies. The real multimodal crack image dataset includes registered visible light images, infrared images and their corresponding pixel-level segmentation masks. Based on a real multimodal crack image dataset, a parametric physical evolution engine is used to generate virtual time-series data covering the entire life cycle of a crack according to a preset physical law model. The engine is based on a preset physical law model including physical driving equations for length evolution, width evolution, and orientation angle evolution. Using a 3D modeling and rendering engine, the virtual temporal data that generates cracks is synthesized and rendered with the background material and texture learned from the real dataset. The output is a registered visible light image, an infrared image, and a segmentation mask, which constitute a multimodal evolution sequence for initial training.

[0009] As a preferred technical solution, the multimodal evolution simulator employs a deep neural network, and its core components include: Multimodal feature extractor: used to extract fused deep features from input visible light and infrared image pairs; Forward and backward cyclic evolution prediction heads: As the dynamics engine of the simulator, they are responsible for predicting future states based on current state characteristics and tracing historical states, respectively, and modeling the temporal evolution law of crack features. Segment Decoder: Connected in parallel with the recurrent evolutionary prediction head, it is responsible for decoding features into pixel-level segmentation masks and establishing the relationship between feature space and physical morphology.

[0010] As a preferred technical solution, the multimodal feature extractor uses a pre-trained ResNet-50 as the backbone network input channel to receive the stitched visible light image and infrared image; through five consecutive convolution and downsampling operations, the input is used for feature extraction and spatial compression to obtain the final output fused feature map.

[0011] As a preferred technical solution, the cyclic evolution prediction head includes a forward cyclic evolution prediction head and a reverse cyclic evolution prediction head; The forward recurrent evolution prediction head is a recurrent neural network consisting of two LSTM layers. The fused features at the current time step are reduced to a 256-dimensional vector through spatial adaptive average pooling and fed into the first LSTM layer, which outputs a 512-dimensional hidden state. The second LSTM layer takes the output of the first LSTM layer as input, outputs a 512-dimensional hidden state, and maps it back to 256 dimensions through a fully connected layer to obtain the predicted feature vector. The predicted feature vector is restored to a 256-dimensional feature map through deconvolution upsampling to obtain the feature map for the next time step. The network structure of the reverse cyclic evolution prediction head is symmetrical and independent of the forward cyclic evolution prediction head. It receives the fused features of the previous time step and outputs the historical state features.

[0012] As a preferred technical solution, the segmentation decoder consists of four upsampling modules connected in series. Each module sequentially performs deconvolution, halving of the number of channels, ReLU activation, and batch normalization operations, outputting a single-channel feature map with the same spatial size as the input image. An activation function is applied to the single-channel feature map to obtain a predicted segmentation mask, where each pixel value represents the probability that the corresponding position belongs to the crack region.

[0013] As a preferred technical solution, the initial training of the multimodal evolution simulator is performed end-to-end by minimizing a comprehensive loss function; the loss function includes: The current frame segmentation loss is based on the real segmentation mask and the predicted segmentation mask, comparing the difference between the predicted result and the real label pixel by pixel; Feature evolution prediction loss is calculated by comparing the error between the future features predicted by the forward recurrent evolution prediction head based on the current state and the actual features at the next moment. The reverse cyclic evolution prediction loss uses the real fusion features at the current moment as the supervision signal to calculate the error of the historical features predicted by the reverse cyclic evolution prediction head based on the real features at the next moment. The future frame segmentation loss constrains the segmentation result after decoding the predicted future features to be consistent with the true future segmentation mask.

[0014] As a preferred technical solution, the generator of the conditional generative adversarial network takes crack image pairs and time steps from a real dataset as input, and aims to generate visually realistic future multimodal images, outputting predicted future image pairs; the discriminator takes a multimodal image as input and outputs a feature map, where each pixel in the feature map represents the probability that a local region in the original image is real.

[0015] As a preferred technical solution, the conditional generative adversarial network generates augmented data that conforms to evolutionary laws, as follows: The synthesized image output by the generator is input into the solidified feature extractor to obtain the generated features; The corresponding real input image is also input into the same feature extractor to obtain the current real features, and then the future features after a time step are inferred through the solidified forward prediction head. The evolutionary consistency loss is calculated based on generated features and future features, and the adversarial network is generated until convergence by combining the adversarial loss training conditions. The generator of a conditional generative adversarial network is used to expand the real seed data to generate an augmented dataset; the augmented dataset is then merged with the initial dataset to form the final training dataset.

[0016] As a preferred technical solution, the online inference stage performs defect diagnosis and localization as follows: Simultaneously acquire visible light and infrared images; optionally acquire environmental vectors, which include dust concentration and light intensity; Visible light and infrared images are input into a trained multimodal evolution simulator to obtain basic fusion features. When environmental vectors are acquired, the environmental vectors are normalized, copied, and tiled into a spatial feature map, which is then concatenated with the basic fusion features in the channel dimension. After that, feature fusion is performed through an additional lightweight convolutional network to obtain the current system state features. When no environmental vectors are acquired, the basic fusion features are directly used as the current system state features. The current system state features are input into a multimodal evolution simulator. The future state is obtained by forward inference through a forward cyclic evolution prediction head, and the historical state is obtained by backward tracing through a backward cyclic evolution prediction head. The feature change rate is calculated based on the future state and the historical state. The feature change rate is calculated. When the feature change rate is lower than the low threshold, the system is determined to be in a stable state. When the feature change rate exceeds the high threshold, an active crack is determined to exist. The basic fused features are input into the solidified segmentation decoder to decode and generate a pixel-level crack segmentation mask, and pixel-level crack localization is performed on the current frame.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1) The proposed method for the prognosis and diagnosis of cracks in major infrastructure based on a multimodal dynamic evolution model realizes a transformation from static, passive crack identification to dynamic, proactive early warning, enabling early perception of crack evolution trends. Relying on the multimodal dynamic evolution model and combining forward extrapolation and backward tracing of crack temporal evolution patterns, it can capture subtle changes in crack initiation, expansion, and extension in advance, achieving advanced prediction of crack development trends. This transforms passive maintenance into proactive early warning, fundamentally reducing the safety hazards of continuous crack expansion leading to facility structural damage and collapse, and significantly improving the foresight and safety of infrastructure operation and maintenance.

[0018] 2) This invention constructs a closed-loop data system that integrates virtual and real data while maintaining physical consistency by combining procedural generation with adversarial enhancement of physical constraints. Based on a physical law model, it generates multimodal evolution sequences covering the complete lifecycle of cracks, building an initial training dataset. Then, using a pre-trained multimodal evolution simulator as physical constraints, it drives a conditional generative adversarial network to generate augmented data that closely matches real evolutionary patterns, ensuring the physical consistency of the generated data. Through complementary fusion of virtual and real data and iterative optimization training, the coverage of the dataset is effectively expanded. This solves the problems of sample scarcity and data distortion, significantly improving the robustness and generalization ability of the model. 3) This invention, through the synergy of forward and backward evolutionary prediction and the calculation of feature change rates, can sensitively and reliably distinguish between steady-state and active states, and achieve precise spatial positioning. Relying on the collaborative work of a bidirectional cyclic evolutionary prediction head, it forward-engineers the future development state of cracks and backward-traces the historical evolutionary characteristics of cracks. By accurately calculating the change rates of current state characteristics and historical and future state characteristics, a quantitative judgment standard is established, enabling sensitive differentiation of different states of cracks, such as stable quiescent, slow expansion, and rapid expansion. This eliminates the problems of misjudging steady-state cracks and missing active cracks, resulting in accurate and reliable judgment results. Once an active crack is identified, a pixel-level crack segmentation mask can be quickly output through a segmentation decoder to accurately pinpoint the spatial location of the crack, achieving the integration of defect state judgment and spatial positioning, and meeting the engineering needs for refined diagnosis of facility cracks. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a method for the prognosis and diagnosis of cracks in major infrastructure based on a multimodal dynamic evolution model, according to the present invention.

[0020] Figure 2 This is a schematic diagram of the network structure of the multimodal evolution simulator of the present invention.

[0021] Figure 3 This is a schematic diagram of the adversarial data augmentation training process based on evolutionary laws according to the present invention.

[0022] Figure 4This is a schematic diagram of the dynamic deduction and diagnosis process in the online reasoning stage of this invention. 。 Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0024] Example 1 like Figure 1 As shown, the overall process of the critical infrastructure crack prognosis and diagnosis method based on a multimodal dynamic evolution model proposed in this invention comprises two closely integrated parts: offline training and online inference. The offline training phase aims to construct and train a core model—a multimodal evolution simulator—that can internalize the physical laws of crack dynamic evolution. The online inference phase utilizes this solidified model to dynamically extrapolate the real-time collected multimodal data, thereby achieving the diagnosis and early warning of crack activity.

[0025] Part 1: Offline Training Phase – Building a Cognitive Model Based on Virtual-Real Fusion

[0026] The core of this phase lies in addressing the bottleneck of scarce panoramic time-series data on cracks in major infrastructure. A hybrid data construction strategy integrating virtual and real data is employed to train a multimodal evolution simulator capable of accurately understanding and predicting crack evolution patterns. This strategy is based on real crack data, driven by a parametric physical model, and uses the trained simulator as a constraint on physical laws. This further drives a generative adversarial network for data augmentation, ultimately forming a self-reinforcing data loop to train a final model with extremely strong generalization capabilities. The specific steps are as follows: S101: Hybrid data construction based on real data guidance and physical law constraints.

[0027] To address the bottleneck of scarce panoramic time-series data on cracks in major infrastructure and to provide physically plausible learning samples for the model, this phase adopts a hybrid data construction strategy, which is based on real data and driven by a parametric physical model.

[0028] (1) Construction of the real crack base dataset: First, a real multimodal crack image dataset covering various typical infrastructure materials (such as concrete and steel structures) and crack morphologies (such as linear and mesh-like) is collected and constructed. This dataset should include registered visible light images, infrared images, and their corresponding pixel-level segmentation masks. This dataset serves as the seed for all subsequent data generation and the basis for statistical analysis of physical laws.

[0029] (2) Generation of parameterized physical evolution sequences: Based on the aforementioned real-world dataset, we developed a parametric physical evolution engine. The core of this engine is to utilize a pre-defined physical law model to programmatically control the morphological evolution of cracks, thereby generating a virtual time-series sequence covering their entire lifecycle.

[0030] Specifically, for a real crack image as seed input, the engine will simulate its subsequent evolution process according to the following physical driving equations: Length evolution: ; Width evolution: ; Evolution of orientation angle: (Used to simulate nonlinear extended behavior).

[0031] Among them, evolution parameters , , , , The preset distribution of such distributions is not arbitrarily set, but is determined based on the statistical analysis results of real crack evolution observation data, thereby ensuring the physical rationality of the evolution law.

[0032] (3) High-fidelity rendering: Using a 3D modeling and rendering engine (such as Blender), the crack morphology generated by the above parametric model is synthesized and rendered with the background material and texture learned from real datasets, outputting a strictly registered visible light image. (3-channel RGB), infrared image (1-channel grayscale) and segmentation mask ,in and These represent the initial height and width of the image, respectively, and together form the multimodal evolution sequence used for initial training. .

[0033] At this point, the initial training dataset has been obtained. It contains both visual features from the real world and dynamic evolutionary information constrained by physical laws.

[0034] S102: Training of a Multimodal Evolution Simulator This step aims to train a multimodal evolution simulator. This allows it to internalize the evolution dynamics of cracks from a multimodal perspective. The simulator like Figure 2 As shown, the multi-modal (mm) feature extractor Two independent cyclic evolutionary prediction heads (forward and backward) and a segmentation decoder constitute.

[0035] (1) Multimodal feature extractor Specific implementation Multimodal feature extractor Using a ResNet-50 pre-trained on ImageNet as the backbone network, its input channel count was increased from 3 to 4 to receive the concatenated multimodal input. The network then performs feature extraction and spatial compression on the input through five consecutive convolutions and downsampling operations, resulting in an output feature map with 2048 channels. The spatial size, compared to the input image, is [height missing]. and width All were reduced to 1 / 32 of their original value, that is... , Then through one The convolutional layer reduces the number of channels in this high-dimensional feature map from 2048 to 256, thus obtaining the final output fused feature map. .

[0036] This feature extractor extracts high-level semantic features through forward propagation of ResNet-50. These semantic features are an abstract representation that integrates visible light texture and infrared thermal information, and can capture the common visual features of cracks and their physical properties (such as thermal anomalies).

[0037] (2) Cyclic Evolution Prediction Head and Specific implementation and training of the forward recurrent evolution prediction head It is a recurrent neural network consisting of two LSTM layers, and its function is to fuse features based on the current time step. Predicting the fusion features at the next time step This allows us to simulate the dynamic evolution of cracks.

[0038] The fusion features at the current moment The hidden state is reduced to 256 dimensions using spatial adaptive average pooling and fed into the first LSTM layer, outputting a 512-dimensional hidden state. The second LSTM layer uses the output of the first LSTM layer as its input for further temporal modeling, also outputting a 512-dimensional hidden state. This hidden state is then mapped back to 256 dimensions through a fully connected layer, temporarily obtaining the predicted feature vector. Dropout is set to 0.3. Finally, the vector is restored to its original state through deconvolution upsampling. Feature map, which is the predicted feature map for the next time step. .

[0039] Reverse cyclic evolution prediction head Network structure and forward recurrent evolution prediction head Symmetrical and independent, receiving and predicting the fused features of the previous time step. It should be noted that the historical state characteristics of the reverse cyclic evolution prediction head output (i.e., The model, by inferring historical state features through the reverse prediction head and co-calculating the rate of change with current and future state features, can effectively amplify the differences between active cracks and steady-state conditions in the feature space, thereby achieving sensitive and robust diagnosis of crack active states. The model achieves reasonable inference of historical states through statistical learning, which already meets the core requirement of distinguishing between steady-state and active cracks in online diagnosis.

[0040] (3) Segment Decoder The specific implementation.

[0041] Segment Decoder It is connected in parallel with the evolution prediction head, and its function is to share features. Decode and restore to pixel-level spatial segmentation results.

[0042] Segment Decoder A lightweight encoder-decoder architecture is employed, consisting of four upsampling modules cascaded together. Each module sequentially performs deconvolution (kernel size 4×4, stride 2), halving of the channel count (sequence: 128, 64, 32), ReLU activation, and batch normalization. The input is... After progressive upsampling through four modules, a single-channel feature map with the same spatial size as the input image is output. Finally, the Sigmoid activation function is applied to the feature map to obtain the predicted segmentation mask. Each pixel value represents the probability that the location belongs to the crack region.

[0043] (4) End-to-end training process The samples used in this training phase, i.e., the serial numbers i The indicated data all originates from the multimodal evolution sequences virtually generated in step S101. .

[0044] For each sample i In time step t Multimodal images enter ,get ,Will enter ,get At the same time from use Decoding the segmentation mask Finally, the segmentation loss is calculated by combining these factors. Let the batch size be... N Then the total loss function is: in, , The current frame segmentation loss is used to ensure that the model can accurately segment the crack region in the current frame. It is calculated using the binary cross-entropy loss function. in, For the true segmentation mask, To predict the segmentation mask, the loss compares the predicted result with the true label pixel by pixel.

[0045] To predict the loss based on feature evolution and ensure that the model can learn the dynamic changes in the feature space, the mean squared error is used for calculation. It should be noted that in the above formula... It serves as the next-moment real feature of the monitoring signal, and it is generated by a fixed feature extractor. From the real next-moment multimodal image pair It is extracted from, that is: Therefore, feature evolution prediction loss The core requirement is that the forward prediction head... Based on the current state Predicted future characteristics Features must be extracted directly from real future images. As close as possible. This allows the prediction head to learn the feature space dynamics that conform to the evolution of real data.

[0046] Reverse cyclic evolution prediction head The training is conducted in a manner symmetrical to the aforementioned forward training process. Specifically, during the training phase, the actual fusion features at the current moment are used... As a supervisory signal, the reverse cyclic evolution prediction head is calculated based on the true features of the next time step. Predicted historical characteristics Error: The inverse cyclic evolution prediction loss and the feature evolution prediction loss Together they constitute a bidirectional temporal consistency constraint, which is added to the total loss function. The model is optimized in two ways to ensure that it can learn the temporal evolution of crack features.

[0047] The core function of the segmentation loss for future frames is to force the feature evolution predicted by the model to correspond to physically reasonable pixel-level morphological evolution: This loss establishes a direct correlation between feature evolution and physical morphological evolution by constraining the segmentation result after predicting future features to be consistent with the actual future segmentation mask.

[0048] By minimizing Update the multimodal feature extractor Forward cyclic evolution prediction head Reverse cyclic evolution prediction head The parameters enable the multimodal evolution simulator It can not only internalize the dynamic characteristics of crack evolution, but also ensure that the predicted future state is real and credible in terms of pixel-level physical morphology.

[0049] S103: Adversarial Data Enhancement Based on Evolutionary Laws This step utilizes a trained evolutionary simulator. As a constraint of physical laws, a conditional generative adversarial network (cGAN) is trained to generate augmented data that conforms to evolutionary laws. This augmented data is used to enhance the training of a multimodal evolutionary simulator, thereby improving its generalization ability in real-world industrial scenarios. This evolution-based adversarial data augmentation process is as follows: Figure 3 As shown.

[0050] (1) Network structure This conditional generative adversarial network consists of a generator and a discriminator. The generator... The U-Net architecture is used, and the input is a pair of real multimodal images. and target time step The output predicted future image pair is denoted as Discriminator D It employs the PatchGAN architecture, taking a 4-channel multimodal image as input. Instead of classifying the entire image as real or fake, it outputs a single... The feature map, where each pixel represents the probability that a local region in the original image is real. This helps the model focus on the realism of local textures. This image can be either a pair of real images or a pair of generated images. Simultaneously, conditional information... It will also be embedded and integrated into the middle layer of the discriminator, making it a conditional discriminator.

[0051] (2) Definition of loss function Discriminator loss function Defined as: in, This represents each sample taken from the dataset, and the loss function drives the discriminator. D Correctly distinguish between real images and generated images.

[0052] Total loss function of the generator Defined as: in, These are hyperparameters used to balance the weights of adversarial loss and evolutionary consistency loss. To combat the loss, the generator is driven to produce synthetic images that can deceive the discriminator with a high probability. To calculate the evolution consistency loss and ensure that the generated image conforms to the physical laws of crack evolution, the solidified feature extractor on which the evolution consistency loss depends is used. With solidification evolution prediction head This refers to the first-generation multimodal evolution simulator obtained from the S102 training step. The parameters of the corresponding components are frozen during the training phase of the conditional generative adversarial network (cGAN) as constant constraints of physical laws.

[0053] The following is the calculation process for evolution consistency loss.

[0054] The synthesized image output by the generator is compared with... Input solidification feature extractor Extract its feature representation: Pair the real input image Input the same solidification feature extractor To obtain the features at the current time. Furthermore, predicting the head through solidification evolution. Deduction Future characteristics after step size: It should be noted that the solidification evolution prediction head By receiving the target time step As a conditional input, or predicted via a loop execution unit. This allows for multi-step deduction.

[0055] The mean squared error between the generated image features and the physically derived features is calculated as the evolutionary consistency loss: in, The Frobenius norm (F-norm) is a classic measure of the overall difference between two matrices in matrix space. This measure can keenly capture the drastic changes in the feature space caused by crack evolution, while the feature changes in stable conditions are relatively gradual. This loss function forces the generator to deceive the discriminator at the pixel level while maintaining consistency with the future state predicted by the physical evolution model in the feature space, thus ensuring that the generated data is both visually realistic and conforms to the inherent physical laws of crack dynamic evolution.

[0056] (3) Alternating training process This invention employs an alternating optimization strategy to train a conditional generative adversarial network, the specific process of which is as follows: Discriminator optimization phase fixed generator The parameters are obtained by minimizing the discriminator loss defined in section (2). Update the discriminator D Network parameters; fixed discriminator during generator optimization phase D The parameters are obtained by minimizing the total generator loss defined in section (2). Update generator Network parameters.

[0057] Through the above iterative training, a generator is finally obtained that can generate multimodal augmented data that is both visually realistic and strictly follows the physical evolution law of cracks.

[0058] (4) Model enhancement and final version generation After the conditional generative adversarial network (GAN) training converges, its generator is used... Batch synthesis of large-scale, high-quality augmented data To augment the dataset Merging with the original virtual dataset, the multimodal evolution simulator was re-engineered. End-to-end training is performed (i.e., the training process of S102 is repeated). In this round of training, the model not only learns the physical laws embedded in the virtual data, but also learns the visual features and noise patterns of real industrial scenes through augmented data.

[0059] After this round of intensive training, the multimodal evolution simulator The generalization ability and robustness are significantly improved. The parameters of this enhanced multimodal evolution simulator are solidified as the final version for use in subsequent online inference and diagnostic stages.

[0060] Part Two: Online Reasoning and Diagnosis Phase – Closed-Loop Intelligent Diagnosis Based on Dynamic Inference In the online phase, the solidified model obtained from offline training is used to dynamically extrapolate and diagnose the real-time collected data.

[0061] S201: Synchronous Acquisition of Multimodal Data Visible light images are acquired synchronously using a hardware synchronization trigger. Infrared images and environment vectors ,in (Unit: mg / m³) 3 () represents dust concentration. (Unit: lux) represents light intensity.

[0062] S202: Multimodal Fusion State Construction Will , Input after curing To obtain basic fusion features Environment vector After normalization, the image is copied and tiled to form a spatial feature map. ,and Concatenated along the channel dimension. This is achieved using a lightweight convolutional network. The refinement process involves two 3x3 convolutional layers, with a ReLU activation function and a batch normalization layer in between, outputting the current system state features: It should be noted that the acquisition of environment vectors is an optional operation. When environment vectors are not acquired, the basic fusion features are directly used. As a characteristic of the current system state Enter the next steps.

[0063] S203: Dynamic simulation and diagnosis based on a multimodal evolution simulator.

[0064] Will Input after curing Forward inference and backward tracing are performed respectively through a cyclic evolution prediction head (the dynamic inference and diagnosis process is as follows). Figure 4 (as shown) Get the future state and historical status .

[0065] Then calculate the characteristic rate of change: Compare this rate of change with a dynamic threshold: like This is determined to be a steady-state operating condition.

[0066] like The crack was identified as an active crack, and the localization process was initiated, using the basic fusion features obtained in S202. Input-fixed segment decoder It directly decodes and generates pixel-level crack segmentation masks to locate active crack regions.

[0067] Dynamic threshold and Based on the statistical determination of the feature change rate of the validation set (including virtual evolution sequences and real industrial samples) during the offline training phase, the goal is to control the false alarm rate of steady-state conditions and the missed alarm rate of active cracks within an acceptable range for engineering.

[0068] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0069] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for diagnosing infrastructure cracks based on a multimodal dynamic evolution model, characterized in that, The method steps include: Offline training phase: Generate multimodal evolution sequences covering the complete life cycle of defects based on physical law models, and construct the initial training dataset; A multimodal evolution simulator is initially trained using an initial training dataset. The multimodal evolution simulator includes a multimodal feature extractor, a recurrent evolution prediction head, and a segmentation decoder. Using a pre-trained multimodal evolution simulator as a constraint of physical laws, a conditional generative adversarial network is driven to generate augmented data that conforms to the laws of evolution. The generated augmented data was used to enhance the training of the multimodal evolution simulator to obtain the final diagnostic model; Online inference phase: Using a multimodal evolution simulator that has undergone reinforcement training, forward inference and backward tracing are performed on real-time collected industrial multimodal data to obtain future state features and historical state features; Calculate the rate of change of features between the current state features and the historical and future state features, and diagnose and locate the active state of the defect based on the rate of change of features.

2. The infrastructure crack diagnosis method based on a multimodal dynamic evolution model according to claim 1, characterized in that, The multimodal evolution sequence is generated as follows: Collect and construct a real multimodal crack image dataset covering various infrastructure materials and crack morphologies. The real multimodal crack image dataset includes registered visible light images, infrared images and their corresponding pixel-level segmentation masks. Based on a real multimodal crack image dataset, a parametric physical evolution engine is used to generate virtual time-series data covering the entire life cycle of a crack according to a preset physical law model. The engine is based on a preset physical law model including physical driving equations for length evolution, width evolution, and orientation angle evolution. Using a 3D modeling and rendering engine, the virtual temporal data that generates cracks is synthesized and rendered with the background material and texture learned from the real dataset. The output is a registered visible light image, an infrared image, and a segmentation mask, which constitute a multimodal evolution sequence for initial training.

3. The infrastructure crack diagnosis method based on a multimodal dynamic evolution model according to claim 1, characterized in that, The multimodal evolution simulator employs a deep neural network, and its core components include: Multimodal feature extractor: used to extract fused deep features from input visible light and infrared image pairs; Cyclic Evolution Prediction Head: As the dynamics engine of the simulator, it is responsible for predicting future states based on current state characteristics and tracing historical states, modeling the temporal evolution law of crack features. Segment Decoder: Connected in parallel with the recurrent evolutionary prediction head, it is responsible for decoding features into pixel-level segmentation masks and establishing the relationship between feature space and physical morphology.

4. The infrastructure crack diagnosis method based on a multimodal dynamic evolution model according to claim 3, characterized in that, The multimodal feature extractor uses a pre-trained ResNet-50 as the backbone network input channel to receive the stitched visible light image and infrared image; it performs feature extraction and spatial compression on the input through five consecutive convolution and downsampling operations to obtain the final output fused feature map.

5. The infrastructure crack diagnosis method based on a multimodal dynamic evolution model according to claim 3, characterized in that, The cyclic evolution prediction head includes a forward cyclic evolution prediction head and a backward cyclic evolution prediction head; The forward recurrent evolution prediction head is a recurrent neural network consisting of two LSTM layers. The fused features at the current time step are reduced to a 256-dimensional vector through spatial adaptive average pooling and fed into the first LSTM layer, which outputs a 512-dimensional hidden state. The second LSTM layer takes the output of the first LSTM layer as input, outputs a 512-dimensional hidden state, and maps it back to 256 dimensions through a fully connected layer to obtain the predicted feature vector. The predicted feature vector is restored to a 256-dimensional feature map through deconvolution upsampling to obtain the feature map for the next time step. The network structure of the reverse cyclic evolution prediction head is symmetrical and independent of the forward cyclic evolution prediction head. It receives the fused features of the previous time step and outputs the historical state features.

6. The infrastructure crack diagnosis method based on a multimodal dynamic evolution model according to claim 3, characterized in that, The segmentation decoder consists of four upsampling modules connected in series. Each module sequentially performs deconvolution, halving of the number of channels, ReLU activation, and batch normalization operations, outputting a single-channel feature map with the same spatial size as the input image. An activation function is applied to the single-channel feature map to obtain the predicted segmentation mask, where each pixel value represents the probability that the corresponding position belongs to the crack region.

7. The infrastructure crack diagnosis method based on a multimodal dynamic evolution model according to claim 3, characterized in that, The initial training of the multimodal evolution simulator is performed end-to-end by minimizing a comprehensive loss function; the loss function includes: The current frame segmentation loss is based on the real segmentation mask and the predicted segmentation mask, comparing the difference between the predicted result and the real label pixel by pixel; Feature evolution prediction loss is calculated by comparing the error between the future features predicted by the forward recurrent evolution prediction head based on the current state and the actual features at the next moment. The reverse cyclic evolution prediction loss uses the real fusion features at the current moment as the supervision signal to calculate the error of the historical features predicted by the reverse cyclic evolution prediction head based on the real features at the next moment. The future frame segmentation loss constrains the segmentation result after decoding the predicted future features to be consistent with the true future segmentation mask.

8. The infrastructure crack diagnosis method based on a multimodal dynamic evolution model according to claim 1, characterized in that, The generator of the conditional generative adversarial network takes crack image pairs from a real dataset and a time step as input, and aims to generate visually realistic future multimodal images, outputting predicted future image pairs; the discriminator takes a multimodal image as input and outputs a feature map, where each pixel in the feature map represents the probability that a local region in the original image is real.

9. The infrastructure crack diagnosis method based on a multimodal dynamic evolution model according to claim 1, characterized in that, The conditional generative adversarial network generates augmented data that conforms to evolutionary patterns, as follows: The synthesized image output by the generator is input into the solidified feature extractor to obtain the generated features; The corresponding real input image is also input into the same feature extractor to obtain the current real features, and then the future features after a time step are inferred through the solidified forward prediction head. The evolutionary consistency loss is calculated based on generated features and future features, and the adversarial network is generated until convergence by combining the adversarial loss training conditions. The generator of a conditional generative adversarial network is used to expand real seed data to generate an augmented dataset; The augmented dataset is merged with the initial dataset to form the final training dataset.

10. The infrastructure crack diagnosis method based on a multimodal dynamic evolution model according to claim 1, characterized in that, The online inference phase diagnoses and locates defects as follows: Simultaneously acquire visible light and infrared images; optionally acquire environmental vectors, which include dust concentration and light intensity; Visible light images and infrared images are input into a trained multimodal evolution simulator to obtain basic fusion features; When collecting environment vectors, the environment vectors are normalized and then copied and tiled into spatial feature maps, which are then concatenated with the basic fusion features in the channel dimension. After that, feature fusion is performed through an additional lightweight convolutional network to obtain the current system state features. When no environmental vectors are collected, the basic fusion features are directly used as the current system state features; The current system state features are input into a multimodal evolution simulator. The future state is obtained by forward inference through a forward cyclic evolution prediction head, and the historical state is obtained by backward tracing through a backward cyclic evolution prediction head. The feature change rate is calculated based on the future state and the historical state. Calculate the characteristic change rate; when the characteristic change rate is lower than a low threshold, the system is determined to be in a stable state. When the feature change rate exceeds a high threshold, it is determined that there is an active crack. The basic fusion features are input into the solidified segmentation decoder to generate a pixel-level crack segmentation mask, and pixel-level crack localization is performed on the current frame.

Images