Rail transit maintenance multi-view image acquisition fault automatic identification method

Abstract

The present application relates to the technical field of data processing, and more particularly to a multi-view image acquisition fault automatic identification method for rail transit maintenance, comprising collecting data and processing the data to generate fusion data; processing the fusion data to generate a virtual synchronous frame sequence; inputting the aligned multi-view image sequence to a meta-learning feature extraction network coupled with physical prior to perform feature fusion and time sequence modeling to generate fault feature representation; constructing a digital twin model to generate a labeled fault simulation sequence, comparing and outputting fault mode recognition results through dynamic time warping; using evidence theory to fuse fault recognition results, similarity scores and sensor time sequence data to calculate a confidence interval, and dynamically adjusting image acquisition parameters through reinforcement learning based on the confidence interval to complete system self-optimization. The present application solves the technical problem of feature time sequence confusion caused by time synchronization error in multi-view image acquisition in a deep learning model, and reduces the false alarm rate of fault dynamic identification.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for automatic fault identification using multi-view image acquisition for rail transit maintenance. Background Technology

[0002] Automatic fault identification technology for rail transit based on multi-view images collects image data of rail transit facilities from different angles by deploying multiple cameras. It then uses multi-view geometric principles and computer vision algorithms to fuse information from each image to generate a more comprehensive 3D scene reconstruction or enhanced feature representation. Since single-view images are easily affected by occlusion or viewpoint deviation, multi-view fusion can provide redundant visual cues, thereby improving the accuracy of fault detection, such as identifying track surface defects or abnormal train components. Automatic fault identification technology for rail transit based on multi-view images relies on image registration, feature extraction, and pattern recognition methods to achieve automated analysis processes, ultimately supporting the intelligent maintenance and safe operation of rail transit systems.

[0003] Existing automatic fault identification technologies for rail transit based on multi-view images suffer from the following technical challenges: When multiple camera arrays acquire images of moving train components, the slight differences in hardware triggering, data transmission, and processing links among the cameras make it difficult to achieve microsecond-level absolute synchronization. This results in millisecond-level time shifts in the same dynamic fault event captured from different viewpoints. When deep learning models perform feature fusion and spatiotemporal modeling on multi-view sequence frames, the asynchronous views present discontinuous states in the fault evolution process. For example, one viewpoint may have captured the transient deformation of a bogie crack propagation, while another viewpoint still shows the relatively normal state from the previous moment. This misalignment of cross-view features on the time axis can cause the model to misclassify different stages of the same fault as multiple independent abnormal events, or fail to correctly associate asynchronous features into a complete fault evolution chain, thus generating a large number of false alarms in dynamic identification, such as misreporting the propagation process of a single crack as multiple discrete defect points. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an automatic fault identification method for multi-view image acquisition in rail transit maintenance, which solves the technical problem of increased false alarm rate in dynamic fault identification caused by feature temporal confusion in deep learning models due to time synchronization errors in multi-view image acquisition.

[0005] To solve the above-mentioned technical problems, the specific contents of the present invention are as follows:

[0006] The present invention provides an automatic fault identification method for multi-view image acquisition in rail transit maintenance, comprising:

[0007] Step 1: Collect event stream data, image data stream, train pose trajectory data, and sensor time series data of rail transit facilities, and add timestamps to the event stream data, image data stream, train pose trajectory data, and sensor time series data to generate fused data with a unified spatiotemporal reference.

[0008] Step 2: Perform dynamic spatiotemporal alignment processing on the fused data, use event stream data to perform motion compensation interpolation between image frames to generate a virtual synchronization frame sequence, and use Lie group theory to optimize the spatiotemporal transformation model between multiple views, outputting the aligned multi-view image sequence and its uncertainty estimate.

[0009] Step 3: The aligned multi-view image sequence and its uncertainty estimate are input into the meta-learning feature extraction network coupled with physical priors. The feature extraction process is constrained by embedding a preset structural dynamics model. Based on the virtual synchronization frame sequence, the attention weights between features from different perspectives are calculated to perform spatial feature fusion. Then, the fused spatial feature sequence is temporally modeled to capture the temporal features of fault evolution, and fault feature representation is generated and output.

[0010] Step 4: Construct digital twin models of train components to generate labeled fault simulation sequences. Perform dynamic time warping comparison between fault feature representations and fault simulation sequences to eliminate temporal differences between fault feature representations and fault simulation sequences. Calculate the similarity between fault feature representations and fault simulation sequences. Based on the similarity, determine and output fault pattern recognition results and similarity scores.

[0011] Step 5: Fuse the fault mode recognition results and similarity scores, and combine the sensor time-series data in the fused data. Based on the multiple fault modes simulated by the digital twin model, calculate the confidence interval of the corresponding fault mode using evidence theory. Based on the confidence interval, generate image acquisition parameter adjustment instructions through reinforcement learning strategy, and send the image acquisition parameter adjustment instructions to the image acquisition device to dynamically adjust the acquisition parameters. When the confidence interval meets the preset conditions, output the fault recognition result.

[0012] Furthermore, in the automatic fault identification method for multi-view image acquisition used in rail transit maintenance of the present invention, step 1 includes:

[0013] Event stream data is generated by capturing pixel-level brightness changes through an event camera and by acquiring images at fixed intervals through a global shutter camera.

[0014] Attach hardware timestamps from the same clock source to event packets in the event stream data and image frames in the image data stream;

[0015] Real-time 3D coordinates and contour point cloud data of the train are obtained through ultra-wideband positioning and laser scanning;

[0016] The three-dimensional coordinates and contour point cloud data are registered to generate the six-degree-of-freedom pose trajectory data of the train. The six-degree-of-freedom pose trajectory data of the train includes spatial position data and attitude direction data. The spatial position data includes the translation coordinate values ​​of the train along the X-axis, Y-axis and Z-axis in the three-dimensional coordinate system, and the attitude direction data includes the rotation matrix of the train's rotation attitude around the X-axis, Y-axis and Z-axis.

[0017] Vibration data of train components are collected using vibration sensors as sensor timing data;

[0018] Transmit event stream data, image data stream, pose trajectory data, and sensor timing data with additional hardware timestamps via time-sensitive networking;

[0019] By associating pose trajectory data, event stream data, image data stream, and sensor time series data with corresponding hardware timestamps, a spatiotemporal index is established to generate fused data.

[0020] Furthermore, in the automatic fault identification method for multi-view image acquisition used in rail transit maintenance of the present invention, step 2 includes:

[0021] Visual odometry features are extracted from the image data stream, and a particle filter algorithm is used to fuse the visual odometry features with the pose trajectory data. Based on the fusion result, a Bayesian estimation model of the train motion state is constructed.

[0022] Based on the train motion state provided by the Bayesian estimation model, and using event stream data, motion compensation is performed between adjacent global shutter image frames through optical flow estimation to generate an interpolated intermediate view.

[0023] The accumulated event stream data is reconstructed into an intensity image, and the intensity image, together with the global shutter image frame and intermediate view, constitutes the initial multi-view sequence.

[0024] For the initial multi-view sequence, a camera extrinsic perturbation model based on Lie algebra is established, and the multi-view geometric relationship is optimized based on the camera extrinsic perturbation model with the goal of minimizing the reprojection error;

[0025] The expectation-maximization algorithm is used to iteratively update the camera extrinsic parameter calibration results, and the uncertainty matrix of the spatiotemporal transformation is calculated. The virtual synchronization frame sequence and uncertainty estimate are then output.

[0026] Furthermore, in the automatic fault identification method for multi-view image acquisition used in rail transit maintenance of the present invention, step 3 includes:

[0027] The aligned multi-view image sequence and its uncertainty estimate are input into the feature extraction network. The feature extraction network constrains the feature extraction process through a pre-set structural dynamics model and maps the extracted features to the physical quantity space.

[0028] A meta-learning framework is used to pre-train the feature extraction network so that the feature extraction network can adapt to the changes in feature distribution under different acquisition conditions;

[0029] Based on the virtual synchronization frame sequence, the attention weights between features from different perspectives are calculated, and spatial features are fused based on the attention weights to obtain the fused spatial feature sequence.

[0030] Temporal modeling is performed on the fused spatial feature sequence to capture the temporal features of fault evolution, and fault feature representation is generated and output.

[0031] Furthermore, in the automatic fault identification method for multi-view image acquisition used in rail transit maintenance of the present invention, step 4 includes:

[0032] A parameterized component model is established through finite element analysis, and an actual operational load spectrum is applied to simulate the dynamic evolution of the fault. The stress-strain cloud map sequence is output as the fault simulation sequence.

[0033] Generative adversarial networks are used to generate synthetic fault image sequences with spatiotemporal annotations;

[0034] The fault simulation sequence is fused with the synthetic fault image sequence to generate an annotated fault simulation sequence.

[0035] Dynamic time warping is performed on the fault feature representation and the labeled fault simulation sequence to find the optimal curved path between the fault feature representation and the labeled fault simulation sequence in order to eliminate the temporal differences between the fault feature representation and the labeled fault simulation sequence.

[0036] The similarity between the fault feature representation after dynamic time warping and the labeled fault simulation sequence is calculated. Based on the similarity, the fault mode recognition result and similarity score are determined and output.

[0037] Furthermore, in the automatic fault identification method for multi-view image acquisition used in rail transit maintenance of the present invention, step 5 includes:

[0038] The basic probability assignment function of DS evidence theory is used to process the fault mode recognition results and similarity scores, and sensor time-series data in the fused data is combined.

[0039] The reliability intervals corresponding to various fault modes simulated by the digital twin model are calculated by synthesizing fault mode recognition results, similarity scores, and sensor time series data through Dempster combination rules.

[0040] An elastic weight consolidation algorithm is used to update the parameters of the feature extraction network based on confidence intervals;

[0041] Based on the confidence interval, image acquisition parameter adjustment instructions are generated using a reinforcement learning strategy with the goal of maximizing information entropy;

[0042] The image acquisition parameter adjustment command is sent to the acquisition device to dynamically adjust the camera sampling frequency and resolution. When the confidence interval reaches the preset threshold, the fault identification result is output.

[0043] Furthermore, the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention also includes:

[0044] The uncertainty estimate output from step 2 is used to adjust the attention weight allocation in the spatial feature fusion process in step 3;

[0045] The train motion state provided by the Bayesian estimation model in step 2 is used to initialize the hidden state of the time series modeling process in step 3;

[0046] In step 3, when the feature extraction network is constrained by the structural dynamics model for feature extraction, it receives the pose trajectory data generated in step 1 and uses it as the boundary condition for the structural dynamics model.

[0047] Furthermore, the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention also includes:

[0048] The fault feature representation output in step 3 serves as the validation benchmark for the digital twin model. The difference between the fault feature representation and the fault simulation sequence is fed back to the meta-learning framework to optimize the feature distribution shift adaptation.

[0049] The activation patterns represented by the fault features output in step 3 are used as training samples for training the generative adversarial network.

[0050] The abnormal patterns identified in step 4 are injected as training samples into the training data stream of the feature extraction network.

[0051] Furthermore, the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention also includes:

[0052] The similarity score output in step 4 serves as the input for the basic probability assignment of the evidence theory in step 5.

[0053] The difference between the fault simulation sequence generated by the digital twin model in step 4 and the fault feature representation output in step 3 is quantified as an uncertainty index and participates in the calculation of the confidence interval in step 5.

[0054] In step 5, the elastic weight consolidation algorithm uses the fault simulation sequence generated in step 4 as an auxiliary task for multi-task learning.

[0055] Furthermore, the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention also includes:

[0056] The image acquisition parameter adjustment command generated in step 5 applies to the acquisition modes of the event camera and the global shutter camera in step 1.

[0057] In step 5, the updated feature extraction network parameters are distributed to the image acquisition device;

[0058] When the confidence interval calculated in step 5 is lower than the preset threshold, the ultra-wideband positioning and laser scanning in step 1 are triggered to collect redundant data.

[0059] Beneficial effects of this invention;

[0060] This invention generates fused data with a unified spatiotemporal reference by collecting event stream data, image data stream, train pose trajectory data, and sensor time-series data and adding timestamps. It uses event stream data to perform motion compensation interpolation between image frames to generate a virtual synchronization frame sequence. Lie group theory is used to optimize the multi-view spatiotemporal transformation model to output an aligned sequence. A meta-learning feature extraction network coupled with physical priors is embedded into a structural dynamics model to constrain the feature extraction process. Attention weights are calculated based on the virtual synchronization frame sequence to perform spatial feature fusion and temporal modeling to capture fault evolution features. A digital twin model is constructed to generate a fault simulation sequence, and similarity scores are calculated by eliminating temporal differences through dynamic time warping comparison. Evidence theory is used to fuse fault identification results, similarity scores, and sensor time-series data to calculate a confidence interval. Based on the confidence interval, image acquisition parameters are dynamically adjusted through reinforcement learning to form a closed-loop self-optimizing system. This solves the problem of feature temporal confusion caused by multi-view image acquisition time synchronization errors in deep learning models and reduces the false alarm rate of dynamic fault identification. Attached Figure Description

[0061] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0062] Figure 1 This is a flowchart of the automatic fault identification method for multi-view image acquisition in rail transit maintenance according to the present invention. Detailed Implementation

[0063] To make the technical solution of the present invention clearer, the present invention will be clearly and completely described below with reference to specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The present invention provided by various embodiments will be described in detail below with reference to the accompanying drawings. To better understand the purpose of the present invention, the present invention will be described in further detail below.

[0064] Please see Figure 1 The present invention provides an automatic fault identification method for multi-view image acquisition in rail transit maintenance, comprising:

[0065] Step 1: Collect event stream data, image data stream, train pose trajectory data, and sensor time series data of rail transit facilities, and add timestamps to the event stream data, image data stream, train pose trajectory data, and sensor time series data to generate fused data with a unified spatiotemporal reference.

[0066] Step 2: Perform dynamic spatiotemporal alignment processing on the fused data, use event stream data to perform motion compensation interpolation between image frames to generate a virtual synchronization frame sequence, and use Lie group theory to optimize the spatiotemporal transformation model between multiple views, outputting the aligned multi-view image sequence and its uncertainty estimate.

[0067] Step 3: The aligned multi-view image sequence and its uncertainty estimate are input into the meta-learning feature extraction network coupled with physical priors. The feature extraction process is constrained by embedding a preset structural dynamics model. Based on the virtual synchronization frame sequence, the attention weights between features from different perspectives are calculated to perform spatial feature fusion. Then, the fused spatial feature sequence is temporally modeled to capture the temporal features of fault evolution, and fault feature representation is generated and output.

[0068] Step 4: Construct digital twin models of train components to generate labeled fault simulation sequences. Perform dynamic time warping comparison between fault feature representations and fault simulation sequences to eliminate temporal differences between fault feature representations and fault simulation sequences. Calculate the similarity between fault feature representations and fault simulation sequences. Based on the similarity, determine and output fault pattern recognition results and similarity scores.

[0069] Step 5: Fuse the fault mode recognition results and similarity scores, and combine the sensor time-series data in the fused data. Based on the multiple fault modes simulated by the digital twin model, calculate the confidence interval of the corresponding fault mode using evidence theory. Based on the confidence interval, generate image acquisition parameter adjustment instructions through reinforcement learning strategy, and send the image acquisition parameter adjustment instructions to the image acquisition device to dynamically adjust the acquisition parameters. When the confidence interval meets the preset conditions, output the fault recognition result.

[0070] During the data acquisition and fusion phase, pixel-level brightness change event streams are asynchronously captured by an event camera, while complete image frames are acquired at fixed intervals by a global shutter camera, generating event stream data and image data streams. Real-time 3D coordinates and contour point cloud data of the train are obtained using ultra-wideband positioning technology and laser scanning technology, and after registration, train pose trajectory data is generated. Vibration data of train components are collected by vibration sensors as sensor time-series data. Hardware timestamps originating from the same precision clock source are appended to all data, and the timestamps are transmitted and associated through a time-sensitive network to establish a spatiotemporal index, thereby generating fused data with a unified spatiotemporal reference. This step provides a time-consistent multi-source data foundation for subsequent processing.

[0071] In the dynamic spatiotemporal alignment processing stage, a particle filter algorithm is used to fuse pose trajectory data and visual odometry features to construct a Bayesian estimation model of train motion state. Utilizing the high temporal resolution of event stream data, motion compensation is performed between adjacent global shutter image frames using optical flow estimation to generate interpolated intermediate views. Accumulated event stream data is reconstructed into intensity images, which, together with global shutter image frames and intermediate views, constitute the initial multi-view sequence. A camera extrinsic parameter perturbation model is established using Lie group theory to optimize the multi-view geometric relationship with the objective of minimizing reprojection error. The uncertainty matrix of spatiotemporal transformation is calculated using the expectation-maximization algorithm, outputting a virtual synchronized frame sequence and uncertainty estimates. This step eliminates temporal asynchrony and spatial bias in the multi-view data, providing aligned input for feature extraction.

[0072] In the feature extraction and fusion stage, the aligned multi-view image sequence and its uncertainty estimate are input into a meta-learning feature extraction network coupled with physical priors. The feature extraction network constrains the feature extraction process by embedding a pre-defined structural dynamics model, mapping the extracted features to a physical quantity space. Based on a virtual synchronization frame sequence, attention weights between features from different perspectives are calculated, and spatial feature fusion is performed to obtain a fused spatial feature sequence. Temporal modeling is then performed on the fused spatial feature sequence to capture the long-term temporal dependencies of fault evolution, generating and outputting fault feature representations. This step extracts spatiotemporal features that conform to physical laws, enhancing the robustness of the feature representation.

[0073] In the fault identification and comparison stage, digital twin models of train components are constructed. The dynamic evolution of the fault is simulated by applying actual operational load spectra through finite element analysis, and the resulting stress-strain contour map sequence serves as the fault simulation sequence. Generative adversarial networks (GANs) are used to generate a sequence of spatiotemporally annotated synthetic fault images, which are then fused with the fault simulation sequence to generate an annotated fault simulation sequence. The fault feature representation and the fault simulation sequence are dynamically time-warped to find the optimal bending path to eliminate temporal differences. The similarity between sequences is calculated, and fault pattern recognition results and similarity scores are output based on the similarity. This step improves the accuracy of fault identification by comparing simulated data with real data.

[0074] In the evidence fusion and self-optimization stage, the basic probability allocation function of DS evidence theory is used to process fault mode recognition results, similarity scores, and sensor time-series data. Multi-source evidence is synthesized using Dempster's combination rule, and the confidence interval corresponding to the simulated fault mode in the digital twin model is calculated. An elastic weight consolidation algorithm is employed to update feature extraction network parameters based on the confidence interval, preventing the forgetting of historical knowledge. Based on the confidence interval, a reinforcement learning strategy is used to maximize information entropy and generate image acquisition parameter adjustment instructions, dynamically adjusting the camera sampling frequency and resolution. When the confidence interval meets preset conditions, the fault recognition result is output, forming a closed-loop control system and achieving system self-optimization. This step improves the reliability and efficiency of fault recognition through multimodal evidence fusion and adaptive optimization.

[0075] Specifically, the automatic fault identification method for multi-view image acquisition in rail transit maintenance according to the present invention includes step 1 as follows:

[0076] Event stream data is generated by capturing pixel-level brightness changes through an event camera and by acquiring images at fixed intervals through a global shutter camera.

[0077] Attach hardware timestamps from the same clock source to event packets in the event stream data and image frames in the image data stream;

[0078] Real-time 3D coordinates and contour point cloud data of the train are obtained through ultra-wideband positioning and laser scanning;

[0079] The three-dimensional coordinates and contour point cloud data are registered to generate the six-degree-of-freedom pose trajectory data of the train. The six-degree-of-freedom pose trajectory data of the train includes spatial position data and attitude direction data. The spatial position data includes the translation coordinate values ​​of the train along the X-axis, Y-axis and Z-axis in the three-dimensional coordinate system, and the attitude direction data includes the rotation matrix of the train's rotation attitude around the X-axis, Y-axis and Z-axis.

[0080] Vibration data of train components are collected using vibration sensors as sensor timing data;

[0081] Transmit event stream data, image data stream, pose trajectory data, and sensor timing data with additional hardware timestamps via time-sensitive networking;

[0082] By associating pose trajectory data, event stream data, image data stream, and sensor time series data with corresponding hardware timestamps, a spatiotemporal index is established to generate fused data.

[0083] During the data acquisition phase, the event camera asynchronously captures event stream data based on pixel-level brightness changes. Each event includes pixel coordinates, a timestamp, and polarity information, thereby generating a high temporal resolution event stream. The global shutter camera captures complete image frames at fixed intervals, generating an image data stream and avoiding rolling shutter distortion. The event stream data and image data stream are synchronized at the microsecond level by adding hardware timestamps from the same precision clock source, providing a unified time reference for multi-source data.

[0084] Ultra-wideband (UWB) positioning technology measures the distance between the train and a fixed base station by transmitting and receiving UWB signals, and calculates the train's real-time 3D coordinates using a polygonal positioning algorithm. Laser scanning technology generates contour point cloud data of the train's surface by emitting a laser beam and receiving the reflected signal. The 3D coordinates and contour point cloud data are registered using an iterative nearest-point algorithm, and spatial transformation parameters are optimized to generate high-precision six-DOF pose trajectory data of the train, including translation and rotation information.

[0085] Vibration sensors are installed on key train components to collect vibration signals as sensor time-series data, reflecting the train's operating status. Time-Sensitive Networks (TSNs) employ a time-aware shaping mechanism to schedule data transmission, ensuring low latency and determinism in the transmission of event stream data, image data streams, pose trajectory data, and sensor time-series data. All data packets are associated with corresponding hardware timestamps, establishing a spatiotemporal index structure that maps timestamps to spatial coordinates, generating fused data with a unified spatiotemporal reference, supporting subsequent efficient querying and processing.

[0086] The timestamp alignment between event stream data and image data stream is based on hardware trigger signals, ensuring precise correspondence between events and image frames on the timeline. The timestamps of pose trajectory data are aligned with the timestamps of sensor time-series data using interpolation methods to compensate for transmission delays. The fused data is stored in a time-series database and indexed by timestamp and spatial location, enabling rapid retrieval and association of multimodal data.

[0087] Specifically, in the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention, step 2 includes:

[0088] Visual odometry features are extracted from the image data stream, and a particle filter algorithm is used to fuse the visual odometry features with the pose trajectory data. Based on the fusion result, a Bayesian estimation model of the train motion state is constructed.

[0089] Based on the train motion state provided by the Bayesian estimation model, and using event stream data, motion compensation is performed between adjacent global shutter image frames through optical flow estimation to generate an interpolated intermediate view.

[0090] The accumulated event stream data is reconstructed into an intensity image, and the intensity image, together with the global shutter image frame and intermediate view, constitutes the initial multi-view sequence.

[0091] For the initial multi-view sequence, a camera extrinsic perturbation model based on Lie algebra is established, and the multi-view geometric relationship is optimized based on the camera extrinsic perturbation model with the goal of minimizing the reprojection error;

[0092] The expectation-maximization algorithm is used to iteratively update the camera extrinsic parameter calibration results, and the uncertainty matrix of the spatiotemporal transformation is calculated. The virtual synchronization frame sequence and uncertainty estimate are then output.

[0093] In the dynamic spatiotemporal alignment processing stage, visual odometry features are first extracted from the image data stream. Feature point detection and tracking algorithms are then used to calculate the relative motion between adjacent image frames, generating visual odometry features. A particle filtering algorithm fuses the visual odometry features with the pose trajectory data. Through importance sampling and resampling, the posterior probability distribution of the train's motion state is estimated, constructing a Bayesian estimation model of the train's motion state and providing a probabilistic description of the train's motion over time.

[0094] Based on the train motion state provided by the Bayesian estimation model, and leveraging the high temporal resolution of event stream data, pixel-level motion vectors are calculated through optical flow estimation. Motion compensation is performed between adjacent global shutter image frames based on the motion vectors, and an interpolation algorithm is used to generate interpolated intermediate views to fill the temporal gaps between image frames, reducing motion blur and temporal inconsistencies.

[0095] Accumulated event stream data is reconstructed into intensity images, and asynchronous events are converted into grayscale images using an event accumulation algorithm. The intensity images, along with global shutter image frames and interpolated intermediate views, constitute the initial multi-view sequence, providing spatiotemporally continuous visual data from multiple perspectives.

[0096] For the initial multi-view sequence, a camera extrinsic parameter perturbation model based on Lie algebras is established. The Lie algebra represents the continuous space of the camera pose, and the perturbation model describes the camera extrinsic parameter calibration error. The multi-view geometric relationship is optimized with the goal of minimizing the reprojection error. A nonlinear optimization algorithm is used to adjust the camera parameters to minimize the error of projecting 3D points onto the 2D image, thereby improving the multi-view alignment accuracy.

[0097] The Expectation-Maximization (EM) algorithm is used to iteratively update the camera extrinsic parameter calibration results. The expectation step estimates the hidden variables, and the maximization step optimizes the parameter estimates. The uncertainty matrix of the spatiotemporal transformation is calculated, and the uncertainty of the transformation parameters is estimated through covariance propagation. The virtual synchronization frame sequence and uncertainty estimate are output, providing alignment and reliability information for subsequent feature extraction.

[0098] Specifically, in the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention, step 3 includes:

[0099] The aligned multi-view image sequence and its uncertainty estimate are input into the feature extraction network. The feature extraction network constrains the feature extraction process through a pre-set structural dynamics model and maps the extracted features to the physical quantity space.

[0100] A meta-learning framework is used to pre-train the feature extraction network so that the feature extraction network can adapt to the changes in feature distribution under different acquisition conditions;

[0101] Based on the virtual synchronization frame sequence, the attention weights between features from different perspectives are calculated, and spatial features are fused based on the attention weights to obtain the fused spatial feature sequence.

[0102] Temporal modeling is performed on the fused spatial feature sequence to capture the temporal features of fault evolution, and fault feature representation is generated and output.

[0103] In the feature extraction and fusion stage, the aligned multi-view image sequence and its uncertainty estimate are input into the feature extraction network. The feature extraction network employs a convolutional neural network architecture, extracting local features from the image through multiple convolutional and pooling operations. The feature extraction network constrains the feature extraction process through a pre-defined structural dynamics model. This model encodes the laws of material mechanics, mapping the features of the intermediate layers of the network to a physical quantity space, such as stress and strain fields, ensuring that the extracted features conform to physical laws. Uncertainty estimation is used to adjust the weight allocation during the feature extraction process, improving feature robustness.

[0104] A meta-learning framework is employed to pre-train the feature extraction network. This framework utilizes a model-independent meta-learning algorithm to handle task sets under multiple different acquisition conditions. During the meta-training phase, the feature extraction network learns the ability to quickly adapt to new tasks, enabling it to adapt to changes in feature distribution under different acquisition conditions, such as illumination variations and viewpoint differences. The meta-learning framework uses support and query sets for episodic training, optimizing the network's initial parameters and improving generalization performance.

[0105] Based on a virtual synchronization frame sequence, attention weights are calculated among features from different perspectives. The attention mechanism calculates feature similarity by querying key-value pairs, generating an attention weight map that highlights fault-related regions from different perspectives. Spatial feature fusion is then performed on features from different perspectives based on the attention weights. A weighted summation method is used to aggregate multi-perspective information, resulting in a fused spatial feature sequence. The spatial feature fusion process preserves complementary information from multiple perspectives, enhancing the integrity of feature representation.

[0106] Temporal modeling is performed on the fused spatial feature sequence, employing a gated recurrent unit (GRU) network to capture the long-term temporal dependencies of fault evolution. The GRU controls information flow by updating and resetting gates, learning dynamic patterns within the sequence. The temporal modeling process analyzes the changing trends of the feature sequence over time, capturing temporal features of fault evolution, such as the gradual process of crack propagation. Finally, a fault feature representation, including spatiotemporal information, is generated and output for subsequent fault identification.

[0107] Specifically, in the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention, step 4 includes:

[0108] A parameterized component model is established through finite element analysis, and an actual operational load spectrum is applied to simulate the dynamic evolution of the fault. The stress-strain cloud map sequence is output as the fault simulation sequence.

[0109] Generative adversarial networks are used to generate synthetic fault image sequences with spatiotemporal annotations;

[0110] The fault simulation sequence is fused with the synthetic fault image sequence to generate an annotated fault simulation sequence.

[0111] Dynamic time warping is performed on the fault feature representation and the labeled fault simulation sequence to find the optimal curved path between the fault feature representation and the labeled fault simulation sequence in order to eliminate the temporal differences between the fault feature representation and the labeled fault simulation sequence.

[0112] The similarity between the fault feature representation after dynamic time warping and the labeled fault simulation sequence is calculated. Based on the similarity, the fault mode recognition result and similarity score are determined and output.

[0113] In the fault simulation sequence generation phase, parametric component models are established through finite element analysis. These models are constructed based on the geometric dimensions and material properties of key train components, and mesh generation and boundary condition settings are performed using finite element analysis software. Actual operational load spectra are applied to simulate the dynamic evolution of the fault. These load spectra are extracted from historical operational data and include vibration loads, thermal loads, and mechanical stress loads. The simulation process simulates common fault types such as crack propagation, wear, and deformation, outputting a sequence of stress-strain contour maps as the fault simulation sequence. Each contour map corresponds to the component state at a specific time point, providing a physical benchmark for fault evolution.

[0114] A generative adversarial network (GAN) is employed to generate a sequence of synthetic fault images with spatiotemporal annotations. The GAN comprises a generator and a discriminator. The generator takes random noise and fault type labels as input to produce synthetic fault images. The discriminator distinguishes between real and generated fault images, and the generation quality is improved through adversarial training. Spatiotemporal annotation is achieved by adding timestamps and spatial coordinates, ensuring that the synthetic image sequence is aligned with the fault simulation sequence in both time and space, thus enhancing the diversity and realism of the data.

[0115] The fault simulation sequence is fused with the synthetic fault image sequence to generate an annotated fault simulation sequence. The fusion process involves data alignment and information integration. A timestamp matching algorithm is used to synchronize the stress-strain contour map sequence and the synthetic image sequence on the time axis. The fused sequence includes physical simulation data and visual data, and the annotation information includes fault type, occurrence time and spatial location, forming a multimodal fault representation that supports subsequent accurate comparison.

[0116] Dynamic time warping is performed between the fault feature representation and the labeled fault simulation sequence to find the optimal curved path between them to eliminate timing discrepancies. The dynamic time warping algorithm calculates the minimum cumulative distance path between the two sequences, allowing the time axis to bend to match similar feature points. The optimal curved path is solved using a dynamic programming algorithm to eliminate timing jitter caused by asynchronous acquisition times, ensuring that fault events are correctly aligned on the time axis.

[0117] The similarity between the dynamically time-warped fault feature representation and the labeled fault simulation sequence is calculated, and distance metrics such as Euclidean distance or cosine similarity are used to evaluate the degree of matching between the sequences. Based on the similarity, fault pattern recognition results and similarity scores are determined and output. The similarity score reflects the consistency between the fault feature representation and the simulation sequence. Fault types are classified according to preset thresholds, and recognition results and confidence scores are output to improve the accuracy and reliability of fault recognition.

[0118] Specifically, in the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention, step 5 includes:

[0119] The basic probability assignment function of DS evidence theory is used to process the fault mode recognition results and similarity scores, and sensor time-series data in the fused data is combined.

[0120] The reliability intervals corresponding to various fault modes simulated by the digital twin model are calculated by synthesizing fault mode recognition results, similarity scores, and sensor time series data through Dempster combination rules.

[0121] An elastic weight consolidation algorithm is used to update the parameters of the feature extraction network based on confidence intervals;

[0122] Based on the confidence interval, image acquisition parameter adjustment instructions are generated using a reinforcement learning strategy with the goal of maximizing information entropy;

[0123] The image acquisition parameter adjustment command is sent to the acquisition device to dynamically adjust the camera sampling frequency and resolution. When the confidence interval reaches the preset threshold, the fault identification result is output.

[0124] In the evidence fusion stage, the basic probability allocation function of DS evidence theory is used to process the fault mode recognition results and similarity scores, and combined with the sensor time-series data in the fused data. The basic probability allocation function transforms the fault mode recognition results into probability quality allocations, and the similarity scores are incorporated into the probability allocations as confidence indicators. The sensor time-series data are transformed into time-series evidence bodies through feature extraction methods, forming multi-source evidence inputs and providing a probabilistic basis for subsequent evidence synthesis.

[0125] The reliability of fault mode recognition results, similarity scores, and sensor time-series data is synthesized using Dempster's combination rules. The reliability intervals for various fault modes simulated by the digital twin model are then calculated. Dempster's combination rules perform orthogonal summation on multi-source evidence to eliminate conflicts between evidence and generate a joint reliability function. The reliability interval includes confidence and likelihood, quantifying the support level and uncertainty of each fault mode and providing a probabilistic basis for decision-making.

[0126] A resilient weight consolidation algorithm is employed to update the parameters of the feature extraction network based on confidence intervals. This algorithm calculates the importance weights of network parameters in historical tasks, using confidence intervals to guide parameter updates. Important parameters are protected against catastrophic forgetting, while less important parameters are adjusted according to the confidence intervals. This allows the feature extraction network to adapt to new failure modes without losing existing knowledge.

[0127] Based on the confidence interval, an image acquisition parameter adjustment instruction is generated using a reinforcement learning strategy with the goal of maximizing information entropy. The reinforcement learning agent uses the confidence interval as its state space and information entropy maximization as its reward function. It employs a policy gradient method to optimize the acquisition strategy, generating instructions to adjust the camera sampling frequency and resolution to maximize information gain and optimize the data acquisition process.

[0128] Image acquisition parameter adjustment commands are sent to the acquisition device to dynamically adjust the camera's sampling frequency and resolution. Upon receiving the commands, the acquisition device adjusts the camera's operating parameters, such as increasing the sampling frequency to improve temporal resolution or adjusting the resolution to enhance spatial detail. When the confidence interval reaches a preset threshold, the system outputs the fault identification result, completing a confidence-based fault decision-making loop.

[0129] Specifically, the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention further includes:

[0130] The uncertainty estimate output from step 2 is used to adjust the attention weight allocation in the spatial feature fusion process in step 3;

[0131] The train motion state provided by the Bayesian estimation model in step 2 is used to initialize the hidden state of the time series modeling process in step 3;

[0132] In step 3, when the feature extraction network is constrained by the structural dynamics model for feature extraction, it receives the pose trajectory data generated in step 1 and uses it as the boundary condition for the structural dynamics model.

[0133] In the spatial feature fusion process of step 3, the uncertainty estimate output in step 2 is used to adjust the attention weight allocation. The uncertainty estimate originates from the spatiotemporal transformation uncertainty matrix calculated in step 2, which quantifies the reliability of each frame in the virtual synchronization frame sequence. When the attention mechanism calculates the similarity weights between features from different perspectives, the uncertainty estimate is incorporated as a moderating factor into the attention weight calculation, causing the model to assign lower attention weights to features with high uncertainty, thereby reducing the impact of unreliable features on the fusion result and improving the robustness of spatial feature fusion.

[0134] In the temporal modeling process of step 3, the train motion state provided by the Bayesian estimation model in step 2 is used to initialize the hidden state. The Bayesian estimation model, based on the particle filter algorithm, fuses visual odometry features and pose trajectory data to provide the probability distribution of the train motion state. The hidden state of the gated recurrent unit network is initialized with the expected value or sampled value of the train motion state, enabling the sequence model to evolve from the current motion state, capturing fault temporal features, and enhancing the physical consistency of the temporal modeling.

[0135] In the feature extraction process of step 3, when the feature extraction network is constrained by the structural dynamics model, it receives the pose trajectory data generated in step 1 as the boundary condition of the structural dynamics model. The feature extraction network is embedded in a preset structural dynamics model, which describes the mechanical behavior of materials based on the finite element method. The pose trajectory data generated in step 1 provides the six degrees of freedom position and attitude information of the train, which serves as the displacement boundary condition of the structural dynamics model for calculating the stress-strain field, thereby mapping the extracted features to the physical quantity space and ensuring that the features conform to physical laws.

[0136] Specifically, the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention further includes:

[0137] The fault feature representation output in step 3 serves as the validation benchmark for the digital twin model. The difference between the fault feature representation and the fault simulation sequence is fed back to the meta-learning framework to optimize the feature distribution shift adaptation.

[0138] The activation patterns represented by the fault features output in step 3 are used as training samples for training the generative adversarial network.

[0139] The abnormal patterns identified in step 4 are injected as training samples into the training data stream of the feature extraction network.

[0140] In the fault verification and optimization phase, the fault feature representation output in step 3 serves as the verification benchmark for the digital twin model. The fault feature representation consists of spatiotemporal features extracted from real data, while the fault simulation sequence is simulated data generated by the digital twin model. The difference between the fault feature representation and the fault simulation sequence is calculated using distance metrics, such as Euclidean distance or cosine similarity, to quantify the distribution shift between real and simulated data. This difference information is fed back to the meta-learning framework, which employs a model-independent meta-learning algorithm. This algorithm uses the difference data to adjust the parameters of the feature extraction network, enabling it to better adapt to the real data distribution, optimize its ability to adapt to feature distribution shifts, and improve the model's generalization performance in real-world environments.

[0141] The Generative Adversarial Network (GAN) uses the activation patterns of the fault feature representations output from step 3 as training samples. These activation patterns are feature maps or activation values ​​from intermediate layers of the feature extraction network, representing key fault regions. The generator of the GAN takes each activation pattern as input and learns to generate more realistic fault feature representations. The discriminator distinguishes between real and generated features, improving the generation quality through adversarial training. The training process uses a min-max optimization algorithm to update the generator and discriminator parameters, enabling the GAN to produce highly realistic synthetic fault data to enhance the training dataset.

[0142] The abnormal patterns identified in step 4 are injected as training samples into the training data stream of the feature extraction network. These abnormal patterns are fault types classified after dynamic time warping, such as cracks or wear. Each abnormal pattern serves as additional training sample, merged with the original training data, and used for retraining the feature extraction network. The training process employs the backpropagation algorithm to update the network weights, enabling the feature extraction network to better identify and extract abnormal features, thereby improving the accuracy and robustness of fault detection and reducing false negatives and false positives.

[0143] Specifically, the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention further includes:

[0144] The similarity score output in step 4 serves as the input for the basic probability assignment of the evidence theory in step 5.

[0145] The difference between the fault simulation sequence generated by the digital twin model in step 4 and the fault feature representation output in step 3 is quantified as an uncertainty index and participates in the calculation of the confidence interval in step 5.

[0146] In step 5, the elastic weight consolidation algorithm uses the fault simulation sequence generated in step 4 as an auxiliary task for multi-task learning.

[0147] In the evidence theory processing stage, the similarity score output in step 4 serves as the basic probability allocation input for the DS evidence theory in step 5. The similarity score derives from the degree of matching between the fault feature representation after dynamic time warping in step 4 and the fault simulation sequence. This score is calculated using distance metrics such as Euclidean distance or cosine similarity, reflecting the level of consistency between real and simulated data. The basic probability allocation function converts the similarity score into probability quality, assigning it to different fault hypotheses, providing an initial probabilistic basis for evidence synthesis and supporting subsequent multi-source evidence fusion.

[0148] In step 4, the difference between the fault simulation sequence generated by the digital twin model and the fault feature representation output in step 3 is quantified as an uncertainty index. The difference quantification process employs statistical methods such as standard deviation or analysis of variance to calculate the degree of deviation between corresponding points in the sequences, generating uncertainty values. This uncertainty index serves as additional evidence input in step 5, participating in the calculation of the confidence interval. Through Dempster's combination rule and fusion with other evidence, the confidence interval of the fault hypothesis is adjusted, improving the accuracy of uncertainty modeling and enhancing the reliability of fault identification.

[0149] In step 5, the elastic weight consolidation algorithm uses the fault simulation sequence generated in step 4 as an auxiliary task for multi-task learning. The elastic weight consolidation algorithm calculates the importance weights of the feature extraction network parameters in historical tasks, and the fault simulation sequence serves as the data source for the auxiliary task, used to train the network to identify simulated fault modes. The multi-task learning framework simultaneously optimizes the objective functions of both the main and auxiliary tasks, enabling the feature extraction network to retain historical knowledge when adapting to new fault types, preventing forgetting, and improving the model's generalization ability and robustness.

[0150] Specifically, the automatic fault identification method for multi-view image acquisition in rail transit maintenance of the present invention further includes:

[0151] The image acquisition parameter adjustment command generated in step 5 applies to the acquisition modes of the event camera and the global shutter camera in step 1.

[0152] In step 5, the updated feature extraction network parameters are distributed to the image acquisition device;

[0153] When the confidence interval calculated in step 5 is lower than the preset threshold, the ultra-wideband positioning and laser scanning in step 1 are triggered to collect redundant data.

[0154] During the system self-optimization phase, the image acquisition parameter adjustment instructions generated in step 5 apply to the acquisition modes of the event camera and the global shutter camera in step 1. These instructions are generated based on the confidence interval and reinforcement learning strategy in step 5 and include adjustments to parameters such as camera sampling frequency and resolution. Each instruction is sent to the event camera and the global shutter camera via the communication interface, directly controlling the camera's operating mode. For example, increasing the sampling frequency to improve temporal resolution or adjusting the resolution to enhance spatial detail optimizes data acquisition quality, forming a closed-loop feedback control.

[0155] In step 5, the updated feature extraction network parameters are distributed to the image acquisition devices. These parameters are updated based on confidence intervals using an elastic weight consolidation algorithm to optimize network weights. The updated parameters are then distributed to the image acquisition devices via a network transmission protocol. Each device possesses edge computing capabilities, enabling it to locally load and execute the feature extraction network. This ensures that the real-time feature extraction process uses the latest network parameters, improving fault identification accuracy and processing efficiency.

[0156] When the confidence interval calculated in step 5 is lower than the preset threshold, the ultra-wideband positioning and laser scanning in step 1 are triggered to perform redundant data acquisition. The confidence interval is calculated using evidence theory to quantify the confidence level of the fault identification result, and the preset threshold is set according to the application scenario. An insufficient confidence interval indicates low data reliability. The system automatically activates the ultra-wideband positioning module and the laser scanning module to perform additional data acquisition, supplementing the train pose trajectory data and contour point cloud data, enhancing data redundancy, providing richer information for subsequent processing, and improving the reliability of system decisions.

[0157] This invention effectively solves the problem of feature timing confusion caused by time synchronization errors in multi-view image acquisition through a multi-step collaborative processing mechanism. In the data acquisition stage, a hybrid array architecture of event cameras and global shutter cameras is adopted, and hardware timestamps originating from the same precision clock source are added to all data sources to generate fused data with a unified spatiotemporal reference, establishing a time synchronization foundation from the source. Train pose trajectory data is acquired through ultra-wideband positioning and laser scanning, providing a spatial reference for subsequent processing.

[0158] In the dynamic spatiotemporal alignment stage, leveraging the high temporal resolution of event stream data, motion compensation interpolation is performed between image frames using optical flow estimation to generate a virtual synchronization frame sequence. Lie group theory is employed to optimize multi-view geometric relationships, and the uncertainty matrix of spatiotemporal transformation is calculated using the expectation-maximization algorithm. This eliminates temporal offsets and spatial deviations caused by hardware triggering differences between different viewpoints, providing spatiotemporally aligned input data for feature extraction.

[0159] In the feature extraction stage, a structural dynamics model is embedded into a meta-learning network coupled with physical priors to constrain the feature extraction process, mapping the extracted features to a physical quantity space. Cross-view attention weights are calculated based on virtual synchronous frame sequences to perform spatial feature fusion, and then temporal modeling is used to capture fault evolution features. This process ensures that the feature representation conforms to physical laws while maintaining temporal consistency, preventing asynchronous features from being misclassified as independent events.

[0160] During the fault identification phase, a labeled fault simulation sequence is generated using a digital twin model. This sequence is then dynamically time-warped with the real feature representation to find the optimal curved path and eliminate timing discrepancies. This comparison mechanism can correctly correlate asynchronous features, identifying different development stages of the same fault as a complete evolution chain rather than discrete events.

[0161] Finally, evidence theory is used to fuse multimodal data to calculate the confidence interval, and reinforcement learning is used to dynamically adjust the acquisition parameters. When the confidence is insufficient, redundant data acquisition is triggered, forming a closed-loop optimization system. The entire method fundamentally solves the problem of feature temporal confusion by unifying spatiotemporal benchmarks, dynamic alignment, physical constraint feature extraction, and temporal regularization comparison, significantly reducing the false alarm rate of dynamic fault identification.

[0162] The specific implementation of this invention is as follows: In the maintenance scenario of rail transit facilities, multi-view image acquisition of operating train components is performed by deploying an event camera array and a global shutter camera array. The event cameras asynchronously capture event stream data based on pixel-level brightness changes. Each event includes pixel coordinates, a timestamp, and polarity information, generating an event stream with microsecond-level resolution. The global shutter cameras expose at fixed time intervals, acquiring complete image frames to generate an image data stream. All data sources are supplemented with hardware timestamps through a precision clock source to establish a unified time reference.

[0163] The ultra-wideband positioning system (UWBS) calculates the real-time 3D coordinates of the train through signal interaction between multiple base stations and train-mounted tags. A laser scanner emits a laser beam and receives reflected signals, generating point cloud data of the train's surface contour. The 3D coordinates and point cloud data are registered using an iterative nearest-point algorithm to generate six-DOF pose trajectory data, including translation and rotation information. Vibration sensors collect vibration signals from key train components as sensor time-series data. A time-sensitive network transmits timestamped multi-source data, establishing a spatiotemporal index to generate fused data.

[0164] In the dynamic spatiotemporal alignment processing stage, visual odometry features are extracted from the image data stream. A particle filter algorithm is used to fuse these features with pose trajectory data to construct a Bayesian estimation model of the train's motion state. Leveraging the high temporal resolution of the event stream data, motion compensation interpolation is performed between global shutter image frames using optical flow estimation to generate a virtual synchronization frame sequence. A camera extrinsic parameter perturbation model is established using Lie group theory. The multi-view geometric relationship is optimized with the goal of minimizing reprojection error, outputting an alignment sequence with uncertainty estimation.

[0165] The feature extraction network employs a meta-learning architecture coupled with physical priors, mapping features to a stress-strain physical quantity space by embedding them into a structural dynamics model. Cross-view attention weights are calculated based on virtual synchronous frame sequences, and spatial feature fusion is performed followed by temporal modeling to capture fault evolution features. The digital twin model applies actual operational load spectra through finite element analysis, simulating fault dynamic processes such as crack propagation to generate labeled fault simulation sequences.

[0166] The Dynamic Time Warping (DTW) algorithm finds the optimal curved path between the fault feature representation and the simulation sequence, and calculates a similarity score after eliminating temporal jitter. The DS evidence theory integrates multi-source evidence to calculate the confidence interval of the fault hypothesis, and dynamically adjusts the camera acquisition parameters through reinforcement learning. When the confidence level falls below a threshold, ultra-wideband and laser redundant acquisition are triggered, forming a closed-loop self-optimizing system. The entire scheme effectively solves the problem of multi-view temporal confusion through spatiotemporal benchmark unification, physical constraint feature extraction, and temporal warping comparison.

[0167] Embodiment 1 of the present invention:

[0168] In the scenario of detecting cracks in train bogies, multi-view image acquisition of the bogies in operation is achieved by deploying an event camera array and a global shutter camera array. The event cameras asynchronously capture event stream data based on pixel-level brightness changes; each event includes pixel coordinates, a timestamp, and polarity information, generating a high temporal resolution event stream. The global shutter cameras acquire complete image frames at fixed intervals, generating an image data stream. All data sources are synchronized at the microsecond level by adding hardware timestamps using the same precision clock source. An ultra-wideband positioning system measures the distance between the train and the base station, and combines this with laser scanning to obtain the three-dimensional coordinates and contour point cloud data of the bogies. After registration, six-DOF pose trajectory data is generated. Vibration sensors collect vibration data from the bogies as sensor time-series data. A time-sensitive network transmits multi-source data with timestamps, establishes a spatiotemporal index to generate fused data, and provides a unified spatiotemporal reference for subsequent processing.

[0169] In the dynamic spatiotemporal alignment processing stage, visual odometry features are extracted from the image data stream. A particle filter algorithm is used to fuse these features with pose trajectory data to construct a Bayesian estimation model of the bogie's motion state. Utilizing the high temporal resolution of the event stream data, motion compensation interpolation is performed between global shutter image frames using optical flow estimation to generate a virtual synchronization frame sequence. Lie group theory is employed to optimize the multi-view geometric relationships, calculate the uncertainty matrix of the spatiotemporal transformation, and output the aligned multi-view sequence and uncertainty estimates, eliminating temporal and spatial offsets caused by hardware trigger differences between different viewpoints.

[0170] In the feature extraction stage, the aligned multi-view sequences are input into a meta-learning feature extraction network coupled with physical priors. The feature extraction process is constrained by embedding a structural dynamics model, mapping the features to a stress-strain physical quantity space. Cross-view attention weights are calculated based on virtual synchronous frame sequences. After spatial feature fusion, temporal modeling is used to capture the temporal features of crack propagation, generating a fault feature representation. The digital twin model applies the actual operational load spectrum through finite element analysis, simulating the dynamic evolution of bogie cracks and outputting an annotated fault simulation sequence.

[0171] In the fault identification phase, the fault feature representation and the fault simulation sequence are dynamically time-warped to find the optimal bending path to eliminate temporal discrepancies, and a similarity score is calculated to output the crack identification result. Evidence theory is used to integrate multimodal data to calculate the confidence interval, and reinforcement learning is used to dynamically adjust the camera sampling frequency and resolution. When the confidence level is insufficient, redundant data acquisition is triggered, forming a closed-loop self-optimizing system that effectively solves multi-view temporal confusion and reduces the false alarm rate of misreporting a single crack propagation as multiple defect points.

[0172] Embodiment 2 of the present invention:

[0173] In the scenario of track surface defect detection, track image data is acquired through a multi-view camera array, event cameras capture the event stream of track surface brightness changes, and global shutter cameras acquire complete image frames. All data are appended with hardware timestamps from the same clock source to generate fused data with a unified spatiotemporal reference. Ultra-wideband positioning and laser scanning acquire the track's 3D coordinates and contour point cloud data, which are then registered to generate pose trajectory data. Vibration sensors collect track vibration data as sensor time-series data. A time-sensitive network transmits the data and establishes a spatiotemporal index.

[0174] Dynamic spatiotemporal alignment processing utilizes event stream data to perform motion compensation interpolation between image frames, generating a virtual synchronization frame sequence. Lie group theory is employed to optimize the multi-view spatiotemporal transformation model, outputting the alignment sequence and uncertainty estimates. The feature extraction network adapts to different lighting conditions through a meta-learning framework, embedding a structural dynamics model to constrain feature extraction and mapping features to a physical quantity space. Attention weights are calculated based on the virtual synchronization frame sequence for spatial feature fusion, and temporal modeling captures track wear evolution characteristics, generating a fault feature representation.

[0175] The digital twin model simulates the track wear process using finite element analysis, outputting a labeled fault simulation sequence. Dynamic time warping compares the real features with the simulation sequence, eliminates timing jitter, calculates a similarity score, and outputs the wear identification result. Evidence theory is used to fuse sensor data to calculate the confidence interval, and reinforcement learning is used to adjust the acquisition parameters. When the confidence level falls below a threshold, redundant acquisition is triggered, forming a closed-loop control that resolves multi-view timing confusion and reduces the false alarm rate of misclassifying progressive wear as discrete defects.

[0176] The train components described in this invention include bogies, wheelsets, axle boxes, spring suspension devices, braking systems, coupler buffer devices, and pantographs, among other key operating components. The multi-view images include pixel-level asynchronous event stream data captured by an event camera, a sequence of synchronous image frames acquired at fixed intervals by a global shutter camera, and a sequence of virtual synchronous frames generated by motion compensation interpolation. All image data together constitute a multimodal visual input with spatiotemporal alignment characteristics.

[0177] The event stream data described in this invention refers to an asynchronous data stream generated by pixel-level brightness changes captured by an event camera, with each event data packet including pixel coordinates, timestamp, and polarity information; the image data stream refers to a complete sequence of image frames acquired at fixed intervals by a global shutter camera; the train pose trajectory data refers to six-degree-of-freedom spatial position and attitude data obtained through ultra-wideband positioning and laser scanning and then registered, including the train's translation coordinates and rotation matrix in a three-dimensional coordinate system; the sensor timing data refers to a sequence of vibration signals from train components acquired by a vibration sensor; and the virtual synchronization frame sequence refers to data generated using event stream data. The image sequence with a unified time reference is generated by motion compensation interpolation between image frames; the structural dynamics model refers to a physical constraint model describing the mechanical behavior of materials based on the finite element method; the digital twin model refers to a parameterized component model that simulates the dynamic evolution of faults by applying actual operational load spectra through finite element analysis; the dynamic time warping refers to an algorithm that finds the optimal bending path between two sequences to eliminate temporal differences; the evidence theory refers to DS evidence theory, which is used to fuse multi-source evidence to calculate the confidence interval; the reinforcement learning strategy refers to an optimization method that generates image acquisition parameter adjustment instructions with the goal of maximizing information entropy.

[0178] The technical features of this invention are explained below:

[0179] The Bayesian estimation model is constructed by fusing visual odometry features and pose trajectory data using a particle filter algorithm. Visual odometry features are extracted from the image data stream, and the relative motion between adjacent image frames is calculated using feature point detection and tracking algorithms. The pose trajectory data is generated by registering the train's 3D coordinates and contour point cloud data obtained from ultra-wideband positioning and laser scanning. The particle filter algorithm employs importance sampling and resampling processes to fuse visual odometry features and pose trajectory data, estimating the posterior probability distribution of the train's motion state. This model leverages the high temporal resolution of event stream data, calculating pixel-level motion vectors through optical flow estimation, and performing motion compensation interpolation between adjacent global shutter image frames to generate intermediate views. The Bayesian estimation model outputs the probability distribution of the train's motion state, providing a hidden state initialization benchmark for the temporal modeling process and a dynamic benchmark for motion compensation.

[0180] The structural dynamics model, based on the finite element method, constrains the feature mapping process of the feature extraction network. This model receives the pose trajectory data generated in step 1 as boundary conditions; the pose trajectory data includes the train's six-degree-of-freedom position and attitude information. The structural dynamics model describes the material's mechanical behavior through finite element analysis, mapping the intermediate layer features of the feature extraction network to the stress-strain physical quantity space. The model employs parametric component modeling, applies actual operational load spectra to simulate the material's mechanical response, and calculates the stress-strain field. Embedded within the feature extraction network, the structural dynamics model constrains the feature extraction process through physical laws, ensuring that the extracted features conform to the material's mechanical properties. The model outputs a feature representation in the physical quantity space, including stress and strain field data, providing a physically-compliant feature benchmark for fault identification.

[0181] Lie group theory is applied to optimize multi-view geometric relationships in the dynamic spatiotemporal alignment stage. The algorithm processes the initial multi-view sequence, including intensity images, global shutter image frames, and intermediate views. Lie group theory is used to establish a camera extrinsic parameter perturbation model, representing the continuous space of the camera pose; this perturbation model describes the camera extrinsic parameter calibration error. A nonlinear optimization algorithm is used to adjust camera parameters, optimizing the multi-view geometric relationships with the goal of minimizing reprojection error. The algorithm outputs the optimized multi-view geometric relationships, a virtual synchronization frame sequence, and uncertainty estimates, providing aligned input data for subsequent feature extraction.

[0182] The particle filter algorithm processes visual odometry features and pose trajectory data during the train motion state estimation stage. Visual odometry features are extracted from the image data stream, while pose trajectory data is acquired through ultra-wideband localization and laser scanning. The algorithm fuses visual odometry features and pose trajectory data through importance sampling and resampling processes to estimate the posterior probability distribution of the train motion state. The particle filter algorithm outputs a Bayesian estimation model of the train motion state, providing a probabilistic motion description used to initialize the hidden states and motion compensation baseline for the temporal modeling process.

[0183] Optical flow estimation processes event stream data and adjacent global shutter image frames during the motion compensation interpolation stage. Event stream data provides high temporal resolution information on brightness variations. Optical flow estimation calculates pixel-level motion vectors and performs motion compensation between adjacent global shutter image frames based on these vectors. An interpolation algorithm is used to generate an interpolated intermediate view, filling the temporal gaps between image frames. Optical flow estimation outputs the interpolated intermediate view, reducing motion blur and temporal inconsistencies, and enhancing the continuity of the virtual synchronization frame sequence.

[0184] The Expectation-Maximization (EM) algorithm handles camera extrinsic perturbation models and reprojection errors during the camera extrinsic calibration phase. The algorithm iteratively updates the camera extrinsic calibration results, estimating hidden variables in the expectation step and optimizing parameter estimates in the maximization step. The EEM algorithm calculates the uncertainty matrix of the spatiotemporal transformation and estimates the uncertainty of the transformation parameters through covariance propagation. The algorithm outputs the camera extrinsic calibration results and the uncertainty matrix of the spatiotemporal transformation, providing credibility information for the virtual synchronization frame sequence.

[0185] The meta-learning framework processes aligned multi-view image sequences during the pre-training stage of the feature extraction network. The framework employs a model-independent meta-learning algorithm to handle task sets under multiple different acquisition conditions. The framework utilizes support and query sets for episodic training to optimize the initial parameters of the feature extraction network. The framework outputs a pre-trained feature extraction network, enabling the network to adapt to changes in feature distribution under different acquisition conditions, thereby improving the model's generalization performance and robustness in fault identification.

[0186] An attention mechanism processes virtual synchronization frame sequences and multi-view feature data during spatial feature fusion. This mechanism calculates the similarity between features from different perspectives by querying key-value pairs, generating an attention weight map that highlights key areas related to faults. Based on the attention weights, a weighted summation operation is performed on the features from different perspectives to achieve spatial feature fusion, outputting a fused spatial feature sequence that enhances the completeness and robustness of the feature representation.

[0187] Gated recurrent unit networks (GRUs) process the fused spatial feature sequences during temporal modeling. The network controls information flow by updating and resetting gates, learning dynamic patterns and long-term dependencies in the sequence to capture the temporal features of fault evolution. The GRU outputs a fault feature representation, incorporating spatiotemporal information, for subsequent fault identification and comparison stages.

[0188] Finite element analysis (FEM) processes parameterized component models and actual operational load spectra during fault simulation sequence generation. Through mesh generation and boundary condition settings, FEM simulates material mechanical behavior, calculates stress-strain distribution, and outputs a stress-strain contour map sequence as the fault simulation sequence. FEM provides a physical benchmark for fault evolution, supporting subsequent dynamic time warping comparisons.

[0189] Generative Adversarial Networks (GANs) handle random noise and fault type labels during the synthetic data generation process. The generator takes noise and labels as input to produce synthetic fault images, while a discriminator distinguishes between real and generated images. Adversarial training improves the quality of the generated images. The GAN outputs a sequence of synthetic fault images with spatiotemporal annotations, enhancing the diversity and realism of the training dataset.

[0190] The Dynamic Time Warping (VTW) algorithm processes fault feature representations and fault simulation sequences during time-series alignment. The algorithm calculates the minimum cumulative distance path between two sequences, allowing the time axis to bend to match similar feature points and eliminating timing jitter caused by asynchronous acquisition. The VTW outputs aligned sequences and similarity scores, used to determine fault mode recognition results and confidence levels.

[0191] DS evidence theory processes fault mode recognition results, similarity scores, and sensor time-series data, transforming each input into a probability quality assignment through a basic probability assignment function. This function categorizes fault mode recognition results into probabilities of different hypotheses, incorporates the similarity score as a confidence index into the probability assignment, and converts the sensor time-series data into a time-series evidence body through feature extraction methods. DS evidence theory outputs basic probability assignments, providing a probabilistic foundation for subsequent evidence synthesis.

[0192] Dempster's combined rule processes the basic probability assignments derived from DS evidence theory, performing orthogonal sum operations on multi-source evidence. The rule calculates the reliability corresponding to the fault mode recognition results, similarity scores, and sensor time-series data, eliminating conflicts between pieces of evidence and generating a joint reliability function. Dempster's combined rule outputs a reliability interval, including confidence and likelihood, quantifying the degree of support and uncertainty of the fault modes simulated by the digital twin model.

[0193] The Elastic Weight Consolidation algorithm processes confidence intervals to calculate the importance weights of feature extraction network parameters in historical tasks. The algorithm guides parameter updates based on confidence intervals, protecting important parameters from catastrophic forgetting, while minor parameters are adjusted according to the confidence intervals. The Elastic Weight Consolidation algorithm outputs updated feature extraction network parameters, enabling the network to adapt to new failure modes without losing existing knowledge.

[0194] The reinforcement learning policy processes the confidence interval and generates image acquisition parameter adjustment instructions with the objective of maximizing information entropy. The policy uses the confidence interval as the state space, maximizes information entropy as the reward function, and employs a policy gradient method to optimize the acquisition strategy. The reinforcement learning policy outputs image acquisition parameter adjustment instructions, dynamically adjusting the camera sampling frequency and resolution to maximize information gain.

Claims

1. A method for automatic fault identification using multi-view image acquisition in rail transit maintenance, characterized in that, include: Step 1: Collect event stream data, image data stream, train pose trajectory data, and sensor time series data of rail transit facilities, and add timestamps to the event stream data, image data stream, train pose trajectory data, and sensor time series data to generate fused data with a unified spatiotemporal reference. Step 2: Perform dynamic spatiotemporal alignment processing on the fused data, use event stream data to perform motion compensation interpolation between image frames to generate a virtual synchronization frame sequence, and use Lie group theory to optimize the spatiotemporal transformation model between multiple views, outputting the aligned multi-view image sequence and its uncertainty estimate. Step 3: The aligned multi-view image sequence and its uncertainty estimate are input into the meta-learning feature extraction network coupled with physical priors. The feature extraction process is constrained by embedding a preset structural dynamics model. Based on the virtual synchronization frame sequence, the attention weights between features from different perspectives are calculated to perform spatial feature fusion. Then, the fused spatial feature sequence is temporally modeled to capture the temporal features of fault evolution, and fault feature representation is generated and output. Step 4: Construct digital twin models of train components to generate labeled fault simulation sequences. Perform dynamic time warping comparison between fault feature representations and fault simulation sequences to eliminate temporal differences between fault feature representations and fault simulation sequences. Calculate the similarity between fault feature representations and fault simulation sequences. Based on the similarity, determine and output fault pattern recognition results and similarity scores. Step 5: Integrate the fault mode recognition results and similarity scores, and combine them with the sensor time series data in the fused data. Based on the multiple fault modes simulated by the digital twin model, use evidence theory to calculate the confidence interval of the corresponding fault mode. Based on the confidence interval, an image acquisition parameter adjustment instruction is generated through a reinforcement learning strategy, and the image acquisition parameter adjustment instruction is sent to the image acquisition device to dynamically adjust the acquisition parameters. When the confidence interval meets the preset conditions, the fault identification result is output.

2. The automatic fault identification method for multi-view image acquisition in rail transit maintenance according to claim 1, characterized in that, Step 1 includes: Event stream data is generated by capturing pixel-level brightness changes through an event camera and by acquiring images at fixed intervals through a global shutter camera. Attach hardware timestamps from the same clock source to event packets in the event stream data and image frames in the image data stream; Real-time 3D coordinates and contour point cloud data of the train are obtained through ultra-wideband positioning and laser scanning; The three-dimensional coordinates and contour point cloud data are registered to generate the six-degree-of-freedom pose trajectory data of the train. The six-degree-of-freedom pose trajectory data of the train includes spatial position data and attitude direction data. The spatial position data includes the translation coordinate values ​​of the train along the X-axis, Y-axis and Z-axis in the three-dimensional coordinate system, and the attitude direction data includes the rotation matrix of the train's rotation attitude around the X-axis, Y-axis and Z-axis. Vibration data of train components are collected using vibration sensors as sensor timing data; Transmit event stream data, image data stream, pose trajectory data, and sensor timing data with additional hardware timestamps via time-sensitive networking; By associating pose trajectory data, event stream data, image data stream, and sensor time series data with corresponding hardware timestamps, a spatiotemporal index is established to generate fused data.

3. The method for automatic fault identification using multi-view image acquisition for rail transit maintenance according to claim 2, characterized in that, Step 2 includes: Visual odometry features are extracted from the image data stream, and a particle filter algorithm is used to fuse the visual odometry features with the pose trajectory data. Based on the fusion result, a Bayesian estimation model of the train motion state is constructed. Based on the train motion state provided by the Bayesian estimation model, and using event stream data, motion compensation is performed between adjacent global shutter image frames through optical flow estimation to generate an interpolated intermediate view. The accumulated event stream data is reconstructed into an intensity image, and the intensity image, together with the global shutter image frame and intermediate view, constitutes the initial multi-view sequence. For the initial multi-view sequence, a camera extrinsic perturbation model based on Lie algebra is established, and the multi-view geometric relationship is optimized based on the camera extrinsic perturbation model with the goal of minimizing the reprojection error; The expectation-maximization algorithm is used to iteratively update the camera extrinsic parameter calibration results, and the uncertainty matrix of the spatiotemporal transformation is calculated. The virtual synchronization frame sequence and uncertainty estimate are then output.

4. The automatic fault identification method for multi-view image acquisition in rail transit maintenance according to claim 3, characterized in that, Step 3 includes: The aligned multi-view image sequence and its uncertainty estimate are input into the feature extraction network. The feature extraction network constrains the feature extraction process through a pre-set structural dynamics model and maps the extracted features to the physical quantity space. A meta-learning framework is used to pre-train the feature extraction network so that the feature extraction network can adapt to the changes in feature distribution under different acquisition conditions; Based on the virtual synchronization frame sequence, the attention weights between features from different perspectives are calculated, and spatial features are fused based on the attention weights to obtain the fused spatial feature sequence. Temporal modeling is performed on the fused spatial feature sequence to capture the temporal features of fault evolution, and fault feature representation is generated and output.

5. The method for automatic fault identification using multi-view image acquisition for rail transit maintenance according to claim 4, characterized in that, Step 4 includes: A parameterized component model is established through finite element analysis, and an actual operational load spectrum is applied to simulate the dynamic evolution of the fault. The stress-strain cloud map sequence is output as the fault simulation sequence. Generative adversarial networks are used to generate synthetic fault image sequences with spatiotemporal annotations; The fault simulation sequence is fused with the synthetic fault image sequence to generate an annotated fault simulation sequence. Dynamic time warping is performed on the fault feature representation and the labeled fault simulation sequence to find the optimal curved path between the fault feature representation and the labeled fault simulation sequence in order to eliminate the temporal differences between the fault feature representation and the labeled fault simulation sequence. The similarity between the fault feature representation after dynamic time warping and the labeled fault simulation sequence is calculated. Based on the similarity, the fault mode recognition result and similarity score are determined and output.

6. The method for automatic fault identification using multi-view image acquisition for rail transit maintenance according to claim 5, characterized in that, Step 5 includes: The basic probability assignment function of DS evidence theory is used to process the fault mode recognition results and similarity scores, and sensor time-series data in the fused data is combined. The reliability intervals corresponding to various fault modes simulated by the digital twin model are calculated by synthesizing fault mode recognition results, similarity scores, and sensor time series data through Dempster combination rules. An elastic weight consolidation algorithm is used to update the parameters of the feature extraction network based on confidence intervals; Based on the confidence interval, image acquisition parameter adjustment instructions are generated using a reinforcement learning strategy with the goal of maximizing information entropy; The image acquisition parameter adjustment command is sent to the acquisition device to dynamically adjust the camera sampling frequency and resolution. When the confidence interval reaches the preset threshold, the fault identification result is output.

7. The method for automatic fault identification using multi-view image acquisition for rail transit maintenance according to claim 6, characterized in that, Also includes: The uncertainty estimate output from step 2 is used to adjust the attention weight allocation in the spatial feature fusion process in step 3; The train motion state provided by the Bayesian estimation model in step 2 is used to initialize the hidden state of the time series modeling process in step 3; In step 3, when the feature extraction network is constrained by the structural dynamics model for feature extraction, it receives the pose trajectory data generated in step 1 and uses it as the boundary condition for the structural dynamics model.

8. The automatic fault identification method for multi-view image acquisition in rail transit maintenance according to claim 7, characterized in that, Also includes: The fault feature representation output in step 3 serves as the validation benchmark for the digital twin model. The difference between the fault feature representation and the fault simulation sequence is fed back to the meta-learning framework to optimize the feature distribution shift adaptation. The activation patterns represented by the fault features output in step 3 are used as training samples for training the generative adversarial network. The abnormal patterns identified in step 4 are injected as training samples into the training data stream of the feature extraction network.

9. The method for automatic fault identification using multi-view image acquisition for rail transit maintenance according to claim 8, characterized in that, Also includes: The similarity score output in step 4 serves as the input for the basic probability assignment of the evidence theory in step 5. The difference between the fault simulation sequence generated by the digital twin model in step 4 and the fault feature representation output in step 3 is quantified as an uncertainty index and participates in the calculation of the confidence interval in step 5. In step 5, the elastic weight consolidation algorithm uses the fault simulation sequence generated in step 4 as an auxiliary task for multi-task learning.

10. The method for automatic fault identification using multi-view image acquisition for rail transit maintenance according to claim 9, characterized in that, Also includes: The image acquisition parameter adjustment command generated in step 5 applies to the acquisition modes of the event camera and the global shutter camera in step 1. In step 5, the updated feature extraction network parameters are distributed to the image acquisition device; When the confidence interval calculated in step 5 is lower than the preset threshold, the ultra-wideband positioning and laser scanning in step 1 are triggered to collect redundant data.

Images