A fault prediction method based on 360° dynamic image detection system

Abstract

The present invention relates to the field of image processing technology, and in particular to a fault prediction method based on a 360° dynamic image detection system. The method comprises the following steps: acquiring the image data of a train, identifying abnormal points in the image according to an image processing algorithm; constructing an associated empirical model, and assigning corresponding weights to different types of abnormal points according to empirical rules; constructing a historical model, identifying and analyzing fault modes and trends based on the historical data of the train, and predicting the life and replacement cycle of components; constructing a transmission model for simulating the propagation process of a fault signal, and quantitatively evaluating the degree of influence of the fault signal on the data during the propagation process through model analysis. The present invention provides a fault prediction method based on a 360° dynamic image detection system, which accurately predicts potential faults through image processing, significantly reduces safety risks during train operation, reduces operational interruptions and maintenance costs caused by faults, and thus improves the overall operational efficiency of the train.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a fault prediction method based on a 360-degree dynamic image detection system. Background Art

[0002] 360° dynamic image detection can continuously and comprehensively collect a large amount of vehicle abnormality data, which records in detail the specific location, component, car number and train number of the abnormal item, providing rich basic data for vehicle fault analysis and prediction. However, despite such a huge data resource, existing technologies are mainly limited to fault detection and cannot effectively predict faults.

[0003] For example, the patent publication number CN118464481A, named "A 360° intelligent detection system for urban rail third rail current-collecting vehicles", includes vehicle entry and exit detection equipment, speedometer, roof linear array scanning module, vehicle bottom line array detection module, vehicle side linear array detection module and current collector detection module. The vehicle entry and exit detection equipment is used to capture images of the entrance and exit areas of the detection area to realize vehicle position judgment and license plate information recognition. The speedometer is used to obtain the vehicle's running speed. The roof linear array scanning module, vehicle bottom line array detection module and vehicle side linear array detection module are used to realize the detection of the roof, bottom and side of the vehicle. The current collector detection module is used to realize the third rail current collector pressure detection and current collector shoe wear detection. The vehicle mileage is used as the horizontal coordinate and the wear amount is used as the vertical coordinate to draw a relationship curve, and the later wear amount is predicted according to the relationship curve. The disadvantage is that it can only predict the later wear amount, and cannot accurately predict potential faults. Summary of the invention

[0004] In view of the problem in the prior art that 360° dynamic image detection can only detect faults but cannot predict faults, the present invention provides a fault prediction method based on a 360° dynamic image detection system. By accurately predicting potential faults, it can significantly reduce the safety risks during train operation, while reducing operational interruptions and maintenance costs caused by faults, thereby improving the overall operational efficiency of the train.

[0005] In order to achieve the above technical objectives, a technical solution provided by the present invention is a fault prediction method based on a 360° dynamic image detection system, comprising the following steps:

[0006] S1, obtain the image data of the train, and identify the abnormal points in the image according to the image processing algorithm;

[0007] S2, build an associated empirical model and assign corresponding weights to different types of abnormal items according to empirical rules;

[0008] S3, builds a historical model to identify and analyze failure modes and trends based on historical train data, and predicts the life and replacement cycle of components;

[0009] S4, construct a transmission model to simulate the propagation process of the fault signal, and quantitatively evaluate the impact of the fault signal on the data during the propagation process through model analysis;

[0010] S5, according to the multi-model fusion method, quantizes the correlation values ​​generated by the associated empirical model, historical model and transfer model, assigns weights to different correlation values ​​and performs weighted summation, and establishes component failure correlation analysis based on multi-modal data.

[0011] In this technical solution, firstly, by acquiring the all-round image data of the train and applying the image processing algorithm, the abnormal points in the image are accurately identified, and the abnormal points are often the precursors of the fault. Then, the method constructs the associated empirical model, and according to the empirical rules, the corresponding weights are assigned to different types of abnormal points, so as to realize the quantitative evaluation of the possibility of fault, making the fault prediction more targeted and accurate. In order to dig deeper into the fault information, a historical model is constructed, based on the massive historical data of the train, to identify and analyze the fault mode and trend, and then predict the life and replacement cycle of the parts, so that the fault prediction is not only limited to the current state, but also can predict the future fault development. In addition, a transmission model is constructed to simulate the propagation process of the fault signal. Through model analysis, the influence of the fault signal on the data during the propagation process is quantitatively evaluated, so as to more accurately judge the scope and severity of the fault. The multi-model fusion technology is used to quantify the correlation values ​​generated by the associated empirical model, the historical model and the transmission model, and weights are assigned to different correlation values ​​and weighted summation is performed, so that the fault prediction results are more comprehensive and accurate, providing strong support for the correlation analysis of component faults.

[0012] The present invention is further configured as follows: in step S5, the establishment of component failure correlation analysis based on multimodal data includes:

[0013] Extract features and perform feature selection on the fused multimodal data;

[0014] Build machine learning models to analyze the fault correlation between components;

[0015] Use network analysis methods to analyze the propagation path and possibility of faults between components;

[0016] Validate machine learning models using historical failure data.

[0017] In this technical solution, by extracting and selecting features from the fused multimodal data, key information closely related to component failures is effectively mined. Next, a machine learning model is constructed and used to deeply analyze the fault correlation between components, revealing the laws and patterns of fault propagation between different components. In addition, network analysis methods are used to further explore the propagation path and possibility of faults between components, providing strong support for predicting and preventing the spread of faults. Finally, the machine learning model is rigorously verified using historical fault data to ensure the accuracy and reliability of the model.

[0018] The present invention is further configured as follows: constructing a machine learning model comprises:

[0019] Synchronously collect image data, vibration signal data and temperature data;

[0020] Process the data and perform feature concatenation to generate a multimodal data format with uniform size;

[0021] According to the fault information, corresponding labels are created for the data, and the processed multimodal data and the corresponding labels are combined to generate a multimodal fault diagnosis feature extraction data set;

[0022] A fault diagnosis neural network is constructed based on the multimodal data set for model training to generate a fault prediction feature extraction model.

[0023] In this technical solution, the image data, vibration signal data and temperature data are collected synchronously to ensure the comprehensiveness and diversity of the information required for fault prediction. Synchronous collection includes time synchronization and space synchronization of data, which ensures the consistency and correlation between data from different sources and improves the accuracy and reliability of data processing. By processing and cascading data, a multimodal data format with the same size is generated, which simplifies the complexity of data processing and makes the data easier to be accepted and processed by the model. Time synchronization is achieved by controlling the triggering of the acquisition instruction to ensure that each sensor collects data almost at the same time, and spatial synchronization is achieved by multi-sensor joint calibration to ensure that the objects collected in a specific area are consistent in spatial position. When constructing a neural network, the input layer accepts 32-bit floating-point data with a size of 640*640*5, the feature extraction layer uses the ResNet152 network pre-trained with the Images public data set to extract rich feature representations, and the output layer is connected to a fully connected layer with a shape of 1*2048 to generate the final fault prediction features.

[0024] The present invention is further configured as follows: the network analysis method comprises:

[0025] The components in the train are represented as nodes in the network, and the fault propagation relationship between the components is represented as edges to construct a fault propagation network;

[0026] Use the shortest path algorithm to analyze the fault propagation path and key nodes in the network;

[0027] Based on the constructed fault propagation network, simulation technology is used to simulate the fault propagation process and predict the components and scope that may be affected by the fault.

[0028] In this technical solution, the construction of a fault propagation network not only makes the fault propagation mode of the train clearly visible, but also provides a solid foundation for subsequent fault analysis and prediction. In the analysis of fault propagation paths and key nodes, the shortest path algorithm is used to quickly and accurately analyze the propagation path of the fault in the network, as well as those nodes that have a key impact on the fault propagation, which is of great significance for understanding the fault diffusion mechanism and formulating effective preventive measures. In addition, based on the constructed fault propagation network, the fault propagation process is simulated using simulation technology, which can not only intuitively show how the fault spreads in the network, but also predict the components and scope that the fault may affect, which is of significant value for timely formulating maintenance plans and reducing the impact of faults on train operations.

[0029] The present invention is further configured as follows: in step S1, the image processing algorithm includes:

[0030] Perform color correction on the continuously acquired 360° images of the train to eliminate the impact of physical environmental changes such as temperature and humidity and light source intensity attenuation on the color temperature and hue of the acquired images;

[0031] Perform histogram equalization and filtering on the color-corrected image to eliminate random noise in the image;

[0032] The anomaly detection deep learning network model is used to detect abnormal points in the preprocessed image, and the category information of the abnormal points, the coordinate values ​​in the image, and the length and width of the minimum circumscribed rectangle of the abnormal area are output.

[0033] In this technical solution, by color correction of the continuously acquired 360° images of the train, the effects of changes in physical environment such as temperature and humidity and attenuation of light source intensity on the color temperature and hue of the image are effectively eliminated, ensuring the accuracy and consistency of the image color, and providing high-quality image data for subsequent processing. The color-corrected image is further processed by histogram equalization to make the image contrast more balanced and the details of the bright and dark areas clearer, which is conducive to the accurate identification of abnormal items in subsequent processing. The filtering process effectively eliminates random noise in the image and improves the purity of the image, laying a solid foundation for subsequent edge detection, feature extraction and other steps. Using the trained anomaly detection deep learning network model, the pre-processed image is deeply analyzed, which can accurately detect abnormal points in the image, including the category of the abnormal point, the precise location (coordinate value) and the size of the abnormal area (the length and width of the minimum enclosing rectangle), providing detailed data support for subsequent fault analysis and prediction.

[0034] The present invention is further configured as follows: in step S3, the identifying and analyzing failure modes and trends and predicting the life and replacement cycle of components include:

[0035] Use knowledge graphs to perform path association analysis and explore potential associations between component process parameters and fault types;

[0036] Based on the mined association rules, a binary graph model of component failure is constructed;

[0037] Collect real-time multi-modal data to form multi-type operating status parameter data sets, and analyze the data sets;

[0038] Through the knowledge graph's path association probability search technology and the Bayesian probability learning method between nodes, the potential fault type, faulty components, and fault timing can be predicted in advance.

[0039] Construct a node association diagram based on components, fault types and system modes to match the fault location and fault signal of potential fault types.

[0040] In this technical solution, firstly, the knowledge graph is used for path association analysis, and the potential association rules between component process parameters and fault types are successfully mined. Secondly, based on the mined association rules, a binary graph model of component faults is constructed, which can intuitively display the propagation path and possible impact range of the fault, and help maintenance personnel to locate and troubleshoot the fault more accurately. In terms of data acquisition and analysis, a multi-type operating status parameter data set is formed by collecting real-time multimodal data, and the XGBoost algorithm is used to conduct in-depth analysis of the data set. For image data, convolutional neural networks can be used to extract features, and then these features are used as inputs for XGBoost. For vibration signals and temperature data, Fourier transforms can be performed to generate one-dimensional features that can be used for XGBoost. It not only improves the accuracy and efficiency of data analysis, but also provides strong data support for subsequent fault prediction. In addition, this method also realizes the early prediction of potential fault types, faulty components and fault timing through the path association probability search technology of the knowledge graph and the Bayesian probability learning method between nodes. It is of great significance for timely formulation of maintenance plans and reducing the impact of faults on train operation. Finally, the method constructs a node association graph based on components, fault types, and system modes, achieving accurate matching of the fault location and fault signal of potential fault types. The implementation of this matching process not only improves the accuracy of fault diagnosis, but also provides maintenance personnel with more specific maintenance guidance.

[0041] The present invention is further configured such that: the empirical rules are based on changes in component appearance in RGB images, changes in geometric shapes in 3D point clouds, and changes in frequency, amplitude and phase in vibration signals.

[0042] In this technical solution, the empirical rules integrate information from multiple data sources, including changes in component appearance in RGB images, changes in geometric shapes in 3D point clouds, and changes in frequency, amplitude, and phase in vibration signals. This comprehensive empirical rule design enables the fault prediction method to more comprehensively capture potential fault signs of train components, thereby improving the accuracy and reliability of predictions. By comprehensively considering the appearance, geometric shape, and vibration characteristics of components, the abnormal state of components can be more accurately identified, providing strong support for timely maintenance measures and ensuring safe operation of trains.

[0043] The present invention is further configured as follows: in the historical model, the life span and replacement cycle of parts are predicted by monitoring the appearance wear of parts in RGB images, the size changes in 3D point clouds, and the energy attenuation in vibration signals.

[0044] In this technical solution, by monitoring the appearance wear of parts in RGB images, it is possible to intuitively capture the signs of wear on the surface of parts due to long-term use. At the same time, combined with the dimensional change analysis in the 3D point cloud, it is possible to accurately measure the dimensional changes of parts and components, and then determine whether there is a risk of deformation or damage. In addition, by monitoring the energy attenuation in the vibration signal, it is also possible to effectively evaluate the working status and performance degradation of parts and components.

[0045] The present invention is further configured as follows: in step S4, the quantitative evaluation indicators include the abnormal area expansion speed in the RGB image, the deformation acceleration rate in the 3D point cloud, and the energy transfer efficiency in the vibration signal to simulate the propagation process of the fault signal.

[0046] In this technical solution, by introducing the abnormal area expansion speed in the RGB image, the diffusion of the fault on the surface of the component can be accurately described, providing an important basis for predicting the further development of the fault. The deformation acceleration rate in the 3D point cloud is also considered, which can reflect the rate of change of the geometric shape of the component, thereby revealing the potential impact of the fault on the component structure. Combined with the energy transfer efficiency in the vibration signal, a quantitative indicator, it can more comprehensively simulate the propagation process of the fault signal in the train system.

[0047] The present invention is further configured as follows: after S5, it also includes S6: generating a warning signal and fault information according to the prediction result after weighted fusion, and pushing them to the management personnel.

[0048] In this technical solution, by generating early warning signals, managers can be notified in advance before potential failures occur, so that they have enough time to react and take corresponding preventive measures. At the same time, detailed fault information also provides managers with an important basis for judging the type of fault and formulating maintenance plans.

[0049] The beneficial effects of the present invention are as follows: (1) by accurately predicting potential faults, it is possible to significantly reduce safety risks during train operation, while reducing operational interruptions and maintenance costs caused by faults, thereby improving the overall operational efficiency of the train; (2) firstly, by acquiring all-round image data of the train and applying image processing algorithms, the abnormal points in the image are accurately identified. The abnormal points are often precursors to faults. Then, the method constructs an associated empirical model and assigns corresponding weights to different types of abnormal points according to empirical rules, thereby achieving a quantitative assessment of the possibility of faults, making fault prediction more targeted and accurate. In order to more deeply mine fault information, a historical model is constructed based on the massive historical data of the train. Data can be used to identify and analyze failure modes and trends, and then predict the life and replacement cycle of components, so that fault prediction is not limited to the current state, but can also predict future fault development. In addition, a transfer model is constructed to simulate the propagation process of fault signals. Through model analysis, the degree of influence of fault signals on data during the propagation process is quantitatively evaluated, so as to more accurately judge the impact scope and severity of the fault. Multi-model fusion technology is used to quantify the correlation values ​​generated by the associated empirical model, historical model and transfer model, and weights are assigned to different correlation values ​​and weighted summed, so that the fault prediction results are more comprehensive and accurate, providing strong support for the correlation analysis of component failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of a fault prediction method based on 360° dynamic image detection of the present invention;

[0051] Figure 2 It is a schematic diagram of the dirty structure in the present invention;

[0052] Figure 3 It is a schematic diagram of the structure of the loosened bolt in the present invention;

[0053] Figure 4 It is a schematic diagram of the structure of the scratch in the present invention. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation method described herein is only an optimal embodiment of the present invention, which is only used to explain the present invention and does not limit the scope of protection of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0055] like Figure 1As shown, as a first embodiment of the present invention, a fault prediction method based on a 360° dynamic image detection system includes the following steps:

[0056] S1, obtain the image data of the train, and identify the abnormal points in the image according to the image processing algorithm;

[0057] S2, build an associated empirical model and assign corresponding weights to different types of abnormal items according to empirical rules;

[0058] S3, builds a historical model to identify and analyze failure modes and trends based on historical train data, and predicts the life and replacement cycle of components;

[0059] S4, construct a transmission model to simulate the propagation process of the fault signal, and quantitatively evaluate the impact of the fault signal on the data during the propagation process through model analysis;

[0060] S5, according to the multi-model fusion method, quantizes the correlation values ​​generated by the associated empirical model, historical model and transfer model, assigns weights to different correlation values ​​and performs weighted summation, and establishes component failure correlation analysis based on multi-modal data.

[0061] In this embodiment, firstly, by acquiring the all-round image data of the train and applying the image processing algorithm, the abnormal points in the image are accurately identified, and the abnormal points are often the precursors of the fault. Then, the method constructs the associated empirical model, and according to the empirical rules, assigns corresponding weights to different types of abnormal points, so as to realize the quantitative evaluation of the possibility of fault, making the fault prediction more targeted and accurate. In order to dig deeper into the fault information, a historical model is constructed, based on the massive historical data of the train, to identify and analyze the fault mode and trend, and then predict the life and replacement cycle of the parts, so that the fault prediction is not only limited to the current state, but also can predict the future fault development. In addition, a transmission model is constructed to simulate the propagation process of the fault signal. Through model analysis, the influence of the fault signal on the data during the propagation process is quantitatively evaluated, so as to more accurately judge the scope and severity of the fault. The multi-model fusion technology is used to quantify the correlation values ​​generated by the associated empirical model, the historical model and the transmission model, and weights are assigned to different correlation values ​​and weighted summation is performed, so that the fault prediction results are more comprehensive and accurate, providing strong support for the correlation analysis of component faults.

[0062] It is understandable that 360° dynamic image detection systems are widely used in the status detection of transportation vehicles such as trains and subway vehicles. They usually include multiple ultra-wide-angle cameras arranged around the vehicle body (roof, side, and bottom) and an image processing unit for receiving image data from each camera and performing distortion restoration, perspective conversion, and image stitching.

[0063] It can be understood that abnormal items include abnormal parts, car number and train number, as well as cracks, corrosion and missing parts.

[0064] It can be understood that the historical data includes fault records, maintenance records and operating parameters.

[0065] It can be understood that the quantification of the correlation values ​​generated for the correlation experience model, the history model and the transfer model includes:

[0066] Use cross-validation methods to evaluate the performance of the associated empirical, historical, and transfer models;

[0067] Based on the model performance evaluation results, a relevance value is generated for each model;

[0068] Assign a preliminary weight to each model based on its performance and applicable scenario factors;

[0069] Optimize and adjust the initial weights through historical data or simulation experiments;

[0070] Use the weighted fusion formula to generate fault prediction results based on the weighted fusion scores.

[0071] It can be understood that the weighted summation formula is:

[0072] FinalScore=w1*x1+w2*x2+w3*x3; where FinalScore is the score after weighted fusion, w1, w2, w3 are the weights of the associated experience model, historical model and transfer model, and x1, x2, x3 are the correlation values ​​of the corresponding models.

[0073] In one embodiment of the present invention, in step S5, the establishing component failure correlation analysis based on multimodal data includes:

[0074] Extract features and perform feature selection on the fused multimodal data;

[0075] Build machine learning models to analyze the fault correlation between components;

[0076] Use network analysis methods to analyze the propagation path and possibility of faults between components;

[0077] Validate machine learning models using historical failure data.

[0078] In this technical solution, firstly, by extracting and selecting features from the fused multimodal data, key information closely related to component failures is effectively mined. Then, a machine learning model is constructed and used to deeply analyze the fault correlation between components, revealing the laws and patterns of fault propagation between different components. In addition, the network analysis method is used to further explore the propagation path and possibility of faults between components, providing strong support for predicting and preventing the spread of faults. Finally, the machine learning model is rigorously verified using historical fault data to ensure the accuracy and reliability of the model.

[0079] It can be understood that the fusion of multimodal data includes data preprocessing and feature extraction and fusion;

[0080] Among them, data preprocessing includes:

[0081] Image data preprocessing: grayscale conversion, filtering and contrast stretching are performed on the train's 360° images to extract key features;

[0082] Vibration signal preprocessing: filtering and noise reduction of vibration signals, and performing spectrum analysis to extract frequency and amplitude characteristics;

[0083] Temperature data preprocessing: smooth the temperature data, eliminate noise, and perform trend analysis;

[0084] Among them, feature extraction and fusion include:

[0085] Feature extraction: Use edge detection, texture analysis, and contour extraction methods to extract features from image data; extract frequency and amplitude features from vibration signals; and extract trend change features from temperature data;

[0086] Feature selection: Select features that are beneficial to fault prediction based on their importance and relevance;

[0087] Feature fusion: The features of different modalities are fused according to the machine learning model to form a unified multimodal feature vector.

[0088] It can be understood that the Min-Max normalization method is used to scale the eigenvalues ​​of each modal data to the same numerical range to ensure that the data of different modalities are numerically comparable and to avoid the fusion effect being affected by the large difference in the numerical range of some modal data.

[0089] It can be understood that the embedding model is trained to generate embedding vectors to transform features of different modalities into a unified, high-dimensional feature space.

[0090] It is understandable that when collecting multimodal data, multiple sensors (such as RGB cameras, lidars, vibration sensors, temperature sensors, etc.) are used to simultaneously collect multi-source data such as images, depth, vibration, and temperature of the train, ensuring that all sensors collect data synchronously in time and space, and trigger the collection instructions through an industrial computer to achieve time synchronization; spatial synchronization is achieved through sensor calibration. It is understandable that in a train system, the failure of a component may be transmitted to other components or subsystems through physical connections or functional dependencies, ultimately leading to performance degradation or failure of the entire system. Establishing component failure correlation analysis based on multimodal data can quantify the degree and speed of such failure propagation, and can be used to predict the scope and severity of the failure.

[0091] It can be understood that in the transfer model, a transfer model score is output, and when the score is higher than a certain threshold, it is determined to be abnormal.

[0092] It can be understood that in step S5, the multi-model fusion quantifies the correlation values ​​generated by the empirical model, the historical model and the transfer model, assigns weights to different correlation values, and finally performs weighted summation.

[0093] In one embodiment of the present invention, constructing a machine learning model includes:

[0094] Synchronously collect image data, vibration signal data and temperature data;

[0095] Process the data and perform feature concatenation to generate a multimodal data format with uniform size;

[0096] According to the fault information, corresponding labels are created for the data, and the processed multimodal data and the corresponding labels are combined to generate a multimodal fault diagnosis feature extraction data set;

[0097] A fault diagnosis neural network is constructed based on the multimodal data set for model training to generate a fault prediction feature extraction model.

[0098] In this technical solution, the image data, vibration signal data and temperature data are collected synchronously to ensure the comprehensiveness and diversity of the information required for fault prediction. Synchronous collection includes time synchronization and space synchronization of data, which ensures the consistency and correlation between data from different sources and improves the accuracy and reliability of data processing. By processing and cascading data, a multimodal data format with the same size is generated, which simplifies the complexity of data processing and makes the data easier to be accepted and processed by the model. Time synchronization is achieved by controlling the triggering of the acquisition instruction to ensure that each sensor collects data almost at the same time, and spatial synchronization is achieved by multi-sensor joint calibration to ensure that the objects collected in a specific area are consistent in spatial position. When constructing a neural network, the input layer accepts 32-bit floating-point data with a size of 640*640*5, the feature extraction layer uses the ResNet152 network pre-trained with the Images public data set to extract rich feature representations, and the output layer is connected to a fully connected layer with a shape of 1*2048 to generate the final fault prediction features.

[0099] It can be understood that during data processing, the vibration signal is processed using the Gram matrix to convert the one-dimensional continuous data into two-dimensional data, whose size matches the width and height of the image; the temperature data is processed into an intensity map equal to the width and height of the image through the Fourier transform conversion method.

[0100] It can be understood that corresponding labels are made for the data according to the fault information, and the processed multimodal data and the labels are combined, and the labels are 0 or 1, where 0 represents a normal state and 1 represents an abnormal state.

[0101] In one embodiment of the present invention, the network analysis method includes:

[0102] The components in the train are represented as nodes in the network, and the fault propagation relationship between the components is represented as edges to construct a fault propagation network;

[0103] Use the shortest path algorithm to analyze the fault propagation path and key nodes in the network;

[0104] Based on the constructed fault propagation network, simulation technology is used to simulate the fault propagation process and predict the components and scope that may be affected by the fault.

[0105] In this technical solution, the fault propagation network is constructed, which not only makes the fault propagation mode of the train clearly visible, but also provides a solid foundation for subsequent fault analysis and prediction. In the analysis of fault propagation paths and key nodes, the shortest path algorithm is used to quickly and accurately analyze the propagation path of the fault in the network, as well as those nodes that have a key impact on the fault propagation, which is of great significance for understanding the fault diffusion mechanism and formulating effective preventive measures. In addition, based on the constructed fault propagation network, the fault propagation process is simulated using simulation technology, which can not only intuitively show how the fault spreads in the network, but also predict the components and scope that the fault may affect, which is of significant value for timely formulating maintenance plans and reducing the impact of faults on train operations.

[0106] It can be understood that when using the shortest path algorithm to analyze the propagation path and key nodes of the fault in the network, each component in the train is regarded as a node in the network. Each node has a unique identifier. According to the physical connection, functional dependency or historical fault propagation data between the components, the fault propagation relationship between the components is defined as an edge. The weight of the edge can represent the probability, speed or impact of the fault propagation; create an adjacency matrix representing the fault propagation network, in which the rows and columns correspond to the nodes in the network respectively, and the matrix elements represent the connection relationship between the nodes and the weight of the edge, and determine the fault starting node, that is, the component where the fault occurs initially. Taking the Dijkstra algorithm as an example, find the shortest propagation path from the initial fault point to other nodes: create a distance array to store the shortest distance from the initial fault point to each node (initially, except for the initial fault point, the distance of the remaining nodes is set to infinity), create a predecessor node array to record the predecessor nodes on the shortest path, and mark the initial fault point as visited. During iterative calculation, all unvisited nodes are traversed, and the node closest to the initial fault point is selected as the current node. For each adjacent node of the current node, the distance from the initial fault point to the adjacent node is calculated (the current node distance + the edge weight from the current node to the adjacent node). If the calculated distance is less than the current value of the adjacent node in the distance array, the distance array and the predecessor node array are updated, and the current node is marked as visited. When all nodes are visited, the algorithm terminates, or if only the shortest path to a specific node needs to be found, it can be terminated early after the shortest path to the node is found. Using the predecessor node array, trace back from the target node to the initial fault point and build the shortest path. Based on the shortest path, identify the key propagation nodes and paths of the fault in the network. Evaluate the components and scope that the fault may affect, and provide a basis for fault prediction and maintenance strategies.

[0107] In step S1, the image processing algorithm includes:

[0108] Perform color correction on the continuously acquired 360° images of the train to eliminate the impact of physical environmental changes such as temperature and humidity and light source intensity attenuation on the color temperature and hue of the acquired images;

[0109] Perform histogram equalization and filtering on the color-corrected image to eliminate random noise in the image;

[0110] The anomaly detection deep learning network model is used to detect abnormal points in the preprocessed image, and the category information of the abnormal points, the coordinate values ​​in the image, and the length and width of the minimum circumscribed rectangle of the abnormal area are output.

[0111] In this technical solution, by color correction of the continuously acquired 360° images of the train, the effects of changes in physical environment such as temperature and humidity and attenuation of light source intensity on the color temperature and hue of the image are effectively eliminated, ensuring the accuracy and consistency of the image color, and providing high-quality image data for subsequent processing. The color-corrected image is further processed by histogram equalization to make the image contrast more balanced and the details of the bright and dark areas clearer, which is conducive to the accurate identification of abnormal items in subsequent processing. The filtering process effectively eliminates random noise in the image and improves the purity of the image, laying a solid foundation for subsequent edge detection, feature extraction and other steps. Using the trained anomaly detection deep learning network model, the pre-processed image is deeply analyzed, which can accurately detect abnormal points in the image, including the category of the abnormal point, the precise location (coordinate value) and the size of the abnormal area (the length and width of the minimum enclosing rectangle), providing detailed data support for subsequent fault analysis and prediction.

[0112] It can be understood that the establishment of anomaly detection deep learning network model includes: collecting a large amount of image data, which should contain samples of normal and abnormal situations; preprocessing the collected image data, including color correction, histogram equalization, filtering and denoising steps, to improve image quality and reduce the difficulty of model training; selecting convolutional neural network or generative adversarial network or autoencoder deep learning architecture according to task requirements and data characteristics; determining the number of network layers, the number of neurons in each layer, activation function, and optimization algorithm parameters; automatically extracting edge and texture local features in the image as useful features from the input image through the designed network structure. During the training process, the network gradually optimizes its parameters by learning the differences between normal and abnormal images to better extract and represent these features; defining a loss function to measure the difference between the model prediction and the actual label; minimizing the loss function through the stochastic gradient descent optimization algorithm; using the preprocessed training data set to train the model. During the training process, the network will learn how to distinguish between normal and abnormal situations based on the input image and the corresponding label; using the validation data set to evaluate the trained model, and adjust the network structure and parameters based on the evaluation results. This process may require multiple iterations until the model performance is satisfactory.

[0113] It can be understood that when edge information in an image is identified according to an edge detection algorithm, texture features of an image are extracted according to a texture analysis method, and abnormal shapes in an image are identified according to a contour extraction method: As a basic step in image processing, edge detection algorithms (such as Canny and Sobel) can identify edge information in an image and provide important clues for subsequent image analysis. However, it is technically difficult to achieve accurate fault detection using edge detection alone, so it needs to be combined with other methods: edge detection is refined into a target detection algorithm to detect specific fault types. For example, a deep learning target detection algorithm YOLOv10 is used, which can quickly and accurately identify target objects in an image. Texture analysis methods (such as gray level co-occurrence matrix and local binary pattern LBP) are used to extract texture features of images. These features are of great significance for distinguishing different materials, surface states, etc., and help improve the accuracy of fault detection. Contour extraction methods (such as Hough transform and edge tracking) are used to identify the contours of objects in an image. By comparing normal shapes with abnormal shapes, abnormal areas in the image can be detected. Use labelimg software to annotate images for faults and form labels. The label format refers to the YoLo format of the Images dataset, including target category and location information. A batch of fault-free train images are collected as a benchmark. Then, image processing software (such as Photoshop) is used to simulate faults (such as removing pipeline clamps) on the copied images, and labelme software is used to annotate the processed parts. The annotation format refers to the VOC semantic segmentation dataset, which contains images before and after processing and corresponding annotation images. The YOLOv10 algorithm is used to train with the constructed target detection dataset. After training, the model is converted to onnx format for easy deployment and inference prediction on different platforms. Change detection algorithms such as Changebind are used to train with the constructed change detection dataset. Similarly, the trained model also needs to be converted to onnx format for deployment and inference. Accurately identify and analyze edge information, texture features and abnormal shapes in images, and then effectively detect specific faults. It not only improves the accuracy and efficiency of fault detection, but also provides strong support for subsequent fault handling and maintenance.

[0114] Preferably, in step S3, the identifying and analyzing failure modes and trends and predicting the life and replacement cycle of components include:

[0115] Use knowledge graphs to perform path association analysis and explore potential associations between component process parameters and fault types;

[0116] Based on the mined association rules, a binary graph model of component failure is constructed;

[0117] Collect real-time multi-modal data to form multi-type operating status parameter data sets, and analyze the data sets;

[0118] Through the knowledge graph's path association probability search technology and the Bayesian probability learning method between nodes, the potential fault type, faulty components, and fault timing can be predicted in advance.

[0119] Construct a node association diagram based on components, fault types and system modes to match the fault location and fault signal of potential fault types.

[0120] In this technical solution, firstly, the knowledge graph is used for path association analysis, and the potential association rules between component process parameters and fault types are successfully mined. Secondly, based on the mined association rules, a binary graph model of component faults is constructed, which can intuitively display the propagation path and possible impact range of the fault, and help maintenance personnel to locate and troubleshoot the fault more accurately. In terms of data acquisition and analysis, a multi-type operating status parameter data set is formed by collecting real-time multimodal data, and the XGBoost algorithm is used to conduct in-depth analysis of the data set. For image data, convolutional neural networks can be used to extract features, and then these features are used as inputs for XGBoost. For vibration signals and temperature data, Fourier transforms can be performed to generate one-dimensional features that can be used for XGBoost. It not only improves the accuracy and efficiency of data analysis, but also provides strong data support for subsequent fault prediction. In addition, this method also realizes the early prediction of potential fault types, faulty components and fault timing through the path association probability search technology of the knowledge graph and the Bayesian probability learning method between nodes. It is of great significance for timely formulation of maintenance plans and reducing the impact of faults on train operation. Finally, the method constructs a node association graph based on components, fault types, and system modes, achieving accurate matching of the fault location and fault signal of potential fault types. The implementation of this matching process not only improves the accuracy of fault diagnosis, but also provides maintenance personnel with more specific maintenance guidance.

[0121] It is understandable that the failure mode is the specific manifestation of the failure, which is an important part of product reliability and safety analysis. By identifying and analyzing the failure mode, the performance and reliability of the product can be further improved.

[0122] It can be understood that in the use of knowledge graphs for path association analysis and mining the potential association between component process parameters and fault types, first, symbolization is performed on the knowledge in the field of train inspection, and a standardized symbol system is used to convert key information such as component name, process parameters (covering materials, dimensions, and processing methods) and fault types into computer-recognizable symbols, which not only improves the data processing efficiency, but also lays a solid foundation for the subsequent knowledge graph construction. Next, the knowledge graph is constructed using the symbolized data. By collecting and sorting the process parameters and fault type data of components, the relevant entities and their relationships are identified, and then the relationship network between entities is constructed. Using graph database technology, this knowledge graph is stored in the form of a graph structure to facilitate subsequent path association analysis. Based on the knowledge graph, path search and scoring are performed. An efficient graph traversal algorithm is used to search for all possible paths from components to fault types, and a scoring mechanism is set for each path based on the path length and the reliability factor of the relationship in the path. By screening the paths with high scores, the potential association between component process parameters and fault types is mined. Based on these association rules, a binary graph model of component faults was constructed, which intuitively shows the propagation path and possible impact range of the fault, and provides accurate fault location and troubleshooting guidance for maintenance personnel. In terms of data collection and analysis, real-time multimodal data is collected to form a multi-type operating status parameter data set. The XGBoost algorithm is used to conduct in-depth analysis of the data set, which improves the accuracy and efficiency of data analysis. In addition, the path association probability search technology of the knowledge graph and the Bayesian probability learning method between nodes are used to achieve early prediction of potential fault types, faulty components, and fault timing. Specifically, according to the relationship between components, process parameters, and fault types, the structure of the Bayesian network is generated, and a conditional probability table is set for each node. By setting evidence nodes and updating the probability distribution of all nodes in the network, the probability distribution of the target node (fault type) is queried to determine the most likely fault type. Finally, a node association graph based on components, fault types, and system modes is constructed. By accurately matching the fault location and fault signal of the potential fault type, not only the accuracy of fault diagnosis is improved, but also more specific maintenance guidance is provided for maintenance personnel.

[0123] The empirical rules are based on component appearance changes in RGB images, geometric shape changes in 3D point clouds, and frequency, amplitude, and phase changes in vibration signals.

[0124] In this technical solution, the empirical rules integrate information from multiple data sources, including changes in component appearance in RGB images, changes in geometric shapes in 3D point clouds, and changes in frequency, amplitude, and phase in vibration signals. This comprehensive empirical rule design enables the fault prediction method to more comprehensively capture potential fault signs of train components, thereby improving the accuracy and reliability of predictions. By comprehensively considering the appearance, geometric shape, and vibration characteristics of components, the abnormal state of components can be more accurately identified, providing strong support for timely maintenance measures and ensuring safe operation of trains.

[0125] Preferably, the appearance change of the component in the RGB image may be dirtiness, referring to Figure 2 ; For changes in bolt appearance, such as signs of looseness, refer to Figure 3 ; can be scratched, reference Figure 4 .

[0126] In the historical model, the life and replacement cycle of parts are predicted by monitoring the appearance wear of parts in RGB images, the dimensional changes in 3D point clouds, and the energy attenuation in vibration signals. By monitoring the appearance wear of parts in RGB images, the signs of wear on the surface of parts due to long-term use can be intuitively captured. At the same time, combined with the dimensional change analysis in the 3D point cloud, the dimensional changes of parts can be accurately measured, and then it can be determined whether there is a risk of deformation or damage. In addition, by monitoring the energy attenuation in the vibration signal, the working status and performance degradation of parts can be effectively evaluated.

[0127] In step S4, the quantitative evaluation indicators include the expansion speed of the abnormal area in the RGB image, the deformation acceleration rate in the 3D point cloud, and the energy transfer efficiency in the vibration signal to simulate the propagation process of the fault signal. By introducing the indicator of the expansion speed of the abnormal area in the RGB image, the diffusion of the fault on the surface of the component can be accurately described, providing an important basis for predicting the further development of the fault. The deformation acceleration rate in the 3D point cloud is also considered, which can reflect the rate of change of the geometric shape of the component, thereby revealing the potential impact of the fault on the component structure. Combined with the quantitative indicator of energy transfer efficiency in the vibration signal, the propagation process of the fault signal in the train system can be more comprehensively simulated.

[0128] Preferably, after S5, S6 is further included: generating an early warning signal and fault information according to the prediction results after weighted fusion, and pushing them to the management personnel. By generating an early warning signal, the management personnel can be notified in advance before the potential fault occurs, so that they have enough time to react and take corresponding preventive measures. At the same time, the detailed fault information also provides the management personnel with an important basis for judging the fault type and formulating a maintenance plan.

[0129] It is understood that the early warning signal includes the type of potential fault, location, estimated time of occurrence and recommended maintenance measures.

[0130] The specific description of the present invention in the above embodiments is only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical engineers in this field may make some non-essential improvements and adjustments to the present invention based on the contents of the above invention, which fall within the scope of protection of the present invention.

Claims

1. A fault prediction method based on a 360° dynamic image detection system, characterized in that: The following steps are involved: S1, obtain the image data of the train, and identify the abnormal points in the image according to the image processing algorithm; S2, constructing an empirical correlation model and assigning corresponding weights to different types of abnormal points according to empirical rules; the empirical rules are based on the appearance changes of parts in RGB images, the geometric shape changes in 3D point clouds, and the frequency, amplitude and phase changes in vibration signals; S3, building a historical model, identifying and analyzing failure modes and trends based on historical data of the train, and predicting the life and replacement cycle of parts; in the historical model, the life and replacement cycle of parts are predicted by monitoring the appearance wear of parts in RGB images, the size change in 3D point clouds, and the energy attenuation in vibration signals; S4, constructing a transmission model to simulate the propagation process of the fault signal, and quantitatively evaluating the impact of the fault signal on the data during the propagation process through model analysis; in step S4, the quantitative evaluation indicators include the abnormal area expansion speed in the RGB image, the deformation acceleration rate in the 3D point cloud, and the energy transfer efficiency in the vibration signal, so as to simulate the propagation process of the fault signal; S5, according to the multi-model fusion method, quantify the correlation values ​​generated by the correlation experience model, the historical model and the transfer model, assign weights to different correlation values ​​and perform weighted summation, and establish component failure correlation analysis based on multi-modal data; when collecting multi-modal data, use multiple sensors to simultaneously collect multi-source data of the train's image, depth, vibration and temperature; In step S5, the establishment of component failure correlation analysis based on multimodal data includes: Extract features and perform feature selection on the fused multimodal data; Build machine learning models to analyze the fault correlation between components; Use network analysis methods to analyze the propagation path and possibility of faults between components; Validate machine learning models using historical failure data.

2. The fault prediction method based on a 360° dynamic image detection system according to claim 1, characterized in that: The construction of the machine learning model includes: Synchronously collect image data, vibration signal data and temperature data; Process the data and perform feature concatenation to generate a multimodal data format with uniform size; According to the fault information, corresponding labels are created for the data, and the processed multimodal data and the corresponding labels are combined to generate a multimodal fault diagnosis feature extraction data set; A fault diagnosis neural network is constructed based on the multimodal data set for model training to generate a fault prediction feature extraction model.

3. The fault prediction method based on a 360° dynamic image detection system according to claim 1, characterized in that: The network analysis method comprises: The components in the train are represented as nodes in the network, and the fault propagation relationship between the components is represented as edges to construct a fault propagation network; Use the shortest path algorithm to analyze the fault propagation path and key nodes in the network; Based on the constructed fault propagation network, simulation technology is used to simulate the fault propagation process and predict the components and scope that may be affected by the fault.

4. The fault prediction method based on a 360° dynamic image detection system according to claim 1, characterized in that: In step S1, the image processing algorithm includes: Perform color correction on the continuously acquired 360° images of the train to eliminate the impact of physical environmental changes such as temperature and humidity and light source intensity attenuation on the color temperature and hue of the acquired images; Perform histogram equalization and filtering on the color-corrected image to eliminate random noise in the image; The anomaly detection deep learning network model is used to detect abnormal points in the preprocessed image, and the category information of the abnormal points, the coordinate values ​​in the image, and the length and width of the minimum circumscribed rectangle of the abnormal area are output.

5. The fault prediction method based on a 360° dynamic image detection system according to claim 4, characterized in that: In step S3, the identification and analysis of failure modes and trends and prediction of the life and replacement cycle of components include: Use knowledge graphs to perform path association analysis and explore potential associations between component process parameters and fault types; Based on the mined association rules, a binary graph model of component failure is constructed; Collect real-time multi-modal data to form multi-type operating status parameter data sets, and analyze the data sets; Through the knowledge graph's path association probability search technology and the Bayesian probability learning method between nodes, the potential fault type, faulty components, and fault timing can be predicted in advance. Construct a node association diagram based on components, fault types and system modes to match the fault location and fault signal of potential fault types.

6. A fault prediction method based on a 360° dynamic image detection system according to claim 1, 2 or 3, characterized in that: After S5, S6 is also included: generating early warning signals and fault information according to the prediction results after weighted fusion, and pushing them to management personnel.

Images