Multi-modal data fusion rail transit vehicle health state evaluation and prediction method

Abstract

The invention relates to the technical field of state prediction, in particular to a multi-modal data fusion rail transit vehicle health state evaluation and prediction method, which comprises the following steps: acquiring multi-modal data and working condition control signals at any moment in the running process of a rail transit vehicle; constructing modal features and identifying the current working condition of the vehicle; calculating the information entropy of each modal feature, and comparing the information entropy with the reference information entropy under the current working condition to obtain the information value of each modal data under the current working condition; calculating a variable coefficient of each modal data information value, and adjusting a temperature parameter of the attention mechanism according to the variable coefficient; performing weighted aggregation on each modal feature by using the temperature parameter and the information value to obtain a multi-modal fusion feature; and inputting the multi-modal fusion features at all moments into an evaluation and prediction model, and outputting a vehicle health state. According to the technical scheme, the vehicle health state can be accurately predicted.

Description

Technical Field

[0001] This application relates to the field of state prediction technology, and in particular to a method for assessing and predicting the health status of rail transit vehicles through multimodal data fusion. Background Technology

[0002] With the rapid development of urban rail transit, the long-term operational safety of rail transit vehicles is of paramount importance. In daily operation, critical components such as the running gear and traction system operate in complex and ever-changing environments, and their health status directly affects overall train safety. To achieve intelligent operation and maintenance of rail transit vehicles, multimodal sensing data, including vibration, temperature, electrical, and acoustic data, are typically collected during vehicle operation for condition monitoring. Therefore, how to fully utilize this heterogeneous data containing equipment status information to accurately predict the health status of rail transit vehicles has become an urgent problem to be solved.

[0003] Currently, the industry commonly uses single-modal data or multimodal fusion to assess vehicle health status. Typically, time-domain or frequency-domain features of each monitoring data point are extracted separately, and then these features are simply aggregated using a fixed-weight concatenation method or a static conventional attention mechanism. Finally, the fused features are input into machine learning models such as Long Short-Term Memory networks for training, thereby outputting the current health status of the vehicle or fault diagnosis results.

[0004] However, since rail transit vehicles frequently experience changes in operating conditions such as stopping at platforms and starting and accelerating during daily operation, the ability of each modal data to represent the vehicle's health status varies greatly under different operating conditions. Traditional static fusion methods ignore the impact of operating condition changes on the amount of information in multimodal data and cannot dynamically quantify and adjust the weight of each modal data during fusion according to specific operating conditions, resulting in inaccurate prediction results of vehicle health status. Summary of the Invention

[0005] To address the technical problem of inaccurate prediction results of vehicle health status, this application provides a multimodal data fusion method for assessing and predicting the health status of rail transit vehicles. This method can fully utilize the complementary information of each modality, thereby achieving accurate and stable prediction of vehicle health status.

[0006] This application provides a method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion. The method includes: collecting multimodal data and operating condition control signals at any time during the operation of the rail transit vehicle; constructing multiple modal features of the multimodal data and identifying the current operating condition of the vehicle based on the operating condition control signals; calculating the information entropy of each modal feature within the current time window and comparing the information entropy with the baseline information entropy under the current operating condition to obtain the information value of each modal data under the current operating condition, wherein the information value is positively correlated with the absolute value of the difference between the information entropy and the baseline information entropy; calculating the coefficient of variation of the information value of each modal data and adaptively adjusting the temperature parameter of the attention mechanism based on the coefficient of variation, wherein the temperature parameter is negatively correlated with the coefficient of variation; weighting and aggregating each modal feature using the temperature parameter and the information value to obtain multimodal fusion features; inputting the multimodal fusion features from multiple time points into a preset assessment and prediction model to output the vehicle health status.

[0007] Information value is defined by the degree of deviation of information entropy from the operating condition baseline, and the attention temperature parameter is adjusted by the coefficient of variation of information value. Under different operating conditions, attention can be automatically focused on the most discriminative mode, which significantly reduces the interference of operating condition changes on the health status assessment results and improves the accuracy of vehicle health status assessment.

[0008] Preferably, after collecting multimodal data and operating condition control signals at any time during the operation of the rail transit vehicle, the method further includes: filling missing values ​​in the multimodal data with different sampling frequencies using linear interpolation, wherein the multimodal data includes at least two of vibration data, temperature data, electrical data, and acoustic data; and aligning the filled modal data in time with a fixed duration as the time window.

[0009] Preferably, constructing multiple modal features of multimodal data includes: extracting time-domain and frequency-domain features of vibration signals to obtain vibration features; extracting temperature rise rate and temperature deviation features of temperature data to obtain temperature features; extracting load fluctuation and harmonic features of electrical parameters to obtain electrical features; extracting spectral envelope features of acoustic signals to obtain acoustic features; and mapping each feature to modal features of the same dimension through a preset fully connected layer.

[0010] For four types of data—vibration, temperature, electrical, and acoustic—features matching their respective faults are extracted and then mapped to the same dimension through a fully connected layer, providing a data foundation for subsequent information entropy calculation and attention fusion.

[0011] Preferably, the current operating condition is any one of platform stopping, starting acceleration, constant speed cruising, and deceleration braking.

[0012] Preferably, the calculation of the information entropy of each modal feature within the current time window includes: performing histogram statistics on each component value of the modal feature to obtain the discrete probability distribution of the feature value falling into each interval; and calculating the information entropy of each modal feature based on the discrete probability distribution.

[0013] Preferably, before obtaining the information value of each modal data under the current operating condition, the method further includes: calculating the average information entropy of each modal feature under different operating conditions in historical operating data to obtain the baseline information entropy corresponding to each operating condition; and calculating the transition frequency between different operating conditions in historical operating records to construct an operating condition transition probability matrix.

[0014] Preferably, obtaining the information value of each modal data under the current operating condition includes: responding to the current time window being in the operating condition transition period, determining the predicted operating condition based on the operating condition transition probability matrix, and weighting the baseline information entropy corresponding to the predicted operating condition with the baseline information entropy under the current operating condition to obtain a calibration baseline; and determining the information value based on the degree of deviation of the information entropy from the calibration baseline.

[0015] During the working condition transition period, the most likely next working condition is determined by the working condition transition probability matrix, and the corresponding benchmark information entropy is weighted and averaged with the current working condition benchmark information entropy to obtain the calibration benchmark, which replaces the single working condition benchmark for calculating information value. This effectively suppresses abnormal fluctuations in information value caused by benchmark jumps during sudden changes in working conditions, and ensures that attention weights remain stable during the working condition switching period, avoiding misjudgments in health status assessment results during the switching period.

[0016] Preferably, the weighted aggregation of modal features using temperature parameters and information value includes: calculating the dot product between any two modal features, adding the ratio of the dot product to the temperature parameter, and adding the bias term corresponding to the information value to obtain a cross-modal attention matrix, wherein the cross-modal attention matrix includes the attention scores between each modal feature; using the cross-modal attention matrix to perform weighted summation on each modal feature to obtain enhanced features; and concatenating and reducing the dimensionality of each enhanced feature to obtain multimodal fusion features.

[0017] The intrinsic correlation between modalities is captured by modal feature dot product, while information value bias term is superimposed, so that the fusion weight takes into account both the semantic similarity between modalities and the actual representation contribution of each modality to the current state. Furthermore, the temperature parameter adjusts the sharpness of the softmax distribution, so that attention is automatically concentrated on the dominant modality when the information value of each modality is significantly differentiated.

[0018] Preferably, the evaluation and prediction model is a long short-term memory network, and the vehicle health status is a value from 0 to 1, where 0 represents health and 1 represents malfunction.

[0019] Preferably, the training steps of the evaluation prediction model include: collecting a sequence of multimodal fusion features from multiple historical moments as training samples, using the real state of the training samples at the next moment as a label, and training the evaluation prediction model using the cross-entropy loss function until the cross-entropy loss function is less than a preset value or the number of iterations reaches the maximum number.

[0020] The supervised training method, which uses historical multi-moment multimodal fusion feature sequences as training samples and the real health status of the next moment as labels, enables the evaluation and prediction model to directly learn the mapping relationship between the fusion feature time series and the evolution of health status.

[0021] The technical solution of this application has the following beneficial technical effects: By simultaneously acquiring multimodal data in four categories—vibration, temperature, electrical, and acoustic—along with operating condition control signals, and calculating the deviation between the information entropy of the current operating condition and the baseline information entropy, the current information value of each mode is quantified. The attention temperature parameter is adaptively adjusted using the coefficient of variation of each mode's information value, driving cross-modal attention fusion to focus on the mode with the most significant state indication at any given time. This yields multimodal fusion features at any given moment, which are then input into a long short-term memory network to obtain the vehicle's health status at the next moment. This system enables dynamic allocation of cross-modal attention based on operating condition information, allowing the vehicle health status assessment results to fully utilize the complementary information of each mode, thus achieving accurate and stable health status prediction. Attached Figure Description

[0022] Figure 1 This is a flowchart of a method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion according to an embodiment of this application.

[0023] Figure 2 This is a schematic diagram of the change curves of the value of each modal data information over time according to the embodiments of this application.

[0024] Figure 3 This is a schematic diagram of a cross-modal attention matrix according to an embodiment of this application. Detailed Implementation

[0025] This application provides a multimodal data fusion method for assessing and predicting the health status of rail transit vehicles, primarily applicable to the assessment and prediction of the health status of urban rail transit vehicles during daily operation. Figure 1 As shown, the multimodal data fusion method for assessing and predicting the health status of rail transit vehicles includes steps S101 to S105, which are described in detail below.

[0026] S101 collects multimodal data and operating condition control signals at any time during the operation of rail transit vehicles.

[0027] In one embodiment, multimodal data refers to various types of sensory data reflecting the vehicle's operating state, including at least two of vibration data, temperature data, electrical data, and acoustic data; operating condition control signals refer to control quantities output in real time by the vehicle control system to characterize the vehicle's current driving operation state, including speed signals and traction or braking command signals. Multimodal data and operating condition control signals together constitute the input information for subsequent health status assessment.

[0028] Since vibration data, temperature data, electrical data, and acoustic data come from different types of sensors and their sampling frequencies differ by orders of magnitude, after the acquisition of multimodal data is completed, it is necessary to perform missing value imputation and time alignment on each modal data. This includes: using linear interpolation to imput missing values ​​in multimodal data with different sampling frequencies; and using a fixed duration as a time window to perform time alignment on each imputed modal data.

[0029] Vibration data is collected by an acceleration sensor installed at the axle box of the running gear; temperature data is collected by a temperature sensor installed at the traction motor and bearing housing; electrical data is collected by a voltage and current sensor at the traction converter; and acoustic data is collected by an acoustic sensor installed at the bottom of the carriage.

[0030] The processing methods of linear interpolation include: for a certain modal data at time... For missing values ​​at a given time, take the two nearest valid sampling points before and after it. and and their corresponding significant values and The interpolated values ​​satisfy the following relationship:

[0031] In the formula, For a moment Missing value imputation results at the location; For a moment Valid sampled values ​​at; For a moment Valid sampled values ​​at; The most recent valid sampling time before the missing point; The most recent valid sampling time after the missing point; This is the moment when the point is missing.

[0032] After missing value imputation, time alignment is performed on the modal data using a fixed duration time window. Specifically, the modal data is divided into several equal-length time windows along the time axis. Within each time window, the complete original sampling point sequence is retained for high-sampling-frequency vibration and acoustic data, while all valid sampling points within the window are retained for low-sampling-frequency temperature and electrical data. This ensures that the modal data form synchronized data segments within the same time window, eliminating time scale inconsistencies caused by different sampling frequencies. Simultaneously, median filtering or Gaussian filtering is used to denoise the modal data.

[0033] In this way, by filling in missing values ​​and aligning with a unified time window for multimodal data with different sampling frequencies, the problem of inconsistent time scales caused by hardware differences in the data of each modality is eliminated, providing a data foundation that is strictly synchronized in the time dimension for the subsequent modal feature extraction and fusion of each modality data.

[0034] S102, construct multiple modal features of multimodal data, and identify the current operating condition of the vehicle based on the operating condition control signal.

[0035] In one embodiment, modal features refer to numerical descriptive vectors extracted from various types of multimodal data that can quantitatively reflect the current state characteristics of the modal data; current operating condition refers to the specific driving operation state category of the vehicle within the current time window.

[0036] The construction of multiple modal features for multimodal data includes: extracting time-domain and frequency-domain features from vibration data to obtain vibration features; extracting temperature rise rate and temperature deviation features from temperature data to obtain temperature features; extracting load fluctuation and harmonic features from electrical data to obtain electrical features; extracting spectral envelope features from acoustic data to obtain acoustic features; and mapping each feature to modal features of the same dimension through a pre-defined fully connected layer.

[0037] For vibration data, time-domain features reflect the overall distribution characteristics of vibration amplitude, while frequency-domain features reflect the distribution pattern of vibration energy in each frequency band. The combination of the two can comprehensively describe the vibration state of the running gear within the current time window. The time-domain features of vibration data include the root mean square value and the peak factor; the frequency-domain features are obtained by performing a fast Fourier transform on the vibration data and calculating the energy proportion of each frequency band; the vibration features are composed of time-domain features and frequency-domain features.

[0038] For temperature data, the rate of temperature rise reflects how fast the temperature changes over time, while the temperature deviation characteristic reflects the degree of deviation of the current temperature from the steady-state temperature. Together, they describe the dynamic trend of temperature change. The temperature characteristic is composed of the rate of temperature rise and the temperature deviation characteristic.

[0039] For electrical data, load fluctuations reflect the amplitude of active power fluctuations within a time window, and harmonic characteristics reflect the harmonic content in the current. Together, they characterize the electrical load state of the traction system. The electrical characteristics are composed of load fluctuations and harmonic characteristics.

[0040] For acoustic data, the spectral envelope features are described using Mel frequency cepstral coefficients. Mel frequency cepstral coefficients are a compact acoustic description obtained after filtering by a Mel filter bank and discrete cosine transform. They are highly sensitive to changes in acoustic features caused by abnormal friction, collisions, etc. of mechanical parts. The acoustic features are the Mel frequency cepstral coefficient vectors.

[0041] After feature extraction from the four types of multimodal data, the resulting vibration, temperature, electrical, and acoustic features have different dimensions, making direct cross-modal computation within the same feature space impossible. Therefore, a pre-defined fully connected layer maps the four types of features to modal features of the same dimension. Preferably, the unified mapping dimension is taken as... The activation function of the fully connected layer uses linear rectified units to preserve the non-negative activation characteristics of the features and suppress invalid feature components. After mapping by the fully connected layer, the vibration data, temperature data, electrical data, and acoustic data correspond to the vibration modal features, respectively. Temperature modal characteristics Electrical modal characteristics Acoustic modal characteristics All of them A dimensional vector. The four types of modal features together constitute the multimodal feature set within the current time window, denoted as... .

[0042] The current operating condition is identified based on the operating condition control signal. The current operating condition is any one of the following: platform stop, start-up acceleration, constant speed cruise, deceleration and braking.

[0043] It should be noted that there is a clear rule-based correspondence between the four current operating conditions and the operating condition control signals. Taking the speed signal as an example... Based on traction or braking command signals, the operating condition identification rules satisfy the following relationship:

[0044] In the formula, Labels for the current operating condition category. Stop at the corresponding platform. Corresponding to startup acceleration, Corresponding to constant speed cruising, Corresponding deceleration and braking; This represents the average speed signal within the current time window, in km / h. The preferred value is the parking speed threshold. km / h, this value is based on the conventional setting of the vehicle control system for stopping judgment, and can be adjusted within the range of 2km / h to 5km / h according to the control system parameters of different models; a positive traction command indicates that the control system outputs a traction force command greater than zero, and a positive braking command indicates that the control system outputs a braking force command greater than zero.

[0045] Thus, by extracting features that match the physical properties of the four types of multimodal data and mapping them uniformly to modal features of the same dimension, a unified expression of multimodal data is achieved; the simultaneous completion of working condition identification provides a clear current working condition label for subsequent information value assessment.

[0046] S103, calculate the information entropy of each modal feature within the current time window, and compare the information entropy with the baseline information entropy under the current operating conditions to obtain the information value of each modal data under the current operating conditions.

[0047] In one embodiment, information entropy refers to an index that measures the degree of uncertainty in the numerical distribution of each component of a modal feature. The higher the information entropy, the richer the numerical fluctuation of the modal data within the current time window, and the greater the amount of information it carries. The baseline information entropy refers to the statistical mean of the information entropy of a modal feature under a specific operating condition in historical normal operation data, reflecting the baseline level of information content when the modality is in a normal state under that operating condition. Information value refers to the degree of deviation of the current information entropy of a modal feature from its baseline information entropy under the current operating condition. The higher the information value, the more the current data fluctuation of the modality deviates from the normal level, and the more likely it is to reflect the real state changes of vehicle components. The information value is positively correlated with the absolute value of the difference between the information entropy and the baseline information entropy.

[0048] After obtaining the modal features and the current operating condition, the relative importance of each modal data under the current operating condition cannot be determined solely by the modal features themselves. If the same fusion weight is assigned to each modal data without distinction, the health status assessment results will be dominated by operating condition interference. Therefore, before calculating the information value, it is necessary to pre-construct the baseline information entropy of each modal feature under different operating conditions based on historical operating data, as well as the operating condition transition probability matrix describing the transition patterns between operating conditions.

[0049] Specifically, the average information entropy of each modal feature under different operating conditions in the historical operation data is statistically analyzed to obtain the baseline information entropy corresponding to each operating condition; the transition frequency between different operating conditions in the historical operation records is statistically analyzed to construct the operating condition transition probability matrix.

[0050] From historical normal operation data, time window samples are selected according to operating condition categories. The information entropy of each modal feature is calculated for each sample. Then, the arithmetic mean of all information entropy samples of the same modality under the same operating condition is taken to obtain the baseline information entropy. ,in For modal indexing, This is a label for the operating condition category.

[0051] Traverse the sequence of operating condition categories for adjacent time windows in the historical operation records and count the categories from the operating conditions. Transfer to working condition Frequency of occurrence The working condition transition probabilities satisfy the following relationship:

[0052] In the formula, To be based on working conditions Transfer to working condition The probability of; For historical records from working conditions Transfer to working condition Frequency; This represents the total number of operating condition categories in this embodiment. ; denominator To be based on working conditions The total frequency of transitions to any working condition ensures that the sum of the probabilities of each row is 1, thus forming... The working condition transition probability matrix It should be noted that when the denominator is 0 (i.e., this condition has not occurred historically), Set to 0.

[0053] After obtaining the baseline information entropy and the working condition transition probability matrix, the information entropy of each modal feature within the current time window is calculated by: performing histogram statistics on each component value of the modal feature to obtain the discrete probability distribution of the feature value falling into each interval; and calculating the information entropy of each modal feature based on the discrete probability distribution.

[0054] Among them, for the first Modal features The specific method for histogram statistics is as follows: of The numerical range of each component is evenly divided into The interval is calculated, and the component values ​​fall within the interval. The number of intervals accounts for a certain percentage of the total number of components. The proportion is used to obtain the discrete probability distribution. ,in The information entropy is calculated based on the discrete probability distribution. The information entropy of each modality feature satisfies the following relationship:

[0055] In the formula, For the first The information entropy of each modality feature within the current time window; Modal features The values ​​of each component fall into the first The probability of each interval; The number of histogram intervals is set to [number] in this embodiment. ;when At that time, it was agreed This is to ensure the numerical stability of the formula in the empty interval.

[0056] Understandably, the range of values ​​for information entropy is... When all component values ​​of a modal feature fall within the same interval, the information entropy reaches its minimum value of 0, indicating that the modal feature carries almost no distribution information within the current window; when the component values ​​are evenly distributed across all intervals, the information entropy reaches its maximum value. This indicates that the modal data exhibits rich fluctuations.

[0057] Furthermore, by comparing the information entropy with the baseline information entropy under the current operating conditions, the influence of the operating conditions themselves on the amount of information is eliminated, thus obtaining the information value that truly reflects changes in health status.

[0058] Under current operating conditions During the non-operating condition transition period, the first Modal data under current operating conditions The information value below Satisfying the relation:

[0059] In the formula, The number calculated within the current time window Information entropy of each modal feature; For the first Each modal feature in working conditions The baseline information entropy is used to normalize the deviation, making the information value between different modes comparable. To prevent the default positive value of denominator 0, the value is set to 0.1.

[0060] Understandably, when the current information entropy of a certain modality... Its baseline information entropy under current operating conditions The greater the deviation, whether it is too high or too low, the more likely it is to reflect the true state changes of the vehicle components, and thus it is given higher information value.

[0061] However, during the transition period when operating conditions change, the baseline information entropy of the current operating condition is insufficient to accurately characterize the evolving data distribution. Directly using the baseline information entropy of a single operating condition as a comparison benchmark can lead to drastic fluctuations in the information value index, thereby affecting the stability of subsequent attention weights. Therefore, in response to the current time window being in the operating condition transition period, a predicted operating condition is determined based on the operating condition transition probability matrix, and the baseline information entropy corresponding to the predicted operating condition is weighted and averaged with the baseline information entropy under the current operating condition to obtain a calibration benchmark; the information value is determined based on the degree of deviation of the information entropy from the calibration benchmark.

[0062] It should be noted that the current operating conditions within the current time window... If the operating conditions are inconsistent with those of the previous time window, the current time window is determined to be in a condition transition period. After confirming that it is in a condition transition period, the operating condition transition probability matrix is ​​used to determine the next condition transition period. The transition target condition with the highest probability in the row is taken as the predicted condition, and the predicted condition satisfies the following relationship:

[0063] In the formula, To predict the category of working conditions; The working condition transition probability matrix is ​​the one from the current working condition. Transfer to working condition The probability of; Indicates taking Largest operating condition category .

[0064] After obtaining the predicted working conditions Then, the baseline information entropy corresponding to the predicted working condition will be determined. Baseline information entropy corresponding to the current operating condition A weighted average is performed to obtain the calibration reference for the transition period of the operating condition. The calibration reference satisfies the following relationship:

[0065] In the formula, During the transition period of the operating condition Calibration benchmarks for each modal feature; For the current working condition The baseline information entropy; To predict working conditions The baseline information entropy; To preset transition weights, reflecting the degree of evolution from the current time window to the next working condition, the preferred weights are... That is, the baseline information entropy of the current working condition and the predicted working condition are assigned equal weights.

[0066] During the transition period of operating conditions, the first The information value of each modal data point is calculated by substituting the calibration benchmark into the benchmark information entropy and using the formula to satisfy the following relationship:

[0067] In the formula, During the transition period of the operating condition The informational value of modal data; The number calculated within the current time window Information entropy of each modal feature; It serves as a calibration benchmark during the transition period of operating conditions, used to normalize deviations and ensure the cross-modal comparability of information value.

[0068] Taking the train's transition from constant speed cruise to deceleration braking as an example: At the moment of transition, the braking current begins to rise, and the information entropy of the electrical mode characteristics increases rapidly. If the baseline information entropy of the constant speed cruise condition is still used as the comparison benchmark, the value of the electrical mode information will be artificially inflated. By introducing the baseline information entropy of the deceleration braking condition for weighted averaging, a smooth transition to the deceleration braking benchmark reference can be achieved, avoiding abnormal jumps in information value indicators from interfering with subsequent attention weight adjustments, and improving the stability of the health status assessment results during the condition transition period. Please see [link to relevant documentation]. Figure 2 This is a schematic diagram of the change curve of the information value of each modal data over time according to the embodiments of this application. The information value of each modal data is different at each moment.

[0069] S104, calculate the coefficient of variation of the information value of each modality, and adaptively adjust the temperature parameter of the attention mechanism based on the coefficient of variation; use the temperature parameter and information value to weight and aggregate the features of each modality to obtain multimodal fusion features.

[0070] In one embodiment, after obtaining the information value of each modality under the current operating condition, the degree of difference between the information values ​​of each modality also carries important information. When the information values ​​of each modality are highly consistent, it indicates that the data of each modality tends to reflect the current state in a balanced way, and the attention mechanism should maintain a relatively even distribution among the modalities. When the information values ​​of each modality differ greatly, it indicates that there is a clear dominant modality, and the attention mechanism should focus on the modality with high information value. Therefore, before performing cross-modal attention fusion, it is necessary to quantify this degree of difference through the coefficient of variation and use this to adaptively adjust the temperature parameter, and then use the temperature parameter and information value to perform weighted aggregation of the features of each modality.

[0071] The weighted aggregation of modal features using temperature parameters and information value includes: calculating the dot product between any two modal features, adding the ratio of the dot product to the temperature parameter, and adding the bias term corresponding to the information value to obtain a cross-modal attention matrix, which includes the attention scores between each modal feature; using the cross-modal attention matrix to perform weighted summation of each modal feature to obtain enhanced features; and concatenating and dimensionality-reducing each enhanced feature to obtain multimodal fusion features.

[0072] Among them, the coefficient of variation of the information value of each modality data is obtained by analyzing the data. The information value set of each modality is obtained through statistical calculation. Indicates the first Modal data under current operating conditions The information value under the current working conditions Coefficient of variation of the information value of each modality data Satisfying the relation:

[0073] In the formula, This is the mean value of all modal information. This represents the standard deviation of the value of all modal information. The coefficient of variation essentially measures the degree of differentiation in the importance pattern of each modal data under the current operating conditions.

[0074] Temperature parameter in attention calculation based on adaptive adjustment of the coefficient of variation; temperature parameter Satisfying the relation:

[0075] In the formula, As the reference temperature parameter, the preferred value is... ; To preset the sensitivity, the preferred value is... It is used to control the degree of influence of the coefficient of variation on temperature parameters.

[0076] Understandably, when As the temperature parameter decreases, the distribution of cross-modal attention scores after softmax normalization becomes more concentrated. The attention score obtained by the high-information-value mode is significantly higher than that of the low-information-value mode, achieving automatic focusing of attention towards the dominant mode. As it approaches zero, the temperature parameter approaches The softmax output tends to be evenly distributed, and each modality receives nearly equal attention weights.

[0077] After obtaining the adaptive temperature parameters, they are used in conjunction with the modal information values ​​to calculate the cross-modal attention matrix. In the... Modal features For the Modal features In the attention score, the first The modal feature is the query for the first modality feature. Attention score for each modality feature Satisfying the relation:

[0078] In the formula, The first layer after mapping by the fully connected layer Modal features; For the first Modal features, For the first Transpose of a modal feature; The dot product of the two modal features reflects the degree of intrinsic similarity between them; For adaptive temperature parameters; For the first Modal data under current operating conditions The information value below; The information value bias coefficient is preferred. ; For the traversal index of normalized summation, Indicates the first Modal data under current operating conditions The information value below For the first Modal features, For the first Transpose of a modal feature For the first The modal features and the first The dot product of modal features. For all Attention scores are calculated for each modality pair, forming a structure as follows: Figure 3 shown The cross-modal attention matrix, the matrix of the first... Line number The elements of the column are .

[0079] Understandably, the information value bias term, on top of the dot product similarity, additionally assigns an attention increment to the high information value modality, ensuring that when the dot product similarity is similar, the modality with higher information value can obtain more fusion weights.

[0080] Based on the cross-modal attention matrix, the features of each modality are weighted and summed to obtain the enhanced features of each modality. Taking the first... Each modal feature is a query, and its enhanced features satisfy the following relation:

[0081] In the formula, For the first The enhanced features are obtained by weighting each modal feature with cross-modal attention, and the dimension is [dimension value missing]. Consistent with the modal feature dimension; The first in the cross-modal attention matrix Line number The column's attention score; For the first Modal features. For Perform the above weighted summation operation on each modality to obtain... One enhanced feature Each enhancement feature incorporates information from other modalities, achieving cross-modal information fusion.

[0082] Will The enhanced features are concatenated along the feature dimension to obtain a dimension of The concatenated vectors are then processed by a fully connected layer for dimensionality reduction, mapping the concatenated vectors to a dimension of 1. Multimodal fusion features.

[0083] Thus, by calculating the coefficient of variation of the information value of each modality and adaptively adjusting the temperature parameter accordingly, the attention mechanism can automatically focus on the dominant modality when the information value of each modality differs significantly, and maintain the fairness of fusion when the information value of each modality is balanced. By introducing the information value bias term and the dot product similarity to jointly determine the cross-modal attention matrix, the fusion weight can take into account both the intrinsic correlation between modalities and the relative importance of each modality under the current working condition.

[0084] S105 inputs the multimodal fusion features from multiple time points into a preset evaluation and prediction model and outputs the vehicle health status.

[0085] In one embodiment, the evaluation prediction model refers to a long short-term memory network that takes a fusion feature sequence composed of multimodal fusion features at multiple times as input and outputs the vehicle health status at the next time step. The vehicle health status refers to the overall operating condition of the vehicle represented by numerical indicators, with values ​​ranging from 0 to 1, where 0 represents health, 1 represents fault, and intermediate values ​​reflect different degrees of degradation.

[0086] The multimodal fusion features from multiple time points are arranged in chronological order to obtain a fusion feature sequence. This fusion feature sequence is then input into a pre-defined evaluation and prediction model to output the health status at the next time point. The number of time points can be set to 10.

[0087] The evaluation and prediction model uses a long short-term memory network. The long short-term memory network selectively retains or forgets historical time-series information through three gating mechanisms: forget gate, input gate, and output gate. It can effectively capture long-term time-series dependencies in the process of vehicle health degradation.

[0088] The training steps of the evaluation and prediction model include: collecting a sequence of multimodal fusion features from multiple historical time points as training samples; using the real state of the training samples at the next time point as the label; and training the evaluation and prediction model using the cross-entropy loss function until the cross-entropy loss function is less than a preset value or the number of iterations reaches the maximum number. The preset value is 0.01, and the maximum number of iterations is 100.

[0089] Thus, by constructing a fused feature sequence from the multimodal fusion features at multiple time points and inputting it into the evaluation and prediction model, accurate prediction of the vehicle's health status at the next time point is achieved.

[0090] It should be noted that the scope of protection of this patent application shall be determined by the appended claims.

Claims

1. A method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion, characterized in that, The method includes: Collect multimodal data and operating condition control signals at any time during the operation of rail transit vehicles; Construct multiple modal features from multimodal data, and identify the current operating condition of the vehicle based on the operating condition control signal; Calculate the information entropy of each modal feature within the current time window, and compare the information entropy with the baseline information entropy under the current operating conditions to obtain the information value of each modal data under the current operating conditions. The information value is positively correlated with the absolute value of the difference between the information entropy and the baseline information entropy. The coefficient of variation of the data information value of each modality is calculated, and the temperature parameter of the attention mechanism is adaptively adjusted according to the coefficient of variation, wherein the temperature parameter is negatively correlated with the coefficient of variation; The multimodal features are weighted and aggregated using temperature parameters and information value to obtain multimodal fusion features. The multimodal fusion features at multiple time points are input into a preset evaluation and prediction model to output the vehicle health status.

2. The method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion according to claim 1, characterized in that, After collecting multimodal data and operating condition control signals at any time during the operation of the rail transit vehicle, the method further includes: Missing values ​​in multimodal data with different sampling frequencies are filled using linear interpolation. The multimodal data includes at least two of the following: vibration data, temperature data, electrical data, and acoustic data. Using a fixed duration as the time window, time alignment is performed on the filled modal data.

3. The method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion according to claim 2, characterized in that, The construction of multiple modal features for multimodal data includes: The vibration characteristics are obtained by extracting the time-domain and frequency-domain features of the vibration signal. Extract the temperature rise rate and temperature deviation characteristics from the temperature data to obtain the temperature characteristics; The load fluctuation and harmonic characteristics of electrical parameters are extracted to obtain the electrical characteristics; The spectral envelope features of the acoustic signal are extracted to obtain the acoustic features; Each feature is mapped to a modal feature of the same dimension through a pre-defined fully connected layer.

4. The method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion according to claim 1, characterized in that, The current operating condition is any one of the following: platform stop, start-up acceleration, constant speed cruise, deceleration and braking.

5. The method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion according to claim 1, characterized in that, The calculation of the information entropy of each modal feature within the current time window includes: performing histogram statistics on each component value of the modal feature to obtain the discrete probability distribution of the feature value falling into each interval; and calculating the information entropy of each modal feature based on the discrete probability distribution.

6. The method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion according to claim 1, characterized in that, Before obtaining the information value of each modal data under the current operating condition, the method further includes: calculating the average information entropy of each modal feature under different operating conditions in historical operating data to obtain the baseline information entropy corresponding to each operating condition; and calculating the transition frequency between different operating conditions in historical operating records to construct an operating condition transition probability matrix.

7. The method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion according to claim 6, characterized in that, The information value of obtaining modal data under the current operating conditions includes: In response to the current time window being in the working condition transition period, the predicted working condition is determined according to the working condition transition probability matrix, and the baseline information entropy corresponding to the predicted working condition is weighted and averaged with the baseline information entropy under the current working condition to obtain the calibration baseline. The information value is determined based on the degree of deviation of the information entropy from the calibration benchmark.

8. The method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion according to claim 1, characterized in that, The weighted aggregation of modal features using temperature parameters and information value includes: Calculate the dot product between any two modal features, add the ratio of the dot product to the temperature parameter, and add the bias term corresponding to the information value to obtain the cross-modal attention matrix, which includes the attention scores between each modal feature; The enhanced features are obtained by weighted summation of the features across modalities using the cross-modal attention matrix. The enhanced features are spliced ​​and dimensionality reduced to obtain multimodal fusion features.

9. The method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion according to claim 1, characterized in that, The evaluation and prediction model is a long short-term memory network, and the vehicle health status is a value from 0 to 1, where 0 represents health and 1 represents malfunction.

10. The method for assessing and predicting the health status of rail transit vehicles based on multimodal data fusion according to claim 9, characterized in that, The training steps of the evaluation and prediction model include: collecting a sequence of multimodal fusion features from multiple historical moments as training samples, using the real state of the training samples at the next moment as a label, and training the evaluation and prediction model using the cross-entropy loss function until the cross-entropy loss function is less than a preset value or the number of iterations reaches the maximum number.

Images