Railway connector workload evaluation system based on multi-modal data analysis

Abstract

The invention relates to the field of railway operation safety monitoring, and discloses a railway connector workload evaluation system based on multi-modal data analysis, which comprises a multi-modal data acquisition module, a data processing module, a modal fusion module, a task optimization module, a time modeling module and an evaluation module, by collecting physiological, behavior and environmental data of railway linkers, performing time alignment and feature extraction, performing fusion to generate joint representation, and performing task correlation optimization and time dynamic modeling on the joint representation, the system can realize real-time classification and evaluation of workload states, the classification results comprise normal, early warning and overload, and the system has the advantages of high efficiency, high reliability and high reliability. When an overload condition is detected, an alarm signal is triggered. The method overcomes the problems of insufficient single-modal analysis, low multi-modal fusion quality, lack of time dynamic modeling and the like in the prior art, has the characteristics of high precision, high real-time performance and high reliability, and provides technical guarantee for safety management of railway operation.

Description

Technical Field

[0001] The present invention relates to the technical field of railway operation safety monitoring, and in particular to a railway coupler workload evaluation system based on multimodal data analysis. Background Art

[0002] Railway couplers undertake high-intensity, high-risk tasks in complex operating environments, and their workload directly impacts the safety and efficiency of railway operations. With the development of the railway transportation industry, effectively monitoring coupler workloads and providing real-time warnings has become a critical technical issue for ensuring railway operational safety and optimizing management.

[0003] Existing workload evaluation technologies are mainly based on single-modal data or multi-modal data analysis. However, single-modal data methods usually rely on only one type of physiological data, behavioral data, or environmental data for load evaluation. For example, physiological load is evaluated through heart rate variability (HRV), behavioral load is analyzed through movement trajectory and behavioral amplitude, or the impact of the external environment on the coupler is judged through environmental parameters such as noise level. Although such methods can reflect one aspect of the workload to a certain extent, due to ignoring the synergy and comprehensiveness between multi-modal data, their evaluation results often lack comprehensiveness and accuracy, making it difficult to meet the monitoring needs of complex load conditions in actual railway operation scenarios.

[0004] To improve the comprehensiveness of evaluation, multimodal data analysis techniques are increasingly being applied to workload assessment, combining physiological, behavioral, and environmental data to obtain richer information. However, existing multimodal data fusion methods often rely on simple concatenation, weighted averaging, or linear combinations. These methods are significantly inadequate for addressing inter-modal heterogeneity (such as the sampling frequency, feature distribution, and expression space of different modalities) and are unable to effectively model complex nonlinear relationships between modalities. Furthermore, the high level of redundant information between modalities and the insufficient representation of key features result in low-quality feature representations in the fusion results, further limiting the performance of classification models.

[0005] On the other hand, most existing workload evaluation technologies use static classification models, predicting load status based solely on feature inputs at a single point in time. This fails to fully account for the dynamic nature of workload changes over time. This makes the classification results susceptible to interference from short-term fluctuations or data acquisition noise, affecting the stability of the evaluation. Furthermore, lacking the support of time series modeling, existing technologies struggle to implement trend analysis and early warning functions for load status, particularly in the early detection of overloaded railway couplers.

[0006] In practical applications, workload evaluation systems must also possess high-precision classification capabilities to distinguish between different workload states (such as "normal," "warning," and "overload"). However, existing classification methods typically fail to optimize the task-relevance of multimodal data features, resulting in poor adaptability of the classification models, particularly in the identification of minority states (such as "overload"). Furthermore, existing technologies often lack comprehensive analysis and trend prediction of overall workload states, and fail to implement real-time alarms and job management support, limiting the practicality of the systems.

[0007] Therefore, the present invention proposes a railway coupler workload evaluation system based on multimodal data analysis to address the shortcomings of the existing technology. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, the present invention provides a railway coupler workload evaluation system and method based on multimodal data analysis to achieve a comprehensive and accurate evaluation of the coupler's workload status, overcoming the problems existing in the existing methods such as insufficient single modal data analysis, low fusion quality, lack of dynamic characteristic modeling, and limited classification performance.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: A railway coupler workload evaluation system based on multimodal data analysis, comprising:

[0010] Data acquisition module, used to obtain multimodal data of railway couplers;

[0011] A data processing module, connected to the data acquisition module, for processing multimodal data;

[0012] A modality fusion module, connected to the data processing module, for fusing the processed multimodal data to generate a joint representation;

[0013] a task optimization module, connected to the modality fusion module, for optimizing the task relevance of the joint representation;

[0014] A temporal modeling module, connected to the modality fusion module, for optimizing the temporal dynamic characteristics of the joint representation;

[0015] An evaluation module is connected to the task optimization module and the time modeling module, and is used to evaluate the workload status of the railway coupler based on the joint representation.

[0016] Preferably, the data acquisition module is used to obtain multimodal data of railway couplers, and the multimodal data includes physiological data, behavioral data and environmental data. The physiological data includes heart rate, blood oxygen, and skin conductance; the behavioral data includes trajectory and movement amplitude; and the environmental data includes temperature, humidity, noise and vibration.

[0017] Preferably, the data processing module includes:

[0018] A time alignment unit, used to synchronize the time series of multimodal data;

[0019] The feature extraction unit is used to extract the features of each modal data.

[0020] Preferably, the modal fusion module generates a joint representation of multimodal data through a deep neural network, and the joint representation is generated based on the optimization of effective information of the multimodal data, and improves the representation quality by weakening redundant information between modalities.

[0021] Preferably, the task optimization module optimizes the support capability of the joint representation for the workload classification task by minimizing the classification loss, wherein the classification task classifies the workload status into normal, warning and overload status.

[0022] Preferably, the time modeling module applies regularization constraints to the time series of the joint representation to optimize the smooth change of the joint representation in the time dimension, so that the workload evaluation has dynamic continuity.

[0023] Preferably, the evaluation module classifies the workload status of the railway coupler based on the optimized joint representation, and triggers an alarm signal when the classification result is an overload state.

[0024] Preferably, the joint representation generated by the modality fusion module maximizes the relevance with each modality data, minimizes the interference of redundant information between modalities, and improves the classification ability of the representation by strengthening task-related information.

[0025] Preferably, a railway coupler workload evaluation method based on multimodal data analysis comprises the following steps:

[0026] Collect multimodal data of railway couplers;

[0027] Processing the multimodal data, including time alignment and feature extraction;

[0028] Generate joint representations of multimodal data through modal fusion;

[0029] Optimizing task relevance of joint representations;

[0030] Optimizing the temporal dynamics of the joint representation;

[0031] The workload status of railway couplers is classified and evaluated based on the optimized joint representation.

[0032] Preferably, a storage medium stores a computer program thereon, and the method described when the computer program is executed by a processor.

[0033] The present invention provides a railway coupler workload evaluation system based on multimodal data analysis.

[0034] It has the following beneficial effects:

[0035] 1. This invention utilizes a modal fusion module based on a deep neural network to integrate multimodal data from physiological, behavioral, and environmental conditions. This technology achieves the technical effect of generating a high-quality joint representation while reducing redundant information between modalities. Compared to existing solutions that rely on a single modality or simply concatenate multimodal data, this approach addresses the issues of insufficient modeling of multimodal data characteristics, high information redundancy, and difficulty in fully reflecting workload conditions.

[0036] 2. This invention utilizes a temporal modeling module to regularize the temporal consistency of the joint representation and perform time series modeling. This achieves the technical effect of optimizing the temporal smoothness of the joint representation and improving the stability of classification results. Compared to existing solutions that ignore dynamic workload changes or rely on static classification models, this solves the problem of insufficient modeling of workload changes in the temporal dimension and significant short-term fluctuations in prediction results.

[0037] 3. This invention utilizes a task optimization module to enhance the task relevance of the joint representation through supervised learning, significantly improving workload state classification accuracy and minority class recognition. Compared to existing solutions that fail to optimize feature representations for specific tasks or whose classification models are insensitive to minority class states (e.g., overload), this approach addresses the issues of low classification accuracy and insufficient model generalization.

[0038] 4. This invention utilizes an evaluation module that combines multimodal data characteristics with temporal modeling results to perform real-time workload classification and alarm triggering. This achieves the technical benefits of accurately classifying connector workloads and providing overload warnings. Compared to existing solutions that only perform simple load analysis or lack warning mechanisms, this solution addresses the limited practicality of classification results and insufficient risk prevention and control capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is the architecture diagram of the railway coupler workload evaluation system based on multimodal data analysis;

[0040] Figure 2This is a flow chart of the railway coupler workload evaluation method based on multimodal data analysis. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] Please see the attached Figure 1 An embodiment of the present invention provides a railway coupler workload evaluation system based on multimodal data analysis. The following describes in detail the various modules of the system of the present invention.

[0043] Data acquisition module, used to obtain multimodal data of railway couplers

[0044] This invention relates to a railway coupler workload assessment system based on multimodal data analysis. The data acquisition module is the foundation of the entire system, capturing multimodal data generated by railway couplers during their work, including physiological, behavioral, and environmental data. By synchronously collecting and initially processing multimodal data, the data acquisition module provides a reliable data source and support for subsequent data processing, modal fusion, and task optimization.

[0045] Railway couplers face complex workloads, involving high-intensity physical activity and diverse operating environments. These factors place high demands on the accuracy and real-time nature of data collection. Therefore, the data acquisition module of this invention utilizes a variety of devices and techniques to comprehensively capture physiological, behavioral, and environmental data, while ensuring synchronization and reliability.

[0046] In this embodiment, the data acquisition module includes the following parts:

[0047] Collection of physiological data

[0048] In one possible implementation, physiological data collection is primarily accomplished through wearable sensing devices. Specifically, wearable devices may include heart rate monitors, smart wristbands, or multi-lead electrode patches, which are used to monitor the railway coupler's physiological signals in real time.

[0049] Heart rate data is collected using photoplethysmography (PPG) or electrocardiogram (ECG) sensors to obtain heart rate HR and its variability (HRV). Key features of physiological signals are extracted through analysis in the time and frequency domains. For example:

[0050] Time domain indicators include average RR interval, standard deviation SDNN (standard deviation of RR interval), etc.

[0051] Frequency domain indicators include low-frequency power (LF), high-frequency power (HF) and their ratio (LF / HF).

[0052] As an option, blood oxygen saturation (SpO2) is collected by a near-infrared optical sensor and preliminarily calculated using the following formula:

[0053]

[0054] HbO2 represents the concentration of oxygenated hemoglobin, and Hb represents the concentration of hemoglobin that is not bound to oxygen.

[0055] Specifically, skin conductance data is collected through a galvanic skin response sensor (GSR) to reflect the stress state of railway couplers. The dynamic changes in skin conductance can be calculated using the following formula:

[0056] ΔGSR=G(t)-G(t0)

[0057] Where G(t) represents the skin conductance value at the current time point, and G(t0) represents the baseline skin conductance value.

[0058] Collection of behavioral data

[0059] In another possible implementation, the behavior data is acquired through a motion capture device, specifically, an inertial measurement unit (IMU) sensor or a camera.

[0060] The inertial measurement unit, comprised of a three-axis accelerometer and gyroscope, records the railroad coupler's trajectory and posture in real time. Trajectory data is calculated from the positional changes of the acquisition points, while the amplitude of movement is determined by the rate of change of joint angles.

[0061] Specifically, the trajectory curvature K can be calculated using the following formula:

[0062]

[0063] Among them, x ′ and y ′ represents the first-order derivative of the trajectory point in time, x ″ and y ″ Represents the second-order derivative of a trajectory point. Alternatively, the camera can be used with computer vision algorithms (such as OpenPose) to identify the railway coupler's posture and behavioral patterns. Keypoint detection and motion vector analysis can be used to extract behavioral characteristics such as movement complexity.

[0064] Collection of environmental data

[0065] In a possible implementation, environmental data is collected by environmental sensors installed in the work area.

[0066] Environmental data includes temperature and humidity.

[0067] External condition parameters such as noise level and vibration.

[0068] Specifically, temperature and humidity data are collected by thermistors and capacitive humidity sensors, and noise data is collected by a sound pressure meter and the sound pressure level (SPL) is recorded. The calculation formula is:

[0069]

[0070] Where P is the measured sound pressure value, and P0 is the reference sound pressure value (usually 20 μPa).

[0071] Vibration data is collected through intensity sensors to record the vibration intensity of the operating equipment.

[0072] By analyzing the time domain and frequency domain characteristics of the vibration signal, its amplitude and frequency characteristics are obtained.

[0073] Synchronous acquisition of multimodal data

[0074] In the technical implementation of the present invention, the data acquisition module also includes a data synchronization unit for achieving time synchronization of multimodal data. Specifically, a timestamp method is used to assign a unified time identifier to each sampling point. Alternatively, a dynamic time warping (DTW) algorithm can be used to time-align asynchronous data from different modalities, ensuring comparability of physiological, behavioral, and environmental data along the same time dimension.

[0075] For example, to align heart rate data with trajectory data, the DTW algorithm can be used to calculate the optimal matching path of the two signals and adjust the data sampling interval to achieve time synchronization.

[0076] Implementation characteristics of data acquisition module

[0077] In some embodiments, the data acquisition module further includes a pre-processing unit for preliminary processing of the acquired signal, including denoising, smoothing, and normalization. For example:

[0078] Apply bandpass filtering to the physiological signal to remove high- and low-frequency noise;

[0079] Eliminate or interpolate abnormal points in behavioral data;

[0080] The environmental data were normalized to eliminate dimensional differences.

[0081] Through this technological implementation, the data acquisition module is able to comprehensively and reliably collect multimodal data from railway couplers, providing high-quality input for subsequent data processing and workload evaluation. The functions and parameters of each component have been optimized, ensuring that the data acquisition module not only has high-precision signal acquisition capabilities but also ensures temporal consistency between different modalities, thus providing a solid foundation for the entire workload evaluation system.

[0082] The data processing module is connected to the data acquisition module and is used to process the multimodal data.

[0083] In this paper, the data acquisition module is a key component of the railway coupler workload evaluation system. This module is responsible for collecting multimodal data generated during the coupler's work, including physiological, behavioral, and environmental data, and provides a fundamental data source for subsequent data processing, modal fusion, and task optimization. Because railway couplers' work scenarios are complex and varied, and their workload is high, the data acquisition module must possess high-precision, multi-dimensional, and real-time synchronous data acquisition capabilities to ensure the accuracy of subsequent processing and evaluation.

[0084] Railway couplers' work involves physiological stress (such as heart rate and blood oxygen levels), behavioral stress (such as movement amplitude and trajectory), and environmental stress (such as noise and vibration). Therefore, the data acquisition module needs to use different types of sensors and acquisition equipment to comprehensively and real-timely acquire multimodal data and ensure time synchronization between the modal data.

[0085] In this embodiment, the data acquisition module includes the following contents:

[0086] In one possible implementation, physiological data is collected through wearable sensor devices. These devices may include heart rate belts, smart bracelets, or multi-lead electrode patches to monitor physiological signals such as the connector's heart rate, blood oxygen saturation, and skin conductance.

[0087] Specifically, heart rate data is collected using photoplethysmography (PPG) or electrocardiogram (ECG) sensors. By analyzing the heart rate signal in the time and frequency domains, the following features can be extracted:

[0088] Time domain characteristics: including the mean of RR intervals and standard deviation (SDNN).

[0089] Frequency domain characteristics: including low-frequency power (LF), high-frequency power (HF) and their ratio (LF / HF).

[0090] As an option, blood oxygen saturation (SpO2) is collected by a near-infrared optical sensor. A preliminary calculation of blood oxygen saturation can be achieved using the following formula:

[0091]

[0092] Here, HbO2 represents the concentration of oxyhemoglobin, and Hb represents the concentration of deoxyhemoglobin. In another possible implementation, skin conductance (GSR) is collected via an electro-responsive sensor, primarily reflecting the stress and fatigue status of the connector. The dynamic changes in skin conductance can be expressed by the following formula:

[0093] ΔGSR=G(t)-G(t0)

[0094] Where G(t) represents the skin conductance value at the current time point, and G(t0) represents the baseline skin conductance value.

[0095] Collection of behavioral data

[0096] Behavioral data is acquired through an inertial measurement unit (IMU) or a camera. The IMU includes an accelerometer and a gyroscope, which are used to collect the movement trajectory and posture information of the linker.

[0097] Specifically, the trajectory data is obtained by calculating the position change of the sampling point. The calculation formula of the trajectory curvature K is as follows:

[0098]

[0099] Among them, x ′ and y ′ They represent the first-order derivative of the trajectory point in time, x ″ and y ″ They represent the second-order derivatives of the trajectory points in time.

[0100] As an option, behavioral data can also be obtained through the use of cameras and computer vision algorithms (such as OpenPose).

[0101] The connector's working posture is analyzed through key point detection and motion vector analysis to extract features such as movement complexity and behavior amplitude.

[0102] In another possible implementation, the behavioral data may also include the frequency and duration of the connector's movements. These features can be used to extract frequency components through Fourier transform of the acceleration signal, thereby identifying high-intensity behavioral patterns.

[0103] Collection of environmental data

[0104] In one possible implementation, environmental data is collected through environmental sensors, mainly including external operating conditions such as temperature and humidity, noise level and vibration.

[0105] Specifically, temperature and humidity data are collected using a thermistor and a capacitive humidity sensor. Noise data is collected using a sound pressure meter, and the sound pressure level (SPL) is calculated using the following formula:

[0106]

[0107] Where P is the measured sound pressure value, and P0 is the reference sound pressure value (usually 20 μPa).

[0108] Vibration data is collected by scaling sensors and used to monitor the vibration intensity of railway operating equipment. The vibration characteristics can be obtained by analyzing the mean and variance of the time-sensing signal and the frequency distribution of the frequency domain signal.

[0109] Synchronous acquisition of multimodal data

[0110] The data acquisition module needs to time synchronize the multimodal data to ensure that physiological, behavioral, and environmental data are aligned on the same timeline.

[0111] In one possible implementation, the data acquisition module uses a unified timestamp recording method to mark each modality's data. Alternatively, a dynamic time warping (DTW) algorithm can be used to time-align the data from different modalities. DTW calculates the optimal matching path between time series to synchronize sampling points.

[0112] For example, when there is a sampling frequency difference between the heart rate signal and the trajectory signal, the DTW algorithm can ensure the alignment of the signals in the time dimension by dynamically adjusting the time step of the two signals.

[0113] Signal preprocessing

[0114] In some embodiments, the data acquisition module also includes a signal preprocessing function for performing denoising, smoothing, and normalization on the collected signal. For example:

[0115] For physiological signals, a bandpass filter is used to remove high-frequency and low-frequency noise;

[0116] For behavioral data, a median filter is used to remove outliers;

[0117] Environmental data are normalized to eliminate dimensional differences.

[0118] Through these technical implementations, the data acquisition module is able to achieve high-precision multimodal data acquisition in complex operating environments, providing comprehensive and reliable foundational data for subsequent workload evaluation. These technical approaches ensure the system's real-time performance and accuracy, while also enhancing data temporal consistency and analytical effectiveness through signal synchronization and preprocessing.

[0119] The modality fusion module is connected to the data processing module and is used to fuse the processed multimodal data to generate a joint representation.

[0120] In the railway coupler workload evaluation system of this invention, the modal fusion module is the core component of the system, responsible for integrating the multimodal features provided by the data processing module. Because physiological, behavioral, and environmental data differ significantly in data distribution, sampling frequency, and feature space, the modal fusion module is designed to address the heterogeneity of multimodal data, enabling collaborative optimization between modalities and generating a joint representation with high task relevance and low redundancy. This module provides high-quality feature input for subsequent processing in the task optimization module and temporal modeling module.

[0121] Multimodal data fusion requires considering both the importance of each modal feature and the interactions between them. This paper introduces a deep neural network to process multimodal features and construct a shared feature space, ensuring that the fused joint representation maximizes the information of the original modalities while reducing redundancy between modalities.

[0122] In this embodiment, the modality fusion module specifically includes the following contents:

[0123] Multimodal feature input and mapping

[0124] In a possible implementation, the modality fusion module receives multimodal feature input provided by the data processing module, including physiological features, behavioral features, and environmental features. i Mapped to a unified feature space, the mapping function can be expressed as:

[0125]

[0126] Among them, X i Represents the feature input of the i-th mode; represents the deep neural network for feature mapping; Z i Represents modality X i Feature representation after mapping.

[0127] Specifically, the mapping function of each modal feature is A multi-layer perceptron (MLP) can be used to implement nonlinear mapping of the feature space by stacking several hidden layers. The activation function of the hidden layer can be ReLU or LeakyRelU to enhance the nonlinear expression ability of the features.

[0128] As an option, the mapping result Z of each modal feature i Normalization can be performed through a normalization layer to eliminate the differences in the range of eigenvalues of different modalities.

[0129] Modality fusion and joint representation generation

[0130] In another possible implementation, the modality fusion module fuses the feature representations of different modalities through a deep neural network to generate the final joint representation Z. The generation of the joint representation Z can be expressed by the following formula:

[0131] Z=g φ (Z1,Z2,…,Z M )

[0132] Among them, g φ represents a fusion network used to integrate multimodal features; M represents the number of modalities; and Z represents the final generated joint representation.

[0133] Specifically, the fusion network g φ The following implementation methods are available:

[0134] Feature splicing: The feature representations of all modalities are Z1, Z2, ..., Z M Direct concatenation is performed and input into a fully connected layer for integration.

[0135] Weighted fusion: assign a learnable weight w to each modality i , the fusion process is expressed as:

[0136]

[0137] where w i represents the weight of the i-th mode.

[0138] Attention mechanism: The self-attention mechanism is introduced to dynamically adjust the fusion weight according to the importance of the modal features. The weight calculation formula is:

[0139]

[0140] Among them, α i represents the weight of the i-th mode, a(Z i ) represents mode Z i Attention score.

[0141] In one possible implementation, the final joint representation Z of the modal fusion module is a low-dimensional feature vector used to characterize the overall workload status of the railway coupler.

[0142] Inter-modal information optimization

[0143] The redundant information of multimodal data phases will lead to a decrease in the quality of joint representation. Therefore, the present invention introduces an information optimization mechanism in the modal fusion process to maximize the effective information while weakening the redundancy between modalities.

[0144] As an alternative, the optimization objective of the joint representation Z can be defined as:

[0145]

[0146] Among them, I(Z;X i ) represents the joint representation Z and modality X i The mutual information of I(X i ;X j ) represents the mode X i and X j λ is the regularization weight of redundant information.

[0147] Specifically, the mutual information I(Z;X i ) can be approximated by using the variational lower bound method, by training a variational distribution q(Z|X i ) is realized. Redundant information I(X i ;X j ) can be suppressed by adding negative correlation constraints.

[0148] Task relevance of joint representation

[0149] In another possible implementation, in order to enhance the adaptability of the joint representation to tasks (such as workload classification), the output Z of the modality fusion module also needs to be adjusted for task relevance by optimizing the classification loss. As an option, the loss function for the classification task can take the form of cross entropy:

[0150]

[0151] in, is the true classification label; P(Y=k|Z) is the predicted probability; C is the number of classification categories.

[0152] By minimizing the classification loss, the classification ability of the joint representation can be further optimized to ensure that it can accurately reflect the workload status of railway connectors.

[0153] Model training and deployment

[0154] In some embodiments, the training of the modality fusion module is achieved by jointly optimizing the following two loss functions:

[0155]

[0156] Where β is the weight coefficient of the classification loss. During the optimization process, the Adam optimizer is used to update the neural network parameters. In actual deployment, the modal fusion module can be deployed on edge computing devices to achieve real-time data processing and joint representation generation.

[0157] Through these technical implementations, the modal fusion module effectively integrates multimodal features to generate a joint representation with high task relevance and low redundancy, providing high-quality feature input for subsequent task optimization and temporal modeling modules. This module ensures the accuracy and robustness of railway coupler workload evaluation through information optimization and enhanced task relevance.

[0158] A task optimization module, connected to the modality fusion module, is used to optimize the task relevance of the joint representation

[0159] In the railway coupler workload evaluation system of the present invention, the task optimization module is responsible for further optimizing the joint representation generated by the modal fusion module to improve its adaptability and discriminative ability for specific tasks (such as workload status classification). Because the joint representation is directly used to evaluate the workload status of railway couplers, the task optimization module strengthens the correlation between the joint representation and the target task through supervised optimization of the classification task, while reducing interference from non-task-related features.

[0160] The joint representation needs to be highly task-relevant so that the system can accurately identify the workload status of railway couplers and make corresponding evaluations. To this end, this paper designs a task optimization module based on supervised learning. By minimizing the classification loss function, the joint representation optimizes the classification ability and ensures that it can adapt to the complex characteristics of multimodal data.

[0161] In this embodiment, the specific implementation of the task optimization module includes the following:

[0162] Definition of classification task

[0163] In one possible implementation, the task optimization module divides workload status into multiple classification tasks. Specifically, workload status includes three categories: "normal," "warning," and "overload." The goal of the classification task is to predict the workload status of railway couplers based on the features of the joint representation Z.

[0164] Specifically, the classification task can be formalized as a multi-class classification problem. Let Y be the target class label (ranging from 1 to C, where C is the number of classes). The classification model outputs the predicted probability P(Y = k | Z) for each class based on the joint representation Z, where k = 1, 2, …, C.

[0165] Construction of classification model

[0166] In another possible implementation, the classification model of the task optimization module is implemented using a fully connected neural network. The classification model receives the joint representation Z as input and outputs the predicted probability of each category after nonlinear mapping through several hidden layers.

[0167] Specifically, the structure of the classification model can be expressed as:

[0168] P(Y|Z)=Softmax(W (L) f (L-1) (f (L-2) (...f (1) (Z))))

[0169] Among them, f (l) represents the nonlinear activation function of the lth layer; W (L) Represents the weight matrix of the output layer; the Softmax function is used to normalize the output value to a probability distribution, which is defined as:

[0170]

[0171] As an option, the hidden layer of the classification model can use the RelU or LeakyRelU activation function to improve the nonlinear expression ability of the network. In addition, to prevent overfitting problems, the dropout operation can also be added to the hidden layer.

[0172] Design of classification loss function

[0173] The task optimization module trains the classification model by minimizing the classification loss function. The classification loss function uses the cross entropy loss, which is defined as:

[0174]

[0175] in, is the one-hot encoding of the true classification label; P(Y=k|Z) is the predicted probability of the kth class; C is the number of classification categories.

[0176] In one possible implementation, if the classification task has an uneven distribution of categories, the category weight ω can be introduced into the loss function. k , the weighted classification loss function is defined as:

[0177]

[0178] Among them, ω k Represents the weight of the kth class, which is usually inversely proportional to the number of samples in that class.

[0179] Task-Relevance Optimization of Joint Representations

[0180] In this embodiment, to further enhance the task relevance of the joint representation Z, the optimization objective of the task optimization module not only includes minimizing the classification loss, but also needs to impose task constraints on the joint representation. In one possible implementation, the task constraint is implemented by adding a task relevance regularization term. Specifically, the task relevance regularization term is defined as:

[0181]

[0182] Among them, Z task represents the task-related target representation; ||·||2 represents the L2 norm.

[0183] By minimizing The joint representation Z can be optimized in the task-related direction to further improve the classification performance.

[0184] Multi-task joint optimization

[0185] In another possible implementation, the task optimization module supports multi-task learning and improves the generalization ability of the model by optimizing the loss functions of multiple tasks simultaneously. For example, in addition to the workload classification task, the workload trend prediction task can also be introduced. Assume that the loss function of the classification task is The loss function of the trend prediction task is Then the multi-task joint loss function can be defined as:

[0186]

[0187] Among them, α and β are weight parameters used to balance the influence of the two tasks.

[0188] Multi-task learning improves the model's adaptability to different tasks by sharing feature representation Z.

[0189] Model training and optimization

[0190] In this embodiment, the training process of the task optimization module includes the following steps:

[0191] First, initialize the parameters of the classification model, using random initialization or pre-trained model parameters.

[0192] Then use the gradient descent method to calculate the loss function or Perform optimization and gradually update the model parameters.

[0193] During the training process, a learning function scheduling strategy (such as cosine annealing or step decay) can be used to speed up the convergence.

[0194] As an option, to improve model stability, the task optimization module can reduce floating-point calculation errors through mixed-precision training methods, which is particularly suitable for resource-constrained computing environments.

[0195] Update and output of joint representation

[0196] The joint representation Z optimized by the task optimization module will serve as the input to the temporal modeling module. After model training is complete, the task optimization module can be directly deployed on edge devices to achieve real-time prediction of the railway coupler's workload status.

[0197] Through these technical implementations, the Task Optimization Module ensures high classification model accuracy while enhancing the task adaptability of the joint representation and guaranteeing the system's real-time and robustness. Working closely with the Modal Fusion and Temporal Modeling modules, the Task Optimization Module enables efficient operation of the railway coupler workload evaluation system.

[0198] A temporal modeling module, connected to the modality fusion module, is used to optimize the temporal dynamic characteristics of the joint representation.

[0199] In the railway coupler workload evaluation system of the present invention, the temporal modeling module is used to perform temporal dynamic modeling of the joint representation generated by the modal fusion module. This module's function is to capture the dynamic changes in workload over time and optimize the temporal consistency and smoothness of the joint representation, thereby improving the continuity and accuracy of workload evaluation. Because railway coupler workloads change dynamically over time and may be affected by nonlinear factors such as the environment and the task, the temporal modeling module must be able to accurately model time series characteristics.

[0200] The temporal modeling module implements temporal modeling through regularization constraints and a dynamic prediction algorithm. Regularization constraints smooth temporal variations in the joint representation, reducing the impact of short-term mutations, while the dynamic prediction algorithm further predicts workload trends over future time periods, thereby enhancing system adaptability.

[0201] In this embodiment, the specific implementation of the time modeling module includes the following:

[0202] Temporal consistency regularization

[0203] In one possible implementation, in order to ensure the smoothness of the joint representation in the time dimension, the temporal modeling module performs a t Apply a time consistency regularization constraint. The goal of time consistency regularization is to make the joint representation Z of each time point t tThe representation of the time points before and after is as close as possible to avoid unnecessary short-term fluctuations in the time series. Specifically, the temporal consistency regularization term is defined as:

[0204]

[0205] Where T represents the total length of the time series; Z t represents the joint representation of time t; δ represents the size of the time window; Avg(Z t-δ ,Z t+δ ) represents the average of the joint representation within the time window.

[0206] The time window size δ can be set based on the dynamic characteristics of the operating environment. For example, for a rapidly changing operating environment, a smaller δ value can capture subtle changes in a short period of time; while for a more stable operating environment, a larger δ value can be selected to enhance the smoothing effect.

[0207] Time Series Dynamic Modeling

[0208] In another possible implementation, in order to further enhance the dynamic modeling capability of the joint representation for the time dimension, the time modeling module uses time series modeling technology (such as recurrent neural networks or Transformer) to dynamically process the joint representation.

[0209] Specifically, the temporal modeling module receives the joint representation sequence {Z1, Z2, …, Z T} as input and generate output representation through time series model

[0210] As an option, a long short-term memory network (LSTM) can be used to model time series, and its recursive formula is:

[0211] h t =LSTM(Z t ,h t-1 )

[0212]

[0213] Among them, h t represents the hidden state at time t; W h and b h Represent the weight and bias of the output layer respectively; is the output representation at time t.

[0214] In another implementation, the temporal modeling module can use the Transformer architecture to model the joint representation sequence through the self-attention mechanism. The calculation formula of the self-attention mechanism is:

[0215]

[0216] Where Q, K, and V represent query, key, and value vectors respectively; d k Indicates the dimension of the key vector.

[0217] Specifically, Transformer can capture the global dependencies of joint representations in time series, thereby improving the ability to model long-term dynamic characteristics.

[0218] Workload trend forecasting

[0219] In this embodiment, the time modeling module cannot predict the workload status in the future time period by using the time series prediction algorithm. Specifically, the prediction model is represented by the joint sequence {Z1, Z2, ..., Z T} is input and outputs the predicted representation of the future time point Where H represents the number of time steps for prediction.

[0220] In one possible implementation, the prediction process of the temporal modeling module can be implemented through a sequence-to-sequence (Seq2Seq) model.

[0221] The Seq2Seq model consists of an encoder and a decoder:

[0222] The encoder receives the historical joint representation sequence as input and generates a context vector C;

[0223] The decoder takes the context vector C and the current decoding state as input and gradually generates predicted representations for future time points.

[0224] As an option, workload state classification constraints can be imposed on the prediction results, and the classification prediction loss can be defined as:

[0225]

[0226] in, represents the true label of the kth category; Represents the predicted classification probability.

[0227] Joint optimization of temporal modeling modules

[0228] In another possible implementation, the optimization goal of the temporal modeling module is to simultaneously minimize the temporal consistency regularization loss and predicted losses

[0229] The total loss function can be defined as:

[0230]

[0231] Among them, α and β are weight parameters used to balance the importance of temporal consistency and prediction task.

[0232] During the optimization process, the gradient descent method can be used to iteratively update the model parameters to simultaneously improve the performance of temporal consistency and smooth prediction.

[0233] Model deployment and application

[0234] In this embodiment, the temporal modeling module can be deployed in either an edge device or a cloud server. In the edge device, the temporal modeling module is used to smooth the temporal dynamics of the workload state in real time; in the cloud server, the temporal modeling module can be used for long-term trend prediction and periodic optimization of the model.

[0235] Through this technical implementation, the temporal modeling module optimizes the continuity and dynamic characteristics of the joint representation in the temporal dimension, significantly improving the stability and robustness of the railway coupler workload evaluation system. Furthermore, the temporal modeling module supports the prediction of workload status in future time periods, providing important insights for operational management and risk control.

[0236] An evaluation module, connected to the task optimization module and the time modeling module, is used to evaluate the workload status of the railway connector based on the joint representation

[0237] In the railway coupler workload evaluation system of the present invention, the evaluation module is the final output module of the system. Its core task is to complete the real-time evaluation and classification of the railway coupler workload status based on the optimized joint representation provided by the task optimization module and the time modeling module. The design of the evaluation module must ensure the accuracy and real-time performance of the evaluation results and be able to distinguish between different workload states (such as "normal", "warning", and "overload"). In addition, when the system detects an overload state, the evaluation module should trigger an alarm signal to provide timely decision-making basis for operation managers.

[0238] The evaluation module extracts the most representative features from the combined representation of multimodal data for workload classification and evaluation. To ensure the reliability of the evaluation results, the evaluation module also smooths out possible short-term fluctuations by incorporating dynamic modeling information provided by the temporal modeling module.

[0239] In this embodiment, the specific implementation of the evaluation module includes the following:

[0240] Evaluation of classification tasks

[0241] In one possible implementation, the evaluation module receives the joint representation Z output by the temporal modeling module. tand its related dynamic characteristic information, and classifies the workload status at each time point based on the classification model.

[0242] Specifically, the evaluation module calculates the predicted probability P(Y t =k|Z t ). Where k = 1, 2, ..., C, C is the number of classification categories. The final classification result It can be determined by the following formula:

[0243]

[0244] in, is the predicted category at time t; P(Y t =k|Z t ) is the predicted probability of the kth class at time t.

[0245] As an option, the evaluation module can calculate the long-term trend of workload status based on the category distribution of the classification results, such as calculating the proportion of different statuses within a certain time window, thereby providing managers with more comprehensive load assessment information.

[0246] Evaluation of classification tasks

[0247] In one possible implementation, the evaluation module receives the joint representation Z output by the temporal modeling module. t and its related dynamic characteristic information, and classifies the workload status at each time point based on the classification model.

[0248] Specifically, the evaluation module calculates the predicted probability P(Y t =k|Z t ). Where k = 1, 2, ..., C, C is the number of classification categories. The final classification result It can be determined by the following formula:

[0249]

[0250] in, is the predicted category at time t; P(Y t =k|Z t ) is the predicted probability of the kth class at time t.

[0251] As an option, the evaluation module can calculate the long-term trend of workload status based on the category distribution of the classification results, such as calculating the proportion of different statuses within a certain time window, thereby providing managers with more comprehensive load assessment information.

[0252] Triggering of alarm signals

[0253] When the system detects that the workload status is "overloaded", the evaluation module will trigger an alarm signal to prompt the operation manager to take corresponding measures.

[0254] Specifically, the evaluation module can perform continuity testing on the classification results. For example, if the classification results within a certain time window continuously exceed a certain proportion, it is considered "overloaded" and an alarm signal is triggered. The alarm triggering condition can be described by the following formula:

[0255]

[0256] Among them, t1 and t2 are the start and end times of the time window; is an indicator function, which takes the value of 1 when the condition is met and 0 otherwise; γ is the alarm trigger threshold, which can usually be set according to the specific working environment and safety requirements.

[0257] As an option, the evaluation module can also combine the future workload status prediction results provided by the time modeling module to issue early warning signals in advance, thereby enhancing the predictability of the system.

[0258] Selection and optimization of classification models

[0259] In this embodiment, the classification model of the evaluation module can be implemented using a multi-layer perceptron (MLP), a random forest (RF), or a lightweight convolutional neural network (CNN). The selection of the classification model needs to be weighed based on the characteristics of the railway coupler workload data and the limitations of the system computing resources.

[0260] Specifically, the training goal of the classification model is to minimize the classification error, and its loss function can be defined as the cross entropy loss:

[0261]

[0262] in, is the true label of the k-th category at time t; P(Y t =k|Z t ) is the predicted probability of the kth class at time t.

[0263] In another possible implementation, in order to enhance the classification model's ability to identify minority classes (such as the "overload" state), the evaluation module can also add a category weight ω to the loss function. k , to increase the attention to the minority class.

[0264] The weighted loss function is defined as:

[0265]

[0266] where ω k It is usually inversely proportional to the sample proportion of the class.

[0267] Comprehensive evaluation within the time window

[0268] In another possible implementation, the evaluation module may perform a comprehensive analysis on the classification results within a certain time window and output an overall workload status evaluation.

[0269] Specifically, the evaluation module can calculate the total load index I based on the classification proportion of each state in the time window. load , defined as follows:

[0270]

[0271] Among them, β k Different weight values are assigned to the load weight corresponding to each state, such as "normal", "warning" and "overload"; t1 and t2 are the start and end times of the time window.

[0272] As an option, the evaluation module can also combine the transition probabilities between different load states to further analyze the fluctuation characteristics of the workload, thereby generating a more comprehensive evaluation report.

[0273] Real-time performance and deployment methods

[0274] In this embodiment, the evaluation module can be deployed on edge computing devices or cloud servers. In edge computing scenarios, the evaluation module is primarily used for real-time classification and alarm triggering; in cloud scenarios, the evaluation module can be integrated with data storage and analysis systems for long-term trend analysis and model optimization.

[0275] Specifically, the real-time requirements of the evaluation module are primarily reflected in the classification and alarm triggering processes, and their computational complexity should be kept within a range suitable for real-time processing. Therefore, the evaluation module can adopt a lightweight classification model and improve computational efficiency through batch or stream processing.

[0276] Through the aforementioned technical implementation, the evaluation module efficiently and accurately classifies and evaluates the workload status of railway couplers. Combined with alarm triggering and trend analysis capabilities, this enhances the system's practicality and early warning capabilities. Furthermore, the evaluation module's close integration with the task optimization and temporal modeling modules ensures high accuracy and dynamic temporal adaptability of classification results, providing reliable technical support for railway operation safety management.

[0277] Please see the attached Figure 2 The present invention also provides a railway coupler workload evaluation method based on multimodal data analysis. The following describes the specific implementation methods of each step in conjunction with the workflow of the method of the present invention.

[0278] S1 collects multimodal data of railway connectors

[0279] In this embodiment, data collection is the starting point of the entire method, and the physiological data, behavioral data and environmental data of the railway coupler during the operation are obtained through the multimodal data acquisition module.

[0280] Specifically, physiological data is collected through wearable devices, including heart rate, blood oxygen saturation and skin conductance; behavioral data is collected through inertial measurement units or cameras to record information such as the connector's movement amplitude and trajectory; environmental data is collected through temperature and humidity sensors, noise meters and vibration sensors to reflect the external conditions of the working environment.

[0281] The collected multimodal data provides a comprehensive and reliable foundation for subsequent data processing and modeling.

[0282] S2 processes the multimodal data, including time alignment and feature extraction

[0283] In this embodiment, data processing includes two parts: time alignment and feature extraction. Time alignment uses the dynamic time warping (DTW) method to align data of different modalities to a unified time axis, solving the problem of inconsistent sampling frequencies of multimodal data.

[0284] In the feature extraction stage, key indicators such as heart rate variability and blood oxygen change rate are extracted from physiological data; dynamic features such as trajectory curvature and movement complexity are extracted from behavioral data; and information such as temperature and humidity change amplitude and noise level are extracted from environmental data.

[0285] The data processing step significantly reduces the dimensionality and redundancy of the data, providing high-quality input for modality fusion.

[0286] S3 generates joint representations of multimodal data through modal fusion

[0287] In this embodiment, modality fusion generates a joint representation of multimodal data using a deep neural network. This joint representation is a low-dimensional representation of physiological, behavioral, and environmental characteristics, encompassing key information from all modalities while improving representation quality by reducing redundancy.

[0288] Specifically, the features of each modality are first mapped into a shared feature space, and then a final joint representation is generated through weighted fusion, feature concatenation, or attention mechanism. The joint representation lays a solid foundation for subsequent task optimization and temporal dynamics modeling.

[0289] S4 optimizes the task relevance of joint representations

[0290] In this embodiment, to improve the adaptability of the joint representation to the workload classification task, the task optimization module further optimizes the joint representation through supervised learning. The goal of the classification task is to classify the workload status into three categories: "normal", "warning", and "overload".

[0291] By minimizing the loss function of the classification model, the task relevance of the joint representation is significantly enhanced. At the same time, in order to improve the classification model's ability to identify minority classes (such as the "overload" state), class weights can be added to the loss function.

[0292] The task optimization step ensures that the joint representation accurately reflects the actual workload status of the connector.

[0293] S5 optimizes the temporal dynamics of the joint representation

[0294] In this embodiment, the temporal modeling module dynamically optimizes the joint representation over time to enhance its smoothness and continuity across the time dimension. The temporal consistency regularization constraint ensures smoother changes in the joint representation over time, avoiding classification errors caused by short-term fluctuations.

[0295] Furthermore, by introducing time series modeling techniques (such as recurrent neural networks or Transformers), the temporal modeling module is able to capture the dynamic characteristics of the joint representation over long time spans. This optimization step provides a more stable and reliable foundation for the final load state classification and evaluation.

[0296] S6 classifies and evaluates the workload status of railway couplers based on the optimized joint representation

[0297] In this embodiment, the evaluation module classifies and evaluates the workload status of railway couplers based on the optimized joint representation provided by the temporal modeling module. The classification model infers the joint representation at each time point and outputs a predicted workload status.

[0298] Specifically, the classification results include "normal," "warning," and "overload." When the classification result is "overload," the system triggers an alarm signal, prompting operations managers to take necessary measures. Furthermore, the evaluation module can perform statistical analysis on the classification results within a specific time window to generate an overall load status evaluation report.

[0299] Through the above classification and evaluation steps, the system can monitor the workload status of railway couplers in real time and provide early warning functions, providing strong support for railway operation safety management.

[0300] Through the above steps, the present invention's railway coupler workload evaluation method based on multimodal data analysis achieves efficient modeling and accurate workload classification throughout the entire process, from data collection to classification and evaluation. The seamless integration and layer-by-layer optimization of each step ensures the reliability and practicality of the evaluation results, providing a systematic solution for monitoring and managing railway coupler work status.

[0301] The present invention also provides a storage medium having a computer program stored thereon, and the computer program, when executed by a processor, executes the above method. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0302] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A railway coupler workload evaluation system based on multimodal data analysis, characterized in that: include: Data acquisition module, used to obtain multimodal data of railway couplers; A data processing module, connected to the data acquisition module, for processing multimodal data; A modality fusion module, connected to the data processing module, for fusing the processed multimodal data to generate a joint representation; a task optimization module, connected to the modality fusion module, for optimizing the task relevance of the joint representation; A temporal modeling module, connected to the modality fusion module, for optimizing the temporal dynamic characteristics of the joint representation; An evaluation module is connected to the task optimization module and the time modeling module, and is used to evaluate the workload status of the railway coupler based on the joint representation.

2. A railway coupler workload evaluation system based on multimodal data analysis according to claim 1, characterized in that: The data acquisition module is used to obtain multimodal data of railway couplers, and the multimodal data includes physiological data, behavioral data and environmental data. The physiological data includes heart rate, blood oxygen, and skin conductance; the behavioral data includes trajectory and movement amplitude; and the environmental data includes temperature, humidity, noise and vibration.

3. The railway coupler workload evaluation system based on multimodal data analysis according to claim 1 is characterized in that: The data processing module includes: A time alignment unit, used to synchronize the time series of multimodal data; The feature extraction unit is used to extract the features of each modal data.

4. The railway coupler workload evaluation system based on multimodal data analysis according to claim 1 is characterized in that: The modality fusion module generates a joint representation of multimodal data through a deep neural network. The joint representation is generated based on the optimization of effective information of the multimodal data and improves the representation quality by reducing redundant information between modalities.

5. The railway coupler workload evaluation system based on multimodal data analysis according to claim 1 is characterized in that: The task optimization module optimizes the support capability of the joint representation for the workload classification task by minimizing the classification loss. The classification task classifies the workload status into normal, warning and overload status.

6. The railway coupler workload evaluation system based on multimodal data analysis according to claim 1 is characterized in that: The temporal modeling module imposes regularization constraints on the time series of the joint representation to optimize the smooth change of the joint representation in the time dimension, so that the workload evaluation has dynamic continuity.

7. The railway coupler workload evaluation system based on multimodal data analysis according to claim 1 is characterized in that: The evaluation module classifies the workload status of the railway coupler based on the optimized joint representation, and triggers an alarm signal when the classification result is an overload state.

8. The railway coupler workload evaluation system based on multimodal data analysis according to claim 4 is characterized in that: The joint representation generated by the modality fusion module maximizes the relevance with each modality data, minimizes the interference of redundant information between modalities, and improves the classification ability of the representation by strengthening task-related information.

9. A railway coupler workload evaluation method based on multimodal data analysis, applied to the evaluation system according to any one of claims 1 to 8, characterized in that: The following steps are involved: Collect multimodal data of railway couplers; Processing the multimodal data, including time alignment and feature extraction; Generate joint representations of multimodal data through modal fusion; Optimizing task relevance of joint representations; Optimizing the temporal dynamics of the joint representation; The workload status of railway couplers is classified and evaluated based on the optimized joint representation.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to claim 9 is implemented.

Images