Liquid cross-attention driven heterogeneous perception rail transit intelligent interaction system and method

Abstract

The application discloses a liquid cross-attention driven heterogeneous perception rail transit intelligent interaction system and method, comprising a data acquisition module, a data preprocessing module, a multi-modal data fusion layer, a unified multi-modal representation module and a decision and output layer; the application overcomes the defects of single modal perception by deeply fusing voice, visual and radar signal data and modeling timing dynamic relationship; through deep and time-aligned multi-modal fusion, precise and robust perception of the driver state, driving environment and voice instruction is realized; finally, reliable technical support is provided for train automatic driving assistance and intelligent human-computer interaction, and the safety and efficiency level of railway driving is comprehensively improved.

Description

Technical Field

[0001] This invention relates to the field of rail safety driving technology, specifically to a heterogeneous sensing intelligent interaction system and method for rail transit driven by liquid cross-attention. Background Technology

[0002] Traditional single-modal systems, such as pure speech recognition or pure visual perception, have inherent limitations in complex and ever-changing real-world environments. For example, in subway driving scenarios, pure speech recognition systems are highly susceptible to interference from environmental noise (such as subway operating noise and announcements), while pure visual perception may be affected by changes in lighting (such as entering and exiting tunnels). These interferences lead to insufficient information and high misjudgment rates. A typical example is the "Lombard effect": when speech recognition is inaccurate, the driver's repetition often fails to effectively correct the error because people unconsciously speak in unusual ways after being misunderstood, which further reduces the accuracy of pure speech recognition. In such cases, if the system can incorporate visual information (such as lip movements), it can assist in understanding the driver's intentions even if the speech signal is severely interfered with, thereby avoiding a negative feedback loop.

[0003] Despite significant progress in multimodal data integration, effective integration still faces major technical challenges, primarily in alignment and fusion. Alignment aims to establish cross-modal semantic relationships, ensuring that representations from different modalities reside in a common space. Fusion, on the other hand, combines this information for unified prediction, addressing noise variability and intermodal reliability differences. Specific challenges include the diversity of feature representations, computational efficiency, data quality (e.g., poorly matched datasets from the internet), and the massive scale of required training data. Speech and video are inherently time-series data, and modal alignment involves not only semantic correspondence but also temporal synchronization. Traditional fusion methods often struggle with dynamic, fine-grained temporal alignment, especially when different modalities have different sampling rates or inherent temporal resolutions.

[0004] Furthermore, traditional recurrent neural networks (RNNs), such as Long Short-Term Memory networks (LSTMs) and gated recurrent units (GRUs), while effective in time-series tasks, are often limited by long-range dependencies, vanishing / exploding gradients, and fixed step-size processing. These limitations make them perform poorly when processing continuous, irregularly sampled, or highly dynamic time-series data. For example, in multimodal data streams, different modalities may be out of sync, and their characteristics (such as speech rate, visual complexity, and noise levels) may change rapidly and unpredictably. Mainstream multimodal fusion methods are typically categorized based on their fusion point (early, late, or hybrid) or their underlying architectural principles, including encoder-decoder fusion, kernel-based fusion, graph fusion, and attention-based fusion. However, due to insufficient interaction between modalities, these simple operations cannot achieve deep and effective fusion.

[0005] Therefore, there is an urgent need in this field for an intelligent interactive system for rail driving that can overcome the limitations of traditional single-modal and multi-modal fusion methods. Summary of the Invention

[0006] The purpose of this invention is to provide a liquid-based cross-attention driven heterogeneous sensing intelligent interaction system and method for rail transit, so as to overcome the limitations of traditional single-modal and multi-modal fusion methods in existing intelligent interaction systems for rail driving.

[0007] To address the aforementioned technical problems, in a first aspect, the present invention provides a liquid-based cross-attention driven heterogeneous sensing intelligent interaction system for rail transit, comprising:

[0008] The data acquisition module is used to collect multimodal raw signal data in the subway driving environment; the multimodal raw signal data includes voice signal data, radar signal data, and visual signal data.

[0009] The data preprocessing module is used to preprocess the acquired multimodal raw signal data, and then extract features from the preprocessed multimodal raw signal data to obtain audio feature vector sequences, visual feature vector sequences, and radar feature vector sequences.

[0010] The multimodal data fusion layer includes an LTC modal encoder and a liquid cross-attention fusion layer; the LTC modal encoder is used to capture the temporal dependencies of each modal feature vector sequence and generate context-aware modal hidden states; the liquid cross-attention fusion layer is used to fuse each modal hidden state to obtain a fused hidden state.

[0011] A unified multimodal representation module is used to identify the fused hidden state and output the driver state analysis result, driving environment perception result, and voice command parsing result;

[0012] The decision and output layer is used to output control commands and provide human-machine interaction feedback based on the driver state analysis results, driving environment perception results, and voice command parsing results.

[0013] Furthermore, the voice signal data includes the driver's voice commands, passenger conversations inside the carriage, and ambient noise;

[0014] The visual signal data includes video of the driver's cab interior and video of the external environment.

[0015] The radar signal data includes distance information and speed information.

[0016] Furthermore, the acquired multimodal raw signal data is preprocessed, including cleaning and synchronization processing.

[0017] The cleaning process includes: filtering the acquired multimodal raw signal data to remove noise and redundant information from each modality's raw signal data;

[0018] The synchronization process includes aligning the timestamps and sampling rates of different modal signal data to ensure that speech signal data, visual signal data, and radar signal data are precisely aligned under a unified timestamp mechanism.

[0019] Furthermore, feature extraction from the preprocessed multimodal raw signal data specifically includes:

[0020] The audio feature vector sequence The extraction process includes: converting the cleaned speech signal data into a Mel spectrogram using an audio encoder, and then extracting audio feature vector sequences that can represent phonemes, intonation, rhythm, and acoustic events using an encoder based on a one-dimensional convolutional neural network and Transformer. ;

[0021] Visual feature vector sequence The extraction process includes: using an image encoder to process synchronized video frames, and then using a spatiotemporal Transformer or a 2D / 3D convolutional neural network to extract visual feature vector sequences that can represent the driver's lip movements, facial expressions, gestures, and external environmental objects, motion states, and colors. The radar feature vector sequence Extraction includes: encoding radar signal data and extracting radar feature vector sequences. .

[0022] Furthermore, the LTC modal encoder models the system dynamics using the following differential equation:

[0023] ;

[0024] in, Indicates the modal hidden state; The time constant representing the mode; Indicates by parameters Neural networks This represents the input feature vector; The bias vector represents the modality.

[0025] Furthermore, the forward propagation of the LTC modal encoder is updated using a fused ODE solver, as follows:

[0026] ;

[0027] in, This indicates the ODE solver step size. It represents the Hadamardi (or Hadama) stack.

[0028] Furthermore, the fusion process of the liquid cross-attention fusion layer is as follows:

[0029] Calculate cross-modal attention weights using a multi-head attention mechanism ;Utilizing the cross-modal attention weights Neural Networks in the Differential Equations of Dynamically Modulated LTC Modal Encoders After adjustment Then Substituting into the differential equation, we get:

[0030] ;

[0031] A bidirectional or multidirectional liquid cross-attention process is used to achieve deep fusion among multiple modalities and obtain the fused hidden state. .

[0032] Furthermore, the unified multimodal representation module includes:

[0033] The driver state analysis module is used to analyze the fused hidden state. Identify driver fatigue, inattention, and unusual behavior;

[0034] The driving environment perception module is used to determine the fused hidden state. Detecting obstacles, identifying traffic lights, determining platform alignment, and estimating distances ahead;

[0035] The voice command parsing module is used to analyze the fusion hidden state. It can recognize voice commands issued by the driver.

[0036] Furthermore, the training strategies employed by this system include:

[0037] Self-supervised pre-training is used to pre-train the LTC modality encoder using an unlabeled multimodal dataset to learn modal feature representations;

[0038] Joint end-to-end training is used to train the liquid cross-attention fusion layer on a dataset with task labels, so as to learn how to perform effective intermodal information interaction in a specific task.

[0039] Fine-tuning involves optimizing the modal encoder and multimodal data fusion layer using labeled datasets to adapt to specific interaction scenarios.

[0040] Secondly, the present invention provides a method for intelligent interaction using the above-mentioned intelligent interaction system for rail driving, comprising:

[0041] Collect multimodal raw signal data in the subway driving environment;

[0042] After preprocessing the acquired multimodal raw signal data, feature extraction is performed on the preprocessed multimodal raw signal data to obtain a sequence of feature vectors for each mode;

[0043] The modal feature vector sequences are input into the multimodal data fusion layer. The time dependency of each modal feature vector sequence is captured by the LTC modality encoder of the multimodal data fusion layer, and a context-aware modal hidden state is generated. Then, the modal hidden states are fused by the liquid cross-attention fusion layer to obtain the fused hidden state.

[0044] The fused hidden state is identified, and the driver state analysis result, driving environment perception result, and voice command parsing result are output.

[0045] It is used to output control commands and provide human-machine interaction feedback based on the driver state analysis results, driving environment perception results, and voice command parsing results.

[0046] The beneficial effects of this invention are as follows: by deeply integrating voice, vision, and radar signal data and modeling temporal dynamic relationships, the defects of single-modal perception are overcome; through deep, time-aligned multimodal fusion, accurate and robust perception of driver status, driving environment, and voice commands is achieved; ultimately, reliable technical support is provided for train automatic driving assistance and intelligent human-machine interaction, comprehensively improving the safety and efficiency of railway driving. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, use the same reference numerals to denote the same or similar parts. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0048] Figure 1 This is a block diagram of the overall architecture of the intelligent interactive system for rail driving that integrates speech recognition and computer vision, as shown in this invention. The diagram provides a macroscopic view of the system from input to decision-making and output, illustrating the main functional modules and their data flow.

[0049] Figure 2 This is a detailed structural diagram of the multimodal data fusion layer in this invention, focusing on the internal working principle of the liquid time constant network (LTC) modal encoder and the liquid cross-attention fusion layer. The diagram details how the audio, visual, and radar LTC encoders process the features of their respective modes, and how the liquid cross-attention mechanism dynamically modulates the ODE dynamics of the LTC to achieve deep intermodal fusion.

[0050] Figure 3 This is a functional diagram of the driver state analysis module in this invention, which shows in detail how it achieves HMI functions such as fatigue detection, attention distraction identification, abnormal behavior analysis, facial recognition, head tracking, gaze detection, emotion detection, and lip reading by fusing multimodal data.

[0051] Figure 4 This is a functional diagram of the driving environment perception module in this invention, which shows in detail how it achieves key perception tasks such as track obstacle detection, signal light recognition, platform alignment judgment, and distance estimation of the train ahead by fusing multimodal data.

[0052] Figure 5 This is a data flow diagram illustrating the interaction between the decision and output layer, the vehicle control system, and the subway signaling system in this invention. It shows how the system generates control commands based on the recognition and prediction results and provides human-machine interaction feedback.

[0053] Figure 6 This diagram illustrates the core dynamics of the Liquid Time Constant Network (LTC) neuron in this invention, demonstrating how its input-dependent variable time constant influences the evolution of the hidden state. The diagram visually presents the differential equations of LTC and the concept of its "liquid" properties. Detailed Implementation

[0054] In a first aspect, the present invention discloses a liquid-based cross-attention driven heterogeneous sensing intelligent interaction system for rail transit, comprising:

[0055] The data acquisition module is used to collect multimodal raw signal data in the subway driving environment; the multimodal raw signal data includes voice signal data, radar signal data and visual signal data;

[0056] The data preprocessing module is used to preprocess the acquired multimodal raw signal data, and then extract features from the preprocessed multimodal raw signal data to obtain audio feature vector sequences, visual feature vector sequences, and radar feature vector sequences.

[0057] The multimodal data fusion layer includes an LTC modal encoder and a liquid cross-attention fusion layer. The LTC modal encoder is used to capture the temporal dependencies of each modal feature vector sequence and generate context-aware modal hidden states. The liquid cross-attention fusion layer is used to fuse the hidden states of each modality to obtain the fused hidden states.

[0058] The unified multimodal representation module is used to identify the fused hidden states and output the driver state analysis results, driving environment perception results, and voice command parsing results.

[0059] The decision and output layer is used to output control commands and provide human-machine interaction feedback based on the driver's state analysis results, driving environment perception results, and voice command parsing results.

[0060] According to one embodiment of this application, the data acquisition module includes a voice acquisition module, a visual acquisition module, and a radar sensor;

[0061] The voice acquisition module collects voice signal data through a high-fidelity microphone array deployed in the driver's cab. This data includes not only explicit voice commands issued by the driver (such as "accelerate to 60 km / h" and "clear passengers at the next station"), but also other sounds that may exist in the driver's cab (such as conversations between the driver and the control center, unusual noises from passengers in the carriages), and key environmental noises (such as wheel and rail noises, station announcements, and emergency alarms).

[0062] The visual acquisition module collects visual signal data through multiple cameras. The camera facing the driver captures video from inside the driver's cab, including facial expressions, head posture, gaze direction, hand gestures, and key indicator light information from the dashboard. The camera facing the external environment captures video images of the track ahead, including track conditions (e.g., presence of foreign objects), signal light status (red, yellow, green), platform information (e.g., alignment with platform screen doors), potential obstacles (e.g., pedestrians or objects encroaching on the clearance), and speed limit signs.

[0063] Radar sensors collect radar signal data, which can provide precise distance and relative speed information about objects ahead, serving as an effective supplement to visual perception. Especially in low visibility environments such as rain, snow, fog, and haze, it can significantly improve the reliability of environmental perception.

[0064] According to one embodiment of this application, the preprocessing of the acquired multimodal raw signal data includes cleaning and synchronization processing;

[0065] The cleaning process includes filtering the acquired multimodal raw signal data to remove noise and redundant information from each modality. For example, noise reduction processing is performed on speech signals to reduce the interference of background noise on speech recognition; and deblurring and illumination normalization are performed on visual images.

[0066] Synchronization processing includes aligning the timestamps and sampling rates of different modal signal data to ensure that speech signal data, visual signal data, and radar signal data are precisely aligned under a unified timestamp mechanism, so as to ensure effective fusion in the future.

[0067] According to one embodiment of this application, feature extraction includes:

[0068] Audio feature vector sequence Extraction involves converting the cleaned speech signal data into a Mel spectrogram using an audio encoder, and then extracting audio feature vector sequences that can represent phonemes, intonation, rhythm, and acoustic events using an encoder based on a one-dimensional convolutional neural network (1D-CNN) and Transformer. ;

[0069] Visual feature vector sequence Extraction involves using an image encoder to process synchronized video frames, employing a spatiotemporal Transformer (such as Video-ViT) or a 2D / 3D convolutional neural network (2D / 3D-CNN) to extract visual feature vector sequences that can characterize the driver's lip movements, facial expressions, gestures, as well as external environmental objects, motion states, and colors. ;

[0070] Radar feature vector sequence Extraction includes encoding radar signal data and extracting radar feature vector sequences. .

[0071] According to one embodiment of this application, the feature vector sequence of each modality ( , , The signal is fed into a dedicated LTC (Liquid Time Constant Network) modal encoder.

[0072] As a special type of continuous-time recurrent neural network, the core advantage of LTC lies in its ability to model system dynamics through differential equations.

[0073] Figure 6 The core dynamics model of the LTC neuron is illustrated schematically. Unlike traditional RNNs that update the state at discrete time steps, the hidden state of LTC evolves continuously, and its core differential equation is:

[0074] ;

[0075] in, Indicates the modal hidden state; The time constant representing the mode; Indicates by parameters Neural networks This represents the input feature vector; The bias vector of the modality. This "fluid" property allows the network to adaptively adjust its response speed and sensitivity to input, making it well-suited for handling scenarios with rapidly changing signal characteristics (such as speech rate and lighting) in subway driving. Through their respective LTC encoders, the system generates context-rich audio hidden states. Visual camouflage state and radar concealment status ,like Figure 2 As shown.

[0076] 1. The audio feature vector sequence Input-based LTC audio modal encoder To capture its unique temporal dependencies and generate context-aware audio hidden states. . The core differential equation for modeling the system dynamics of a linear first-order dynamical system using nonlinear interconnected gating modulation is:

[0077] ;

[0078] in, Indicates that the audio is hidden. It is determined by parameters Neural networks It is audio input (corresponding to this) ), It is the time constant of the audio mode. It is the bias vector of the audio modality.

[0079] Its inherent stability, expressive power, and adaptability to different input rates make it particularly suitable for modeling the complex, continuous temporal dynamics of speech signals and generating context-aware audio hidden states. .

[0080] 2. Visual feature vector sequence Input-based LTC-based visual modal encoder To capture its unique temporal dependencies and generate context-aware visual hidden states. . The core differential equation for modeling the system dynamics of a linear first-order dynamical system using nonlinear interconnected gating modulation is:

[0081] ;

[0082] in, Indicates a visually hidden state. It is determined by parameters Neural networks It is visual input (corresponding to this) ), It is the time constant of the visual modality. It is the bias vector of the visual modality. It can capture unique spatial-temporal dependencies in visual flow and generate context-aware visual hidden states. .

[0083] The above-mentioned core differential equation is used to continuously process time-series signals. and Dynamically adjust the time constant (such as when the light suddenly changes) (Reduced to 50ms), it can eliminate feature misalignment caused by traditional hard alignment, realize asynchronous fusion of heterogeneous sampling data of speech (16kHz) and vision (30fps), and the timing error can be less than 50ms, which is much smaller than the traditional timing error of 300ms.

[0084] 3. The radar feature vector sequence Input to LTC-based radar mode encoder This allows for the capture of its unique temporal dependencies and the generation of context-aware radar hidden states. .

[0085] The core differential equation for modeling the system dynamics of a linear first-order dynamical system using nonlinear interconnected gating modulation is:

[0086] ;

[0087] in, Indicates radar is in a hidden state. It is determined by parameters Neural networks It is radar input (i.e.) ), It is the time constant of the radar mode. It is the bias vector of the radar mode. It can capture the temporal dependencies of radar data and generate context-aware radar hidden states. .

[0088] According to one embodiment of this application, the forward propagation of the LTC modal encoder is updated using a fused ODE solver, as shown below:

[0089] ;

[0090] in, This indicates the ODE solver step size. It represents the Hadamardi (or Hadama) stack.

[0091] According to one embodiment of this application, the fusion process of the liquid cross-attention fusion layer is as follows:

[0092] First, a multi-head attention mechanism is used to calculate cross-modal attention weights. ;

[0093] Then, the cross-modal attention weights are utilized. Neural Networks in the Differential Equations of Dynamically Modulated LTC Modal Encoders After adjustment Then Substituting into the differential equation, we get:

[0094] ;

[0095] Where MLP(·) is a multilayer perceptron;

[0096] This process defines the hidden state derivative and the liquid time constant itself (as...). neural networks It now explicitly relies on cross-modal attention output. This allows the “speed” and “coupling sensitivity” of each LTC neuron to be directly influenced by dynamic alignment information from another modality. For example, if the visual stream instructs the driver’s lip movements (lip reading), the time constant of the audio LTC may become more sensitive to phoneme-level changes, thus aiding speech recognition in noisy environments.

[0097] Finally, a bidirectional or multidirectional liquid cross-attention process is performed to achieve deep fusion among multiple modalities and obtain the fused hidden state. This process is bidirectional or multidirectional, meaning that different modalities pay attention to each other (e.g., audio pays attention to vision and radar, vision pays attention to audio and radar, and radar pays attention to audio and vision). The outputs of these cross-attention mechanisms (cross-modal attention weights) are then considered. The combined (e.g., summed or spliced) states are fed back into the dynamics of their respective LTCs or input into a dedicated fusion LTC layer. Ultimately, the fusion hidden states from the liquid cross-attention layers (e.g., the output of the "fusion LTC" or the combined states of the modulated LTCs) form a comprehensive, dynamically evolving multimodal representation. This representation captures the independent temporal dynamics of each modality and its complex, adaptively aligned cross-modal dependencies.

[0098] According to one embodiment of this application, a multi-head attention mechanism is used to calculate cross-modal attention weights. Time; specific calculation methods include:

[0099] First, generate the query ( ),key( ) and value ( (Vector). In a multi-head attention mechanism, the LTC hidden state of one modality acts as the query, while the LTC hidden states of other modalities act as the key and value; for example, for an audio modality, attention is paid to a visual modality:

[0100] Audio hidden state Act as a query ( ).

[0101] Visual Hidden State Act as a key ( ) and value ( ).

[0102] Specifically, this is represented as the projection of the h-th attention head:

[0103] ;

[0104] in, Indicates the hidden state of the query modality. This represents the hidden state of the key / value modality. , , Let be the projection matrix of the h-th attention head.

[0105] Then, the scaled dot product attention is calculated to generate cross-modal attention weights. Attention calculations are performed in parallel across multiple attention heads, and then the results from each head are concatenated.

[0106] ;

[0107] in, For the key / query dimension; this will produce another modality of attention-weighted representation, such as and .

[0108] According to one embodiment of this application, the unified multimodal representation module includes:

[0109] The driver state analysis module is used to analyze the fused hidden state. Identify driver fatigue, inattention, and unusual behavior, such as... Figure 3 As shown; for example, by analyzing the driver's facial features (such as eye opening and closing, yawning frequency) and head posture (such as looking down, shaking), driver fatigue can be identified; by analyzing whether the driver's gaze deviates from the path for a long time, distraction can be identified; by analyzing the driver's gestures and emotions, abnormal behavior can be identified.

[0110] The driving environment perception module is used to determine the hidden state based on the fusion. Detecting obstacles, recognizing traffic lights, determining platform alignment, estimating distances ahead, etc. Figure 4 As shown; for example, by fusing visual and radar information, all-weather track obstacle detection can be achieved; visual information can accurately identify the status of traffic lights, accurately determine the alignment of the platform when entering the station, and estimate the safe distance to the train in front in real time;

[0111] The voice command parsing module is used to analyze the fusion hidden state. It recognizes voice commands issued by the driver, such as "open door", "close door", "accelerate", "decelerate", and "emergency brake", and outputs the parsing results in a structured format.

[0112] According to one embodiment of this application, the decision and output layer includes a decision and control module and a human machine interface (HMI) module.

[0113] The decision and control module generates control commands based on the analysis results, which are used to control the on-board control system (such as train speed, braking, and door opening and closing) and the subway signaling system (such as emergency stop signals).

[0114] The Human-Machine Interface (HMI) module provides drivers with interactive information (such as warning messages, operation confirmations, and system status) through voice and visual (screen display) methods, enabling intelligent interaction.

[0115] According to one embodiment of this application, the training strategy employed by the system includes:

[0116] Self-supervised pre-training utilizes large-scale unlabeled audiovisual or multimodal datasets (e.g., lip-reading datasets such as LRS3 and VoxCeleb, video action recognition datasets such as Kinetics and ActivityNet, or datasets specifically designed for audiovisual event detection) to learn modality-specific representations through self-supervised objectives such as contrastive learning, mask modeling, or cross-modal generation, in order to learn a robust initial feature extractor.

[0117] Joint end-to-end training is performed using task-specific loss functions (e.g., cross-entropy for classification, mean squared error for regression, negative log-likelihood for sequence generation) to train the entire multimodal LTC architecture, which includes an LTC modality encoder, liquid cross-attention fusion, and a downstream task head.

[0118] Fine-tuning is performed on a smaller, task-specific labeled dataset to optimize the pre-trained LTC modality encoder and fusion layer to adapt them to specific subway driving interaction scenarios.

[0119] Through this training strategy, the system can achieve enhanced temporal modeling, improved adaptability to noise and variable input rates, increased interpretability potential, and robust alignment, thereby improving the safety and efficiency of subway driving.

[0120] Through the aforementioned technical solutions, this system achieves enhanced temporal modeling, improved adaptability to noise and variable input rates, potential for increased interpretability, and robust alignment. The continuous-time characteristics and adaptive time constants of LTC allow for more accurate and flexible modeling of the complex, asynchronous temporal dynamics inherent in audiovisual data, outperforming fixed-step RNNs. Its "liquid" nature enables the network to dynamically adjust its internal dynamics and responsiveness based on the quality and characteristics of the incoming multimodal data stream, resulting in greater robustness in noisy or variable real-world environments. The explicit time constants in LTC characterize the "velocity and coupling sensitivity" of the ODE. Analyzing how these time constants change in response to cross-modal attention in a multimodal context provides insights into how the network prioritizes and integrates information from different modalities, potentially leading to more transparent and trustworthy AI systems. The liquid cross-attention mechanism provides a more deeply integrated and adaptive alignment strategy than traditional attention, which typically operates on static feature representations, by modulating ODE dynamics. These advantages collectively improve the safety and efficiency of subway driving, opening new avenues for building smarter, more robust, and more human-like AI systems.

[0121] Secondly, the present invention discloses a method for intelligent interaction using the above-mentioned intelligent interaction system for rail driving, comprising:

[0122] Collect multimodal raw signal data in the subway driving environment;

[0123] After preprocessing the acquired multimodal raw signal data, feature extraction is performed on the preprocessed multimodal raw signal data to obtain the feature vector sequence of each mode;

[0124] The feature vector sequences of each modality are input into the multimodal data fusion layer. The LTC modality encoder of the multimodal data fusion layer captures the temporal dependencies of each feature vector sequence and generates context-aware modal hidden states. Then, the liquid cross-attention fusion layer fuses the hidden states of each modality to obtain the fused hidden states.

[0125] The system identifies the hidden states and outputs driver state analysis results, driving environment perception results, and voice command parsing results.

[0126] It is used to output control commands and provide human-machine interaction feedback based on the driver's state analysis results, driving environment perception results, and voice command parsing results.

[0127] The fusion method employed in this invention offers significant advantages over existing multimodal fusion techniques. Compared to early / late fusion, this method provides deep, dynamic interactions between modalities throughout the processing, avoiding the limitations of early fusion (premature fusion of unaligned features) and late fusion (loss of fine-grained interactions). Unlike traditional attention-based fusion, this method integrates it into the dynamics of the recurrent unit, making the attention mechanism itself "fluid" and adaptive, unlike standard attention mechanisms that typically apply static weights to features. Compared to vanilla neural ODEs, LTC is an evolution of neural ODEs, offering superior stability and expressiveness due to its specific time constant formulation. The architecture proposed in this invention further extends this to achieve multimodal interaction. The core innovation is not merely applying LTC to multimodal data, but fundamentally changing the way fusion and alignment are performed by leveraging its "fluid" properties. By dynamically modulating the ODE and its time constant through cross-modal attention, a continuous, adaptive, and intrinsically aligned fusion process is introduced, representing a paradigm shift from discrete, static fusion points to fluid, dynamic, and context-aware integration of multimodal temporal information. This signifies a major leap forward in the ability of AI systems to process complex real-world perception streams.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A heterogeneous sensing intelligent interaction system for rail transit driven by liquid cross-attention, characterized in that, include: The data acquisition module is used to collect multimodal raw signal data in the subway driving environment; the multimodal raw signal data includes voice signal data, radar signal data, and visual signal data. The data preprocessing module is used to preprocess the acquired multimodal raw signal data, and then extract features from the preprocessed multimodal raw signal data to obtain audio feature vector sequences, visual feature vector sequences, and radar feature vector sequences. A multimodal data fusion layer includes an LTC modality encoder and a liquid cross-attention fusion layer. The LTC modality encoder is used to capture the temporal dependencies of each modality feature vector sequence and generate context-aware modality hidden states. The liquid cross-attention fusion layer is used to fuse the hidden states to obtain fused hidden states. The fusion process of the liquid cross-attention fusion layer is as follows: Calculate cross-modal attention weights using a multi-head attention mechanism ; Utilizing the cross-modal attention weights Neural Networks in the Differential Equations of Dynamically Modulated LTC Modal Encoders After adjustment Then Substituting into the differential equation, we get: ； Where MLP(·) is a multilayer perceptron; A bidirectional or multidirectional liquid cross-attention process is used to achieve deep fusion among multiple modalities and obtain the fused hidden state. ; A unified multimodal representation module is used to identify the fused hidden state and output the driver state analysis result, driving environment perception result, and voice command parsing result; The decision and output layer is used to output control commands and provide human-machine interaction feedback based on the driver state analysis results, driving environment perception results, and voice command parsing results.

2. The liquid-state cross-attention driven heterogeneous sensing rail transit intelligent interaction system according to claim 1, characterized in that, The voice signal data includes the driver's voice commands, passenger conversations in the carriage, and ambient noise; The visual signal data includes video of the driver's cab interior and video of the external environment. The radar signal data includes distance information and speed information.

3. The liquid-state cross-attention driven heterogeneous sensing rail transit intelligent interaction system according to claim 2, characterized in that, The preprocessing of the acquired multimodal raw signal data includes cleaning and synchronization processing; The cleaning process includes: filtering the acquired multimodal raw signal data to remove noise and redundant information from each modality's raw signal data; The synchronization process includes aligning the timestamps and sampling rates of different modal signal data to ensure that speech signal data, visual signal data, and radar signal data are precisely aligned under a unified timestamp mechanism.

4. The liquid-phase cross-attention driven heterogeneous sensing intelligent interaction system for rail transit according to claim 3, characterized in that, Feature extraction from preprocessed multimodal raw signal data specifically includes: The audio feature vector sequence The extraction process includes: converting the cleaned speech signal data into a Mel spectrogram using an audio encoder, and then extracting audio feature vector sequences that can represent phonemes, intonation, rhythm, and acoustic events using an encoder based on a one-dimensional convolutional neural network and Transformer. ; The visual feature vector sequence The extraction process includes: using an image encoder to process synchronized video frames, and then using a spatiotemporal Transformer or a 2D / 3D convolutional neural network to extract visual feature vector sequences that can represent the driver's lip movements, facial expressions, gestures, and external environmental objects, motion states, and colors. ; The radar feature vector sequence Extraction includes: encoding radar signal data and extracting radar feature vector sequences. .

5. The liquid-based cross-attention driven heterogeneous sensing intelligent interaction system for rail transit according to any one of claims 1-4, characterized in that, The LTC modal encoder models the system dynamics using the following differential equation: ； in, Indicates the modal hidden state; The time constant representing the mode; Indicates by parameters Neural networks This represents the input feature vector; The bias vector represents the modality.

6. The liquid-phase cross-attention driven heterogeneous sensing rail transit intelligent interaction system according to claim 5, characterized in that, The forward propagation of the LTC modal encoder is updated using a fused ODE solver, as follows: ； in, This indicates the ODE solver step size. It represents the Hadamardi (or Hadama) stack.

7. The liquid-state cross-attention driven heterogeneous sensing rail transit intelligent interaction system according to claim 1, characterized in that, The unified multimodal representation module includes: The driver state analysis module is used to analyze the fused hidden state. Identify driver fatigue, inattention, and unusual behavior; The driving environment perception module is used to determine the fused hidden state. Detecting obstacles, identifying traffic lights, determining platform alignment, and estimating distances ahead; The voice command parsing module is used to analyze the fusion hidden state. It can recognize voice commands issued by the driver.

8. The liquid-phase cross-attention driven heterogeneous sensing intelligent interaction system for rail transit according to claim 1, characterized in that, The training strategies employed by this system include: Self-supervised pre-training is used to pre-train the LTC modality encoder using an unlabeled multimodal dataset to learn modal feature representations; Joint end-to-end training is used to train the liquid cross-attention fusion layer on a dataset with task labels, so as to learn how to perform effective intermodal information interaction in a specific task. Fine-tuning involves optimizing the modal encoder and multimodal data fusion layer using labeled datasets to adapt to specific interaction scenarios.

9. A method for intelligent interaction using the intelligent interactive system for rail transit according to any one of claims 1-8, characterized in that, include: Collect multimodal raw signal data in the subway driving environment; After preprocessing the acquired multimodal raw signal data, feature extraction is performed on the preprocessed multimodal raw signal data to obtain a sequence of feature vectors for each mode; The modal feature vector sequences are input into the multimodal data fusion layer. The time dependency of each modal feature vector sequence is captured by the LTC modality encoder of the multimodal data fusion layer, and a context-aware modal hidden state is generated. Then, the modal hidden states are fused by the liquid cross-attention fusion layer to obtain the fused hidden state. The fused hidden state is identified, and the driver state analysis result, driving environment perception result, and voice command parsing result are output. It is used to output control commands and provide human-machine interaction feedback based on the driver state analysis results, driving environment perception results, and voice command parsing results.

Images