A vehicle lane-changing recognition method and system based on multi-modal data fusion

Abstract

The application belongs to the field of intelligent transportation, and discloses a vehicle lane-changing recognition method and system based on multi-modal data fusion. In the application, eye movement tracking and multi-modal fusion technology of vehicle kinematics are combined to analyze vehicle motion data, and the statistical characteristics of the vehicle motion data are combined to extract a feature vector A representing the driving habit and stability of the driver. Meanwhile, the eye movement features and the feature vector A are spliced to extract a multi-modal feature vector B focusing on capturing lane-changing. A group classification mechanism based on the experience of the driver is further proposed, the feature vectors A and B are fused, and then the fused features are input into a meta-fusion gate network. The dynamic weight is output through the gate network to realize the classification of the driver based on the experience, and the most suitable expert recognition model is automatically weighted for each driver at each driving moment, so that more accurate and personalized recognition of lane-changing is realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and relates to a vehicle lane change recognition method and system based on multimodal data fusion. Background Technology

[0002] Lane changing is often considered one of the most unpredictable driving tasks and a major cause of traffic congestion and accidents. However, in assessing driver experience, past research has found that even novice drivers may possess sufficient skill to control the vehicle, but sometimes their higher-order cognition is not fully developed, which can lead to accidents. Since cognitive processes are reflected in several aspects of eye-tracking behavior, it is necessary to study the differences in eye movements between drivers of varying experience. Existing research primarily uses eye-tracking data to identify lane-changing intentions. While previous studies have shown differences in eye-tracking patterns between novice and experienced drivers, these studies have not yet explored how these differences manifest specifically in identifying lane-changing intentions between the two types of drivers.

[0003] With the rapid development of intelligent connected vehicle technology, intelligent lane-changing behavior has become a core function of Advanced Driver Assistance Systems (ADAS) and Autonomous Driving Systems (AD), and a key link in realizing human-machine co-driving and intelligent transportation. However, existing models are static and rigid, unable to adapt to the behavioral differences of drivers with different experience levels, and have weak generalization ability. In addition, the existing model fusion mechanism is shallow and fails to deeply explore the temporal coordination relationship between eye movement and vehicle control, limiting the ability of early recognition and prediction. Therefore, there is an urgent need for a lane-changing behavior recognition method that can dynamically perceive driver characteristics and deeply integrate cognitive and executive information to meet the comprehensive requirements of safety, personalization, and efficiency of next-generation intelligent transportation systems.

[0004] In summary, existing vehicle lane change recognition methods still have the following shortcomings:

[0005] 1. Ignoring Individual Driver Differences: Existing lane change recognition methods do not adequately consider the differences in individual driving experience among drivers. In reality, drivers with varying levels of experience make decisions differently when changing lanes. Novice drivers, due to their lack of experience, experience a higher cognitive load and make more cautious decisions when changing lanes. Experienced drivers, on the other hand, with their rich driving experience, will make more decisive lane changes based on road conditions and their own vehicle handling skills. Ignoring these differences between novice and experienced drivers leads to inaccurate lane change recognition and consequently, poor generalization ability of the lane change model.

[0006] 2. Limited data sources: Existing vehicle lane change recognition methods often rely excessively on single indicators such as eye movements or vehicle features for data collection and analysis. Judging whether a vehicle has changed lanes based solely on these limited feature data often leads to significant biases, easily resulting in inaccurate judgments or even serious misjudgments.

[0007] 3. Limited Recognition Accuracy: Existing lane change recognition methods have significant limitations in accuracy, making it difficult to meet the practical application requirements of advanced driver assistance systems (ADAS). Despite employing advanced ensemble learning algorithms, the current best recognition accuracy hovers only around 79%. This is due, in part, to the relatively outdated learning algorithms used, and in part to the neglect of necessary driver group classification based on driving experience. This low recognition accuracy makes it difficult for related models to move from academic research to practical engineering deployment, failing to provide a reliable technical foundation for precise, personalized driver assistance. Summary of the Invention

[0008] The purpose of this invention is to propose a vehicle lane change recognition method based on multimodal data fusion. This method constructs an adaptive meta-fusion network for vehicle lane change recognition. In the adaptive meta-fusion network, this invention integrates eye-tracking and vehicle kinematics data and introduces a real-time dynamic driver experience classification mechanism. By deeply integrating driver experience classification and lane change recognition into an end-to-end adaptive learning framework, the most suitable expert model can be automatically weighted for each driver at each driving moment, thereby achieving more accurate and personalized lane change recognition.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A vehicle lane change recognition method based on multimodal data fusion includes the following steps:

[0011] Step 1. Synchronously collect vehicle motion and driver eye movement data, and perform preprocessing operations;

[0012] Step 2. Construct an adaptive meta-fusion network for vehicle lane change recognition, which includes a global empirical feature extraction branch, a lane change feature extraction branch, a feature fusion module, a gating network, a dynamic expert selection module, and an adaptive weighted fusion module;

[0013] The global empirical feature extraction branch is used to receive multi-dimensional vehicle motion time-series data input, and uses a lightweight temporal convolutional network TCN to extract features, followed by global average pooling, and then integrates vehicle statistical features to generate a global empirical feature vector A.

[0014] Vehicle statistical characteristics are directly calculated from multidimensional vehicle motion time-series data;

[0015] The lane change feature extraction branch receives eye-tracking features and global empirical feature vector A and concatenates them. It then uses the Transformer encoder to extract features to capture the long-range dependence between eye-tracking and vehicle movement. Finally, it performs global average pooling to output feature vector B.

[0016] Eye movement features are obtained from preprocessed eye movement data through feature extraction.

[0017] The feature fusion module concatenates the global empirical feature vector A and feature vector B to obtain the fused feature, and inputs the fused feature into the gating network to generate expert dynamic weight vectors α1 and α2, where α1+α2=1;

[0018] Simultaneously, vehicle statistical features and eye-tracking features are concatenated and input into the dynamic expert selection module, and predictions are made using novice expert models and experienced expert models respectively to obtain preliminary predicted lane-changing probabilities P1 and P2.

[0019] The adaptive weighted fusion module calculates the lane-changing probability after dynamic weighted fusion based on parameters α1, α2, P1, and P2, and then compares it with a preset lane-changing probability threshold to achieve vehicle lane-changing recognition.

[0020] Furthermore, based on the aforementioned vehicle lane change recognition method based on multimodal data fusion, this invention also proposes a corresponding vehicle lane change recognition system based on multimodal data fusion, which adopts the following technical solution:

[0021] A vehicle lane change recognition system based on multimodal data fusion includes the following modules:

[0022] The data acquisition and preprocessing module is used to simultaneously collect vehicle motion and driver eye movement data and perform preprocessing operations.

[0023] The prediction module constructs an adaptive meta-fusion network for vehicle lane change recognition, which includes a global empirical feature extraction branch, a lane change feature extraction branch, a feature fusion module, a gating network, a dynamic expert selection module, and an adaptive weighted fusion module.

[0024] The global empirical feature extraction branch is used to receive multi-dimensional vehicle motion time-series data input, and uses a lightweight temporal convolutional network TCN to extract features, followed by global average pooling, and then integrates vehicle statistical features to generate a global empirical feature vector A.

[0025] Vehicle statistical features are calculated from preprocessed vehicle motion data;

[0026] The lane change feature extraction branch receives eye-tracking features and global empirical feature vector A and concatenates them. It then uses the Transformer encoder to extract features to capture the long-range dependence between eye-tracking and vehicle movement. Finally, it performs global average pooling to output feature vector B.

[0027] Eye movement features are obtained from preprocessed eye movement data through feature extraction.

[0028] The feature fusion module concatenates the global empirical feature vector A and feature vector B to obtain the fused feature, and inputs the fused feature into the gating network to generate expert dynamic weight vectors α1 and α2, where α1+α2=1;

[0029] Simultaneously, vehicle statistical features and eye-tracking features are concatenated and input into the dynamic expert selection module, and predictions are made using novice expert models and experienced expert models respectively to obtain preliminary predicted lane-changing probabilities P1 and P2.

[0030] The adaptive weighted fusion module calculates the lane-changing probability after dynamic weighted fusion based on parameters α1, α2, P1, and P2, and then compares it with a preset lane-changing probability threshold to achieve vehicle lane-changing recognition.

[0031] Furthermore, based on the aforementioned vehicle lane change recognition method based on multimodal data fusion, this invention also proposes a computer device comprising a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it implements the steps of the aforementioned vehicle lane change recognition method based on multimodal data fusion.

[0032] Furthermore, based on the aforementioned vehicle lane change recognition method based on multimodal data fusion, this invention also proposes a computer-readable storage medium storing a program thereon, which, when executed by a processor, is used to implement the steps of the aforementioned vehicle lane change recognition method based on multimodal data fusion.

[0033] The present invention has the following advantages:

[0034] As described above, this invention discloses a vehicle lane change recognition method and system based on multimodal data fusion. This method constructs an adaptive meta-fusion network (AMF-Net) for vehicle lane change recognition, introducing a real-time dynamic driver experience classification mechanism to seamlessly couple driver experience classification with vehicle lane change recognition. First, a lightweight temporal convolutional network analyzes vehicle kinematic data in real time. A dual-branch structure extracts feature vector A representing driver operating habits and stability, and feature vector B focusing on lane change detection. Then, feature vectors A and B are fused, and the fused features are input into a meta-fusion gating network to achieve experience-based driver classification. This dynamically determines whether the driver's current state is "novice" or "experienced," and adaptively adjusts the expert dynamic weight vector. Dynamic experience modeling helps overcome the limitations of static recognition, resulting in higher recognition accuracy. Furthermore, this invention employs a pre-trained expert database + online dynamic selection strategy, significantly reducing online computation compared to traditional deep learning models while maintaining decision accuracy. This invention employs a triple-layered protection mechanism: a dynamic expert selection module for initial lane-change probability prediction, an adaptive weighted fusion module for probability fusion, and a high-confidence threshold. This ensures that lane-change decisions are triggered only when sufficient evidence is available, and multi-level verification guarantees decision reliability and enhanced security. The method of this invention, through a technical path of dynamic experience perception → multimodal deep fusion → intelligent expert selection → end-to-end optimization, achieves a paradigm shift in vehicle lane-change recognition from "static and singular" to "dynamic and personalized," establishing an intelligent recognition system capable of understanding driver states, adapting to complex scenarios, and balancing accuracy and efficiency. This provides a reliable technical foundation for the practical deployment of advanced driver assistance systems, autonomous driving systems, and vehicle-to-infrastructure (V2I) environments. Attached Figure Description

[0035] Figure 1 This is a flowchart of a vehicle lane change recognition method based on multimodal data fusion in an embodiment of the present invention;

[0036] Figure 2 This is a processing block diagram of the adaptive meta-fusion network AMF-Net for vehicle lane change recognition in an embodiment of the present invention;

[0037] Figure 3 This is a structural diagram of the TCN network in an embodiment of the present invention;

[0038] Figure 4 This is a network structure diagram of a single residual block in an embodiment of the present invention;

[0039] Figure 5 This is a flowchart of the lane change feature extraction branch in an embodiment of the present invention. Detailed Implementation

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0041] Example 1

[0042] This embodiment 1 describes a vehicle lane change recognition method based on multimodal data fusion. This method constructs an adaptive meta-fusion network (AMF-Net) for vehicle lane change recognition. AMF-Net introduces a group classification mechanism based on driver experience, deeply integrating driver experience classification and lane change recognition into an end-to-end, adaptive learning framework. Specifically, this invention combines eye-tracking and vehicle kinematics multimodal fusion technology. It analyzes vehicle motion data through a lightweight temporal convolutional network and, combined with the statistical characteristics of the vehicle motion data, extracts a feature vector A representing the driver's operating habits and stability. Simultaneously, it fuses eye-tracking features with feature vector A to extract a multimodal feature vector B that focuses on capturing lane changes, providing a data foundation for subsequent output of weights for different types of drivers and lane change probabilities for different groups. Then, feature vectors A and B are fused, and the fused features are input into a meta-fusion gating network. The gating network outputs weight coefficients α=[α1,α2], ∑α i =1, i=1,2, to achieve experience-based driver classification: when the operation is unfamiliar and the eye movement is hesitant, the weight of the novice model is significantly increased (α1 is the largest), and the driver is judged to be close to the novice category; when the operation is smooth and the eye movement is effective, the weight of the skilled model is significantly increased (α2 is the largest), and the driver is judged to be close to the skilled category. This allows for the dynamic and automatic weighting of the most suitable expert recognition model for each driver at each driving moment, thereby achieving more accurate and personalized recognition of vehicle lane changes.

[0043] like Figure 1 As shown, the vehicle lane change recognition method based on multimodal data fusion in this embodiment 1 includes the following steps:

[0044] Step 1. Synchronously collect vehicle motion and driver eye movement data, and perform preprocessing operations.

[0045] Step 1.1. Data Acquisition.

[0046] This embodiment preferably uses a combination of a driving simulator and an eye tracker to simultaneously collect vehicle motion and driver eye movement data.

[0047] The driving simulator system used mainly includes a vehicle control system (including accelerator pedal, steering wheel, and gear shift lever), a three-screen scene display, and an audio system. It can record vehicle operation data (such as speed, acceleration, yaw rate, etc.) and driver operation data (such as steering wheel angle, throttle opening, etc.) in real time at a sampling frequency of 50Hz.

[0048] Eye-tracking data was collected using the Tobii Pro Glasses 3, a device that features wireless connectivity and a lightweight design to minimize interference with the subjects and records eye movements at a frequency of 50Hz.

[0049] All driver participants met the following criteria before the trial: no history of medication use within the past month, no alcohol consumption within 24 hours, and no consumption of coffee, strong tea, or energy drinks within 12 hours. Prior to the formal trial, a questionnaire was used to record basic information about all driver participants, including gender, age, napping habits, and driving experience.

[0050] The driving scenario was constructed using Silab software, simulating a 20-kilometer-long, six-lane, two-way circular highway with a standard lane width of 3.75 meters, consisting of straight sections, transition curves, and circular curve sections.

[0051] To achieve precise time synchronization of multimodal data, a same-frequency (50Hz) acquisition strategy was adopted, and the HRT visualization platform was used to receive data streams from different sources in real time, ensuring that the time steps of vehicle data and eye-tracking data were the same.

[0052] Each person collected data for 30 minutes, and the final standardized vehicle dataset is shown in Table 1.

[0053] HRT (Human Research Tool) Visualization Platform: HRT includes modules for eye-tracking data analysis and motion trajectory analysis, and can integrate third-party data for analysis. It can visualize vehicle data and eye-tracking data.

[0054] The collected data is divided into offline data and real-time data. Offline data: The experiment recruited 33 participants with valid driver's licenses and good health, including 24 men and 9 women.

[0055] The participants were aged between 23 and 60 years old (mean = 29.03, standard deviation = 8.35) and had driving experience between 3 and 30 years (mean = 5.97, standard deviation = 4.35). Based on driving experience, 21 participants with 3 years or less of driving experience were divided into a novice driver group, and the remaining 12 participants with more than 4 years of driving experience were divided into an experienced driver group, which was used to train the offline expert database.

[0056] Table 1 Vehicle Data

[0057]

[0058] Step 1.2. Data preprocessing, including data straightening and alignment.

[0059] I.1. Data Repair.

[0060] During data acquisition, factors such as momentary sensor malfunction, signal transmission interference, or extreme driver actions can lead to missing values ​​(recorded as NaN or Null) and outliers (values ​​significantly exceeding reasonable physical ranges) in the data. These defects can severely interfere with model training and must be corrected; therefore, outlier identification and removal are necessary.

[0061] This invention sets reasonable upper and lower thresholds based on the physical meaning of the data. Data points exceeding the upper and lower thresholds can be regarded as outliers. These identified outlier points will be marked as data points to be repaired, just like missing values.

[0062] I.2. Linear interpolation repair.

[0063] For all points to be repaired (including missing values ​​and removed outliers), linear interpolation is used for filling. It is assumed that the change of physical quantities is approximately linear within a short time interval, and the value of the repaired point is estimated using adjacent normal data points.

[0064] The linear interpolation formula is as follows: assuming at the timestamp and ( < There are normal data values ​​at this location. and For timestamps Missing values ​​at Its estimated value for: .

[0065] Among them, y i Time point to be repaired The estimated value; , They are respectively with The values ​​of the adjacent preceding and following normal data points; , respectively with , The corresponding timestamp, .

[0066] II. Timing synchronization, multimodal data alignment based on hardware timestamps.

[0067] This invention relates to two separate data acquisition devices: a driving simulator (vehicle data) and a Tobii eye tracker (eye-tracking data). Although they operate at the same sampling frequency (50Hz), their data streams are not synchronized at the initial moment, resulting in an initial time difference. To achieve accurate fusion of cross-modal features, strict timing synchronization between the two is necessary.

[0068] This invention employs a hardware timestamp mechanism to achieve high-precision synchronization. At the start of the experiment, a unified hardware clock simultaneously sends a precise start pulse signal to both the driving simulator data acquisition system and the Tobii eye tracker. Upon receiving this signal, both systems record this absolute start time point (i.e., the hardware timestamp) in their respective data streams.

[0069] All collected data points are marked with their offset (in milliseconds or microseconds) relative to the hardware's start timestamp, thereby unifying the timelines of the two independent systems onto the same absolute time base.

[0070] On a unified timeline, for each point in time in the eye-tracking data, the vehicle data values ​​at the same or nearest time are precisely located. The HRT visualization platform allows for real-time monitoring and ensures the accuracy of this alignment process.

[0071] Step 2. Build an Adaptive Meta-Fusion Network (AMF-Net) for vehicle lane change recognition. AMF-Net is an end-to-end deep learning architecture designed for personalized temporal data recognition tasks.

[0072] AMF-Net dynamically integrates multiple pre-trained expert models through meta-learning and adaptively assigns weights to each expert in the final decision based on the real-time features of the input data, thereby achieving accurate and robust recognition.

[0073] like Figure 2 As shown, the AMF-Net network in this embodiment includes a global empirical feature extraction branch, a lane-changing feature extraction branch, a feature fusion module, a gating network, a dynamic expert selection module, and an adaptive weighted fusion module.

[0074] The feature extraction layer adopts a dual-branch parallel structure to process input modalities with different physical meanings. The first branch is the global empirical feature extraction branch, and the second branch is the lane-change feature extraction branch.

[0075] The global empirical feature extraction branch is used to receive multi-dimensional vehicle motion time series data input.

[0076] The global experience feature extraction branch uses a lightweight temporal convolutional network (TCN) to extract deep features that characterize the driver's operational stability and habits. It is then followed by global average pooling and fused with vehicle statistical features to generate a global experience feature vector A.

[0077] Among them, vehicle statistical characteristics can be directly calculated from multidimensional vehicle motion time series data.

[0078] The lane change feature extraction branch receives eye-tracking features and global empirical feature vector A and concatenates them. It uses the self-attention mechanism of the Transformer encoder to capture long-range temporal dependencies across modalities and extracts instantaneous collaborative features that directly represent the germination of intent to capture the long-range dependencies between eye-tracking and vehicles. This is followed by global average pooling to output feature vector B.

[0079] The eye movement features are obtained from the preprocessed eye movement data through feature extraction.

[0080] The feature fusion module concatenates the global empirical feature vector A and feature vector B to obtain the fused feature, and inputs the fused feature into the gating network to generate expert dynamic weight vectors α1 and α2, where α1+α2=1.

[0081] Where α1 and α2 represent the probabilities that the real-time input data sample belongs to a novice driver and an experienced driver, respectively.

[0082] Simultaneously, vehicle statistical features and eye-tracking features are concatenated and input into the dynamic expert selection module, and predictions are made using novice expert models and experienced expert models respectively to obtain preliminary predicted lane-changing probabilities P1 and P2.

[0083] Where P1 and P2 are the lane-changing probabilities of real-time input data samples as novice and experienced drivers, respectively.

[0084] The adaptive weighted fusion module calculates the lane-changing probability after dynamic weighted fusion based on parameters α1, α2, P1, and P2, and then compares it with a preset lane-changing probability threshold to achieve vehicle lane-changing recognition.

[0085] This invention deeply integrates the two coupled tasks of driver real-time experience classification and lane change recognition into a unified, end-to-end learning framework. Through its internal dual-branch encoder, it extracts long-term features characterizing the stability of driving behavior and instantaneous multimodal features predicting the emergence of lane changes. Furthermore, through its meta-fusion gating network, it can infer the driver's state tendency in real time based on these features and dynamically adjust the discourse power of different expert models in subsequent recognition tasks.

[0086] This mechanism makes the AMF-Net network proposed in this invention no longer a single, static model, but an intelligent agent that can "differentiate itself" and "change with time," thereby overcoming the inherent defects of existing technologies that ignore individual differences among drivers and have weak model generalization ability. Ultimately, it achieves a significant improvement in lane change recognition accuracy and robustness, providing more accurate and reliable personalized decision support for fields such as advanced driver assistance systems (ADAS), autonomous driving, and vehicle-road cooperation.

[0087] The following provides a detailed description of each module in the AMF-Net network in this embodiment.

[0088] The processing flow of the global empirical feature extraction branch is as follows:

[0089] First, input multiple vehicle kinematics time-series variables, with the data format being a data set of length L and dimension D. in Multivariate time series segment, where L is the time step length, D in Let X be the input feature dimension, mathematically represented as X∈R. (L × Din ) .

[0090] In this embodiment, the input dimension D in =19, which refers to the 19 vehicle kinematic time-series variables listed in Table 1. The data format is a string with length L=90000 and dimension D. in = 19 multivariate time series segment.

[0091] At a sampling frequency of 50Hz, there are 50 sampling points per second. Assuming that the real-time data acquisition time in this embodiment is 30 minutes, 30*60*50=90000, that is, L=90000.

[0092] X is the input of multidimensional vehicle motion time series data as the global empirical feature extraction branch.

[0093] X is input into the TCN network to extract features, capturing driver features hierarchically. For example... Figure 3 As shown, TCN is designed to be lightweight, so the number of channels is kept small and the number of residual blocks is small to reduce computational complexity.

[0094] Temporal Convolutional Network (TCN) is a convolutional neural network architecture specifically designed for processing temporal data. It effectively applies the powerful feature extraction capabilities of traditional CNNs in image processing to one-dimensional temporal signals through special structural design.

[0095] The specific parameters of the TCN network are as follows: the initial convolutional layer output channels are C=64, the number of residual blocks is N=4, and the kernel size is k=3. All convolutional layers use ReLU activation, weight normalization (WeightNorm), and residual connections use identity mapping, requiring no additional convolution to adjust dimensions.

[0096] The input to TCN is a segment of multivariate time series data with dimensions L×D. in The input data is X∈R (L × Din) .

[0097] The processing flow of the TCN network is as follows:

[0098] The input projection layer of TCN uses a one-dimensional causal convolutional layer with a kernel size of k=3, a dilation coefficient of d=1, and 64 output channels C. Puffing p=(k-1)×d=2 is applied to the left of the beginning of the sequence to ensure the output sequence length matches the input length, with a total length of 90000. Here, p represents the padding size, i.e., how many zeros or dummy data points are added to the left of the input sequence to ensure the length of the output sequence after causal convolution is the same as the length of the input sequence. The calculation formula is as follows:

[0099] ;

[0100] in, ∈R 90000×64 It is the output sequence.

[0101] ∈R 64×19×3 It is the convolution weight, b (0) ∈R 64 It is the bias; t is the time index. This represents the value of the input data to be read at time step t - i·d when calculating the output at time t, where t = 1, 2, ..., 90000.

[0102] The TCN contains N=4 residual blocks, each with an index i=1,2,3,4. Each residual block has the same structure but a different inflation coefficient d. i =2i-1, that is, d1=1, d2=2, d3=4, d4=8, and the number of input and output channels for each residual block is C=64.

[0103] like Figure 4 As shown, the operation of a single residual block (the i-th residual block) is taken as an example:

[0104] Residual block input Z (i-1) ∈R 90000×64 (Output from the previous layer, initially Z) (0) =Y (0) The specific steps are as follows:

[0105] First dilated causal convolutional layer:

[0106] Kernel size k=3; Dilation coefficient d i Left-filling p=(k-1)×d i =2; Calculation formula:

[0107] ;

[0108] in ∈R 64×64×3 It is the weight, b (i,1) ∈R 64It's the bias, the output. ∈R 90000×64 , Weight normalization is a technique for reparameterizing the weights of a neural network.

[0109] The second dilated causal convolutional layer has the same parameters as the first dilated causal convolutional layer, and its calculation formula is as follows:

[0110] ;

[0111] in ∈R 64×64×3 b (i,2) ∈R 64 Output ∈R 90000×64 .

[0112] Residual connection, input Z (i-1) With output The formula for addition is as follows:

[0113] ;

[0114] Output ∈R 9000×64 .

[0115] After 4 residual blocks, the final output is... ∈R 90000×64 .

[0116] After extracting the TCN deep features using TCN, global average pooling (GAP) is then used to average the time dimension L, aggregating the deep features extracted by the TCN network into a fixed-length vector V. gap .

[0117] Of course, the TCN network in this embodiment can also be replaced with recurrent neural networks such as LSTM, Bi-LSTM (bidirectional LSTM), and GRU, which can achieve the same purpose. It will not be described in detail here.

[0118] Global Average Pooling (GAP) is an extremely simple pooling operation that compresses the spatial or temporal dimensions of the entire feature map, directly converting it into a single scalar value.

[0119] Global average pooling GAP Averaging is performed to aggregate the temporal features into a fixed-length vector, preserving overall pattern information. The calculation formula is as follows: .

[0120] Z t (4) ∈R64 It is the feature vector at time t, and the output is V. gap ∈R 64 .

[0121] Then, for each variable in the input X, four statistics are calculated, including the maximum, minimum, mean, and variance, and interpretable statistical features V are further extracted. stats This is to supplement the deep features of TCN.

[0122] Maximum value The calculation formula is: ;

[0123] Minimum value The calculation formula is: ;

[0124] mean The calculation formula is: ;

[0125] variance The calculation formula is: .

[0126] Where i represents the time index, with a value range from 1 to L; j represents the variable index, with a value range from 1 to 19; This represents the original data value, which is the instantaneous measurement value of the vehicle variable at the i-th time point.

[0127] In this embodiment, four statistical measures are calculated for each variable in the input X using MATLAB software. First, the vehicle's time-series data, sampling frequency, and feature index to be extracted are input. Then, the time-domain features of the signal are calculated, including maximum, minimum, mean, and variance. Finally, the corresponding feature values ​​and their names are output according to the feature index. The 4 × 19 = 76 feature values ​​are concatenated into a vector in variable order, forming a 76-dimensional interpretable statistical feature V. stats .

[0128] Finally, the vector V output by global average pooling is... gap With statistical characteristics V stats By concatenating the vectors, we obtain the global empirical feature vector A:

[0129] ;

[0130] The final output feature vector A∈R 140 (64 + 76 = 140 dimensions).

[0131] In this embodiment, the eye tracker collects eye movement variables based on the Tobii Pro Glasses 3 device. The eye movement variables are time-series data and do not inherently distinguish between lane changing and lane keeping states of the vehicle.

[0132] Based on this, the present invention utilizes Tobii Pro Lab software to import the variable data collected by the device along with synchronized eye movement trajectory video into the software. The lane-changing and lane-keeping phases are then manually labeled within the software based on the eye movement trajectory video.

[0133] To determine the lane-changing time window, a 10-second window was established, extending five seconds forward and five seconds backward from the lane ID switching moment in the driving simulator as a reference point. Based on this method, nine eye-tracking features were extracted for each participant from the lane-changing and lane-keeping time windows, as detailed in Table 2.

[0134] Table 2 Tobii's eye movement characteristics

[0135]

[0136] The nine eye-tracking features obtained are used to form an eye-tracking feature sequence E by feature duplication and splicing.

[0137] The lane change feature extraction branch takes eye-tracking features and global empirical features A as inputs and concatenates them. A Transformer encoder is selected, and a self-attention mechanism is used to capture the long-range dependence of eye-tracking on the vehicle. The output feature vector B focuses on the lane change signal.

[0138] like Figure 5 As shown, the processing flow of the lane change feature extraction branch is as follows:

[0139] First, input the global empirical feature vector A and the eye-tracking feature sequence E. Then, copy the global empirical feature vector A in the time dimension to expand it into a global empirical feature sequence A of the same length as E. seq .

[0140] Then A seq E is concatenated with E along the feature dimension to form a unified, multimodal input sequence X. fused .

[0141] Based on manually labeled lane changing and lane keeping phases, nine eye movement features were extracted from the raw eye movement data for each sample, forming a time series. For a sequence of length L... X The sample size = 500 is denoted as E∈R. (500×9) .

[0142] Both the driving simulation test platform and the Tobii Pro Glasses3 have a sampling frequency of 50Hz, meaning they collect 50 data points per second. Therefore, for each 10-second time window, the sequence length L... x The calculation formula is: L x =10 seconds × 50 Hz = 500.

[0143] The system operates in real time using a sliding window approach. For each 10-second short window to be analyzed, a feature vector A is calculated from the current and previous long sequences, and then fused with the eye-tracking features of the current 10-second window to perform vehicle lane-changing recognition.

[0144] The global empirical feature vector A is a fixed-length vector A∈R 140 .

[0145] The global empirical feature vector A encodes the driver's operational stability and habits throughout the entire time window. To effectively fuse the global feature A with the eye-tracking time sequence E, a feature duplication and concatenation method is employed.

[0146] Specifically, the global feature vector A is copied along the time dimension and expanded into a sequence A of the same length as the eye-tracking sequence E. seq The calculation formula is as follows: A seq =Repeat(A,500)∈R (500×140) .

[0147] The copied global empirical feature sequence A is then... seq The sequence is concatenated with the eye-tracking feature sequence E along the feature dimension to form a unified, multimodal input sequence, which is ultimately input to X. fused =Concat(A seq ,E)∈R (500×(140+9)) =R (500×149) .

[0148] Capture using the Transformer encoder to capture X fused Linear projection onto high-dimensional space Z0∈R (Lx×Dmodel) By adding positional encoding PE, a comprehensive feature H0 containing both feature and temporal information is obtained, providing a unified feature representation and temporal information for the model.

[0149] The hidden dimension D of the Transformer encoder model model It is a configurable hyperparameter whose value needs to be balanced between the model's representational ability and computational complexity. Typically, values ​​that are powers of 2, such as 256, 384, or 512, are chosen.

[0150] The Transformer encoder is a deep learning architecture that utilizes self-attention to efficiently process sequential data. It can directly capture the global dependencies between any two elements in a sequence and has extremely high parallel computation efficiency.

[0151] In the AMF-Net of this invention, the Transformer encoder is used as the core of the multimodal intent feature encoder, responsible for extracting feature vector B rich in long-range, cross-modal semantic information from the fused eye-tracking and vehicle data.

[0152] Setting D model =256, which ensures high-precision intent recognition while also having good computational efficiency, making it suitable for practical deployment.

[0153] The synthesized feature H0 is fed into a Transformer consisting of N layers of encoders stacked together.

[0154] In each layer, the self-attention mechanism performs the following key operation: for each time point in the sequence, it obtains an attention score by calculating the similarity between its query vector and the key vectors of all time points in the sequence.

[0155] After multiple layers of processing, the Transformer encoder outputs sequence H. N ∈R (Lx×Dmodel) The output sequence H N Each feature at any given time point contains a deeply enhanced representation of the entire sequence context.

[0156] The final output sequence H of the Transformer encoder N Along the time dimension L x Perform global average pooling to compress the complex dynamic information of the entire sequence into a single vector v. global :

[0157] .

[0158] This operation compresses the complex dynamic information of the entire sequence into a single vector v. global ∈R (Dmodel) It represents global lane change information that integrates overall driving experience with instantaneous eye-tracking patterns.

[0159] The global vector v global The final feature vector B is obtained by performing dimensionality reduction and nonlinear activation through a fully connected layer.

[0160] Next, an expert database needs to be built. Before system deployment, the expert model database needs to be built offline in advance.

[0161] To establish a high-quality static expert database, this invention collected eye movement data from 33 drivers (including 22 novices and 11 experienced drivers) during simulated driving, and extracted a total of 30,140 valid eye movement feature samples.

[0162] Among them, there were 20,682 novice drivers and 9,458 experienced drivers.

[0163] Based on the above samples, three high-performance machine learning models, namely XGBoost, LightGBM and CatBoost, were trained for the two types of drivers to evaluate their performance in the lane change intention recognition task.

[0164] XGBoost is a scalable, end-to-end tree boosting system. Through parallel processing and regularization techniques, it pushes the performance and efficiency of gradient boosting frameworks to the extreme, becoming one of the de facto standard models in machine learning competitions and industry.

[0165] LightGBM, developed by Microsoft, is a gradient boosting framework focused on faster training speeds and lower memory consumption. Its design goal is to address the efficiency challenges of model training on large-scale datasets.

[0166] CatBoost is a gradient boosting algorithm specifically designed for efficient, preprocessing-free handling of categorical features.

[0167] The model performance evaluation results show that: on the novice driver data, the classification accuracy of the XGBoost model is 96.42%, LightGBM is 82.35%, and CatBoost is 86.11%; on the experienced driver data, the accuracy of the XGBoost model is 90.62%, LightGBM is 85.29%, and CatBoost is 87.50%.

[0168] Overall, XGBoost performed best across both driver groups and was therefore selected as the base model for building the expert database. During the online inference phase, the expert database parameters were kept frozen to ensure the stability and consistency of the system's decisions.

[0169] Two expert models (a novice expert and a skilled expert based on XGBoost) perform parallel predictions on real-time input samples, respectively outputting the initial lane-changing probability P corresponding to their experience categories. i .

[0170] The offline training process for the novice expert model and the skilled expert model in this embodiment is as follows:

[0171] First, an expert model library is built offline in advance, using the XGBoost model as the base model for building the expert model library.

[0172] Vehicle motion data and driver eye movement data were collected from multiple novice and experienced drivers during the simulated driving process, and the synchronously collected vehicle motion data and driver eye movement data were preprocessed.

[0173] In this embodiment, the offline training library data also undergoes the aforementioned data preprocessing and feature extraction processes. The difference is that real-time data is collected in a real driving environment and also requires data preprocessing.

[0174] The statistical features of the vehicle and the eye movement features of the driver were extracted from the preprocessed vehicle motion data and driver eye movement data respectively, and the two were concatenated. Based on the concatenated features, the dataset was divided into a novice driver dataset and a skilled driver dataset.

[0175] The XGBoost model was trained using a novice driver dataset and a skilled driver dataset to obtain a trained novice expert model and a skilled expert model, respectively. The two expert models were then deployed in practice.

[0176] During the inference phase, the vehicle statistical features and eye-tracking features collected and processed in real time are concatenated and input into the novice expert model and the skilled expert model, respectively, to obtain the lane-changing probabilities P1 and P2 predicted by the novice and skilled expert models.

[0177] Where P1 and P2 represent the lane-changing probabilities of real-time input data as novice drivers and skilled experts, respectively.

[0178] When the system is running in real time, for each new segment of driving data, the following steps are performed:

[0179] The feature vectors A and B obtained from the global empirical feature extraction branch and the lane change feature extraction branch are concatenated, and F... fused =CONCAT(A,B), which will set F fused The input is fed into the gating network to obtain the expert weight vector α=[α1,α2].

[0180] Where α1 and α2 represent the probabilities that the real-time input data belongs to a novice driver and an experienced driver, respectively.

[0181] A gating network is a special type of neural network structure whose core function is to act as a "dynamic decision-maker" or "intelligent weight allocator." By analyzing the current input features, it automatically learns and generates a set of weight coefficients. This set of weights determines which (or a combination of) of the multiple downstream expert models or processing paths has the greatest influence on the final decision. In this invention, the gating network is the core computational unit of the dynamic expert selection module.

[0182] The gated network takes the concatenated fused feature vector [A, B] as input and outputs a normalized weight vector α=[α1,α2], where ∑α i =1. At the meta-learning level, this weight vector represents the probability that a real-time input sample belongs to a novice driver or an experienced driver, thus determining the confidence level of the corresponding expert model.

[0183] The extracted features (a concatenation of eye-tracking features and vehicle statistical features) are input into offline, prepared novice and expert expert models, respectively. The two experts then perform forward inference and output their respective lane-changing probabilities.

[0184] The output probability of a novice expert is: P1 = P(lane change | novice model), and the output probability of a skilled expert is: P2 = P(lane change | skilled model).

[0185] The adaptive weighted fusion module, based on the input dynamic weights α and the predicted probabilities of the current time-series data from the expert database, ultimately obtains the output lane-changing probability P through dynamic weighted fusion. final :P final =α1×P1+α2×P2.

[0186] The lane-changing probability threshold is set to β. The lane-changing probability threshold β is a configurable parameter, and its value is determined by the performance of the model on the validation set. It is used to balance the false alarm rate and the false alarm rate of the system in order to achieve the optimal early warning effect.

[0187] In this embodiment, β=0.65 can be set based on model verification.

[0188] If the probability of changing lanes is P final If the value is ≥β, it is determined that the vehicle is changing lanes; otherwise, it is determined that the vehicle is maintaining its lane.

[0189] Drivers with different experience levels exhibit systematic differences in lane-changing decisions. By using dynamic weights α1 and α2, the contributions of the two expert models can be adaptively adjusted based on the real-time identification of driver experience tendencies.

[0190] In addition, weighted averaging can balance the prediction uncertainties of each model and reduce the systemic risk caused by misjudgment by a single model.

[0191] This embodiment combines the dynamic weights representing experience categories generated by the gating network with the lane-changing prediction P of each experience category generated by the expert database. i The final lane change recognition probability is generated by performing weighted summation. The fundamental reason for this design is that the driver's experience level is not a static label, and their behavior pattern will change dynamically depending on the state and scenario.

[0192] The AMF-Net network uses a meta-fusion gating network to evaluate the matching degree between behavioral patterns and expert domains in real time. Then, at the weighted decision layer, it organically integrates the opinions of two domain experts, giving the expert who best understands the current driving style the greatest say. This process achieves the goal of dynamically allocating the lane-changing probabilities of the most relevant experts for joint decision-making based on the driver's real-time state.

[0193] Compared with traditional methods, the method of the present invention has at least the following advantages:

[0194] 1. An adaptive decision-making mechanism based on real-time dynamic driver experience classification.

[0195] This invention abandons the static or pre-classification mode and uses a lightweight temporal convolutional network (TCN) to perform millisecond-level analysis on vehicle yaw rate, acceleration and other kinematic data, infer the driver's experience level (such as novice or expert) in real time, and output dynamic weights α1 and α2, thus realizing the leap from "one policy for a thousand people" to "a thousand policies for a thousand people".

[0196] This invention can instantly identify and adapt to different driving styles. For example, when unstable operation is detected, the weight α1 representing the "novice model" is automatically increased, making the system decision more conservative and warning-oriented; otherwise, the "experienced model" is trusted.

[0197] 2. A multimodal feature extraction network structure that deeply integrates eye-tracking and vehicle time-series data.

[0198] This invention designs a dedicated dual-branch feature extractor:

[0199] One branch focuses on extracting implicit empirical features representing operational stability from vehicle data; the other branch deeply fuses eye-tracking features and feature vector A to capture visual attention features during lane changes, thus solving the limitation of a single data source.

[0200] By capturing the deep patterns of "hand-eye" coordination, it is possible to perceive the driver's decision-making motivation earlier and more accurately.

[0201] 3. Gating network design for “Dynamic Expert Selector”.

[0202] This invention designs an intelligent switch-gating network that takes fused features as input and dynamic weight α as output.

[0203] This gating network is essentially a small neural network that learns mapping relationships that automatically and appropriately allocate decision weights between two "experts" based on the current multimodal context. This achieves transparency and optimization of the decision-making process. The design ensures that the weight allocation is not fixed but rather a complex function based on the actual driving situation.

[0204] For example, on highways, experienced experts may be relied upon more heavily; while on complex ramps, the opinions of novice experts are given higher weight. This flexible decision-making mechanism ensures that the weighted fusion accuracy remains consistently above 92% across various scenarios.

[0205] 4. Methods for constructing and calling the "Expert Database".

[0206] In the offline phase, based on eye-tracking data from 30,140 samples from 33 drivers, XGBoost models specifically designed for novice and experienced drivers were trained (with accuracies of 96.42% and 90.62%, respectively), thus constructing a high-quality static expert database.

[0207] During the online phase, two experts work in parallel, and their predicted lane-change probabilities P1 and P2 are weighted and integrated based on dynamic weights. This balances the model's powerful performance with efficient and stable inference. Offline training ensures the high accuracy of the expert model itself, while parameter freezing during online invocation guarantees the real-time performance and reliability of the system. This "offline refinement, online fusion" strategy enables millisecond-level inference speeds while avoiding the instability risks that may arise from online model learning.

[0208] 5. An end-to-end architecture that seamlessly couples driver experience classification with lane change recognition.

[0209] The AMF-Net architecture integrates all modules, including the global experience feature extraction branch, lane-changing feature extraction branch, feature fusion module, gating network, dynamic expert selection module, and adaptive weighted fusion module, into a unified architecture. From multimodal data input to feature extraction, experience classification, weight calculation, expert library invocation, and finally generating fusion decision, the entire process is seamlessly connected within the end-to-end framework, thereby achieving closed-loop optimization from perception to decision.

[0210] In addition, the tight coupling between modules minimizes information loss, and the system can be optimized as a whole rather than a simple splicing of multiple independent parts. This not only improves the robustness of the overall performance, but also greatly facilitates the training and deployment of the model. This adaptive fusion mechanism can significantly improve the generalization ability and robustness of the final model.

[0211] This invention addresses the problems of lagging lane change behavior recognition and lack of differentiated response to driver cognitive state and experience level in existing methods by using multimodal data fusion and driver experience modeling for vehicle lane change recognition. Furthermore, it achieves the following objectives:

[0212] 1. High-precision, early-cycle lane change recognition:

[0213] By analyzing the coordinated changes in visual characteristics such as driver gaze and saccades with kinematic parameters such as vehicle yaw rate and acceleration, lane-changing trends can be predicted in advance, providing a reliable decision-making basis for ADAS and autonomous driving systems.

[0214] 2. Construct an experience-adaptive co-driving interaction mechanism: Dynamically adjust the assistance strategy according to the driver's experience level to achieve proactive intervention for novice drivers and minimal interference for experienced drivers, thereby improving the trust and comfort of human-machine co-driving.

[0215] 3. Support for lane-changing behavior optimization under vehicle-road cooperation:

[0216] In a connected environment, anonymously sharing vehicle lane-changing behavior data to roadside systems supports micro-control and macro-traffic guidance, alleviates traffic conflicts in key areas such as merging zones and construction sections, and improves road network efficiency.

[0217] 4. Promote the personalized and intelligent development of intelligent transportation systems: Deeply integrate driver cognitive state with vehicle behavior to provide a scalable and adaptable behavioral understanding and decision-making framework for next-generation ADAS and autonomous driving systems.

[0218] Example 2

[0219] This embodiment 2 describes a vehicle lane change recognition system based on multimodal data fusion, which is based on the same inventive concept as the vehicle lane change recognition method based on multimodal data fusion in embodiment 1 above.

[0220] The vehicle lane change recognition system based on multimodal data fusion in this embodiment includes the following modules:

[0221] The data acquisition and preprocessing module is used to simultaneously collect vehicle motion and driver eye movement data and perform preprocessing operations.

[0222] The prediction module constructs an adaptive meta-fusion network for vehicle lane change recognition, which includes a global empirical feature extraction branch, a lane change feature extraction branch, a feature fusion module, a gating network, a dynamic expert selection module, and an adaptive weighted fusion module.

[0223] The global empirical feature extraction branch is used to receive multi-dimensional vehicle motion time-series data input, and uses a lightweight temporal convolutional network TCN to extract features, followed by global average pooling, and then integrates vehicle statistical features to generate a global empirical feature vector A.

[0224] Vehicle statistical features are calculated from preprocessed vehicle motion data;

[0225] The lane change feature extraction branch receives eye-tracking features and global empirical feature vector A and concatenates them. It then uses the Transformer encoder to extract features to capture the long-range dependence between eye-tracking and vehicle movement. Finally, it performs global average pooling to output feature vector B.

[0226] Eye movement features are obtained from preprocessed eye movement data through feature extraction.

[0227] The feature fusion module concatenates the global empirical feature vector A and feature vector B to obtain the fused feature, and inputs the fused feature into the gating network to generate expert dynamic weight vectors α1 and α2, where α1+α2=1;

[0228] Simultaneously, vehicle statistical features and eye-tracking features are concatenated and input into the dynamic expert selection module, and predictions are made using novice expert models and experienced expert models respectively to obtain preliminary predicted lane-changing probabilities P1 and P2.

[0229] The adaptive weighted fusion module calculates the lane-changing probability after dynamic weighted fusion based on parameters α1, α2, P1, and P2, and then compares it with a preset lane-changing probability threshold to achieve vehicle lane-changing recognition.

[0230] It should be noted that any content not mentioned in the above-described functional modules of the system described in this embodiment can be referred to the step description of the corresponding method in Embodiment 1 above, and will not be repeated in detail here.

[0231] Example 3

[0232] This embodiment 3 describes a computer device including a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it implements the steps of the vehicle lane change recognition method based on multimodal data fusion described in embodiment 1 above.

[0233] In this embodiment, the computer device can be any device or apparatus with data processing capabilities, and will not be described in detail here.

[0234] Example 4

[0235] This embodiment 4 describes a computer-readable storage medium storing a program that, when executed by a processor, is used to implement the steps of the vehicle lane change recognition method based on multimodal data fusion in embodiment 1 above.

[0236] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc.

[0237] This invention achieves high-precision lane change recognition by fusing eye-tracking and vehicle kinematics data and adaptively modeling based on the driver's experience level. It can be directly integrated into Advanced Driver Assistance Systems (ADAS), autonomous driving systems, and vehicle-to-infrastructure (V2I) environments to enable early and accurate recognition and decision support for driver lane change behavior. By analyzing the duration and scanning pattern of the driver's gaze area, combined with vehicle data such as speed, acceleration, and yaw rate, the system can accurately determine the lane change status and trend. Based on this, the system can provide warnings for inexperienced drivers or provide autonomous driving systems with precise lane change timing and trajectory planning decisions, significantly improving the safety and smoothness of the lane change process. In scenarios where human drivers and autonomous driving systems share vehicle control, this invention can serve as a core interaction bridge. By identifying the driver's experience level in real time, the system dynamically adjusts the intervention strategy and interaction method of the assistance system. For example, for novice drivers, the system can take earlier and more proactive assistance interventions; while for experienced drivers, the system may only provide minimal prompts, respecting their driving habits. This personalized interaction based on cognitive state and driving ability can greatly improve the acceptance, trust, and comfort of the co-driving system. In a vehicle-road cooperative environment, this invention can be extended to road and traffic management centers. Through network communication technology, vehicles can anonymously share their lane-changing decisions and behavioral states with surrounding vehicles and roadside units. The traffic management system can then use this information to perform micro-control and macro-guidance on lane-changing behavior in specific road sections (such as ramp merging areas and highway construction sections), effectively alleviating traffic bottlenecks caused by frequent and conflicting lane-changing behaviors by suggesting optimal lane-changing timing and intervals, thereby improving the overall traffic efficiency and stability of the road network.

[0238] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A lane-changing recognition method based on multi-modal data fusion, characterized in that, Comprising the following steps: Step 1. Synchronously collect vehicle motion and driver eye movement data and perform preprocessing operations; Step 2. Build an adaptive meta-fusion network for vehicle lane changing recognition, which includes a global experience feature extraction branch, a lane changing feature extraction branch, a feature fusion module, a gating network, a dynamic expert selection module, and an adaptive weighted fusion module; The global experience feature extraction branch is used to receive multi-dimensional vehicle motion time series data input and extract features using a lightweight time series convolution network (TCN), followed by global average pooling, and fuse vehicle statistical features to generate a global experience feature vector A; The vehicle statistical features are directly calculated from the multi-dimensional vehicle motion time series data; The lane changing feature extraction branch is used to receive eye movement features and global experience feature vector A and concatenate them, extract features using a Transformer encoder to capture the long-range dependencies of eye movement and vehicle, followed by global average pooling, and output a feature vector B; The eye movement features are obtained from the preprocessed eye movement data through feature extraction; The feature fusion module concatenates the global experience feature vector A and the feature vector B to obtain the fusion features, and inputs the fusion features into the gating network to generate dynamic expert weight vectors α1 and α2, α1+α2=1; Meanwhile, the vehicle statistical features and the eye movement features are concatenated and input into the dynamic expert selection module, and the novice expert model and the skilled expert model are used for prediction respectively to obtain the preliminary predicted lane changing probabilities P1 and P2; The adaptive weighted fusion module calculates the dynamically weighted and fused lane changing probability based on parameters α1, α2, P1, and P2, and compares it with the preset lane changing probability threshold to realize vehicle lane changing recognition.

2. The vehicle lane changing recognition method based on multi-modal data fusion according to claim 1, wherein In the step 2, the processing flow of the global experience feature extraction branch is as follows: First, a plurality of vehicle kinematic time series variables are input, and the data form is a multivariate time series segment with a length of L and a dimension of D in , L is the length of the time step, D in is the input feature dimension, and its mathematical representation is X∈R (L × Din ) ; X is input as the multi-dimensional vehicle motion time series data of the global experience feature extraction branch; X is input to the TCN network for feature extraction to obtain a deep feature of the TCN, and then global average pooling is used to average the time dimension L, and the deep feature extracted by the TCN network is aggregated into a fixed-length vector V gap ; Then four statistics, including maximum, minimum, mean and variance, are calculated for each variable in the input X, respectively, and further extract interpretable statistical features V stats to complement the deep features of TCN; Finally, the vector V output by the global average pooling is obtained gap and the statistical feature V stats are spliced to obtain the global experience feature vector A.

3. The vehicle lane changing recognition method based on multi-modal data fusion according to claim 1, wherein In the step 2, the vehicle motion time series data includes vehicle speed, vehicle longitudinal acceleration, steering wheel angle, vehicle remaining time away from the current lane, distance to the front vehicle, distance to the front vehicle on the left lane, distance to the rear vehicle on the right lane, front vehicle speed, left lane front vehicle speed, right lane rear vehicle speed, vehicle lateral speed, vehicle lateral acceleration, pre-crash time, distance to the front vehicle on the right lane, distance to the rear vehicle, distance to the rear vehicle on the left lane, right lane front vehicle speed, rear vehicle speed, and left lane rear vehicle speed.

4. The vehicle lane changing recognition method based on multi-modal data fusion according to claim 1, wherein In the step 1, a driving simulator and an eye tracker are combined to synchronously collect vehicle motion and driver eye movement data; The eye tracker collects eye movement variables based on the Tobii Pro Glasses 3 device, which are time series data, and uses the Tobii Pro Lab software to import the variable data collected by the device together with the synchronized eye movement trajectory video into the software. In the software, according to the eye movement trajectory video, the lane changing and lane keeping stages are manually labeled, and the lane ID switching time in the driving simulator is taken as the reference point, and a 10-second lane changing time window is established by extending five seconds forward and backward. Nine eye movement features are extracted from the lane changing and lane keeping time window, including the average duration of fixation, the average pupil diameter of all fixation samples, the average peak velocity of saccade, the minimum peak velocity of saccade, the maximum peak velocity of saccade, the standard deviation of all peak velocities of saccade, the average amplitude of all saccades, the minimum amplitude of saccade and the maximum amplitude of saccade.

5. The vehicle lane changing recognition method based on multi-modal data fusion according to claim 1, characterized in that, In step 2, the processing flow of the lane changing feature extraction branch is as follows: First, input the global experience feature vector A and the eye movement feature sequence E, copy the global experience feature vector A in the time dimension, and expand it into a global experience feature sequence A with the same length as E seq ; Subsequently, A seq is concatenated with E in the feature dimension, forming a unified, multi-modal input sequence X fused ; Then the input sequence X is captured by the Transformer encoder, and the output is a comprehensive feature H0 that contains both feature and timing information. fused Linear projection to high-dimensional space Z0, and add position encoding PE, get comprehensive features H0 that contain both feature and timing information. Then, the comprehensive feature H0 is sent to a Transformer stacked with N layers of encoders; In each layer, the self-attention mechanism performs the following key operations: for each time point in the sequence, it obtains an attention score by calculating the similarity between its Query vector and the Key vectors of all time points in the sequence; After multiple layers of processing, the Transformer encoder outputs a sequence H N , where H is the output sequence N Global average pooling is performed along the time dimension L, compressing the complex dynamic information of the entire sequence into a single vector v global ; The global vector v global The final feature vector B is obtained by a fully connected layer for dimension reduction and nonlinear activation.

6. The vehicle lane changing recognition method based on multi-modal data fusion according to claim 1, characterized in that, In step 2, the offline training process of the novice expert model and the skilled expert model is as follows: First, an expert model library is constructed offline, and an XGBoost model is used as the base model for constructing the expert model library; Vehicle motion data and driver eye movement data of multiple novice and skilled drivers during the simulation driving process are collected respectively, and the synchronously collected vehicle motion data and driver eye movement data are preprocessed; From the preprocessed vehicle motion data and driver eye movement data, vehicle statistical features and eye movement features are extracted respectively and spliced, and the data set is divided into a novice driver data set and a skilled driver data set based on the spliced features; The XGBoost model is trained using the novice driver data set and the skilled driver data set respectively, and trained novice expert model and skilled expert model are obtained, and the two expert models are actually deployed; In the inference stage, the spliced vehicle statistical features and eye movement features obtained by real-time collection and processing are input into the novice expert model and the skilled expert model respectively, and the lane changing probabilities predicted by the novice and skilled expert models are obtained.

7. The vehicle lane changing recognition method based on multi-modal data fusion according to claim 1, characterized in that, In step 2, the processing process of the adaptive weighted fusion module is as follows: The expert dynamic weight vectors α1, α2 and the prediction probabilities P1, P2 of the novice and skilled expert models for the current time series data are input. Wherein α1, α2 respectively represent the probability that the real-time input data belongs to the novice and skilled driver; P1, P2 respectively represent the lane changing probability of the real-time input data as a novice driver and a skilled expert; Based on the parameters a1, a2, P1, P2, dynamic weighted fusion is performed, and a lane changing probability P is output final The formula is as follows: P final = α1x P1+ α2x P2; Set the lane change probability threshold to β; if P final ≥ β, determine that the vehicle is changing lanes; otherwise, determine that the vehicle is maintaining the lane.

8. A lane changing recognition system for vehicles based on multi-modal data fusion, characterized in that, It includes the following modules: The data acquisition and preprocessing module is used for synchronously collecting vehicle motion and driver eye movement data and performing preprocessing operations. The prediction module is configured to build an adaptive multi-fusion network for vehicle lane-changing recognition, which includes a global experience feature extraction branch, a lane-changing feature extraction branch, a feature fusion module, a gating network, a dynamic expert selection module, and an adaptive weighted fusion module. The global experience feature extraction branch is configured to receive multi-dimensional vehicle motion time series data input, extract features by using a lightweight time series convolution network (TCN), and then connect a global average pooling, and fuse vehicle statistical features to generate a global experience feature vector A. The vehicle statistical features are calculated from pre-processed vehicle motion data. The lane-changing feature extraction branch is configured to receive eye movement features and the global experience feature vector A and concatenate them, extract features by using a Transformer encoder to capture long-range dependencies between eye movements and vehicles, connect a global average pooling, and output a feature vector B. The eye movement features are obtained by feature extraction from pre-processed eye movement data. The feature fusion module concatenates the global experience feature vector A and the feature vector B to obtain a fusion feature, and inputs the fusion feature into the gating network to generate dynamic expert weight vectors α1 and α2, where α1+α2=1. The vehicle statistical features and the eye movement features are concatenated and input into the dynamic expert selection module, and the novice expert model and the skilled expert model are used for prediction to obtain preliminary predicted lane-changing probabilities P1 and P2. The adaptive weighted fusion module calculates the dynamically weighted and fused lane-changing probability based on the parameters α1, α2, P1, and P2, and compares it with a preset lane-changing probability threshold to realize vehicle lane-changing recognition.

9. A computer device comprising a memory and one or more processors; characterized in that, Executable code is stored in the memory, and when the processor executes the executable code, the steps of the vehicle lane-changing recognition method based on multi-modal data fusion according to any one of claims 1 to 7 are implemented.

10. A computer readable storage medium having stored thereon a program; characterized in that, The program is executed by the processor to implement the steps of the vehicle lane-changing recognition method based on multi-modal data fusion according to any one of claims 1 to 7.

Images