Efficient abnormal driving state monitoring method and system based on multi-modal memory fusion

Abstract

The invention discloses an efficient abnormal driving state monitoring method and system based on multi-modal memory fusion. The states of a driver and a vehicle are monitored through a long and short term memory network and a multi-query attention mechanism; the method comprises the following steps: firstly, acquiring vehicle body data by using an IMU sensor and a CAN bus, and preprocessing and filtering the vehicle body data; secondly, establishing a monitoring model based on a long and short term memory network and a multi-query attention mechanism method; and finally, in the driving process, based on real-time data, judging whether an abnormal driving behavior occurs or not through the established monitoring model. The invention provides a lightweight abnormal driving state monitoring system, which effectively reduces the detection cost, increases the detection flexibility and reduces the calculation amount.

Description

Technical Field

[0001] The present invention relates to the technical field of driving behavior monitoring, and in particular to a method and system for efficiently monitoring abnormal driving conditions based on multimodal memory fusion. Background Art

[0002] With the development of the automobile industry and the increase in the number of motor vehicles, road traffic safety issues have become increasingly prominent. Abnormal driving behaviors during driving, such as sudden braking, sudden acceleration, and sharp turns, are often important precursors to accidents and seriously threaten people's lives and property safety. Timely and accurate monitoring and identification of these abnormal driving states can effectively prevent accidents and improve the overall safety of road traffic. At present, there are two types of monitoring methods for abnormal driving states: wearable devices and non-wearable devices. Wearable devices will affect the driver's comfort, and the purchase and maintenance costs of the equipment are relatively high, which limits their popularity. Non-wearable devices include cameras, infrared sensors, and environmental detection sensors installed in the car. The in-car camera determines the driver's driving state by analyzing the driver's facial expressions, including the degree of opening and closing of the mouth; the infrared sensor monitors the driver's eye closure state to identify drowsiness or inattention; the environmental detection sensor assists in evaluating the driver's reaction ability and safety risks by monitoring the surrounding traffic conditions, weather and road conditions. However, the monitoring data of these non-wearable devices are mostly in the form of images, with large data volumes, and are also affected by the performance of vehicle-mounted equipment. Even if data compression algorithms such as GAN are used, the real-time performance is still poor, and the monitoring effect is single and inefficient. They can only monitor the driver's status but not the vehicle status. Summary of the invention

[0003] In view of the problems that wearable devices are too expensive and cannot be deployed on a large scale, and non-wearable devices have poor real-time performance, large amount of calculation, and single monitoring effect; the present invention provides an efficient abnormal driving state monitoring method and system based on multimodal memory fusion, which is expanded in the direction of non-wearable devices. In order to read the hidden state at each moment, a long short-term memory network is used to process continuous sequence data and extract information; in order to reduce the amount of calculation and increase the real-time performance of the system, a multi-query attention mechanism is used to reduce the number of parameters in the calculation process, improve the stability of the system, and effectively monitor the driver's abnormal driving behavior, reduce the detection cost, increase the detection flexibility, and reduce the amount of calculation.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] An efficient abnormal driving state monitoring method based on multimodal memory fusion, the method comprising the following steps:

[0006] Step 1) Read the data of IMU sensor and CAN bus in real time while the car is driving, and perform real-time monitoring and evaluation regularly.

[0007] Step 2) preprocessing the collected data, the preprocessing steps include outlier processing based on the Z-Score method and down-sampling data alignment of data of different frequencies.

[0008] Step 3) Adaptively perform extended Kalman filtering on the data collected by the IMU sensor, and use second-order Butterworth filtering on the data collected by the CAN bus.

[0009] Step 4) Extract features from the accelerator, pedal and steering wheel data after filtering on the CAN bus.

[0010] Step 5) The extracted accelerator pedal opening mean, accelerator pedal opening standard deviation, accelerator pedal opening sample entropy, brake pedal opening mean, brake pedal opening standard deviation, brake pedal opening sample entropy, vehicle steering frequency, vehicle speed mean data and three-axis angular velocity and three-axis acceleration data collected by IMU are used to establish a monitoring model for abnormal vehicle driving behavior using convolutional neural network CNN, long short-term memory network LSTM, and multi-query attention mechanism MQA.

[0011] Step 6) During actual vehicle operation, real-time CAN bus data and IMU data are used as monitoring model inputs to obtain abnormal state judgment results; if the current vehicle is in an abnormal state, the data is reported and recorded to the online platform.

[0012] Execute steps 1)-6) to realize an efficient abnormal driving state monitoring method and system based on multimodal memory fusion to timely prompt and predict the driver's abnormal driving state.

[0013] Furthermore, in the step 1), the IMU sensor and CAN bus data are read, the data collected by the IMU include X-axis acceleration, Y-axis acceleration, Z-axis acceleration, X-axis angular velocity, Y-axis angular velocity, and Z-axis angular velocity, the data transmitted by the CAN bus include timestamp, vehicle braking index, vehicle throttle index, vehicle steering index, and vehicle speed, the IMU sensor sampling frequency is 100 Hz, the CAN bus data acquisition frequency is 1000 Hz, and monitoring is performed every 2 seconds.

[0014] Furthermore, in step 2), the preprocessing is: outlier processing based on the Z-Score method. Data alignment includes aligning the timestamp of the IMU sensor data with the timestamp of the data collected by the CAN. Since the CAN bus acquisition frequency is greater than the IMU frequency, the CAN data is downsampled.

[0015] Furthermore, in the step 3), the adaptive extended Kalman filter for IMU is described, wherein the three-axis acceleration is subjected to adaptive Kalman rate filtering; after the three-axis angular velocity is converted into a quaternion, an adaptive extended Kalman filter is performed; and the filtering method for the data collected by the CAN bus is a Butterworth low-pass filter.

[0016] Furthermore, in step 4), the extended data is to extract features from the filtered data to generate the following data:

[0017] Mean accelerator pedal opening, standard deviation of accelerator pedal opening, sample entropy of accelerator pedal opening, mean brake pedal opening, standard deviation of brake pedal opening, sample entropy of brake pedal opening, vehicle turning frequency, and mean vehicle speed.

[0018] Furthermore, in the step 5), the data generated in step 4 is annotated with a data set in an abnormal state before modeling, and then the data is used to extract data features with the help of CNN; the feature data output by CNN enters the LSTM layer to further capture the body data dependency in the long-term and short-term series during the vehicle driving process. The specific operations include: sequentially passing through the input gate, forget gate and output, and gradually generating a high-level feature representation useful for abnormal state discrimination; using the MQA layer after the LSTM layer to make up for the global dependency during the vehicle driving process; and finally the result F t The data is input into the classifier to determine whether there is sudden braking, sharp turning, sudden acceleration, rollover and fatigue.

[0019] The beneficial effects of the present invention are mainly manifested in that abnormal behaviors during driving, including sudden braking, sharp turns, sudden acceleration, rollover and fatigue driving, are judged by processing the data in the CAN bus and IMU sensors. Compared with the existing evaluation methods, this method is convenient for integrating multiple sensor information, and uses a unified CNN-LSTM-MQA evaluation model to monitor multiple abnormal behaviors at the same time, reducing detection costs, increasing detection flexibility, and reducing calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of the abnormal driving state monitoring system based on multimodal memory fusion;

[0021] Figure 2 This is the CAN data processing diagram;

[0022] Figure 3 This is a comparison chart of standard deviation before and after outlier processing;

[0023] Figure 4 This is a comparison chart of longitudinal acceleration before and after filtering;

[0024] Figure 5This is a comparison chart of the angular velocity change curve around the z-axis;

[0025] Figure 6 Before and after comparison of using different filters for CAN data;

[0026] Figure 7 These are some data processing diagrams and monitoring result diagrams during the actual vehicle process;

[0027] Figure 8 It is a physical picture of the device;

[0028] Fig. 9 It is the system module structure diagram. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings.

[0030] Reference Figures 1 to 9 , an efficient abnormal driving state monitoring method and system based on multimodal memory fusion, comprising the following steps:

[0031] Step 1) read the IMU sensor and CAN bus data; the data collected by the IMU include X-axis acceleration, Y-axis acceleration, Z-axis acceleration, X-axis angular velocity, Y-axis angular velocity, and Z-axis angular velocity; the data transmitted by the CAN bus include timestamp, vehicle braking index, vehicle throttle index, vehicle steering index, and vehicle speed; the sampling frequency of the IMU sensor is 100 Hz, the data acquisition frequency of the CAN bus is 1000 Hz, and monitoring is performed every 2 seconds.

[0032] Step 2) Outlier processing based on Z-Score method:

[0033]

[0034] Where X is a data point, μ is the mean of the data set, and σ is the standard deviation of the data set. If |Z|>2, then X is considered an outlier.

[0035] Data alignment: The timestamp of the IMU sensor data is aligned with the timestamp of the data collected by CAN, mainly for downsampling of CAN data:

[0036] Given the timestamp t of the IMU data imu , find the closest one that is no later than t imu Time point t CAN , select t CAN CAN data at a point in time.

[0037] The comparison results of standard deviation before and after data cleaning are shown in the figure below: Figure 3 shown.

[0038] Step 3) Adaptive extended Kalman filtering for IMU, in which adaptive Kalman rate filtering is performed on the three-axis acceleration, prediction steps:

[0039] X K|K-1 =X K-1 (2)

[0040] P K|K-1 =P K-1 +Q K-1 (3)

[0041] Where X K-1 is the predicted value of the state at time K-1, X K|K-1 is the estimated value at time K based on the predicted value at time K-1, P K-1 is the covariance matrix at K-1 time, Q K-1 is the system error at K-1 moment; P K|K-1 It is the estimated value at moment K based on the predicted covariance matrix value at moment K-1.

[0042] Update steps:

[0043] X K =X K-1 +M(z K -X K|K-1 ) (4)

[0044]

[0045] P K =(IM)P K|K-1 (6)

[0046] Q K+1 =Q K +α((z K -X K|K-1 )(z K -X K|K-1 ) T -Q K ) (7)

[0047] where z K is the observed value at time K, X K is the estimated value at time K, M is the Kalman gain, and R is the observation noise; α is an adaptation factor less than 1, which is used to control the adaptive step size; Q K is the system error at time K; Q K+1 is the system error at time K+1; X K|K-1 It is the estimated value at moment K based on the predicted value at moment K-1.

[0048] like Figure 4As shown in the figure, after the longitudinal acceleration is filtered by the adaptive Kalman filter, the acceleration is more stable than the original data.

[0049] After converting the three-axis angular velocity into quaternion, adaptive extended Kalman filtering is performed, and the prediction steps are:

[0050]

[0051] where q K-1 is the predicted value at time K-1, q K|K-1 is the estimated value at time K based on the predicted value at time K-1, ω K-1 is the angular velocity at the K-1 moment. K-1 is the covariance matrix at time K-1, and Q is the system error. K|K-1 is the estimated value at time K based on the predicted value at time K-1. K is the Jacobian matrix of the state transfer matrix at time K:

[0052]

[0053] Observation steps:

[0054] M=P K|K-1 H T (HP K|K-1 H T +R) -1 (11)

[0055] q K|K =q K|K-1 +M(z K -Hq K|K-1 ) (12)

[0056] P K =(I-MH)P K|K-1 (13)

[0057]

[0058] where z K is the observed value at time K, X K is the estimated value at time K, M K is the Kalman gain at time K, R is the observation noise; α is an adaptation factor less than 1, which is used to control the adaptive step size; Q K is the system error at time K; Q K+1 is the system error at time K+1; X K|K-1 is the estimated value at time K based on the predicted value at time K-1; H K is the Jacobian matrix of the observation transfer matrix at time K, which describes the first-order partial derivative of the observation function with respect to the state vector:

[0059]

[0060] where h is the observation function, q K is the state vector.

[0061] like Figure 5 As shown in the figure, after the angular velocity around the z-axis passes through the adaptive extended Kalman filter, the angular velocity is more stable than the original data and the ordinary Kalman filter.

[0062] The filtering method for the data collected by the CAN bus is a second-order Butterworth low-pass filter:

[0063] y[n]=αx[n]+(1-2α)y[n-1]+αy[n-2] (16)

[0064] Where x[n] is the input signal, y[n] is the output signal after filtering, and α is the coefficient of the filter, which is related to ω 0 and sampling period T S The calculation formula is as follows:

[0065]

[0066] where ω 0 is the filter’s angular cutoff frequency, and the base frequency f c related:

[0067] ω 0 =2πf c (18)

[0068] like Figure 6 As shown in the figure, the data filtered by the second-order Butterworth filter is smoother than the original data, and compared with the first-order Butterworth filter, it avoids overfitting.

[0069] Step 4) Extract features from the filtered data to generate the following data:

[0070] The accelerator pedal opening mean, accelerator pedal opening standard deviation, accelerator pedal opening sample entropy, brake pedal opening mean, brake pedal opening standard deviation, brake pedal opening sample entropy, vehicle steering frequency, and vehicle speed mean. The specific generation method is as follows:

[0071] Accelerator pedal average:

[0072]

[0073] Accelerator pedal standard deviation:

[0074]

[0075] Gas pedal sample entropy:

[0076]

[0077]

[0078] Where n is the total number of data points, x i is the accelerator pedal position of the i-th data point; n i is the data point x i The number of times it appears in the data set; k is the number of different accelerator pedal openings in the data set.

[0079] Brake pedal average:

[0080]

[0081] Brake pedal standard deviation:

[0082]

[0083] Brake pedal sample entropy:

[0084]

[0085] Where m is the total number of data points, y i is the accelerator pedal position of the i-th data point; m i is the data point y i The number of times it appears in the data set; k is the number of different brake pedal openings in the data set.

[0086] Vehicle steering frequency:

[0087]

[0088] Where n is the number of turns and T is the total time.

[0089] Average speed:

[0090]

[0091] where v i is the speed of the sampling points. Since the sampling frequency of the CAN bus is 10 times that of the IMU, 10 sampling points are taken to calculate the mean.

[0092] Step 5) The extracted accelerator pedal opening mean, accelerator pedal opening standard deviation, accelerator pedal opening sample entropy, brake pedal opening mean, brake pedal opening standard deviation, brake pedal opening sample entropy, vehicle steering frequency, vehicle speed mean data and three-axis angular velocity and three-axis acceleration data collected by IMU are used to extract data features using CNN. The CNN process includes convolution and pooling operations, and the calculation process is:

[0093] X=[x 1 ,x 2 ,…,x 8 ] (29)

[0094] Y[f]=ReLU(W·X[f:f+3]+b) (30)

[0095] ReLU(x)=max(0,x) (31)

[0096] Z[f]=max(Y[f:f+2]) (32)

[0097] where x 1 ,x 2 ,…,x 8 Represents the eigenvalues ​​of the input; f represents the feature item to be convolved; b represents the bias item; W is the convolution kernel; Y[f] is the result of the convolution and activation function, which is an element of a feature map; Z[f] represents the result after CNN feature extraction.

[0098] The feature data output by CNN enters the LSTM layer to further capture the dependency of vehicle body data in long-term and short-term time series during vehicle driving. The specific operations include: sequentially passing through the input gate, forget gate and output, and gradually generating high-level feature representations that are useful for abnormal state discrimination. The calculation process is:

[0099] i t =σ(W i ·[h t-1 ,x t ]+b i ) (33)

[0100] f t =σ(W f ·[h t-1 ,x t ]+b f ) (34)

[0101] ο t =σ(W ο ·[h t-1 ,x t ]+b ο ) (35)

[0102]

[0103] h t =ο t tanh(C t ) (38)

[0104]

[0105] where x t is the current input; h t-1 is the hidden state at the previous moment; C t-1 is the cell state at the previous moment; i t ,f t ,ο t They are the input gate, forget gate, and output gate outputs; W i , W f , W ο , W C are the weight matrices of the input gate, forget gate, output gate, and cell state; b i 、b f 、b ο 、b C are the input gate, forget gate, output gate and bias term of cell state respectively; tanh is the hyperbolic tangent function; σ is the sigmoid activation function. At this time, the output of the LSTM layer is:

[0106] H=[h 1 ,h 2 ] (40)

[0107] The MQA layer is used after the LSTM layer to make up for the global dependency during vehicle driving, and the hidden state H including the mean accelerator pedal opening, standard deviation of accelerator pedal opening, sample entropy of accelerator pedal opening, mean brake pedal opening, standard deviation of brake pedal opening, sample entropy of brake pedal opening, vehicle turning frequency, and mean vehicle speed is converted into a t Operate with different weight matrices to generate corresponding query, key, and value matrices to obtain the output of the sub-attention. The calculation process is:

[0108] Q t =H t ·W Q (41)

[0109] K t =H t ·W K (42)

[0110] V t =H t ·W V (43)

[0111]

[0112] F t =LayerNorm(H t +O t ) (46)

[0113] Among them, H t is the output of LSTM at time t; WQ , W K , W V is the query weight matrix, key weight matrix, and value weight matrix; Q t , K t 、V t is the corresponding query, key, and value matrix; d k refers to the dimension of the key, here it is 2, which is used to scale the score; S t,t′ is the attention score, which measures the relationship between the vehicle features at the current time t and the vehicle features at the time t′, and is subsequently converted into the attention weight matrix W through the normalization function. t,t′ ;O t is the multi-query attention output, which contains the vehicle status at multiple time steps; F t Output O for multiple query attention t and the original output H t The final output result after residual normalization and concatenation.

[0114] Finally, the result F t Input to the binary classifier to determine whether there is sudden braking, sharp turning, sudden acceleration, rollover and fatigue. First, the high-dimensional vector F t Different classification heads are converted into separate scalars, and then further expressed as probability values ​​through activation functions, and finally compared with pre-defined thresholds to determine whether they are in the corresponding abnormal state. The calculation is as follows:

[0115] y=w T x+b (47)

[0116]

[0117] O=Sigmoid(y) (49)

[0118] Where x is the output of MQA; O is the probability of the final output being in an abnormal state, which is determined by comparing with the threshold; b represents the bias term, which depends on the situation.

[0119] Step 6) During actual vehicle operation, the real-time CAN bus data and IMU data are used as monitoring model inputs to obtain abnormal state judgment results.

[0120] like Figure 7 , Figure 7 The following is a set of real-time longitudinal acceleration data before and after processing and monitoring results, including real-time speed, real-time acceleration and timestamp. The sampling frequency is 100Hz. After the longitudinal acceleration is processed by the adaptive Kalman filter, the longitudinal acceleration at time 1 and time 2 is still high. When the actual vehicle speed is 100km / h, according to the conventional threshold judgment method:

[0121] a>=2m / s 2

[0122] Acceleration a is greater than or equal to 2m / s 2 If it is regarded as a sudden acceleration, then time 1 and time 2 are sudden acceleration. Time 3 and time 4 are normal driving, which is the same as the output result of the monitoring model.

[0123] If the current vehicle is in an abnormal state, the data and records will be reported to the online platform.

[0124] Iteratively execute steps 1)-4) and 6) to achieve real-time evaluation and warning of the driver. Accordingly, this example also provides an efficient abnormal driving state monitoring method and system based on multimodal memory fusion, which is used to execute the above method.

[0125] Please refer to Figure 8 , Figure 8 This is a real picture of the device. The device is small in size, connected to the vehicle body by wireless wire, easy to use, and easy to install and disassemble.

[0126] Please refer to Fig. 9 , Fig. 9 The schematic diagram of the device module is shown in the figure. As shown in the figure, the efficient abnormal driving state monitoring method and system module based on multimodal memory fusion includes: first, the device collects vehicle vibration information through sensors and data acquisition modules; second, the data preprocessing module and the filtering module process and filter the collected data, and then pass the data as input to the evaluation model module to output the driving behavior; then, the reporting module is used to issue alarms and prompts; finally, the vehicle status and running time can be observed in real time using the online platform.

[0127] The above embodiments are only preferred embodiments of the present invention and are not limitations of the technical solutions of the present invention. Any technical solution that can be implemented on the basis of the above embodiments without creative work should be deemed to fall within the scope of protection of the patent of the present invention.

Claims

1. An efficient abnormal driving state monitoring method based on multimodal memory fusion, characterized by: The method comprises the following steps: Step 1) Reading the data of IMU sensor and CAN bus in real time while the car is driving; Step 2) preprocessing the collected data, the preprocessing step includes outlier processing based on the Z-Score method and down-sampling data alignment of data of different frequencies; Step 3) Adaptively perform extended Kalman filtering on the data collected by the IMU sensor, and use second-order Butterworth filtering on the data collected by the CAN bus; Step 4) extracting features from the accelerator, pedal and steering wheel data after filtering by the CAN bus; Step 5) The extracted accelerator pedal opening mean, accelerator pedal opening standard deviation, accelerator pedal opening sample entropy, brake pedal opening mean, brake pedal opening standard deviation, brake pedal opening sample entropy, vehicle steering frequency, vehicle speed mean data and three-axis angular velocity and three-axis acceleration data collected by IMU are used to establish a monitoring model for abnormal vehicle driving behavior using a convolutional neural network CNN, a long short-term memory network LSTM, and a multi-query attention mechanism MQA; Step 6) During actual vehicle operation, real-time CAN bus data and IMU data are used as monitoring model inputs to obtain abnormal state judgment results; if the current vehicle is in an abnormal state, the data is reported and recorded to the online platform.

2. The efficient abnormal driving state monitoring method based on multimodal memory fusion according to claim 1, characterized in that: In the step 1), the data read from the IMU sensor includes X-axis acceleration, Y-axis acceleration, Z-axis acceleration, X-axis angular velocity, Y-axis angular velocity, and Z-axis angular velocity; The data read from the CAN bus includes a timestamp, a vehicle braking index, a vehicle throttle index, a vehicle steering index and a vehicle speed.

3. The efficient abnormal driving state monitoring method based on multimodal memory fusion according to claim 1, characterized in that: In the step 2), the outlier processing based on the Z-Score method is as follows: Where X is a data point, μ is the mean of the data set, and σ is the standard deviation of the data set; if |Z|>2, then X is considered an outlier; The down-sampling data alignment of the different frequency data includes aligning the timestamp of the IMU sensor data with the timestamp of the data collected by the CAN, which is the down-sampling of the CAN data: Given the timestamp t of the IMU data imu , find the closest one that is no later than t imu Time point t CAN , select t CAN CAN data at a point in time.

4. The efficient abnormal driving state monitoring method based on multimodal memory fusion according to claim 1, characterized in that: In step 3), the adaptive extended Kalman filter for IMU, wherein the three-axis acceleration is adaptively Kalman rate filtered, is as follows: X K|K-1 =X K-1 (2) P K|K-1 =P K-1 +Q K-1 (3) Where X K-1 is the predicted value of the state at time K-1, X K|K-1 is the estimated value at time K based on the predicted value at time K-1, P K-1 is the covariance matrix at K-1 time, Q K-1 is the system error at K-1 moment; P K|K-1 It is the estimated value at moment K based on the predicted covariance matrix value at moment K-1. Update steps: X K =X K-1 +M(z K -X K|K-1 ) (4) P K =(I-M)P K|K-1 (6) Q K+1 =Q K +α((z K -X K|K-1 )(z K -X K|K-1 ) T -Q K ) (7) where z K is the observed value at time K, X K is the estimated value at time K, M is the Kalman gain, and R is the observation noise; α is an adaptation factor less than 1, which is used to control the adaptive step size; Q K is the system error at time K; Q K+1 is the system error at time K+1; X K|K-1 It is the estimated value at time K based on the predicted value at time K-1; After converting the three-axis angular velocity into quaternion, adaptive extended Kalman filtering is performed. The steps are as follows: where q K-1 is the predicted value at time K-1, q K|K-1 is the estimated value at time K based on the predicted value at time K-1, ω K-1 is the angular velocity at K-1, P K-1 is the covariance matrix at K-1 time, Q K is the system error at K-1 time, P K|K-1 is the estimated value at time K based on the predicted value at time K-1, and F K is the Jacobian matrix of the state transfer matrix at time K: Observation steps: M=P K|K-1 H T (HP K|K-1 H T +R) -1 (11) q K|K =q K|K-1 +M(z K -Hq K|K-1 ) (12) P K =(I-MH)P K|K-1 (13) Q K+1 =Q K +α((z K -X K|K-1 )(z K -X K|K-1 ) T -Q K ) (14) where z K is the observed value at time K, X K is the estimated value at time K, M K is the Kalman gain at time K, R is the observation noise; α is an adaptation factor less than 1, which is used to control the adaptive step size; Q K is the system error at time K; Q K+1 is the system error at time K+1; X K|K-1 is the estimated value at time K based on the predicted value at time K-1; H K is the Jacobian matrix of the observation transfer matrix at time K, which describes the first-order partial derivative of the observation function with respect to the state vector: where h is the observation function, q K is the state vector; Filtering method for data collected by CAN bus: y[n]=αx[n]+(1-2α)y[n-1]+αy[n-2] (16) Where x[n] is the input signal, y[n] is the output signal after filtering, α is the coefficient of the filter, and ω0 and the sampling period T S The calculation formula is as follows: Where ω0 is the filter’s angular cutoff frequency, and the base frequency f c related: ω0=2πf c (18)。 5. The efficient abnormal driving state monitoring method based on multimodal memory fusion according to claim 1, characterized in that: In the step 4), feature extraction is performed on the filtered data, including: The mean value of accelerator pedal opening, the standard deviation of accelerator pedal opening, the sample entropy of accelerator pedal opening, the mean value of brake pedal opening, the standard deviation of brake pedal opening, the sample entropy of brake pedal opening, the vehicle turning frequency, and the mean value of vehicle speed; The specific generation method is as follows: Average accelerator pedal opening: Accelerator pedal opening standard deviation: Accelerator pedal opening sample entropy: Where n is the total number of data points, x i is the accelerator pedal position of the i-th data point; n i is the data point x i The number of times it appears in the data set; k is the number of different accelerator pedal openings in the data set; Average brake pedal opening: Brake pedal opening standard deviation: Brake pedal opening sample entropy: Where m is the total number of data points, y i is the accelerator pedal position of the i-th data point; m i is the data point y i The number of times it appears in the data set; k is the number of different brake pedal openings in the data set. Vehicle steering frequency: Where n is the number of turns and T is the total time; Average speed: where v i is the speed of the sampling point.

6. The efficient abnormal driving state monitoring method based on multimodal memory fusion according to claim 1, characterized in that: In the step 5), the accelerator pedal opening mean, accelerator pedal opening standard deviation, accelerator pedal opening sample entropy, brake pedal opening mean, brake pedal opening standard deviation, brake pedal opening sample entropy, vehicle steering frequency, and vehicle speed mean are marked as data sets in abnormal states before modeling, and CNN is used to extract data features. The calculation process is: X=[x1,x2,…,x8] (29) Y[f]=ReLU(W·X[f:f+3]+b) (30) ReLU(x)=max(0,x) (31) Z[f]=max(Y[f:f+2]) (32) Where x1, x2,…, x8 represent the eigenvalues ​​of the input; f represents the feature item to be convolved; b represents the bias item; W is the convolution kernel; Y[f] is the result of convolution and activation function, which is an element of a feature map; Z[f] represents the result after CNN feature extraction; The feature data output by CNN enters the LSTM layer to further capture the dependency of vehicle body data in long-term and short-term series during vehicle driving. The specific operations include: sequentially passing through the input gate, forget gate, and output gate to gradually generate high-level feature representations that are useful for abnormal state discrimination. The calculation process is as follows: i t =σ(W i ·[h t-1 ,x t ]+b i ) (33) f t =σ(W f ·[h t-1 ,x t ]+b f ) (34) the t =σ(W ο ·[h t-1 ,x t ]+b ο ) (35) h t =ο t ·tanh(C t ) (38) where x t is the current input; h t-1 is the hidden state at the previous moment; C t-1 is the cell state at the previous moment; i t ,f t ,ο t They are the input gate, forget gate, and output gate outputs; W i , W f , W ο , W C are the weight matrices of the input gate, forget gate, output gate, and cell state respectively; b i , b f , b ο , b C are the input gate, forget gate, output gate, and bias term of the cell state; tanh is the hyperbolic tangent function; σ is the sigmoid activation function; the output of the LSTM layer is: H=[h1,h2] (40) The MQA layer is used after the LSTM layer to make up for the global dependency of the vehicle during driving, and the hidden state H including the mean value of accelerator pedal opening, standard deviation of accelerator pedal opening, sample entropy of accelerator pedal opening, mean value of brake pedal opening, standard deviation of brake pedal opening, sample entropy of brake pedal opening, vehicle turning frequency, and mean speed is converted into a t Operate with different weight matrices to generate corresponding query, key, and value matrices to obtain the output of the sub-attention. The calculation process is: Q t =H t ·W Q (41) K t =H t ·W K (42) V t =H t ·W V (43) F t =LayerNorm(H t +O t ) (46) Among them, H t is the output of LSTM at time t; W Q , W K , W V is the query weight matrix, key weight matrix, and value weight matrix; Q t , K t 、V t is the corresponding query, key, and value matrix; d k refers to the dimension of the key, used to scale the scores; S t,t′ is the attention score, which measures the relationship between the vehicle features at the current time t and the vehicle features at the time t′, and is subsequently converted into the attention weight matrix W through the normalization function. t,t′ ;O t is the multi-query attention output, which contains the vehicle status at multiple time steps; F t Output O for multi-query attention t and the original output H t The final output result after residual normalization and concatenation; Finally, the result F t Input into the binary classifier to determine whether there is sudden braking, sharp turning, sudden acceleration, rollover and fatigue state; first, the high-dimensional vector F t Different classification heads are converted into separate scalars, and then further represented by probability values ​​through activation functions. Finally, they are compared with pre-defined thresholds to determine whether they are in the corresponding abnormal state. The calculation is as follows: y=w T x+b (47) O=Sigmoid(y) (49) Where x is the output of MQA; O is the probability of the final output being in an abnormal state, which is determined by comparing with the threshold; b represents the bias term.

7. An efficient abnormal driving state monitoring system based on multimodal memory fusion, characterized by: Contains the following modules: Data acquisition module: used to collect vehicle body data; Data preprocessing module: used to process outliers based on the Z-Score method and align downsampled data of different frequencies; Filtering module: used to filter the processed data in different ways; Evaluation model module: used to generate corresponding vehicle driving abnormal status through input filtered data in real-time vehicle operation; Data reporting module: If the driver is in a fatigued state or the vehicle is in an abnormal state, the device will report the data and record it to the online platform; Online platform module: used to record and display an overview of vehicle abnormal status and driver driving status.

Images