Intelligent assisted driving method, system and edge device based on multimodal data

Abstract

The present invention discloses an intelligent assisted driving method, system and edge device based on multimodal data. The intelligent assisted driving method based on multimodal data includes: acquiring physiological data of a driver and vehicle driving data; fusing the physiological data and the vehicle driving data to obtain multimodal fusion data; performing intention recognition based on the multimodal fusion data to obtain the driver's driving operation intention; performing behavior prediction based on the multimodal fusion data and the driving operation intention to obtain a driving behavior prediction result; and controlling the vehicle to drive based on an assisted driving mode based on the driving behavior prediction result.

Description

Technical Field

[0001] The present invention relates to the technical field of driving assistance, and in particular to an intelligent assisted driving method, system and edge device based on multimodal data. Background Art

[0002] With the development of intelligent vehicles, driving assistance technology has received extensive attention from researchers. However, how to provide drivers with high-precision driving assistance has become the focus of research.

[0003] The Advanced Driving Assistance System (ADAS) proposed by related technologies mainly provides driving assistance with information from the vehicle's external sensors, such as cameras, radars, and lidars. However, the system focuses more on the external environment and ignores the driver's internal state, such as attention, cognitive load, and physiological state, which limits the accuracy and adaptability of the assistance system to a certain extent. Summary of the invention

[0004] The embodiments of the present invention aim to solve at least one of the technical problems in the related art to a certain extent. To this end, one object of the present invention is to propose an intelligent assisted driving method, system, edge device and storage medium based on multimodal data to improve the accuracy of driving assistance.

[0005] An embodiment of the present invention provides an intelligent assisted driving method based on multimodal data. The intelligent assisted driving method based on multimodal data includes: acquiring physiological data of a driver and vehicle driving data; fusing the physiological data and the vehicle driving data to obtain multimodal fusion data; performing intention recognition based on the multimodal fusion data to obtain the driver's driving operation intention; performing behavior prediction based on the multimodal fusion data and the driving operation intention to obtain a driving behavior prediction result; and controlling the vehicle to drive based on an assisted driving mode based on the driving behavior prediction result.

[0006] Exemplarily, the physiological data and the vehicle driving data are fused to obtain multimodal fusion data, including: fusing the physiological data and the vehicle driving data based on a Kalman fusion method to obtain multimodal fusion data.

[0007] Exemplarily, the multimodal fusion data includes a feature vector, and the physiological data and the vehicle driving data are fused based on the Kalman fusion method to obtain the multimodal fusion data, including: for the multimodal data composed of physiological data and vehicle driving data, a state equation is established for each modal data to obtain multiple state equations; based on the multiple state equations, a joint state equation is obtained, wherein the joint state equation includes weight parameters corresponding to each modal data; based on the joint state equation, each modal data is updated to obtain a feature vector, wherein the weight parameter can be dynamically adjusted based on the Kalman gain of each modal data.

[0008] Exemplarily, performing intention recognition based on multimodal fusion data to obtain the driver's driving operation intention includes: inputting the multimodal fusion data into a classification model to perform intention recognition to obtain the driver's driving operation intention.

[0009] Exemplarily, the multimodal fusion data includes time series data, and behavior prediction is performed based on the multimodal fusion data and driving operation intention to obtain a driving behavior prediction result, including: inputting the multimodal fusion data into a time series prediction model to perform driving behavior prediction, and using the driving operation intention as a reference in the prediction process to obtain a driving behavior prediction result.

[0010] Exemplarily, the intelligent assisted driving method based on multimodal data also includes: obtaining the driver's driving status data based on physiological data; based on the driving behavior prediction results, controlling the vehicle to drive based on the assisted driving mode, including: based on the driving behavior prediction results and the driving status data, controlling the vehicle to drive based on the assisted driving mode.

[0011] Exemplarily, the physiological data includes at least one of eye movement data and EEG data, the driving status data includes at least one of visual concentration data obtained based on the eye movement data and cognitive load data obtained based on the EEG data; the assisted driving mode includes a driving warning mode; based on the driving behavior prediction result and the driving status data, the vehicle is controlled to drive based on the assisted driving mode, including: when the driving behavior prediction result indicates driving acceleration and at least one of the visual concentration data and the cognitive load data indicates physiological abnormality, the vehicle is controlled to drive based on the driving warning mode.

[0012] Exemplarily, at least one of the visual concentration data and the cognitive load data indicates a physiological abnormality, including: obtaining a first physiological result based on the visual concentration data and a first weight, and obtaining a second physiological result based on the cognitive load data and a second weight, wherein the first weight and the second weight can be dynamically adjusted based on the environment; obtaining a comprehensive physiological result based on the first physiological result and the second physiological result; comparing the comprehensive physiological result with a preset physiological threshold to obtain a comparison result, wherein the comparison result is used to indicate the physiological abnormality.

[0013] Exemplarily, the physiological data includes electrocardiogram data, the driving status data includes fatigue status data, the assisted driving mode includes a lane keeping assist mode, and based on the driving behavior prediction result and the driving status data, the vehicle is controlled to drive based on the assisted driving mode, including: when the driving behavior prediction result indicates a lane change and the fatigue status data indicates that the current fatigue level is lower than a first preset fatigue level, controlling the vehicle to drive based on the lane keeping assist mode.

[0014] Exemplarily, the physiological data includes electrocardiogram data, the driving status data includes fatigue status data, the assisted driving mode includes a driving warning mode, and based on the driving behavior prediction result and the driving status data, the vehicle is controlled to drive based on the assisted driving mode, including: when the driving behavior prediction result indicates driving acceleration and the fatigue status data indicates that the current fatigue level is lower than a second preset fatigue level, the vehicle is controlled to drive based on the driving warning mode.

[0015] Another embodiment of the present invention provides an intelligent assisted driving system based on multimodal data. The intelligent assisted driving system based on multimodal data includes: an acquisition module, used to acquire the driver's physiological data and vehicle driving data; a fusion module, used to fuse the physiological data and the vehicle driving data to obtain multimodal fusion data; an intention recognition module, used to perform intention recognition based on the multimodal fusion data to obtain the driver's driving operation intention; a behavior prediction module, used to perform behavior prediction based on the multimodal fusion data and driving operation intention to obtain a driving behavior prediction result; and a control module, used to control the vehicle to drive based on an assisted driving mode based on the driving behavior prediction result.

[0016] Another embodiment of the present invention provides an edge device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of any one of the above embodiments when executing the computer program.

[0017] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method of any one of the above embodiments are implemented.

[0018] In the above implementation, the intelligent assisted driving method based on multimodal data includes: obtaining the driver's physiological data and vehicle driving data; fusing the physiological data and vehicle driving data to obtain multimodal fusion data; performing intention recognition based on the multimodal fusion data to obtain the driver's driving operation intention; performing behavior prediction based on the multimodal fusion data and driving operation intention to obtain a driving behavior prediction result; based on the driving behavior prediction result, controlling the vehicle to drive based on the assisted driving mode. The above method realizes accurate recognition of the driver's intention through multimodal data fusion, reduces the risk of accidents caused by distraction or fatigue, and by combining vehicle driving data, can predict the driver's operating behavior and provide driving assistance to the driver, improve the accuracy of driving assistance, and realize more comprehensive driver status monitoring and personalized assistance functions.

[0019] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flow chart of an intelligent assisted driving method based on multimodal data provided by an embodiment of the present invention;

[0021] Figure 2 A block diagram of an intelligent assisted driving system based on multimodal data provided by another embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the edge device structure provided for another embodiment of the present invention. DETAILED DESCRIPTION

[0023] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0024] With the development of intelligent vehicles, driving assistance technology has received extensive attention from researchers. However, how to provide drivers with high-precision driving assistance has become the focus of research.

[0025] The Advanced Driving Assistance System (ADAS) proposed by related technologies mainly provides driving assistance with the help of the vehicle's external sensors, such as cameras, radars, and lidars. However, the system pays more attention to the external environment and ignores the driver's internal state, such as attention, cognitive load, and physiological state. This limits the accuracy and adaptability of the assistance system to a certain extent, and it is unable to respond to the driver's true intentions and attention changes in a timely manner. In addition, it lacks the integration of multimodal physiological signals and cannot provide comprehensive monitoring of the driver's cognitive state.

[0026] To this end, an embodiment of the present invention provides an intelligent assisted driving method based on multimodal data. Through multimodal data fusion, accurate recognition of the driver's intention is achieved, reducing the risk of accidents caused by distraction or fatigue. By combining vehicle driving data, the driver's operating behavior can be predicted and driving assistance can be provided to the driver, thereby improving the accuracy of driving assistance and achieving more comprehensive driver status monitoring and personalized assistance functions.

[0027] Figure 1 A flow chart of an intelligent assisted driving method based on multimodal data provided in an embodiment of the present invention.

[0028] like Figure 1 As shown, the intelligent assisted driving method 100 based on multimodal data includes steps S110 to S150.

[0029] Step S110, acquiring the driver's physiological data and vehicle driving data.

[0030] Exemplarily, the vehicle is operated in different driving situations (such as high-speed driving and crowded urban roads) to collect various physiological data of the driver and vehicle driving data; physiological data include, for example, electroencephalogram (EEG), eye movement signal (EOG), electrocardiogram (ECG), etc. EEG signals use non-invasive EEG acquisition equipment to record signals of frequency bands such as α waves (8-13Hz) and β waves (13-30Hz), which are mainly used to monitor the driver's attention and cognitive load. Eye movement signals use eye tracking equipment to collect the driver's line of sight, gaze point, blinking frequency and gaze duration, which are used to analyze the concentration of visual attention and the object of attention. Electrocardiogram signals use electrocardiogram sensors to collect the driver's heart rate and heart rate variability (HRV) to assess physiological stress and fatigue status. Vehicle driving data include, for example, operation information such as accelerator pedal, brake, and steering wheel rotation.

[0031] Step S120, performing data fusion on the physiological data and the vehicle driving data to obtain multimodal fusion data.

[0032] Exemplarily, based on the collected physiological data and vehicle driving data, data preprocessing (including signal denoising and data synchronization) is performed on them, and the EEG signal uses bandpass filtering (0.5-30Hz) to remove myoelectric and environmental noise; the eye movement signal is smoothed to reduce jitter; the ECG signal is extracted from the RR (RR interval) interval through the R peak detection algorithm, and then the multimodal data is timestamped using the edge device to ensure the synchronous collection of each signal, and the interpolation method is used to fill the missing data points, and the preprocessed data is input into the corresponding model (such as the Kalman filter) to obtain the fused multimodal data.

[0033] Step S130, performing intention recognition based on the multimodal fusion data to obtain the driver's driving operation intention.

[0034] Exemplarily, the driving operation intention includes categories such as acceleration, deceleration or lane change, and the obtained multimodal fusion data is input into a machine learning model (such as a support vector machine (SVM) model) to identify the driver's driving operation intention in real time.

[0035] Step S140, performing behavior prediction based on the multimodal fusion data and the driving operation intention to obtain a driving behavior prediction result.

[0036] Exemplarily, the multimodal fusion data and the obtained real-time driving operation intention are input into a neural network model (such as a long short-term memory network (LSTM) model) to predict the driver's next operation. The next operation may include acceleration, deceleration, or lane change.

[0037] Step S150, based on the driving behavior prediction result, controlling the vehicle to drive based on the assisted driving mode.

[0038] For example, based on the driving behavior prediction results, combined with the environmental information detected by the camera and lidar and the driver's physiological data, the driver's driving behavior is dynamically adjusted to adapt to changes in pedestrians, obstacles, etc., so that it can meet safety needs.

[0039] In the above embodiment, multimodal data fusion is used to accurately identify the driver's intention, reduce the risk of accidents caused by distraction or fatigue, and by combining vehicle driving data, the driver's operating behavior can be predicted and driving assistance can be provided to the driver, thereby improving the accuracy of driving assistance and achieving more comprehensive driver status monitoring and personalized assistance functions.

[0040] The physiological data and the vehicle driving data are fused to obtain multimodal fusion data, including: fusing the physiological data and the vehicle driving data based on a Kalman fusion method to obtain multimodal fusion data.

[0041] Exemplarily, the multimodal fusion data includes a feature vector, and the physiological data and the vehicle driving data are fused based on the Kalman fusion method to obtain the multimodal fusion data, including: for the multimodal data composed of physiological data and vehicle driving data, a state equation is established for each modal data to obtain multiple state equations; based on the multiple state equations, a joint state equation is obtained, wherein the joint state equation includes weight parameters corresponding to each modal data; based on the joint state equation, each modal data is updated to obtain a feature vector, wherein the weight parameter can be dynamically adjusted based on the Kalman gain of each modal data.

[0042] For example, for a discrete-time linear dynamic system, the state equation is usually expressed as formula (1):

[0043] x k =F k x k-1 +B k u k +w k (1)

[0044] Among them, x k is the system state vector at time k, F k is the state transition matrix, x k-1 is the state at the previous moment, B k is the control input matrix (if there is an external control input u k ), w k is the process noise, with mean 0 and covariance Q k , that is, w k ~N(0,Q k ).

[0045] State equations are established for physiological data and vehicle driving data respectively, and the state equations are expanded to obtain a joint state equation. The Kalman gain of each modal data is calculated respectively to obtain multiple weight parameters. The Kalman gain of each modal data is dynamically adjusted to obtain the Kalman gain of the joint state equation. The joint state equation is updated according to formula (2) to obtain the fused feature vector F:

[0046] F t =K t ×X t +(1-K t )×F t-1 (2)

[0047] Among them, K t is the Kalman gain of the joint state equation, X t is the current state observation value of the joint state equation, F t-1 is the fused state value of the joint state equation at the previous moment.

[0048] The feature vector F contains: visual attention index (based on eye movement data), cognitive load index (based on EEG α / β ratio), stress / fatigue index (based on HRV parameters), vehicle status data (speed, steering wheel angle, etc.).

[0049] The visual attention index is calculated based on the eye movement data. By statistically distributing the driver's gaze points, a gaze heat map is generated to represent the driver's visual focus area. The calculation method is shown in formula (3):

[0050]

[0051] In formula (3), A v is the visual attention index, f i is the fixation frequency, d i is the distance between the fixation point and the target area, and T is the total fixation time.

[0052] The cognitive load index is calculated based on the ratio of EEG α wave to β wave, and the calculation method is shown in formula (4):

[0053]

[0054] In formula (4), C e is the cognitive load index.

[0055] For the stress / fatigue index, the standard deviation of NN intervals (SDNN) is calculated based on HRV (heart rate variability) to assess the driver's stress. The smaller the SDNN, the greater the stress.

[0056] For vehicle status data: the vehicle's current speed data is collected in real time through the on-board system; the driver's control of the steering wheel is collected and the steering wheel rotation angle is recorded to understand the driver's control intention; the strength of the accelerator pedal and the brake pedal is recorded to analyze whether the driver is accelerating or decelerating.

[0057] In the above embodiment, the fusion of multimodal data is achieved based on the Kalman method, so that the data has low noise and high time series synchronization, providing a stable data basis for driving intention recognition and behavior prediction. The Kalman method can quickly process sensor data and immediately output accurate fusion results with strong real-time performance.

[0058] Intention recognition is performed based on multimodal fusion data to obtain the driver's driving operation intention, including: inputting the multimodal fusion data into a classification model to perform intention recognition to obtain the driver's driving operation intention.

[0059] Exemplarily, the classification model includes a support vector machine (SVM) model, and the fused multimodal feature vector F is input into the classification model SVM, which is mapped to the driving intention category through the SVM, and the driver's current operation intention, such as acceleration, deceleration or lane change, is output and calculated in real time on the edge device. The SVM model is trained offline with vehicle driving data and updated regularly, and based on multiple sets of vehicle driving data and physiological data, the SVM classifier is trained to optimize the decision boundary, the driver's operation intention is judged according to the classification boundary, and the intention result is used as a reference for behavior prediction.

[0060] For example, the intention categories include: acceleration intention: the driver actively accelerates; deceleration intention: the driver is preparing to brake or release the accelerator pedal; lane change intention: the driver adjusts the lane position through the steering wheel; keep driving in a straight line: no obvious operation intention.

[0061] The core principle of the SVM classification model is to find an optimal hyperplane to divide data of different categories. In the binary classification problem, assuming there are two types of data points, labeled as positive (+1) and negative (-1), the hyperplane can be represented by the equation ω.x+b=0, where ω is the normal vector of the hyperplane, which determines the direction of the hyperplane, and b is the intercept.

[0062] Before training the SVM model, multimodal data was collected from actual driving experiments to mark the driver's actual operating intentions. The data set was divided into a training set (80%) and a test set (20%). During training, the SVM classification algorithm was used to map multimodal features to different intent categories. The model optimization goal was to minimize the classification error, and cross-validation was used to adjust hyperparameters (such as kernel function type and regularization coefficient).

[0063] In the above embodiment, intention recognition is performed based on the SVM model, which has strong generalization ability, fewer parameters and is more flexible. By inputting the fused feature vector into the SVM model, the driver's operation intention can be accurately and real-time identified, especially in scenarios such as distraction, with a high recognition rate.

[0064] The multimodal fusion data includes time series data. Behavior prediction is performed based on the multimodal fusion data and driving operation intention to obtain a driving behavior prediction result, including: inputting the multimodal fusion data into a time series prediction model to perform driving behavior prediction, and using the driving operation intention as a reference in the prediction process to obtain a driving behavior prediction result.

[0065] Exemplarily, the feature vector F of a period of time (such as the past few seconds) is input into the time series prediction model (Long-Short Term Memory (LSTM)) in the form of a time series, and the output is the driver's next operation (such as turning, slowing down or changing lanes).

[0066] Long-Short Term Memory (LSTM) is a special type of Recurrent Neural Network (RNN). RNN has the problem of gradient vanishing or gradient exploding when processing sequence data, and it is difficult to effectively process long sequence data. LSTM can remember information in a longer sequence, so as to better model the sequence data.

[0067] The LSTM model processes the behavior in time steps. The LSTM model processes the input data in time steps per second to continuously predict future driving behavior. For each time step, the LSTM model calculates the driver's possible next action and outputs it in the form of a label, such as "about to accelerate" or "about to change lanes", and temporarily caches the prediction results on the edge device.

[0068] The loss function used by the LSTM model in behavior prediction is the mean squared error (MSE). The mean squared error is an indicator widely used in many fields such as statistics, machine learning, and signal processing to measure the error between the predicted value and the true value, and is used to evaluate the quality of the model prediction effect. The loss function is shown in formula (5):

[0069]

[0070] In formula (5), y i is the actual behavior label, is the model's predicted value.

[0071] Based on the environmental data detected by the camera and lidar, the prediction results are dynamically adjusted on the edge device to adapt to changes such as pedestrians and obstacles.

[0072] In the above embodiment, a long short-term memory network (LSTM) model is used to predict driving behavior, combining the driver's physiological signals and driving status data to predict the driver's next operation. Through the time series processing capability of the LSTM model, high-precision prediction of future driving behaviors (such as acceleration, steering, lane change, etc.) is achieved.

[0073] The intelligent assisted driving method based on multimodal data also includes: obtaining the driver's driving status data based on physiological data; controlling the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result, including: controlling the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result and the driving status data.

[0074] Exemplarily, the physiological data includes at least one of eye movement data and EEG data, the driving status data includes at least one of visual concentration data obtained based on the eye movement data and cognitive load data obtained based on the EEG data; the assisted driving mode includes a driving warning mode; based on the driving behavior prediction result and the driving status data, the vehicle is controlled to drive based on the assisted driving mode, including: when the driving behavior prediction result indicates driving acceleration and at least one of the visual concentration data and the cognitive load data indicates physiological abnormality, the vehicle is controlled to drive based on the driving warning mode.

[0075] Specifically, at least one of the visual concentration data and the cognitive load data indicates a physiological abnormality, including: obtaining a first physiological result based on the visual concentration data and a first weight, and obtaining a second physiological result based on the cognitive load data and a second weight, wherein the first weight and the second weight can be dynamically adjusted based on the environment; obtaining a comprehensive physiological result based on the first physiological result and the second physiological result; comparing the comprehensive physiological result with a preset physiological threshold to obtain a comparison result, wherein the comparison result is used to indicate the physiological abnormality.

[0076] For example, based on the visual concentration data and cognitive load data, the comprehensive physiological result W is calculated as shown in formula (6):

[0077] W=k1·C e +k2·A v (6)

[0078] In formula (6), k1 is the second weight, k2 is the first weight, C e is cognitive load data, A v It is the visual concentration data. If W exceeds the preset physiological threshold, an early warning is triggered and the vehicle is controlled to drive based on the driving early warning mode.

[0079] The driving warning mode includes: issuing visual or auditory warnings when an abnormal driver state is detected.

[0080] Exemplarily, the physiological data includes electrocardiogram data, the driving status data includes fatigue status data, the assisted driving mode includes a lane keeping assist mode, and based on the driving behavior prediction result and the driving status data, the vehicle is controlled to drive based on the assisted driving mode, including: when the driving behavior prediction result indicates a lane change and the fatigue status data indicates that the current fatigue level is lower than a first preset fatigue level, controlling the vehicle to drive based on the lane keeping assist mode.

[0081] For example, ECG data collects the driver's heart rate and heart rate variability (HRV) through an ECG sensor, calculates HRV (heart rate variability) parameters based on ECG signals, and uses the standard deviation of normal heart beat intervals (SDNN) in HRV as a fatigue index to analyze the driver's fatigue state in real time. The fatigue index calculation method is shown in formula (7):

[0082]

[0083] When FI exceeds a certain threshold (first preset fatigue level), the system determines that the driver is in a fatigue state. At this time, when the driving behavior prediction result indicates driving lane change, the vehicle is controlled to drive based on the lane keeping assist mode.

[0084] The lane keeping assist mode includes: when it is predicted that the driver's next operation may cause the vehicle to deviate from the lane and the system detects that the driver is in a state of fatigue, the lane keeping assist function is automatically activated. The lane keeping assist system keeps the vehicle in the center of the lane through light steering control. According to the driver's actual operation and vehicle position, the system adjusts the steering force in real time to ensure that the vehicle drives safely in the lane.

[0085] Exemplarily, the physiological data includes electrocardiogram data, the driving status data includes fatigue status data, the assisted driving mode includes a driving warning mode, and based on the driving behavior prediction result and the driving status data, the vehicle is controlled to drive based on the assisted driving mode, including: when the driving behavior prediction result indicates driving acceleration and the fatigue status data indicates that the current fatigue level is lower than a second preset fatigue level, the vehicle is controlled to drive based on the driving warning mode.

[0086] For example, the ECG data collects the driver's heart rate and heart rate variability (HRV) through the ECG sensor, calculates the HRV (heart rate variability) parameters based on the ECG signal, and uses the standard deviation of normal heart beat intervals (SDNN) in HRV as the fatigue index for real-time analysis of the driver's fatigue state. The fatigue index calculation method is shown in formula (7). When FI exceeds a certain threshold (the second preset fatigue level), the system determines that the driver is in a fatigue state. At this time, when the driving behavior prediction result indicates driving acceleration, the vehicle is controlled to drive based on the driving warning mode.

[0087] Among them, according to the individual differences of different drivers, the fatigue threshold can be dynamically adjusted to adapt to the different physiological states of the drivers.

[0088] The driving warning mode includes: when the system predicts the driver's intention to accelerate and detects fatigue, the system will make a sound or display a prompt to remind the driver to stay alert; if the driver tries to accelerate while fatigued, the system triggers an acceleration reminder on the edge device and reminds the driver through the on-board display or voice notification, reminding the driver to avoid unnecessary acceleration when fatigued to reduce the risk of accidents.

[0089] In another example, during driving, the strength of the assistance can be dynamically adjusted according to the driver's state. For example, as the driver's fatigue state increases, the strength of the lane keeping assistance will be increased accordingly to provide more stable support.

[0090] In the above embodiment, driving assistance control is performed based on behavior prediction and fatigue state detection, providing personalized assistance functions such as lane keeping assistance and acceleration reminder, and dynamically adjusting the assistance intensity. Through real-time driving behavior prediction combined with driver fatigue state detection, personalized assistance and warning are provided when necessary to enhance driving safety and comfort.

[0091] In another example, an intelligent assisted driving system based on multimodal data achieves personalized model optimization through incremental learning, so that the intention recognition and behavior prediction models can adapt to the driver's long-term driving habits.

[0092] For example, based on the adaptive gradient optimization (AdaGrad) algorithm, the model parameters are updated in real time through accumulated data, allowing the system to gradually adapt to the habits of individual drivers.

[0093] Adaptive Gradient Algorithm (AdaGrad) is an optimization algorithm used to update the parameters of a machine learning model (especially a neural network) when training it. It belongs to the family of gradient-based optimization algorithms and was proposed to solve the problem of low efficiency of traditional gradient descent algorithms when processing different features with different scales or sparse data.

[0094] In another example, edge devices record multimodal data and system interaction data for subsequent analysis and model optimization.

[0095] In the above embodiment, personalized model optimization is achieved through incremental learning, so that the intention recognition and behavior prediction models can adapt to the driver's long-term driving habits and be more flexible. Based on the adaptive gradient optimization (AdaGrad) algorithm, the model parameters are updated in real time through accumulated data to dynamically optimize the accuracy of driving intention and behavior prediction.

[0096] Figure 2 A block diagram of an intelligent assisted driving system based on multimodal data provided in another embodiment of the present invention.

[0097] The embodiment of the present invention provides an intelligent assisted driving system 200 based on multimodal data. Figure 2 The intelligent assisted driving system 200 based on multimodal data includes: an acquisition module 210, a fusion module 220, a first acquisition module 230, a second acquisition module 240, and a control module 250.

[0098] Exemplarily, the acquisition module 210 is used to acquire the driver's physiological data and the vehicle driving data.

[0099] Exemplarily, the fusion module 220 is used to perform data fusion on the physiological data and the vehicle driving data to obtain multimodal fusion data.

[0100] Exemplarily, the first obtaining module 230 is used to perform intention recognition based on multimodal fusion data to obtain the driver's driving operation intention.

[0101] Exemplarily, the second obtaining module 240 is used to perform behavior prediction based on the multimodal fusion data and the driving operation intention to obtain a driving behavior prediction result.

[0102] Exemplarily, the control module 250 is used to control the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result.

[0103] It can be understood that for the specific description of the driving assistance control system 200 based on multimodal data fusion, reference can be made to the above description of the intelligent assisted driving method based on multimodal data, which will not be repeated here.

[0104] Exemplarily, the fusion module 220 is further used to: fuse the physiological data and the vehicle driving data based on a Kalman fusion method to obtain multimodal fusion data.

[0105] Exemplarily, the multimodal fusion data includes a feature vector, and the Kalman fusion method is used to fuse the physiological data and the vehicle driving data to obtain the multimodal fusion data, including: for the multimodal data composed of the physiological data and the vehicle driving data, a state equation is established for each modal data to obtain multiple state equations; based on the multiple state equations, a joint state equation is obtained, wherein the joint state equation includes weight parameters corresponding to each modal data; based on the joint state equation, each modal data is updated to obtain the feature vector, wherein the weight parameter can be dynamically adjusted based on the Kalman gain of each modal data.

[0106] Exemplarily, the first obtaining module 230 is further used to: input the multimodal fusion data into a classification model for intention recognition to obtain the driver's driving operation intention.

[0107] Exemplarily, the multimodal fusion data includes time series data, and the second acquisition module 240 is also used to: input the multimodal fusion data into the time series prediction model to perform driving behavior prediction, and use the driving operation intention as a reference in the prediction process to obtain a driving behavior prediction result.

[0108] Exemplarily, the intelligent assisted driving system 200 of multimodal data also includes: a third acquisition module, used to obtain the driver's driving status data based on the physiological data; the control module 250 is also used to: control the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result and the driving status data.

[0109] Exemplarily, the physiological data includes at least one of eye movement data and EEG data, the driving status data includes at least one of visual concentration data obtained based on the eye movement data and cognitive load data obtained based on the EEG data; the assisted driving mode includes a driving warning mode; controlling the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result and the driving status data includes: when the driving behavior prediction result indicates driving acceleration and at least one of the visual concentration data and the cognitive load data indicates physiological abnormality, controlling the vehicle to drive based on the driving warning mode.

[0110] Exemplarily, at least one of the visual concentration data and the cognitive load data indicates a physiological abnormality, including: obtaining a first physiological result based on the visual concentration data and a first weight, and obtaining a second physiological result based on the cognitive load data and a second weight, wherein the first weight and the second weight can be dynamically adjusted based on the environment; obtaining a comprehensive physiological result based on the first physiological result and the second physiological result; comparing the comprehensive physiological result with a preset physiological threshold to obtain a comparison result, wherein the comparison result is used to indicate the physiological abnormality.

[0111] Exemplarily, the physiological data includes electrocardiogram data, the driving status data includes fatigue status data, the assisted driving mode includes a lane keeping assist mode, and controlling the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result and the driving status data includes: when the driving behavior prediction result indicates a lane change and the fatigue status data indicates that a current fatigue level is lower than a first preset fatigue level, controlling the vehicle to drive based on the lane keeping assist mode.

[0112] Exemplarily, the physiological data includes electrocardiogram data, the driving status data includes fatigue status data, the assisted driving mode includes a driving warning mode, and controlling the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result and the driving status data includes: when the driving behavior prediction result indicates driving acceleration and the fatigue status data indicates that the current fatigue level is lower than a second preset fatigue level, controlling the vehicle to drive based on the driving warning mode.

[0113] Figure 3 A schematic diagram of the edge device structure provided for another embodiment of the present invention.

[0114] like Figure 3 As shown, the edge device of this embodiment includes a processor 301 and a memory 302, wherein the memory 302 stores programs or instructions that can be run on the processor 301, and when the programs or instructions are executed by the processor 301, the steps of any of the above-mentioned embodiments of the intelligent assisted driving method based on multimodal data are implemented.

[0115] Intelligent assisted driving methods based on multimodal data are all completed in the vehicle edge devices (such as NVIDIA Jetson or high-performance embedded ARM modules) to ensure millisecond response. NVIDIA Jetson is a series of development boards and modules launched by NVIDIA for embedded edge computing and artificial intelligence applications. It has powerful computing power and can efficiently handle complex artificial intelligence tasks. Each module is a complete modular system that integrates components such as CPU, GPU, power management chip, memory and flash memory; ARM module usually refers to a hardware module with specific functions and interfaces built by a processor based on the ARM architecture. It is a processor architecture based on the reduced instruction set (RISC).

[0116] When implementing intelligent assisted driving methods based on multimodal data, the on-board edge devices follow the following order: multimodal data acquisition and preprocessing → feature fusion → intention recognition (SVM) → behavior prediction (LSTM) → personalized assistance.

[0117] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method of any one of the above embodiments are implemented.

[0118] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of the present invention, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0119] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0120] In the description of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In the present invention, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0121] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0122] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of the present invention, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0123] In the present invention, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "connected" and "fixed" etc. in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific implementation situation.

[0124] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0125] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An intelligent assisted driving method based on multimodal data, characterized in that: The method comprises: Obtain driver's physiological data and vehicle driving data; Performing data fusion on the physiological data and the vehicle driving data to obtain multimodal fusion data; Performing intention recognition based on the multimodal fusion data to obtain the driver's driving operation intention; Performing behavior prediction based on the multimodal fusion data and the driving operation intention to obtain a driving behavior prediction result; Based on the driving behavior prediction result, the vehicle is controlled to drive based on the assisted driving mode.

2. The method according to claim 1, characterized in that The step of fusing the physiological data and the vehicle driving data to obtain multimodal fusion data includes: The physiological data and the vehicle driving data are fused based on the Kalman fusion method to obtain multimodal fusion data.

3. The method according to claim 2, characterized in that The multimodal fusion data includes a feature vector, and the physiological data and the vehicle driving data are fused based on the Kalman fusion method to obtain the multimodal fusion data, including: For the multimodal data composed of the physiological data and the vehicle driving data, a state equation is established for each modal data to obtain a plurality of state equations; Based on the multiple state equations, a joint state equation is obtained, wherein the joint state equation includes a weight parameter corresponding to each modal data; Each modal data is updated based on the joint state equation to obtain the characteristic vector, wherein the weight parameter can be dynamically adjusted based on the Kalman gain of each modal data.

4. The method according to claim 1, characterized in that: The performing intention recognition based on the multimodal fusion data to obtain the driver's driving operation intention includes: The multimodal fusion data is input into a classification model for intention recognition to obtain the driver's driving operation intention.

5. The method according to claim 1, characterized in that The multimodal fusion data includes time series data, and the behavior prediction based on the multimodal fusion data and the driving operation intention is performed to obtain a driving behavior prediction result, including: The multimodal fusion data is input into the time series prediction model to perform driving behavior prediction, and the driving operation intention is used as a reference during the prediction process to obtain a driving behavior prediction result.

6. The method according to any one of claims 1 to 5, characterized in that: The method further includes: obtaining driving state data of the driver based on the physiological data; and controlling the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result, including: Based on the driving behavior prediction result and the driving state data, the vehicle is controlled to drive based on the assisted driving mode.

7. The method according to claim 6, characterized in that The physiological data includes at least one of eye movement data and electroencephalogram data; the driving state data includes at least one of visual concentration data obtained based on the eye movement data and cognitive load data obtained based on the electroencephalogram data; the assisted driving mode includes a driving warning mode; and controlling the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result and the driving state data includes: When the driving behavior prediction result indicates driving acceleration and at least one of the visual concentration data and the cognitive load data indicates physiological abnormality, the vehicle is controlled to drive based on the driving warning mode.

8. The method according to claim 7, characterized in that At least one of the visual concentration data and the cognitive load data indicates a physiological abnormality, including: Obtaining a first physiological result based on the visual concentration data and a first weight, and obtaining a second physiological result based on the cognitive load data and a second weight, wherein the first weight and the second weight can be dynamically adjusted based on the environment; Obtaining a comprehensive physiological result based on the first physiological result and the second physiological result; The comprehensive physiological result is compared with a preset physiological threshold to obtain a comparison result, wherein the comparison result is used to indicate physiological abnormality.

9. The method according to claim 6, characterized in that The physiological data includes electrocardiogram data, the driving state data includes fatigue state data, the assisted driving mode includes a lane keeping assist mode, and the controlling the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result and the driving state data includes: When the driving behavior prediction result indicates a lane change and the fatigue status data indicates that the current fatigue level is lower than a first preset fatigue level, the vehicle is controlled to drive based on the lane keeping assist mode.

10. The method according to claim 6, characterized in that The physiological data includes electrocardiogram data, the driving state data includes fatigue state data, the assisted driving mode includes a driving warning mode, and the controlling the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result and the driving state data includes: When the driving behavior prediction result indicates driving acceleration and the fatigue status data indicates that the current fatigue level is lower than a second preset fatigue level, the vehicle is controlled to drive based on the driving warning mode.

11. An intelligent assisted driving system based on multimodal data, characterized in that: The system comprises: An acquisition module, used to acquire the driver's physiological data and vehicle driving data; A fusion module, used for fusing the physiological data and the vehicle driving data to obtain multimodal fusion data; An intention recognition module, used to perform intention recognition based on the multimodal fusion data to obtain the driver's driving operation intention; A behavior prediction module, used to perform behavior prediction based on the multimodal fusion data and the driving operation intention to obtain a driving behavior prediction result; The control module is used to control the vehicle to drive based on the assisted driving mode based on the driving behavior prediction result.

12. An edge device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images