Driver watching behavior safety reminding method and device suitable for intelligent cabin
By using hybrid deep learning models in the smart cockpit to predict the driver's gaze area and issue safety reminders, the problem of distraction when performing touch screen interactive tasks is solved, driving safety is improved, and driver gaze is achieved Active safety monitoring and reminder of behavior.
Patent Information
- Application Number
- CN202410982957.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-22
AI Technical Summary
The existing driver gaze behavior monitoring methods cannot accurately predict the gaze area when the driver performs interactive tasks with full touch screen, resulting in distraction of the driver's attention and increasing the risk of traffic accidents. Especially in smart cockpit environments, there is a lack of active safety reminder function.
Adopting a hybrid depth model based on BiLSTM, CNN and attention mechanism, by constructing a human-computer interaction data set for intelligent cockpit touch screen, extracting time series data features, preprocessing and building a data matrix, training a hybrid deep learning model, and predicting driver gaze and send safety prompts when necessary to actively remind drivers to pay attention to the road.
Accurate prediction and safety reminder of driver gaze behavior is achieved, driving risks are reduced, drivers are improved in the smart cockpit environment, and potential conflicts in human-computer interaction are effectively avoided.
Smart Images

Figure CN118918569A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method and a device for safely reminding a driver's gaze behavior in a smart cockpit, and belongs to the technical field of road traffic safety. Background Art
[0002] Driver gaze behavior is a key concern in the field of road traffic safety. So far, there are methods for monitoring driver gaze behavior, but these methods monitor driver gaze behavior in real time, and have never predicted driver gaze behavior and actively interacted to improve safety. If the autonomous driving system can accurately predict the driver's gaze area when the driver performs a full touch screen interactive task, and actively issue safety prompts when necessary, it can remind the driver to shift his attention to the road area in time to reduce potential driving risks. On the other hand, in the future human-machine co-driving stage, predicting the driver's gaze area will help the intelligent driving system make the right decision on when to take over control.
[0003] With the intelligence of vehicles, large touch screens have become the main interface for human-vehicle interaction. This development has created more possibilities for drivers to perform visual interaction tasks. When the driver performs touch screen interaction tasks, his attention will switch back and forth between the touch screen and the road. At the same time, the driver also needs to physically interact by clicking or sliding the controls on the touch screen to keep his operations consistent with the visual focus. This increase in visual and manual workload will aggravate visual and manual distractions, causing the driver to be unable to perceive potential dangers on the road, thereby causing serious traffic accidents. Therefore, the present invention focuses on the above-mentioned problems and proposes a driver gaze behavior safety reminder method and device suitable for a smart cockpit, which aims to achieve accurate prediction of the driver's attention area, gain insight into the driver's behavior in advance, and actively interact with the driver at the right time and send safety reminder information. Summary of the invention
[0004] Purpose of the invention: In view of the deficiencies in the prior art, the present invention provides a method and device for safety reminder of driver's gaze behavior applicable to a smart cockpit, so as to solve the problems mentioned in the above background technology.
[0005] Technical solution: A driver gaze behavior safety reminder method applicable to a smart cockpit, comprising the following steps:
[0006] S1: Constructing a human-computer interaction dataset for smart cockpit touch screens;
[0007] S2: time series data feature extraction;
[0008] S3: data preprocessing and constructing time series matrix;
[0009] S4: Construct a hybrid deep model CNN-BiLSTM-Attention based on BiLSTM, CNN and attention mechanism;
[0010] S5: CNN-BiLSTM-Attention training, testing, tuning and integration;
[0011] S6: Execute the gaze behavior safety reminder method based on the prediction result of the gaze area.
[0012] The step S1 is specifically as follows:
[0013] S1.1: Select N0 models with full touch screen in smart cockpit;
[0014] S1.2: Design a touch screen interaction test and select N1 interactive tasks based on the vehicle's interactive functions;
[0015] S1.3: Select N2 drivers with different genders, driving experience and ages; each driver performs each similar interactive task N3 times; design N4 actual driving routes, each of which has N5 different vehicle speeds; the total number of interactive task samples finally obtained is N0×N1×N2×N3×N4×N5=N T ;
[0016] S1.4: The facial image and hand image of the driver are collected respectively through the dual-source camera monitoring system; the vehicle data collector is used to collect the driver's operation data and vehicle motion state parameters, and save them as text data.
[0017] The step S2 is specifically as follows:
[0018] S2.1: embedding visual feature extraction models into the microprocessor unit - token-based head pose estimator TokenHPE, gaze region recognition model iTracker and hand key point extraction model MediaPipe-Hands;
[0019] S2.2: Extract N from the original data of step S1 T Interactive task process fragment;
[0020] S2.3: using the data acquisition unit to collect and store data, and sending it to the microprocessing unit, where the head posture, gaze area, and hand key points are respectively processed for each interaction sample;
[0021] S2.4: In the microprocessor unit, a token-based head posture estimator TokenHPE is used to extract the driver's head posture time series, where the head posture includes the head pitch angle Y, the head yaw angle P, and the head roll angle R;
[0022] S2.5: In the microprocessing unit, a gaze area estimation model is used to extract a gaze area label G; the value of G is 0 and 1, 0 represents no gaze on the full touch screen, and 1 represents gaze on the full touch screen;
[0023] S2.6: In the microprocessing unit, the hand key point extraction model MediaPipe-Hands is used to extract the driver's hand key point time series;
[0024] S2.7: In the microprocessor unit, hand motion parameters are extracted from the key points of the driver's hand, including hand movement speed Hms, hand lateral position Hx, hand longitudinal position Hy, and finger bending angle Fb.
[0025] The step S3 is specifically as follows:
[0026] S3.1: Construct an interaction sample parameter matrix. Each interaction sample is represented by a parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}; where V, δ, L l They represent the vehicle speed, steering wheel angle, and vehicle lateral position extracted from the vehicle data acquisition unit;
[0027] S3.2: Use the Savitzky-Golay algorithm to smooth the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll};
[0028] S3.3: Use the maximum and minimum method to normalize the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll};
[0029] S3.4: Generate feature matrix X for input model using sliding window method i and predicted region label G i ;
[0030] In a certain time interval Δt, the cumulative duration of the interactive task is represented by T, the start of the task is represented by t0, and the end of the task is represented by te, T = te-t0;
[0031] The frequency of the time series data is 30 Hz, which means that there are 30 sets of different feature data in a one-second time interval; if the sliding window step length S = 1s, and the prediction time interval Δt = 0.5s, it means that the input feature data length of the model is 30, and it can predict the driver's gaze area G after 0.5s, and the values of S and Δt satisfy S≥2Δt; X i It is a matrix with a length of 30 and a width of 11, 30 is the time length, 11 is the parameter dimension, G i is a number with a value of 0 or 1, X i and Gi The time interval is Δt.
[0032] The step S4 is specifically as follows:
[0033] S4.1: Extract the underlying features of the input multidimensional time series data based on the CNN convolutional neural network model;
[0034] The formula is:
[0035]
[0036]
[0037] where f cnn represents the convolutional neural network module, represents the network layer in the convolution module, l = 1, 2, 3 represents the number of convolution layers, b represents the network layer bias, W represents the parameter weight, X l represents the output value of each layer, d represents the convolution kernel parameter, t represents the convolution kernel step size, is the input of the convolutional neural network model, is the output of the convolutional neural network model, the first layer of convolution Input
[0038] S4.2: Capturing long-term causal relationships based on the BiLSTM bidirectional long-short-term network model;
[0039] The calculation process of BiLSTM is as follows:
[0040]
[0041]
[0042] i t =σ(X t W xi +H t-1 W hi +b i )
[0043] f t =σ(X t W xf +H t-1 W hf +b f )
[0044] O t =σ(X t W xo +H t-1 W ho +b o )
[0045]
[0046]
[0047] h t =O t ⊙tanh(C t )
[0048] Where W and b correspond to parameter weight and bias respectively, i, f, O represent input gate, forget gate and output gate respectively, C and H represent candidate memory unit and forget memory unit respectively;
[0049] S4.3: Combined with the ability of the Attention mechanism to allocate weights, the model can adaptively allocate training weights. The calculation formula is as follows:
[0050] y=f attention (X)
[0051] X=[x1,x2,x3,....,x n ]
[0052]
[0053] s(x i ,q)=v T tanh(wx i +uq)
[0054]
[0055] Among them, X is the input, α is the feature importance weight, w and q are model parameters, and y is the final output result;
[0056] S4.4: Use different BiLSTM modules to extract temporal information from the output of each convolutional layer in the CNN model;
[0057] S4.5: Add respective attention mechanism modules to the forward and backward hidden layers of each parallel BiLSTM module to make the relevant information within each BiLSTM module more concentrated and reduce the potential interference between key information;
[0058] Add attention f to the front and back directions attention The process is as follows:
[0059]
[0060]
[0061]
[0062] in and are the forward and backward hidden layer outputs, n represents the number of hidden layers, and U, w, and b represent parameter weights, respectively;
[0063] After calculations from S4.1 to S4.5, the final architecture HDLM of hybrid deep learning can be built. HDLM contains three layers of CNN and Each layer of CNN is connected to an independent BiLSTM module Then each BiLSTM module is connected to the bidirectional attention module Finally, through the shallow neural network f nn Output the gaze area prediction result G. The above calculation process is expressed by the formula:
[0064]
[0065]
[0066]
[0067] G=f nn (y 1 +y 2 +y 3 )
[0068] Where: y 1 is the output result of the first layer attention module; y 2 is the output result of the second layer attention module; y 3 is the output result of the third-layer attention module; G is the prediction result of the gaze area.
[0069] The step S5 is specifically as follows:
[0070] S5.1: With {X i , G i} Perform cyclic training and testing on the HDLM model as a sample, set the number of iterations to 200, the batch size to 64, the learning rate to 0.001, and save the results of each round of training and testing after the training starts; repeat step S5.1 until the model test accuracy reaches more than 98% to reach the expected target, and save the model parameter weights under the test accuracy;
[0071] S5.2: Build a model test hardware-in-the-loop test platform, including a driving simulator, a virtual simulation environment, a computing server, a six-degree-of-freedom vehicle platform, a vehicle data acquisition unit, a dual-source camera, a driver status monitoring system, a data acquisition unit, a microprocessor and a logic judgment unit, and recruit Nx drivers to verify the effect;
[0072] During the test, the driver sits on a six-degree-of-freedom vehicle platform and drives in a simulated environment, operating the central control screen while driving. The dual-source camera collects real-time facial and hand images of the driver, and the vehicle data collector collects vehicle operation and running data. The data mobile phone unit transmits the collected data to the microprocessor unit to calculate the real-time gaze area prediction results.
[0073] S5.3: The Likert scale satisfaction is calculated for Nx drivers. When the satisfaction rate reaches above 95%, it is determined that the proposed method meets the design standards and the algorithm is integrated into the microprocessor unit.
[0074] 7. The driver gaze behavior safety reminder method applicable to the smart cockpit according to claim 1, characterized in that the step S6 is specifically:
[0075] S6.1: Set the visual distraction duration threshold TD0, rear-end collision time threshold TTC0, and vehicle deviation time distance TLC0 according to the driver distraction theory and safe driving theory;
[0076] S6.2: Obtain the final HDLM model parameters from step S5.1;
[0077] S6.3: The system starts working, and the data acquisition unit collects data in real time; the DMS system determines the driver’s current T X Check whether to start controlling the full touch screen at the moment, if yes, proceed to the next step;
[0078] S6.4: Data acquisition unit collects T X to T X +S time period data {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}, real-time construction of the model input parameter matrix {X i , G i};
[0079] S6.5: The parameter matrix {X i , G i} is sent to the HDLM algorithm module in the microprocessor to X +S+Δt time to predict the fixation area G;
[0080] S6.6: In the microprocessor, calculate T X + The total time TD that the driver continuously stares at the entire touch area at time S;
[0081] Record T X to T X At +S, the model HDLM predicts the number of times G=1. If the number is NG, the fixation time is NG / 30;
[0082] S6.7: Calculate T in the microprocessor X + The TTC between the host vehicle and the leading vehicle at time S
[0083] TTC = (vf - vs) / d, where vf and vs represent the vehicle speeds of the leading vehicle and the host vehicle respectively, and d represents the distance between the leading vehicle and the host vehicle
[0084] S6.8: Calculate T in the microprocessor X + The time-to-lane-change TLC of the host vehicle at time S
[0085] TLC = y / LV, where y represents the lateral distance of the vehicle center from the lane line, and LV represents the lateral speed of the vehicle
[0086] S6.9: When the TD at time T X + S > 2 s, and the TTC at time T X + S < TTC0, and the TLC at time T X + S < TLC0, and the G at time T X + S + Δt = 1, the microprocessing unit controls the decision to give a safety reminder
[0087] S6.10: The microprocessor controls the in-vehicle unit to pop up a prompt message at the central upper position of the touch screen interface; meanwhile, the microprocessor controls the in-vehicle unit to emit a warning sound from the audio unit
[0088] S6.11: After the DMS system determines that the driver is driving normally, the system ends its operation
[0089] A device for safety reminder of the driver's gaze behavior applicable to an intelligent cockpit, comprising:
[0090] Dual-source camera monitoring system: The dual-source camera monitoring system consists of two RGB cameras installed at different positions. One camera captures the facial image of the driver, and the other camera is used to capture the hand image of the driver; during operation, the dual-source cameras are respectively connected to the data acquisition unit
[0091] DMS system: It can obtain the driver's behavior state in real time, determine whether the driver starts to perform a touch screen interaction task through the DMS system, is connected to the data acquisition unit through a signal harness, and transmits data to the data acquisition unit
[0092] Vehicle data collector: The vehicle data collector is a hardware unit for collecting the vehicle motion state and the driver operation state; during operation, it collects data in real time and transmits the data to the data acquisition unit through a signal harness
[0093] Data acquisition unit: The data acquisition unit is connected to the vehicle data acquisition unit and the dual-source camera monitoring system; it collects and stores the data matrix within a period of time, waiting for the call of the microprocessor unit;
[0094] Microprocessing unit: One end of the microprocessing unit is connected to the data acquisition unit through an input data interface, and the other end is connected to the safety prompt device through an output data interface;
[0095] The microprocessing unit includes an algorithm module and a logic judgment module. The algorithm module integrates the HDLM model to predict the driver's gaze area. The logic judgment module also includes a function to output a safety warning signal based on the driver's gaze area, the driver's continuous gaze time and the driving risk index.
[0096] Safety reminder device: includes a visual reminder module and a voice reminder module, both modules are integrated with the vehicle-mounted assembly; for the visual reminder module, when a safety reminder is needed, the reminder information will appear in a small window on the full touch screen, and the content will be displayed in the form of a pop-up information interface; for the voice reminder module, when a safety reminder is needed, it will emit a reminder sound in the voice module of the full touch screen.
[0097] Beneficial effects: The present invention develops a hybrid deep learning model, takes the driver's vehicle control state, vehicle movement state, driver's head posture, driver's hand movement state, and driver's gaze area as model input parameters, and takes the driver's gaze area as model output. The driver's gaze behavior prediction is achieved through model training, testing and integration; then, according to the driver's gaze area prediction results, distraction theory and safe driving theory, the timing of sending safety prompt information is determined, and the driver is actively interacted with through visual information and voice information to prompt the driver to regulate his gaze behavior. Compared with traditional methods, this method is aimed at the full touch screen interaction scene of the smart cockpit, can predict the driver's gaze area, actively interact with the driver to improve driving safety, and can effectively avoid human-machine conflicts. BRIEF DESCRIPTION OF THE DRAWINGS
[0098] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0099] Figure 1 This is a flow chart of the driver's gaze behavior safety reminder method of the present invention.
[0100] Figure 2 This is a diagram of the input and output feature construction method of the present invention.
[0101] Figure 3 It is a hybrid deep learning algorithm unit.
[0102] Figure 4 FIG. 4 is a diagram of a gaze area prediction device according to the present invention. DETAILED DESCRIPTION
[0103] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0104] In the description of the present invention, it is necessary to understand that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0105] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.
[0106] like Figures 1 to 4 As shown, a driver gaze behavior safety reminder method applicable to a smart cockpit includes the following steps:
[0107] S1: Constructing a human-computer interaction dataset for smart cockpit touch screens;
[0108] S1.1: Select N0 models with full touch screen in smart cockpit;
[0109] S1.2: Design a touch screen interaction test and select N1 interactive tasks based on the vehicle's interactive functions;
[0110] S1.3: Select N2 drivers with different genders, driving experience and ages; each driver performs each similar interactive task N3 times; design N4 actual driving routes, each of which has N5 different vehicle speeds; the total number of interactive task samples finally obtained is N0×N1×N2×N3×N4×N5=N T ;
[0111] S1.4: The facial image and hand image of the driver are collected respectively through the dual-source camera monitoring system; the vehicle data collector is used to collect the driver's operation data and vehicle motion state parameters, and save them as text data.
[0112] S2: time series data feature extraction;
[0113] S2.1: embedding the visual feature extraction model TokenHPE token-based head pose estimator, iTracker gaze region recognition model and MediaPipe-Hands hand key point extraction model into the microprocessor;
[0114] S2.2: Extract N from the original data of step S1 T Interactive task process fragment;
[0115] S2.3: using the data acquisition unit to collect and store data, and sending it to the microprocessing unit, where the head posture, gaze area, and hand key points are respectively processed for each interaction sample;
[0116] S2.4: In the microprocessor unit, a token-based head posture estimator TokenHPE is used to extract the driver's head posture time series, where the head posture includes the head pitch angle Y, the head yaw angle P, and the head roll angle R;
[0117] S2.5: In the microprocessing unit, a gaze area estimation model is used to extract a gaze area label G; the value of G is 0 and 1, 0 represents no gaze on the full touch screen, and 1 represents gaze on the full touch screen;
[0118] S2.6: In the microprocessing unit, the hand key point extraction model MediaPipe-Hands is used to extract the driver's hand key point time series;
[0119] S2.7: In the microprocessor unit, hand motion parameters are extracted from the key points of the driver's hand, including hand movement speed Hms, hand lateral position Hx, hand longitudinal position Hy, and finger bending angle Fb.
[0120] S3: data preprocessing and constructing time series matrix;
[0121] S3.1: Construct an interaction sample parameter matrix. Each interaction sample is represented by a parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}; where V, δ, L l They represent the vehicle speed, steering wheel angle, and vehicle lateral position extracted from the vehicle data acquisition unit;
[0122] S3.2: Use the Savitzky-Golay algorithm to smooth the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll};
[0123] S3.3: Use the maximum and minimum method to normalize the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll};
[0124] S3.4: Generate feature matrix X for input model using sliding window method i and predicted region label G i ;
[0125] In a certain time interval Δt, the cumulative duration of the interactive task is represented by T, the start of the task is represented by t0, and the end of the task is represented by te, T = te-t0;
[0126] The frequency of the time series data is 30Hz, which means there are 30 sets of different feature data in a one-second time interval. If the sliding window step length S = 1s, and the prediction time interval Δt = 0.5s, it means that the input feature data length of the model is 30, and it can predict the driver's gaze area G after 0.5s. The values of S and Δt satisfy S≥2Δt; X i It is a matrix with a length of 30 and a width of 11, 30 is the time length, 11 is the parameter dimension, G i is a number with a value of 0 or 1, X i and G i The time interval is Δt.
[0127] S4: Construct a hybrid deep model CNN-BiLSTM-Attention based on BiLSTM, CNN and attention mechanism;
[0128] S4.1: Extract the underlying features of the input multidimensional time series data based on the CNN convolutional neural network model;
[0129] The formula is:
[0130]
[0131]
[0132] where f cnnrepresents the convolutional neural network module, represents the network layer in the convolution module, l = 1, 2, 3 represents the number of convolution layers, b represents the network layer bias, W represents the parameter weight, X l represents the output value of each layer, d represents the convolution kernel parameter, t represents the convolution kernel step size, is the input of the convolutional neural network model, is the output of the convolutional neural network model, the first layer of convolution Input
[0133] S4.2: Capturing long-term causal relationships based on the BiLSTM bidirectional long-short-term network model;
[0134] The calculation process of BiLSTM is as follows:
[0135]
[0136]
[0137] i t =σ(X t W xi +H t-1 W hi +b i )
[0138] f t =σ(X t W xf +H t-1 W hf +b f )
[0139] O t =σ(X t W xo +H t-1 W ho +b o )
[0140]
[0141]
[0142] h t =O t ⊙tanh(C t )
[0143] Where W and b correspond to the parameter weight and bias respectively, i, f, O represent the input gate, forget gate and output gate respectively, C and H represent the candidate memory unit and forget memory unit respectively.
[0144] S4.3: Combined with the ability of the Attention mechanism to allocate weights, the model can adaptively allocate training weights. The calculation formula is as follows:
[0145] y=f attention (X)
[0146] X=[x1,x2,x3,....,x n ]
[0147]
[0148] s(x i ,q)=v T tanh(wx i +uq)
[0149]
[0150] Among them, X is the input, α is the feature importance weight, w and q are model parameters, and y is the final output result.
[0151] S4.4: Use different BiLSTM modules to extract temporal information from the output of each convolutional layer in the CNN model;
[0152] S4.5: Add respective attention mechanism modules to the forward and backward hidden layers of each parallel BiLSTM module to make the relevant information within each BiLSTM module more concentrated and reduce the potential interference between key information;
[0153] Add attention f to the front and back directions attention The process is as follows:
[0154]
[0155]
[0156]
[0157] in and They are the forward and backward hidden layer outputs respectively, n represents the number of hidden layers, and U, w, and b represent the parameter weights respectively.
[0158] After calculations from S4.1 to S4.5, the final architecture HDLM of hybrid deep learning can be built. HDLM contains three layers of CNN and Each layer of CNN is connected to an independent BiLSTM module Then each BiLSTM module is connected to the bidirectional attention module Finally, through the shallow neural network fnn Output the gaze area prediction result G. The above calculation process is expressed by the formula:
[0159]
[0160]
[0161]
[0162] G=f nn (y 1 +y 2 +y 3 )
[0163] Where: y 1 is the output result of the first layer attention module; y 2 is the output result of the second layer attention module; y 3 is the output result of the third-layer attention module; G is the prediction result of the gaze area.
[0164] S5: CNN-BiLSTM-Attention training, testing, tuning and integration;
[0165] S5.1: With {X i , G i} Perform cyclic training and testing on the HDLM model as a sample, set the number of iterations to 200, the batch size to 64, the learning rate to 0.001, and save the results of each round of training and testing after the training starts; repeat step S5.1 until the model test accuracy reaches more than 98% to reach the expected target, and save the model parameter weights under the test accuracy;
[0166] S5.2: Build a model test hardware-in-the-loop test platform, including a driving simulator, a virtual simulation environment, a computing server, a six-degree-of-freedom vehicle platform, a vehicle data acquisition unit, a dual-source camera, a driver status monitoring system, a data acquisition unit, a microprocessor, and a logic judgment unit, and recruit Nx drivers to verify the effect;
[0167] During the test, the driver sits on a six-degree-of-freedom vehicle platform and drives in a simulated environment, operating the central control screen while driving. The dual-source camera collects real-time facial and hand images of the driver, and the vehicle data collector collects vehicle operation and running data. The data mobile phone unit transmits the collected data to the microprocessor unit to calculate the real-time gaze area prediction results.
[0168] S5.3: Conduct a Likert scale satisfaction questionnaire survey on Nx drivers. When the satisfaction rate reaches above 95%, it is determined that the proposed method meets the design standards and the algorithm is integrated into the microprocessor unit.
[0169] S6: executing a gaze behavior safety reminder method based on the prediction result of the gaze area;
[0170] S6.1: Set the visual distraction duration threshold TD0, rear-end collision time threshold TTC0, and vehicle deviation time distance TLC0 according to the driver distraction theory and safe driving theory;
[0171] S6.2: Obtain the final HDLM model parameters from step S5.1;
[0172] S6.3: The system starts working, and the data acquisition unit collects data in real time; the DMS system determines the driver’s current T X Check whether to start controlling the full touch screen at the moment, if yes, proceed to the next step;
[0173] S6.4: Data acquisition unit collects T X to T X +S time period data {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}, real-time construction of the model input parameter matrix {X i , G i};
[0174] S6.5: The parameter matrix {X i , G i} is sent to the HDLM algorithm module in the microprocessor to X +S+Δt time to predict the fixation area G;
[0175] S6.6: In the microprocessor, calculate T X + The total time TD that the driver continuously stares at the entire touch area at time S;
[0176] Record T X to T X At +S, the model HDLM predicts the number of times G=1. If the number is NG, the fixation time is NG / 30;
[0177] S6.7: In the microprocessor, calculate T X + TTC of the vehicle in front and the vehicle in front at the moment S;
[0178] TTC = (vf-vs) / d, where vf and vs represent the speeds of the preceding vehicle and the vehicle itself, respectively, and d represents the distance between the preceding vehicle and the vehicle itself;
[0179] S6.8: In the microprocessor, calculate T X + The departure time distance TLC of the vehicle at time S;
[0180] TLC=y / LV, y represents the lateral distance between the center of the vehicle and the lane line, and LV represents the lateral speed of the vehicle.
[0181] S6.9: When T X + TD at the moment of S > 2 s, and T X + TTC at the moment of S < TTC0, and T X + TLC at the moment of S < TLC0, and T X + G at the moment of S + Δt = 1, the micro - processing unit controls the decision - making to give a safety reminder;
[0182] S6.10: The micro - processor controls the vehicle head unit to pop up a prompt message "Please pay attention to the road surface in time!" at the central upper position of the touch - screen interface; meanwhile, the micro - processor controls the vehicle head unit to emit a "beep beep beep" sound from the audio unit.
[0183] S6.11: After the DMS system determines that the driver is driving normally, the system ends its operation.
[0184] A device for safety reminder of a driver's gaze behavior applicable to an intelligent cockpit, comprising:
[0185] Dual - source camera monitoring system: The dual - source camera monitoring system is composed of two RGB cameras with different installation positions. One camera captures the facial image of the driver, and the other camera is used to capture the hand image of the driver; during the working process, the dual - source cameras are respectively connected to the data acquisition unit;
[0186] DMS system: It can obtain the driver's behavior state in real time, determine whether the driver starts to execute a touch - screen interaction task through the DMS system, and transfer the data to the data acquisition unit;
[0187] Vehicle data collector: The vehicle data collector is a hardware unit for collecting the vehicle motion state and the driver's operation state; during the working process, it collects data in real time and transfers the data to the data acquisition unit through a signal harness;
[0188] Data acquisition unit: The data acquisition unit is connected to the vehicle data acquisition unit and the dual - source camera monitoring system; it collects and stores a data matrix within a period of time, waiting for the call of the micro - processing unit;
[0189] Micro - processing unit: One end of the micro - processing unit is connected to the data acquisition unit through an input data interface, and the other end is connected to the safety reminder device through an output data interface;
[0190] The micro - processing unit includes an algorithm module and a logic judgment module. The algorithm module integrates the core algorithm HDLM of the present invention to realize the prediction of the driver's gaze area;
[0191] The function of the logic judgment module is to output a safety reminder signal according to the driver's gaze area, the driver's continuous gaze time, and the driving risk index;
[0192] Safety reminder device: includes a visual reminder module and a voice reminder module, both modules are integrated with the vehicle-mounted assembly; for the visual reminder module, when a safety reminder is needed, the reminder information will appear in a small window on the full touch screen, and the content will be displayed in the form of a pop-up information interface; for the voice reminder module, when a safety reminder is needed, it will emit a reminder sound in the voice module of the full touch screen.
[0193] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0194] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A driver gaze behavior safety reminder method applicable to a smart cockpit, characterized in that: The following steps are involved: S1: Constructing a human-computer interaction dataset for smart cockpit touch screens; S2: time series data feature extraction; S3: data preprocessing and constructing time series matrix; S4: Construct a hybrid deep model CNN-BiLSTM-Attention based on BiLSTM, CNN and attention mechanism; S5: CNN-BiLSTM-Attention training, testing, tuning and integration; S6: Execute a gaze behavior safety reminder method based on the prediction result of the gaze area.
2. The driver gaze behavior safety reminder method applicable to the smart cockpit according to claim 1 is characterized in that: The step S1 is specifically as follows: S1.1: Select N0 models with full touch screen in smart cockpit; S1.2: Design a touch screen interaction test and select N1 interactive tasks based on the vehicle's interactive functions; S1.3: Select N2 drivers with different genders, driving experience and ages; each driver performs each similar interactive task N3 times; design N4 actual driving routes, each of which has N5 different vehicle speeds; the total number of interactive task samples finally obtained is N0×N1×N2×N3×N4×N5=N T ; S1.4: The facial image and hand image of the driver are collected respectively through the dual-source camera monitoring system; the vehicle data collector is used to collect the driver's operation data and vehicle motion state parameters, and save them as text data.
3. The driver gaze behavior safety reminder method applicable to the smart cockpit according to claim 1 is characterized in that: The step S2 is specifically as follows: S2.1: embedding visual feature extraction models into the microprocessor unit - token-based head pose estimator TokenHPE, gaze region recognition model iTracker and hand key point extraction model MediaPipe-Hands; S2.2: Extract N from the original data of step S1 T Interactive task process fragment; S2.3: using the data acquisition unit to collect and store data, and sending it to the microprocessing unit, where the head posture, gaze area, and hand key points are respectively processed for each interaction sample; S2.4: In the microprocessor unit, a token-based head posture estimator TokenHPE is used to extract the driver's head posture time series, where the head posture includes the head pitch angle Y, the head yaw angle P, and the head roll angle R; S2.5: In the microprocessing unit, a gaze area estimation model is used to extract a gaze area label G; the value of G is 0 and 1, 0 represents no gaze on the full touch screen, and 1 represents gaze on the full touch screen; S2.6: In the microprocessing unit, the hand key point extraction model MediaPipe-Hands is used to extract the driver's hand key point time series; S2.7: In the microprocessor unit, hand motion parameters are extracted from the key points of the driver's hand, including hand movement speed Hms, hand lateral position Hx, hand longitudinal position Hy, and finger bending angle Fb.
4. The driver gaze behavior safety reminder method applicable to the smart cockpit according to claim 1 is characterized in that: The step S3 is specifically as follows: S3.1: Construct an interaction sample parameter matrix. Each interaction sample is represented by a parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}; where V, δ, L l They represent the vehicle speed, steering wheel angle, and vehicle lateral position extracted from the vehicle data acquisition unit; S3.2: Use the Savitzky-Golay algorithm to smooth the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}; S3.3: Use the maximum and minimum method to normalize the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}; S3.4: Generate feature matrix X for input model using sliding window method i and predicted region label G i ; In the time interval Δt, the cumulative duration of the interactive task is represented by T, the start of the task is represented by t0, and the end of the task is represented by te, T = te-t0; The frequency of the time series data is 30 Hz, which means there are 30 different sets of feature data within a time interval of one second. The sliding window step size S is set, and the prediction time interval Δt is set. The values of S and Δt satisfy S≥2Δt.
5. The driver gaze behavior safety reminder method applicable to the smart cockpit according to claim 1 is characterized in that: The step S4 is specifically as follows: S4.1: Extract the underlying features of the input multidimensional time series data based on the CNN convolutional neural network model; The formula is: where f cnn represents the convolutional neural network module, represents the network layer in the convolution module, l = 1, 2, 3 represents the number of convolution layers, b represents the network layer bias, W represents the parameter weight, X l represents the output value of each layer, d represents the convolution kernel parameter, t represents the convolution kernel step size, is the input of the convolutional neural network model, is the output of the convolutional neural network model, the first layer of convolution Input S4.2: Capturing long-term causal relationships based on the BiLSTM bidirectional long-short-term network model; The calculation process of BiLSTM is as follows: i t =σ(X t W xi +H t-1 W hi +b i ) f t =σ(X t W xf +H t-1 W hf +b f ) O t =σ(X t W xo +H t-1 W ho +b o ) h t =O t ⊙tanh(C t ) Where W and b correspond to parameter weight and bias respectively, i, f, O represent input gate, forget gate and output gate respectively, C and H represent candidate memory unit and forget memory unit respectively; S4.3: Combined with the ability of the Attention mechanism to allocate weights, the model can adaptively allocate training weights. The calculation formula is as follows: y=f attention (X) X=[x1,x2,x3,....,x n ] s(x i ,q)=v T tanh(wx i +uq) Among them, X is the input, α is the feature importance weight, w and q are model parameters, and y is the final output result; S4.4: Use different BiLSTM modules to extract temporal information from the output of each convolutional layer in the CNN model; S4.5: Add respective attention mechanism modules to the forward and backward hidden layers of each parallel BiLSTM module to make the relevant information within each BiLSTM module more concentrated and reduce the potential interference between key information; Add attention f to the front and back directions attention The process is as follows: in and are the forward and backward hidden layer outputs, n represents the number of hidden layers, and U, w, and b represent parameter weights, respectively; After calculations from S4.1 to S4.5, the final architecture HDLM of hybrid deep learning can be built. HDLM contains three layers of CNN and Each layer of CNN is connected to an independent BiLSTM module Then each BiLSTM module is connected to the bidirectional attention module Finally, through the shallow neural network f nn Output the gaze area prediction result G. The above calculation process is expressed by the formula: G=f nn (y 1 +y 2 +y 3 ) Where: y 1 is the output result of the first layer attention module; y 2 is the output result of the second layer attention module; y 3 is the output result of the third-layer attention module; G is the prediction result of the gaze area.
6. The driver gaze behavior safety reminder method applicable to the smart cockpit according to claim 1 is characterized in that: The step S5 is specifically as follows: S5.1: With {X i , G i }Use it as a sample to perform cyclic training and testing on the HDLM model, set the number of iterations to 200, the batch size to 64, the learning rate to 0.001, and save the results of each round of training and testing after the training starts; Repeat step S5.1 until the model test accuracy reaches 98% or more, reaching the expected target, and save the model parameter weights at the test accuracy; S5.2: Build a model test hardware-in-the-loop test platform, including a driving simulator, a virtual simulation environment, a computing server, a six-degree-of-freedom vehicle platform, a vehicle data acquisition unit, a dual-source camera, a driver status monitoring system, a data acquisition unit, a microprocessor and a logic judgment unit, and recruit Nx drivers to verify the effect; During the test, the driver sits on a six-degree-of-freedom vehicle platform and drives in a simulated environment, operating the central control screen while driving. The dual-source camera collects real-time facial and hand images of the driver, and the vehicle data collector collects vehicle operation and running data. The data mobile phone unit transmits the collected data to the microprocessor unit to calculate the real-time gaze area prediction results. S5.3: The Likert scale satisfaction is calculated for Nx drivers. When the satisfaction rate reaches above 95%, it is determined that the proposed method meets the design standards and the algorithm is integrated into the microprocessor unit.
7. The driver gaze behavior safety reminder method applicable to the smart cockpit according to claim 1 is characterized in that: The step S6 is specifically as follows: S6.1: Set the visual distraction duration threshold TD0, rear-end collision time threshold TTC0, and vehicle deviation time distance TLC0 according to the driver distraction theory and safe driving theory; S6.2: Obtain the final HDLM model parameters from step S5.1; S6.3: The system starts working, and the data acquisition unit collects data in real time; the DMS system determines the driver’s current T X Check whether to start controlling the full touch screen at the moment, if yes, proceed to the next step; S6.4: Data acquisition unit collects T X to T X +S time period data {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}, real-time construction of the model input parameter matrix {X i , G i }; S6.5: The parameter matrix {X i , G i } is sent to the HDLM algorithm module in the microprocessor to X +S+Δt time to predict the fixation area G; S6.6: In the microprocessor, calculate T X + The total time TD that the driver continuously stares at the entire touch area at time S; Record T X to T X At +S, the model HDLM predicts the number of times G=1. If the number is NG, the fixation time is NG / 30; S6.7: In the microprocessor, calculate T X + TTC of the vehicle in front and the vehicle in front at the moment S; TTC = (vf-vs) / d, where vf and vs represent the speeds of the preceding vehicle and the vehicle itself, respectively, and d represents the distance between the preceding vehicle and the vehicle itself; S6.8: In the microprocessor, calculate T X + The departure time distance TLC of the vehicle at time S; TLC = y / LV, y represents the lateral distance between the center of the vehicle and the lane line, and LV represents the lateral speed of the vehicle; S6.9: When T X + TD at time S > 2 s, and T X + TTC at time S < TTC0, and T X + TLC at time S < TLC0, and T X + G at time S + Δt = 1, the microprocessing unit controls the decision to give a safety reminder; S6.10: The microprocessor controls the vehicle machine assembly to pop up a prompt message at the upper center of the touch screen interface; at the same time, the microprocessor controls the vehicle machine assembly to emit a warning sound from the audio unit; S6.11: After the DMS system determines that the driver is driving normally, the system stops working.
8. A device for providing safety reminders of driver gaze behavior in a smart cockpit, characterized in that: include: Dual-source camera monitoring system: The dual-source camera monitoring system consists of two RGB cameras installed in different positions, one of which captures the driver's facial image and the other captures the driver's hand image. During operation, the dual-source cameras are connected to the data acquisition unit respectively. DMS system: It can obtain the driver's behavior status in real time, determine whether the driver starts to perform touch screen interaction tasks through the DMS system, connect with the data acquisition unit through the signal harness, and transmit data to the data acquisition unit; Vehicle data collector: A vehicle data collector is a hardware unit used to collect vehicle motion status and driver operation status; During the working process, it collects data in real time and transmits the data to the data acquisition unit through the signal harness; Data acquisition unit: The data acquisition unit is connected to the vehicle data acquisition unit and the dual-source camera monitoring system; it collects and stores the data matrix within a period of time, waiting for the call of the microprocessor unit; Microprocessing unit: One end of the microprocessing unit is connected to the data acquisition unit through an input data interface, and the other end is connected to the safety prompt device through an output data interface; The microprocessor unit includes an algorithm module and a logic judgment module. The algorithm module integrates the HDLM model to predict the driver's gaze area. The logic judgment module is used to output safety warning signals according to the driver's gaze area, the driver's continuous gaze time and the driving risk index; Safety reminder device: includes a visual reminder module and a voice reminder module, both modules are integrated with the vehicle-mounted assembly; for the visual reminder module, when a safety reminder is needed, the reminder information will appear in a small window on the full touch screen, and the content will be displayed in the form of a pop-up information interface; for the voice reminder module, when a safety reminder is needed, it will emit a reminder sound in the voice module of the full touch screen.
Citation Information
Patent Citations
Interaction method and device based on vehicle-mounted digital human, and storage medium
CN110728256A
Driving fatigue detection method and system combining pseudo 3D convolutional neural network and attention mechanism
CN111428699A
Distraction monitoring triggering type vehicle collision early warning system and control method
CN111845736A
Lithium ion battery health state prediction method based on CNN-BiLSTM-AT hybrid model
CN114325450A
Android malicious program detection method based on deep learning
CN115510445A
Cited By
Safety detection method and system based on camera at automotive trim and storage medium
CN119206680A
Intelligent driving test data acquisition and alignment method in multi-vehicle-type mixed scene
CN119296066A
A method for collecting and aligning intelligent driving test data in mixed scenarios with multiple vehicle models
CN119296066B
